
Lifecycle Experiments
Part of Lifecycle experiments and holdouts
Choosing a holdout size before launching a journey test
Choose a journey holdout using eligible volume, baseline outcomes, the smallest useful effect and the time available for a fair comparison.
undefined
Base the holdout size on the outcome you need to detect, the eligible audience you expect and the time available. A convenient percentage cannot replace a prospective sample calculation. A smaller holdout means fewer customers miss the proposed journey, but it usually gives a less precise comparison.
Define the test before the split
Define what the holdout misses: one promotional email or the whole marketing journey. Name the primary outcome and whether it belongs to a person or an account. For an account purchase, assign and count accounts, not every contact as an independent purchase opportunity.
Estimate the baseline outcome rate from a population resembling those who will qualify. For a numerical outcome, use its mean and variation. Whole-site traffic can misstate both the volume and outcome behaviour of a narrow journey audience. Estimate how many new eligible units will arrive while the journey, offer and outcome definition can remain stable.
Set the smallest effect that would change the decision; do not size the test to detect a gain too small to justify the journey. Agree the analysis method, confidence level and power before launch. For a binary outcome, use Select Statistical Consultants' Comparing Two Proportions calculator, which returns the minimum sample per group from the group proportions, confidence level and power (Select Statistical Consultants).
For a numerical outcome, Statsig's Power Analysis Calculator uses historical mean, variance and traffic volume to model the trade-off among minimum detectable effect, duration, exposures and allocation (Statsig).
Steps to Determine Optimal Holdout Size
- Define the primary outcomeSpecify whether it's a user-level or account-level event (e.g., purchase, sign-up). Use account-based counting for account purchases.
- Estimate baseline rateUse historical data from a population similar to the eligible journey audience—not whole-site traffic.
- Set minimum detectable effectChoose the smallest change that would justify launching the journey—do not aim to detect trivial gains.
- Calculate required sample sizeUse Select Statistical Consultants’ calculator for proportions or Statsig’s Power Analysis Tool for numerical outcomes.
Compare feasible allocations
Compare required group counts with eligible arrivals and available time, then choose the smallest holdout share that supplies the required comparison count and keeps enough journey units. An even split generally estimates a difference more precisely than a small holdout. A smaller holdout may suit the customer cost of withholding the journey, but cannot answer the question without enough observations. Statsig's Power Analysis Calculator models allocation, detectable effect, duration and exposures (Statsig).
Record the eligible-unit rule and expected arrivals, the baseline rate (or numerical mean and variation), the worthwhile effect, allocation, analysis settings and outcome follow-up. Use a population resembling journey qualifiers, not whole-site traffic.
For proportions, enter the journey's baseline and the treatment rate implied by the smallest worthwhile effect, plus confidence and power, in Select Statistical Consultants' two-proportion calculator (Select Statistical Consultants). For numerical outcomes, enter historical mean, variance and eligible traffic in Statsig's Power Analysis Calculator; compare exposure and duration estimates across allocations (Statsig). Record the resulting counts in both groups, not just the holdout percentage.
Select Statistical Consultants gives a marketing-promotion example requiring 248 customers who received the promotion and 248 who did not, at a 95% confidence level and 80% power, to detect a difference of this size (Select Statistical Consultants). That equal-count example is an even 50/50 split, not a journey default: use your own baseline and worthwhile effect to obtain the counts for your test.
Calculate the required units in each group and the likely calendar duration. Recruitment and outcome follow-up both take time: the last entrant may still need observation after the last journey message is sent.
For cumulative feature-impact holdouts, Statsig recommends a low single-digit holdout percentage, such as 1% to 2%; the effect is measured over months. That is a platform recommendation for measuring cumulative launches, not a default for one marketing journey (Statsig).
Holdout Size vs. Test Precision and Customer Impact
- Small Holdout (e.g., 1–2%)Lower customer impact; fewer users miss the journey. Less precise comparison; may lack power to detect meaningful effects.
- Even Split (50/50)Higher precision in estimating differences; optimal statistical power. More customers miss the journey, increasing opportunity cost.
- Larger Holdout (e.g., 10–20%)Improved statistical confidence; better for detecting small effects. High customer impact; significant number excluded from journey.
Pros and Cons of Small vs. Large Holdouts
- Small Holdout (1–2%)Pros: Minimises customer impact; aligns with platform recommendations for cumulative feature launches. Cons: May not achieve sufficient statistical power; risk of underpowered test.
- Large Holdout (≥10%)Pros: Increases statistical reliability; better at detecting subtle changes. Cons: Higher customer opportunity cost; more users excluded from journey.
Decide what happens if the test cannot finish
If the required sample cannot arrive before the offer, product or audience changes materially, record that limit before launch. Options include a larger still-relevant population, a more common outcome that genuinely supports the decision, or a larger minimum effect worth detecting. If none makes the test feasible, use other evidence for a narrower decision. Do not declare an underpowered result decisive.
Keep assignment and outcome definitions fixed during the test. Investigate a statistically meaningful, persistent mismatch between intended and observed allocation. Record opt-outs, skipped sends and competing campaigns so a weak treatment contrast is visible at readout.
There is no universal holdout percentage. Choose a split that respects the cost of withholding the journey and can resolve the stated decision within the available time.



