Lifecycle Experiments
Part of Lifecycle experiments and holdouts
Distinguishing a message effect from natural customer progression
Use an eligible holdout and a shared outcome window to separate what a lifecycle message may add from progress customers make anyway.
A customer completing a task after a lifecycle message does not show that the message caused it; some eligible customers would have progressed anyway. To estimate what offering the message added, compare outcomes for customers randomly assigned to it with outcomes for eligible customers assigned to a genuine no-message comparison over the same window.
Identify progress without the message
Start with one eligibility event and assign groups before the message is due. The comparison group should miss the specific marketing intervention under study while retaining necessary account and support communication. If another journey sends it the same prompt, the contrast no longer represents that prompt's absence.
The holdout outcome rate estimates how often the defined population progressed under the comparison experience. The difference between assigned-group rates estimates the added effect of offering the message under the tested conditions, subject to sampling uncertainty and checks that the groups and outcome records remained comparable. It is not a count of individual customers whom the message persuaded.
Setting up a clean holdout comparison
- Define one eligibility eventAnchor the comparison to a single, pre-specified entry point into the journey.
- Assign groups before the message is dueRandomly allocate eligible customers to the message or to a genuine no-message comparison before any send occurs.
- Withhold only the intervention under studyThe comparison group still receives necessary account and support communication.
- Check for duplicate promptsIf another journey sends the same prompt, the contrast no longer represents that prompt's absence.
- Keep every assigned unit in its groupOpt-outs, delivery failures and early completions stay in the message-assigned group for the primary comparison.
- Report delivery and skips separatelyShow how much of the intended treatment customers actually received, apart from the assigned-group rates.
- Compare over the same outcome windowBoth groups need equal follow-up opportunity for the difference in rates to mean anything.
Avoid selecting people by what they did later
People who opened an email and people who did not are selected after the journey began. An engaged customer may be more likely both to open and to finish the task. Likewise, app visitors may be more likely to see an in-product prompt and complete the task. Those comparisons describe association, even when the rate gap is large.
A before-and-after change can also reflect a changed product, audience, season or competing campaign. It can help monitor a journey, but cannot by itself separate the message from those changes. A platform's attributed conversion count applies a reporting rule to outcomes associated with a message; it does not answer what would have happened without that message.
Keep all originally assigned eligible units in their groups for the primary comparison. A customer who opts out, encounters a delivery failure or completes the task before a delayed send stays in the message-assigned group. Report delivery and skips separately to show how much of the intended treatment customers actually received.
What each type of comparison can tell you
- Randomised holdout versus assigned message groupEstimates the added effect of offering the message under the tested conditions, subject to sampling uncertainty. It is not a count of individual customers whom the message persuaded.
- Openers versus non-openersSelected after the journey began. An engaged customer may be more likely both to open and to finish the task, so this describes association only.
- Prompt seen versus prompt not seenApp visitors may be more likely both to see an in-product prompt and to complete the task, even when the rate gap is large.
- Before versus after a campaignCan reflect a changed product, audience, season or competing campaign. Useful for monitoring a journey, but cannot by itself separate the message from those changes.
- Platform attributed conversionsApplies a reporting rule to outcomes associated with a message. It does not answer what would have happened without that message.
Interpret the estimate narrowly
Check assignment counts, identifiers, outcome definitions and equal follow-up opportunity, then inspect whether the intended experiences differed: did account members cross groups, did another campaign reach the holdout, did sales or support treat groups differently, did the offer change? Each answer limits what the comparison can mean.
Report the outcome count and rate in each assigned group, the absolute difference and an uncertainty interval. A positive estimate with a wide interval may still allow no useful gain or harm. A sufficiently narrow interval can rule out an improvement large enough to justify the message under these conditions. Interpret the interval against the decision set before the result was seen.
When random assignment is unavailable, describe the comparison as observational and identify likely differences in motivation, timing and exposure. Use it to form a testable question rather than claiming that later progress was caused by the message.
Checks before you interpret the estimate
- Assignment counts reconcile with the eligible population
- Identifiers match across assignment and outcome records
- Outcomes are defined the same way in both groups
- Both groups had equal follow-up opportunity
- No account members crossed between groups
- No other campaign reached the holdout
- Sales and support treated both groups the same way
- The offer did not change during the test
- Absolute difference reported with an uncertainty interval
- Interval judged against the decision set agreed before the result was seen
- Where random assignment was unavailable, the comparison is labelled observational
