Comparing journey exposure with a suitable holdout group

To assess a journey's contribution, compare outcomes for eligible customers assigned to receive it with those for eligible customers assigned to a suitable holdout.

Assign groups before the journey can influence behaviour, measure both over the same window, and keep necessary service communication available to both.

This primary comparison concerns the effect of offering the journey; actual message exposure is a separate fact to inspect.

Start with the same eligible population

Define who could enter at a specific decision point. Record the entry rule, exclusions, counting unit and assignment time. Draw the holdout from that population. A group of all customers who never received a message may include people who were never eligible.

Choose an assignment unit that fits the outcome. If several contacts can influence one account purchase, assigning different contacts at the same account to opposite groups can mix the experiences. Account-level assignment may fit that decision if account IDs are reliable. An individual onboarding task may call for person-level assignment.

Steps to Fairly Compare Journey and Holdout Groups

Separate assignment, delivery and viewing

Record whether a unit was assigned to the journey or holdout, whether a message was attempted or delivered, and whether viewing can be observed at all. A delivery failure, opt-out or completion before a scheduled step can leave an assigned customer with little or none of the journey. Keep that customer in the primary assigned-group comparison; removing them after assignment can change the populations being compared.

Comparing only viewers with non-viewers is descriptive. Viewing may depend on behaviour that also predicts the outcome: customers who return to an app can both see an in-app prompt and complete a task more often. Their difference alone cannot be credited to the prompt.

Specify what the holdout withholds. It may omit promotional journey messages while retaining account notices and customer care. Check whether another campaign delivers the same offer, whether sales handles the groups differently and whether a product change affects one group during follow-up.

Compare a shared verified outcome

Choose the primary outcome and follow-up period before reading results. For each group, count assigned eligible units and those with a verified outcome inside the window.

The difference between outcome rates estimates the effect of offering this journey to this eligible population if assignment was fair, the intended treatment difference was maintained and outcome records are comparable. Report uncertainty; a numerical difference alone does not establish a reliable benefit or harm.

Inspect relevant exceptions too, such as wrong sends after completion, opt-outs and unresolved support issues. Keep recent cohorts pending until both groups have had the full observation period.

Audit the allocation against the intended split and verify stable identifiers. Separately check delivery and exposure logs. An unexpected group-size imbalance can reveal assignment or logging problems; delivery counts can also differ because the journey's rules legitimately skip messages. Investigate the cause before interpreting a reported lift.

If fair assignment was unavailable, label matched or historical comparisons observational and explain how the groups may differ. An attributed conversion report or higher open rate does not establish incremental effect. Holdout size and power belong to experiment planning before launch.

Journey Exposure vs Holdout Group: Key Comparison Factors

Pros and Cons of Using a Holdout Group

Key Metrics for Holdout Comparison