section 01
Use holdouts for journey value
Holdouts are most useful for onboarding, activation, trial conversion, retention, and winback journeys. They show whether the automation adds value beyond what users would have done anyway.
guide
Holdout Groups for Email
A split test compares message variants. A holdout compares sending against not sending. Growth teams need holdouts when the question is whether a journey changes behavior, not whether one subject line gets more clicks.
last updated 2026-06-12 4 sections
section 02
Pick the unit of randomization
B2B SaaS often needs account-level holdouts so teammates do not contaminate each other. Consumer and newsletter programs can usually randomize by user.
| unit | fit | risk |
|---|---|---|
| User | Individual lifecycle journeys. | Account teammates can influence each other. |
| Account | B2B SaaS activation and renewal. | Needs stable account membership. |
| Domain | Enterprise routing or sales-assist journeys. | Can be too coarse. |
section 03
Measure the business event
Do not stop at opens or clicks. Holdouts should measure activation, conversion, renewal, recovery, or usage. Secondary metrics should catch complaints and unsubscribes.
section 04
Keep the holdout stable
A holdout that changes membership mid-test is hard to trust. Store assignment, assignment time, journey ID, and eligibility state.
- ok Assign once per journey version.
- ok Exclude users who were already exposed.
- ok Keep the analysis window fixed before launch.
reading this as SaaS product teams
A SaaS product sends on behalf of its customers, so the deciding factors are tenant isolation, whether suppressions can be scoped rather than global, and whether support can answer why one specific account did not receive one specific email. Lifecycle automation and transactional sending usually need to coexist.
Applied to holdout groups for email, that means weighing multi-tenant support, event stream, the suppressions API, and operating track record ahead of the rest, against trial expiry notices, dunning, onboarding sequences, and per-tenant transactional mail.