Guides
Holdouts
Measure what all your experiments add up to.
Each experiment measures one change. A holdout measures them all together: a small share of visitors is kept out of every experiment, and of every winner shipped while the holdout runs. After a few months, comparing them with everyone else shows what experimentation added up to, including effects that don't simply add up, or fade after the first weeks.
How it works#
| Topic | What happens |
|---|---|
| Who is held out | A share of the environment's visitors (5% is typical), by a hash of their id and the holdout's own salt, so the same visitors stay held out. |
| What they see | Experiments' rules are skipped for them, and so are winners shipped while the holdout runs: they get the flag's other rules or its default. |
| Who is counted | Visitors who reach an experiment, in both groups: the SDK logs which group each of them is in, once. Visitors who never meet a test aren't counted. |
| Results | The holdout's goals, held-out visitors as the baseline against everyone else, with the same statistics, goal directions and caps as an experiment. |
| One at a time | One holdout runs per environment. Its share is fixed once it starts: changing it would move visitors in or out. |
| Ending | Stopping it releases its visitors: they join experiments again and get every winner shipped meanwhile. |
In the dashboard#
Under Holdouts in the project, name it, pick the environment, the share and the goals, and start it. Its page shows how much better everyone else does than the visitors kept out, goal by goal.
With the API#
POST /v1/projects/shop/holdouts
{ "key": "h2026", "name": "2026 holdout", "environment": "production", "share": 500,
"goals": [{ "eventKey": "purchase", "type": "conversion" }, { "eventKey": "order", "type": "mean" }],
"start": true }
GET /v1/projects/shop/holdouts/h2026/report
POST /v1/projects/shop/holdouts/h2026/stop