LiftCanary

Guides

Holdouts

Measure what all your experiments add up to.

Each experiment measures one change. A holdout measures them all together: a small share of visitors is kept out of every experiment, and of every winner shipped while the holdout runs. After a few months, comparing them with everyone else shows what experimentation added up to, including effects that don't simply add up, or fade after the first weeks.

How it works#

TopicWhat happens
Who is held outA share of the environment's visitors (5% is typical), by a hash of their id and the holdout's own salt, so the same visitors stay held out.
What they seeExperiments' rules are skipped for them, and so are winners shipped while the holdout runs: they get the flag's other rules or its default.
Who is countedVisitors who reach an experiment, in both groups: the SDK logs which group each of them is in, once. Visitors who never meet a test aren't counted.
ResultsThe holdout's goals, held-out visitors as the baseline against everyone else, with the same statistics, goal directions and caps as an experiment.
One at a timeOne holdout runs per environment. Its share is fixed once it starts: changing it would move visitors in or out.
EndingStopping it releases its visitors: they join experiments again and get every winner shipped meanwhile.

In the dashboard#

Under Holdouts in the project, name it, pick the environment, the share and the goals, and start it. Its page shows how much better everyone else does than the visitors kept out, goal by goal.

With the API#

Start a 5% holdout measuring purchases and revenue
POST /v1/projects/shop/holdouts
{ "key": "h2026", "name": "2026 holdout", "environment": "production", "share": 500,
  "goals": [{ "eventKey": "purchase", "type": "conversion" }, { "eventKey": "order", "type": "mean" }],
  "start": true }

GET /v1/projects/shop/holdouts/h2026/report
POST /v1/projects/shop/holdouts/h2026/stop