Guides
Multi-armed bandits
Move traffic to the winning variation while the test runs.
A fixed split keeps every variation's share until the test ends: the fastest way to learn whether B is better than A. A multi-armed bandit moves traffic toward the variations that are winning while the test runs, so fewer visitors see a losing one. Use it when the result matters more during the test than after it: a seasonal banner, a sale headline, an offer you will only show for a few weeks.
How it works#
A bandit starts with an even split. Once every variation has 100 visitors, it looks at the primary goal on a schedule (every hour, every 6 hours or once a day) and sets each variation's share to its chance of being the best one (Thompson sampling). Every variation keeps at least 5%, so a slow starter can still catch up and the losing variations keep enough visitors for the results to prove the winner.
| Topic | What happens |
|---|---|
| Visitors keep their variation | Browsers and apps remember the variation a visitor first got, so a returning visitor never switches when the split changes. |
| Fair comparisons | Each period between two changes is compared on its own and the periods are added up the same way for every variation. A variation that got most of its visitors on a busy day isn't mistaken for a better one. |
| Calling the winner | A variation wins when its chance to be the best reaches the experiment's confidence level (95% by default). If the original reaches it, every variation is behind. In simulations, with no real difference a winner showed up at some point in about 5% of tests; a real 16% improvement was found in about 80% of them, more often than a fixed split proves it. |
| Why not significance | A bandit sends so few visitors to the losing variations that an always-valid interval almost never excludes zero, so bandits are judged by the chance to be best instead. Guardrails still use always-valid significance, since they're checked every few minutes. |
| Goals | The primary goal must be a conversion or value goal. Secondary goals and guardrails are reported as usual; guardrails can still stop the test. |
| Ending | Stop it and ship the winner like any experiment: the winner becomes a rollout for the whole audience. |
In the dashboard#
In the experiment wizard, choose Multi-armed bandit under Traffic split in the goals step, and how often it re-weighs traffic. A draft can switch between a fixed split and a bandit in its Setup tab. While it runs, the Results tab shows the split right now, each variation's chance to be best and every change the bandit made; Update now re-weighs it at once.
With the API#
POST /v1/projects/shop/experiments
{ "kind": "changes", "name": "Hero offer", "environment": "production",
"changes": { "include": [{ "type": "simple", "value": "/" }],
"variations": [{ "name": "Free shipping", "css": "", "js": "" }] },
"primaryMetric": { "eventKey": "purchase", "type": "conversion" },
"bandit": { "updateEveryHours": 6 }, "start": true }POST /v1/projects/:project/experiments/:id/bandit/update re-weighs a running bandit now. The experiment shows bandit with its settings, every change (updates: time, weights, chances) and nextCheckAt. PATCH with bandit (or null) changes a draft.