LiftCanary

Guides

Multi-armed bandits

Move traffic to the winning variation while the test runs.

A fixed split keeps every variation's share until the test ends: the fastest way to learn whether B is better than A. A multi-armed bandit moves traffic toward the variations that are winning while the test runs, so fewer visitors see a losing one. Use it when the result matters more during the test than after it: a seasonal banner, a sale headline, an offer you will only show for a few weeks.

How it works#

A bandit starts with an even split. Once every variation has 100 visitors, it looks at the primary goal on a schedule (every hour, every 6 hours or once a day) and sets each variation's share to its chance of being the best one (Thompson sampling). Every variation keeps at least 5%, so a slow starter can still catch up and the losing variations keep enough visitors for the results to prove the winner.

TopicWhat happens
Visitors keep their variationBrowsers and apps remember the variation a visitor first got, so a returning visitor never switches when the split changes.
Fair comparisonsEach period between two changes is compared on its own and the periods are added up the same way for every variation. A variation that got most of its visitors on a busy day isn't mistaken for a better one.
Calling the winnerA variation wins when its chance to be the best reaches the experiment's confidence level (95% by default). If the original reaches it, every variation is behind. In simulations, with no real difference a winner showed up at some point in about 5% of tests; a real 16% improvement was found in about 80% of them, more often than a fixed split proves it.
Why not significanceA bandit sends so few visitors to the losing variations that an always-valid interval almost never excludes zero, so bandits are judged by the chance to be best instead. Guardrails still use always-valid significance, since they're checked every few minutes.
GoalsThe primary goal must be a conversion or value goal. Secondary goals and guardrails are reported as usual; guardrails can still stop the test.
EndingStop it and ship the winner like any experiment: the winner becomes a rollout for the whole audience.

In the dashboard#

In the experiment wizard, choose Multi-armed bandit under Traffic split in the goals step, and how often it re-weighs traffic. A draft can switch between a fixed split and a bandit in its Setup tab. While it runs, the Results tab shows the split right now, each variation's chance to be best and every change the bandit made; Update now re-weighs it at once.

With the API#

A bandit on a no-code test
POST /v1/projects/shop/experiments
{ "kind": "changes", "name": "Hero offer", "environment": "production",
  "changes": { "include": [{ "type": "simple", "value": "/" }],
               "variations": [{ "name": "Free shipping", "css": "", "js": "" }] },
  "primaryMetric": { "eventKey": "purchase", "type": "conversion" },
  "bandit": { "updateEveryHours": 6 }, "start": true }

POST /v1/projects/:project/experiments/:id/bandit/update re-weighs a running bandit now. The experiment shows bandit with its settings, every change (updates: time, weights, chances) and nextCheckAt. PATCH with bandit (or null) changes a draft.