LiftCanary

Guides

Multivariate tests

Test several parts of a page at once, and what each version adds.

An A/B test compares whole pages. A multivariate test changes a few parts of a page at once, say the headline, the hero image and the button, each with its original and one to three new versions. Every combination is tested. The report then says which version of each part works, and how the combinations compare.

How it works#

TopicWhat happens
CombinationsEvery combination of the parts' versions is a variation, the all-original one being the baseline. 2 parts with 2 and 3 versions make 2 × 3 = 6 combinations.
SplitAlways even across the combinations, so every version of a part meets every version of the others equally often. That's what makes each part's result fair, so a multivariate test can't be a bandit.
Limits2 to 4 parts, up to 3 new versions each, and at most 16 combinations: each one splits the traffic further.
What visitors seeTheir combination's versions together: each version's CSS one after the other, its JavaScript in its own function (one failing doesn't stop the others), and its visual changes.
EditingEach version is edited on its own, in the visual editor or as CSS and JavaScript on the test's Setup tab. Every combination that shows it is rebuilt and published at once. The parts and versions are set when the test is created.
ShippingStop the test and ship a combination, like any variation.

Results by part#

Each part's versions are compared with its original using every visitor in the test: everyone who saw the bold headline, whatever button they saw, against everyone who saw the original headline. Because the split is even, the other parts are balanced between the two groups, so the difference is the version's effect averaged over the other parts (its main effect). With 6 combinations, a headline version is measured on half the visitors instead of a sixth, so it gets an answer far sooner: in simulations, a 20% lift was found 78% of the time from the parts and 5% of the time from the combination alone, on the same traffic.

  • The numbers are the combinations' added up, so parts get the same statistics as any experiment: always-valid intervals, CUPED, value caps and goal directions.
  • One Bonferroni family covers every version of every part: with 95% and three new versions in all, each needs 98.3%. Looking at every part together keeps the confidence level.
  • The report suggests the best combination so far: each part's winning version, the others at their original.
  • Parts can work differently together (an interaction): a bold headline may only help with the big button. The results by combination show each one against the original page; check the suggested combination there before shipping it.
  • Percentile goals (p50 to p99) have no results by part, since percentiles of combined groups don't add up. Their combinations still have results.

How long it takes#

The wizard's estimate counts the part with the most versions: each of its versions needs the visitors an A/B test's variation would, at the stricter confidence that comes from comparing every version of every part. Comparing the combinations themselves takes far longer; keep them few when that matters.

With the API#

Headline × button, on the home page
POST /v1/projects/shop/experiments
{ "kind": "mvt", "name": "Hero MVT", "environment": "production",
  "primaryMetric": { "eventKey": "purchase", "type": "conversion" },
  "mvt": {
    "include": [{ "type": "simple", "value": "/" }],
    "factors": [
      { "name": "Headline", "versions": [{ "name": "Bold", "css": "h1 { font-weight: 800 }" }] },
      { "name": "Button", "versions": [{ "name": "Green" }, { "name": "Big" }] }
    ]
  },
  "start": true }

# A version's name, CSS or JavaScript; every combination showing it is rebuilt
PATCH /v1/projects/shop/flags/mvt_hero_mvt/mvt
{ "factors": [{ "key": "button", "levels": [{ "key": "green", "css": ".cta { background: #16a34a }" }] }] }

# Each goal's result has "factors": every part's versions against its original
GET /v1/projects/shop/experiments/{id}/report