LiftCanary

Guides

Results and statistics

How results are computed: always-valid intervals, significance, SRM and Bonferroni.

Most A/B tools give you results that are only valid if you look once, at a sample size planned in advance. Everyone looks every day. LiftCanary's results are always valid: you can check them as often as you like, and stop as soon as they're clear, without inflating false positives.

Reading the results#

ColumnMeaning
VisitorsExposed users in the variation (users who saw more than one variation are left out).
Conversions / valueUsers with the event, or the total value, counting only events after each user's first exposure.
Rate / value per visitorConversion rate, value per user (mean goals) or value per event (ratio goals).
ImprovementRelative change versus the baseline (the first variation): +12% means 12% higher.
Confidence intervalThe range of improvements consistent with the data, at the experiment's confidence level. Always valid.
Significance1 − the adjusted, always-valid p-value. At or above the confidence level, the variation is a winner or a loser.
VerdictBetter, worse or inconclusive, taking the goal's direction into account.

The experiment's confidence level (80, 90, 95 or 99%; default 95%) is the chance of not declaring any false winner or loser across all its variations.

Statuses#

StatusMeaning
collectingFewer than 100 users in some variation: no verdict yet.
runningEnough data, nothing significant so far.
significantAt least one variation is a clear winner or loser.
sample_ratio_mismatchThe observed split doesn't match the configured weights. Overrides every other status: something is wrong with assignment or tracking, so the numbers can't be trusted.

The method#

Always-valid inference (mSPRT)#

Intervals and p-values come from the mixture sequential probability ratio test (Johari, Pekelis & Walsh, 2017), applied to the logarithm of the ratio treatment ÷ control, with a delta-method variance and a normal mixing distribution (τ = 0.1). Intervals are reported as relative improvement. On the log scale the variance doesn't grow with the size of the effect, so large regressions, like errors ×4, are caught after hundreds of users rather than thousands.

In simulation (1,000 A/A experiments, results checked 40 times each):

ApproachFalse positives
mSPRT (LiftCanary)0.9%
Classic 95% interval, checked at every look29.9%

The price is some extra data compared with a fixed-horizon test that is looked at exactly once. Median users per variation to call a lift on a 10% baseline conversion rate (300 simulated runs each):

True liftUsers per variation
+10%about 13,800
+50%about 1,300
×4 (+300%)about 300

Planned duration#

Being able to look at any time has a price: an always-valid interval is wider than a classic one, so a test needs about twice the visitors to show the same lift. When you can decide the length before the start, choose Planned duration under Analysis. The error budget (1 − confidence level) is then split in two, as in Datadog's and Eppo's hybrid designs:

  • While the test runs, the always-valid test checks it every day at half the budget (97.5% for a 95% test), so a clear winner can still be called early.
  • On the last planned day, one fixed-horizon analysis of exactly the planned days decides with the other half. Its interval is much tighter. A variation wins if either test says so; by the union bound, the false-positive rate stays under the budget.
  • Visitors after the planned days aren't counted in the goals (guardrails keep watching), and the length can't change once the test starts: a final analysis is only valid for a length chosen before any result.

For a 5% conversion rate and a +10% lift at 95% and 80% power, a variation needs 61,330 visitors always-valid and 37,859 planned (−38%). In simulation (2,000 A/A tests of 14 days, checked every day) planned tests called a false winner 2.5% of the time; with the planned visitors they found the lift 81% of the time, against 61% for an always-valid test with the same visitors. The duration estimate in the wizard follows the choice.

Variance reduction (CUPED)#

Conversion and mean goals use CUPED (Deng et al., 2013): each person's same goal in the 14 days before their first exposure is used to remove noise that has nothing to do with the test. A returning buyer who spends a lot before the test usually spends a lot during it; CUPED takes that out of the comparison. Because the pre-period can't be affected by the variation, results stay unbiased and always valid, and the intervals get narrower.

How much it helps depends on how predictable your visitors are. In simulation, with revenue per visitor and 30% returning buyers, CUPED removed 80% of the variance: a real +8% lift was found in 47% of tests instead of 5% at the same traffic, with false positives still under 1%. New visitors have no pre-period, so sites with mostly first-time traffic gain little. The report shows the reduction in the Variance reduction box, or why CUPED wasn't used (for example “No pre-test history” when every visitor is new); add cuped=false to the results API to see the plain analysis. Ratio goals (average per event) aren't adjusted.

Multiple variations and guardrails#

With more than one treatment, the error rate is split across comparisons (Bonferroni): at 95% with three variations (two comparisons), each comparison uses 97.5%. Guardrails split it again across guardrails.

Sample ratio mismatch#

A χ² test compares the observed split with the rule's weights; p < 0.001 raises the warning. Typical causes: a redirect or a crash that stops one variation from reporting, bots, or code that only decides for some users.

Metric types#

  • Conversion and mean goals are per-user averages; their variance is computed over users.
  • Ratio goals (average per event) are Σ value ÷ Σ events per variation; the variance uses the delta method over users, the unit of randomization, so heavy users don't make results look more certain than they are.
  • Percentile goals (p50, p90, p95, p99 of the value per event, e.g. latency) compare the variations' quantiles. Their variance comes from the share of events below the quantile (a ratio over users, by the delta method) divided by the density there, so users with many events don't make results look more certain either. In simulations the estimated variance was within 2% of the real one. Percentiles have no breakdown yet.

Outliers in value goals#

One very large order can decide a revenue test on its own and blow up its interval. Value goals are therefore capped (winsorized): a visitor's total above the cap counts as the cap. New value goals use the 99th percentile of buyers (computed over the visitors with the event, across the experiment), and you can choose the 95th, a fixed amount, or no cap on the goal. Nobody's data is dropped, and the goal's results say where the cap landed and how many visitors it touched.

Chance to beat the baseline#

Next to the significance, each variation shows its chance to beat the baseline: how likely it is to be better given the results so far, a Bayesian reading with a flat prior on the same numbers. It's easy to read, but it doesn't account for looking at the results many times, so it can be 95% while the test is still inconclusive. Winners are called by the always-valid significance (bandits by the chance to be best).

Multivariate tests#

A multivariate test's combinations are compared like any variations. Each part also gets its own result: its versions against its original, from the combinations' numbers added up, so every visitor counts. The split is even, which balances the other parts between the groups (the version's main effect). One Bonferroni correction covers every version of every part. See Multivariate tests.

Breakdown by segment#

Under the goals, Breakdown splits a goal's results by one attribute: device, browser, operating system, country, new or returning visitor, traffic source (utm_source, utm_medium, utm_campaign), referrer, language, or any attribute you send yourself (a plan, a customer tier). It answers "who does it work for?": a variation can be flat overall but win on mobile and lose on desktop.

  • Each visitor is counted in the segment they were in when they first saw the experiment; visitors without the attribute are under Not set.
  • The 8 largest segments are shown; smaller ones are added up under Other (exactly: the statistics are sums).
  • Looking at many segments finds "significant" differences by chance. Each segment is therefore tested at the experiment's confidence level split across the segments (Bonferroni): with 95% and 4 segments, a segment needs 98.75%. Segments also need 100 visitors per variation before a verdict.
  • Even so, treat a segment winner as a lead for a follow-up test (targeted at that audience) rather than a reason to ship to everyone.

Data quality rules#

  • Events and exposures are deduplicated by their uuid, so SDK retries never count twice.
  • Results count people: an anonymous id linked to a user at sign-in is the same person, so a landing-page exposure and a conversion inside the app belong together. See From landing page to app.
  • Only events at or after a user's first exposure count.
  • Users exposed to more than one variation are excluded and reported as excludedUsers.
  • Device clocks more than 5 minutes ahead are replaced with server time; events older than 30 days are rejected.

API#

HTTP
GET /v1/projects/:project/experiments/:id/report
GET /v1/projects/:project/experiments/:id/results?metric=purchase&type=conversion
GET /v1/projects/:project/flags/:flag/environments/:env/results?metric=purchase&type=conversion|mean|ratio
    [&rule=<experiment rule>] [&control=<variation>] [&from=<ISO|ms>] [&to=<ISO|ms>]

The report returns every goal at once (primary first) with, per variation, users, converters, total, value, lift, interval, significance, difference (absolute change, in the metric's units) and verdict, plus the funnel and guardrail results.