LiftCanary

Guides

Experiments

Create, start, stop and ship A/B tests, from the dashboard or the API.

An experiment is an experiment rule on a flag, in one environment, plus what you want to learn: a name, a hypothesis, metrics, a confidence level, guardrails and an optional funnel. Targeting stays in the flag's rule, so your code doesn't change between a rollout and an experiment.

Experiment types#

TypeUse it forCode needed
A/B, without codeVariations made of CSS and JavaScript on chosen pages. See No-code tests.No: the snippet applies them
A/B, in your codeVersions your site, app or server chooses between: a checkout flow, a paywall price, an onboarding screen, a model.Yes: read the flag
Split URLSend visitors of a page to different URLs, e.g. / vs /next. See Redirect tests.No: the snippet redirects
MultipageOne variation across several pages (a funnel), with or without code.Optional
A/ATwo identical originals, to check tracking and the split. See No-code tests.No
PersonalizationNot an experiment: show a change to an audience, with no comparison. It has its own Personalizations page. See No-code tests.No

Create one in the dashboard#

Experiments → New experiment walks through five steps:

  1. What are you testing? The type (A/B, Split URL, Multipage or A/A) and, for A/B and Multipage, whether the variations are made without code or in your code.
  2. Which variations? Without code: the pages and each variation's CSS/JavaScript. In code: pick a flag (only flags with at least two variations besides “off” are listed) or create one here, and set the split.
  3. Where, and who takes part? The pages (any page, or a saved location), the audience and the share of it that enters the experiment.
  4. How do you measure success? The primary goal, secondary goals, an optional funnel and the confidence level. Choose goals picks several saved goals at once and marks the primary one.
  5. What must not get worse? Up to five guardrails and what happens when one is breached.

Save it as a draft or start it right away.

Lifecycle#

ActionWhat happens to the flag
Create (draft)An experiment rule is added at 0% traffic: nobody is enrolled yet.
StartThe rule gets its traffic and the flag is switched on in that environment. startedAt is recorded.
Stop and ship a variationThe rule becomes a 100% rollout of that variation, for the same audience, in the same position. Everyone who would have entered now gets the winner.
StopThe rule is removed; users fall back to the flag's other rules.

The experiment and the flag editor stay in sync: adding an experiment rule in the flag editor creates an experiment, raising a draft's traffic starts it, and removing the rule stops it. The rule as it ran is stored with the experiment, so results stay readable after it's gone.

Final results#

Events sent late (by an app that was offline, or a server that sends in batches) still count for 30 days after an experiment ends. Then its results are saved: the totals per variation of every goal and guardrail, the funnel, and the breakdowns. From then on the report is computed from them, so it no longer changes and stays after the raw events are deleted, 400 days after they happen. A very long experiment is saved sooner, before its oldest data expires. The report says which: Final on 9 Dec while late events still count, Final results, saved … after.

What you can change while it runs#

Name, hypothesis, metrics, confidence level, guardrails and funnel can be edited at any time: results are recomputed from the raw events. The split and the audience live in the flag rule; changing them mid-test changes who enters, so prefer stopping and starting a new experiment. Once an ended experiment's results are saved, its goals can be removed and brought back, and its confidence level changed, but a goal it never measured can't be added.

How long will it take?#

Under the confidence level, the wizard estimates how many visitors each variation needs and how many days that takes. Enter the goal's current conversion rate and the smallest lift worth finding (relative: 10% means 5% → 5.5%); visitors a day start from the environment's last 24 hours, and the share of the audience in the test is taken into account. The estimate uses LiftCanary's own always-valid test at your confidence level, 80% power and the number of variations, so it holds even if you check results every day.

Conversion rateLift to findVisitors per variation (A/B, 95%)
5%+10%about 61,000
5%+20%about 16,000
10%+10%about 29,000
30%+10%about 7,400

More than eight weeks is flagged: test a bolder change, put more traffic in, or pick a goal more visitors reach. Under a week, run it a full week anyway so every weekday is in the results. A lift bigger than the one you entered is usually called sooner.

Schedule a start or an end#

In the wizard, When → Schedule starts the experiment by itself at a set time (it's saved as a draft until then), and an optional end time stops it by itself, without shipping a variation. The same works for personalizations: turn a campaign banner on at 9:00 on Black Friday and off at midnight. Later, the Schedule card on the experiment's page changes or clears the times.

  • Times are entered in your browser's time zone and checked every 30 seconds by the API, so a start or end happens within half a minute.
  • Only a draft can get a start time; a running experiment can still get (or move) its end. Starting or stopping by hand clears the schedule.
  • A scheduled start and end show as Starts 12 Nov, 09:00 or Live until … in the lists, and in the audit log as scheduled-start / scheduled-stop by system:schedule.
  • If the API was down at the time, it catches up as soon as it's back.

Preview before you start#

On a test's page, Preview on your site (with the visual editor for no-code tests, Preview and edit on your site) lists every variation. Enter the address of a page the test runs on, then Preview opens it showing that variation, or Copy link gives you a link to send to a colleague or client. It works for drafts and running tests, for no-code, split URL and flag tests read by the website snippet.

  • The link adds ?liftcanary_preview=<flag>:<variation> (several pairs separated by commas). The tab keeps the preview while you browse the site.
  • Nothing is sent while previewing (no exposures, no events), so previews never count in results.
  • A banner at the bottom left shows the variation, switches to another one and ends the preview (Exit preview, or open a page with ?liftcanary_preview=off).
  • Anyone with the link sees the variation, as with every variation's content: it's in the public datafile anyway. Don't put secrets in variations.

For apps and servers, pin a user to a variation with an override on the flag instead.

Who gets counted#

  • A user counts once they are exposed: decided into one of the experiment's variations by an SDK. Users the rule didn't take don't count.
  • Only events at or after the user's first exposure count, so earlier purchases don't bias a variation.
  • Users who saw more than one variation (for example, a shared device) are left out and reported separately.
  • With a triggered location, only visitors who reach the trigger are exposed.

Live log#

The Live tab of a test lists its latest views and conversions as they arrive, newest first: when, which variation, the device and browser, the country, where the visit came from, whether the visitor is new or returning, and for a conversion the goal and its value. It refreshes every 2, 5 or 15 seconds while you look at it, pauses on request, and shows how many visitors each variation got in the last hour, so a lopsided split stands out early.

  • Check your own visit. Open the site, run liftcanary.userId() in the browser console and search for that id: your view, and your conversion when you make one, appear within seconds with the variation you got. Ids linked by a sign-in count as one visitor.
  • It covers the last 24 hours, up to 100 visits, and counts conversions the way results do: from a visitor's first view, for people who saw one variation.
  • Only built-in attributes are shown; custom ones you send may carry anything, so they stay out. Admins see visitor ids; other members see a stable alias per visitor.
  • With hybrid deployment there's no live log: it's made of rows about visitors, and those never leave your data plane.

Results#

The experiment page shows, per goal and variation: visitors, conversions or value, the improvement over the baseline with an always-valid confidence interval, the statistical significance and a verdict (better, worse, inconclusive) that respects whether the goal should go up or down. A sample ratio mismatch warning appears when the split doesn't match the configured weights, because then the results can't be trusted. How it all works: Results and statistics.

  1. 1The verdict on the primary goal, in plain words. A variation that breaks a guardrail is never called the winner.
  2. 2The improvement over the baseline, with its always-valid interval: the true effect lies in this range.
  3. 3Statistical significance, against the confidence level you chose for the experiment.
  4. 4The traffic-split check. A mismatch means something loses visitors, and the results can't be trusted.

Create one with the API#

POST /v1/projects/web/experiments
{
  "name": "One-page checkout",
  "hypothesis": "Fewer steps means more completed orders.",
  "flag": "new_checkout",
  "environment": "production",
  "audience": { "attribute": "country", "op": "in", "value": ["TR", "DE"] },
  "traffic": 5000,
  "allocation": [
    { "variation": "control", "weight": 5000 },
    { "variation": "one_page", "weight": 5000 }
  ],
  "primaryMetric": { "eventKey": "purchase", "type": "conversion" },
  "secondaryMetrics": [{ "eventKey": "purchase", "type": "mean" }],
  "confidenceLevel": 95,
  "guardrails": [{ "eventKey": "checkout_error", "type": "conversion" }],
  "guardrailAction": "stop",
  "start": true,
  "schedule": { "stopAt": "2026-12-01T00:00:00+03:00" }
}

Body

namerequired
string
Shown in the dashboard.
environmentrequired
key
Where the experiment runs.
kind
"flag" | "redirect" | "changes" | "aa"
Default flag. For redirect, send redirect instead of flag; for changes and aa, send changes (see No-code tests).
flag
key
Existing flag to test. Or send newFlag to create one.
newFlag
{ key, name, variations: [{ key, name? }] }
Creates the flag with the experiment (2–10 variations); an off variation is added.
redirect
object
Redirect test settings and destinations: see Redirect tests.
hypothesis
string
Up to 2,000 characters.
ruleKey
key
Key of the experiment rule; generated when omitted.
audience
condition
Who can enter. Omitted = everyone. See conditions.
traffic
1–10000
Share of the audience that enters, in buckets. Default 10000 (100%).
allocation
[{ variation, weight }]
Split across variations, weights sum to 10000. Default: even across every variation except the off variation.
primaryMetric
{ eventKey, type }
type: conversion (default), mean or ratio.
secondaryMetrics
metric[]
Up to 10.
confidenceLevel
80 | 90 | 95 | 99
Default 95.
guardrails
[{ eventKey, type, maxDegradation? }]
Up to 5. See Guardrails.
guardrailAction
"alert" | "stop"
Default alert.
funnel
[{ eventKey }]
2 to 8 ordered steps, or none.
start
boolean
Start right away instead of saving a draft. Default false.
schedule
{ startAt?, stopAt? }
ISO times (with a time zone) or epoch milliseconds: start a draft and/or end it by itself. startAt needs start: false; PATCH changes them, null clears one.

Then:

HTTP
POST /v1/projects/web/experiments/:id/start
POST /v1/projects/web/experiments/:id/stop      { "ship": "one_page" }   // or { "ship": null } to just end it
GET  /v1/projects/web/experiments/:id/report    // every goal, funnel and guardrail at once; frozenAt once its results are saved

On LiftCanary Cloud's Free plan two experiments run at a time: starting a third answers 402 with error: "plan_limit" and limit: "runningExperiments" until one ends. Drafts don't count. See Plans and billing.