LiftCanary

Guides

Guardrails

Metrics an experiment must not hurt, checked automatically, with an optional automatic stop.

Guardrails are metrics an experiment must not make worse: errors, latency, cost, revenue, unsubscribes. They turn an experiment into a safe release: if the new version hurts one of them, you hear about it, or the experiment stops on its own.

Add guardrails#

In the wizard's step 5 (“What must not get worse?”) or later with Edit, add up to five goals. Each can have a tolerance: “10% worse is acceptable” means only a degradation beyond 10% counts as a breach.

ActionOn a breach
alertThe breach is recorded, the experiment is flagged in the dashboard, and an audit-log entry is written by system:guardrails.
stopEverything above, and the experiment ends: the baseline (first variation) is shipped to the audience, so nobody gets the harmful variation anymore.

How breaches are decided#

  • Checked automatically while the experiment runs, every 5 minutes by default (GUARDRAIL_INTERVAL_SECONDS on self-hosted installs; 0 turns it off). POST …/experiments/:id/guardrails/check checks one right away.
  • No false alarms from looking often. Each check uses the same always-valid intervals as the results, so checking every few minutes doesn't raise the false-alarm rate.
  • Confidence is split across guardrails and variations (Bonferroni): at 95% with two guardrails, each is checked at 97.5%.
  • A treatment breaches when it is significantly worse than the baseline in the goal's direction, beyond the tolerance even at the favourable end of its interval.

A guardrail on revenue per visitor with a 5% tolerance. A variation breaches only when it's significantly worse and even the favourable end of its interval is past the tolerance: B is worse, but could be within 5%; C is at least 8% worse.

Good guardrails#

  • An error event your code tracks (checkout_error, llm.error) as a conversion goal with direction decrease.
  • Latency or cost per call as a ratio goal, with a tolerance such as 10%.
  • Revenue per visitor (mean), when testing something that isn't meant to change revenue.

API#

JSON
{
  "guardrails": [
    { "eventKey": "llm.error", "type": "conversion" },
    { "eventKey": "llm.latency_ms", "type": "ratio", "maxDegradation": 0.1 }
  ],
  "guardrailAction": "stop"
}
guardrails[].eventKeyrequired
string
The event; its goal supplies the name and direction.
guardrails[].type
"conversion" | "mean" | "ratio"
Default conversion.
guardrails[].maxDegradation
0–1
Tolerance as a fraction: 0.1 = 10% worse is acceptable.
guardrailAction
"alert" | "stop"
Default alert.

Send these when creating an experiment or with PATCH …/experiments/:id. The report's guardrails section shows each check's result.