Guides
Guardrails
Metrics an experiment must not hurt, checked automatically, with an optional automatic stop.
Guardrails are metrics an experiment must not make worse: errors, latency, cost, revenue, unsubscribes. They turn an experiment into a safe release: if the new version hurts one of them, you hear about it, or the experiment stops on its own.
Add guardrails#
In the wizard's step 5 (“What must not get worse?”) or later with Edit, add up to five goals. Each can have a tolerance: “10% worse is acceptable” means only a degradation beyond 10% counts as a breach.
| Action | On a breach |
|---|---|
alert | The breach is recorded, the experiment is flagged in the dashboard, and an audit-log entry is written by system:guardrails. |
stop | Everything above, and the experiment ends: the baseline (first variation) is shipped to the audience, so nobody gets the harmful variation anymore. |
How breaches are decided#
- Checked automatically while the experiment runs, every 5 minutes by default (
GUARDRAIL_INTERVAL_SECONDSon self-hosted installs;0turns it off).POST …/experiments/:id/guardrails/checkchecks one right away. - No false alarms from looking often. Each check uses the same always-valid intervals as the results, so checking every few minutes doesn't raise the false-alarm rate.
- Confidence is split across guardrails and variations (Bonferroni): at 95% with two guardrails, each is checked at 97.5%.
- A treatment breaches when it is significantly worse than the baseline in the goal's direction, beyond the tolerance even at the favourable end of its interval.
A guardrail on revenue per visitor with a 5% tolerance. A variation breaches only when it's significantly worse and even the favourable end of its interval is past the tolerance: B is worse, but could be within 5%; C is at least 8% worse.
Good guardrails#
- An error event your code tracks (
checkout_error,llm.error) as a conversion goal with direction decrease. - Latency or cost per call as a ratio goal, with a tolerance such as 10%.
- Revenue per visitor (mean), when testing something that isn't meant to change revenue.
API#
{
"guardrails": [
{ "eventKey": "llm.error", "type": "conversion" },
{ "eventKey": "llm.latency_ms", "type": "ratio", "maxDegradation": 0.1 }
],
"guardrailAction": "stop"
}guardrails[].eventKeyrequiredstring | The event; its goal supplies the name and direction. |
guardrails[].type"conversion" | "mean" | "ratio" | Default conversion. |
guardrails[].maxDegradation0–1 | Tolerance as a fraction: 0.1 = 10% worse is acceptable. |
guardrailAction"alert" | "stop" | Default alert. |
Send these when creating an experiment or with PATCH …/experiments/:id. The report's guardrails section shows each check's result.