LLM experiments
Is the new model better for your users?
Put two models, prompts or settings in front of real traffic. See what each does to the outcome you care about, and what it costs in money and time. Ship the winner from the dashboard, without a deploy.
Any provider, any model. Node.js, Python, Go, Java, PHP, Ruby and .NET. Prompts and answers never leave your servers.
Support assistant · 14 days · 18,240 users
Challenger winscurrent
large-model · t=0.7
challenger
small-model · t=0.3 · new prompt
| Goal | Current | Challenger | Change |
|---|---|---|---|
| Tickets resolved without a human | 61.4% | 66.0% | +7.5% |
| Cost per call | $0.0072 | $0.0011 | −84.7% |
| Latency per call | 1,204 ms | 742 ms | −38.4% |
| Answers left unfinished | 6.8% | 4.1% | −39.7% |
| Answers rated 👍 | 30.2% | 31.0% | +2.6% |
Guardrail on errors holding · example data
Evals are not enough
Evals say a model is good. Only an experiment says it's better.
Keep your evals and your tracing: they catch broken answers before release and explain single calls. An experiment answers the question they can't: what happens to your users, and your bill, when you switch.
| LiftCanary | Evals | Tracing | |
|---|---|---|---|
| Scores answers before release | |||
| Traces and logs every call | |||
| Compares versions on real users, at random | |||
| Measures business goals: conversions, resolved tickets, retention | |||
| Cost and latency per version, with intervals | |||
| Tells you when a difference is real (always-valid statistics) | |||
| Rolls a bad version back by itself | |||
| Changes model or prompt without a deploy |
Evals: offline test sets scored before release. Tracing: LLM observability tools. Categories, not products: many teams use all three together.
What you can test
Every change to an LLM feature is a hypothesis
A cheaper or newer model
Does the small model hold up for your users? Find out what it saves and what it costs you in quality, before you switch everyone.
A rewritten prompt
Shorter answers, a different tone, a step-by-step instruction. Prompts are variables, so marketing and product can change them too.
Parameters and retrieval
Temperature, max tokens, how many documents to retrieve, which tools an agent may use: anything you pass in params.
Whether AI helps at all
Test the AI feature against no AI feature. Sometimes the plain search box converts better.
Ten lines of code
Works with the model you already call
- 1
Create an AI config
A flag with model, prompt and params, and two variations: Current and Challenger.
- 2
Wrap your call
Read the config with aiConfig and wrap the call in measureLLM, or a streamed answer in measureLLMStream. Your code still calls the provider.
- 3
Start the experiment
Pick the goal that matters as primary, and add cost, latency and errors as guardrails.
Python too: ai_config and measure_llm, sync, async and streamed.
import { createClient, aiConfig, measureLLM, trackFeedback } from "@liftcanary/node";
const liftcanary = createClient({
sdkKey: process.env.LIFTCANARY_SDK_KEY!,
modelPrices: { "small-model": { input: 0.25, output: 1.25 }, "large-model": { input: 3, output: 15 } },
});
const fallback = { model: "large-model", prompt: "You are a helpful assistant." };
export async function answer(req) {
const user = liftcanary.createUser({ id: req.user.id });
// Model, prompt and params of this user's variation
const cfg = aiConfig(user, "support_bot", fallback);
// Your own provider call; latency, tokens, cost and errors are recorded
return measureLLM(user, cfg,
() => llm.chat({ model: cfg.model, system: cfg.prompt, ...cfg.params, messages: req.messages }),
(r) => ({ inputTokens: r.usage.input_tokens, outputTokens: r.usage.output_tokens }));
}
// When the user rates an answer (thumbs up or down)
export function rate(req, positive) {
const user = liftcanary.createUser({ id: req.user.id });
trackFeedback(user, aiConfig(user, "support_bot", fallback), positive);
}Safe by default
Try bold changes on a few users, not on all of them
Automatic rollback
Guardrails on errors, latency, cost or negative feedback. If the challenger is significantly worse, everyone goes back to the current version on its own.
Look whenever you like
Always-valid statistics: check results every hour without inventing winners. Big regressions show up after a few hundred users.
Cost you can trust
Cost per call from your own price list and real token counts, compared with an interval, not a guess from a spreadsheet.
Prompts stay with you
Only latency, token counts, cost, errors and the model name are sent. Prompts, answers and API keys never leave your servers. Hybrid deployment keeps even that in your cloud.
FAQ
Questions from AI teams
Which providers and models does it work with?
All of them. LiftCanary gives your code the model name, prompt and parameters for each user; your code makes the call with whatever SDK you use, including self-hosted models.
Do you see our prompts or users' messages?
No. The SDK sends the experiment, the variation, the model name and numbers: latency, tokens, cost and whether the call failed. The prompt template lives in your flag; the filled-in prompt and the answer stay on your servers.
How many users do we need?
It depends on how big the difference is. A large regression, like errors four times as often, shows after a few hundred users per variation. A few percent on a business goal needs thousands; the experiment wizard estimates how long it will take from your traffic.
Can the same user get different models on different requests?
No. Assignment is by user, so each person keeps their variation, and their experience and feedback can be compared fairly.
What does it cost?
Experiments are priced by tested visitors a month, with a free plan. Flags, rollouts and the AI configs themselves are free and unlimited.
Your next model switch, measured.
Free to start. Your first AI experiment can run today.