LLM experiments

Is the new model better for your users?

Put two models, prompts or settings in front of real traffic. See what each does to the outcome you care about, and what it costs in money and time. Ship the winner from the dashboard, without a deploy.

Any provider, any model. Node.js, Python, Go, Java, PHP, Ruby and .NET. Prompts and answers never leave your servers.

Support assistant · 14 days · 18,240 users

Challenger wins

current

large-model · t=0.7

challenger

small-model · t=0.3 · new prompt

GoalCurrentChallengerChange
Tickets resolved without a human61.4%66.0%+7.5%
Cost per call$0.0072$0.0011−84.7%
Latency per call1,204 ms742 ms−38.4%
Answers left unfinished6.8%4.1%−39.7%
Answers rated 👍30.2%31.0%+2.6%

Guardrail on errors holding · example data

Evals are not enough

Evals say a model is good. Only an experiment says it's better.

Keep your evals and your tracing: they catch broken answers before release and explain single calls. An experiment answers the question they can't: what happens to your users, and your bill, when you switch.

LiftCanaryEvalsTracing
Scores answers before release
Traces and logs every call
Compares versions on real users, at random
Measures business goals: conversions, resolved tickets, retention
Cost and latency per version, with intervals
Tells you when a difference is real (always-valid statistics)
Rolls a bad version back by itself
Changes model or prompt without a deploy

Evals: offline test sets scored before release. Tracing: LLM observability tools. Categories, not products: many teams use all three together.

What you can test

Every change to an LLM feature is a hypothesis

  • A cheaper or newer model

    Does the small model hold up for your users? Find out what it saves and what it costs you in quality, before you switch everyone.

  • A rewritten prompt

    Shorter answers, a different tone, a step-by-step instruction. Prompts are variables, so marketing and product can change them too.

  • Parameters and retrieval

    Temperature, max tokens, how many documents to retrieve, which tools an agent may use: anything you pass in params.

  • Whether AI helps at all

    Test the AI feature against no AI feature. Sometimes the plain search box converts better.

Ten lines of code

Works with the model you already call

  1. 1

    Create an AI config

    A flag with model, prompt and params, and two variations: Current and Challenger.

  2. 2

    Wrap your call

    Read the config with aiConfig and wrap the call in measureLLM, or a streamed answer in measureLLMStream. Your code still calls the provider.

  3. 3

    Start the experiment

    Pick the goal that matters as primary, and add cost, latency and errors as guardrails.

Python too: ai_config and measure_llm, sync, async and streamed.

support.ts
import { createClient, aiConfig, measureLLM, trackFeedback } from "@liftcanary/node";

const liftcanary = createClient({
  sdkKey: process.env.LIFTCANARY_SDK_KEY!,
  modelPrices: { "small-model": { input: 0.25, output: 1.25 }, "large-model": { input: 3, output: 15 } },
});

const fallback = { model: "large-model", prompt: "You are a helpful assistant." };

export async function answer(req) {
  const user = liftcanary.createUser({ id: req.user.id });
  // Model, prompt and params of this user's variation
  const cfg = aiConfig(user, "support_bot", fallback);

  // Your own provider call; latency, tokens, cost and errors are recorded
  return measureLLM(user, cfg,
    () => llm.chat({ model: cfg.model, system: cfg.prompt, ...cfg.params, messages: req.messages }),
    (r) => ({ inputTokens: r.usage.input_tokens, outputTokens: r.usage.output_tokens }));
}

// When the user rates an answer (thumbs up or down)
export function rate(req, positive) {
  const user = liftcanary.createUser({ id: req.user.id });
  trackFeedback(user, aiConfig(user, "support_bot", fallback), positive);
}

Safe by default

Try bold changes on a few users, not on all of them

  • Automatic rollback

    Guardrails on errors, latency, cost or negative feedback. If the challenger is significantly worse, everyone goes back to the current version on its own.

  • Look whenever you like

    Always-valid statistics: check results every hour without inventing winners. Big regressions show up after a few hundred users.

  • Cost you can trust

    Cost per call from your own price list and real token counts, compared with an interval, not a guess from a spreadsheet.

  • Prompts stay with you

    Only latency, token counts, cost, errors and the model name are sent. Prompts, answers and API keys never leave your servers. Hybrid deployment keeps even that in your cloud.

FAQ

Questions from AI teams

Which providers and models does it work with?

All of them. LiftCanary gives your code the model name, prompt and parameters for each user; your code makes the call with whatever SDK you use, including self-hosted models.

Do you see our prompts or users' messages?

No. The SDK sends the experiment, the variation, the model name and numbers: latency, tokens, cost and whether the call failed. The prompt template lives in your flag; the filled-in prompt and the answer stay on your servers.

How many users do we need?

It depends on how big the difference is. A large regression, like errors four times as often, shows after a few hundred users per variation. A few percent on a business goal needs thousands; the experiment wizard estimates how long it will take from your traffic.

Can the same user get different models on different requests?

No. Assignment is by user, so each person keeps their variation, and their experience and feedback can be compared fairly.

What does it cost?

Experiments are priced by tested visitors a month, with a free plan. Flags, rollouts and the AI configs themselves are free and unlimited.

Your next model switch, measured.

Free to start. Your first AI experiment can run today.