Guides
AI configs and LLM experiments
Test models, prompts and parameters on latency, cost and real outcomes.
A new model or prompt can look better in a playground and still be slower, more expensive or less helpful for real users. AI configs let you test model, prompt and parameter changes on real traffic, measured on latency, cost, errors, feedback and your business goals, with the same always-valid statistics as any experiment.
Create an AI config flag
Flags → New flag → AI config asks for the current model and prompt (and optionally a challenger model). It creates the variables
model(text),prompt(text) andparams(JSON), and the variations Off, Current and Challenger. Edit the challenger's values to whatever you want to test: another model, a rewritten prompt, a different temperature.Add the LLM goals
Goals → Add LLM goals creates:
Metric Goal type Better when Measures llm.latency_msaverage per event lower The whole call, until the last token of a streamed answer llm.ttft_msaverage per event lower Time to first token: how long the user waits before a streamed answer starts llm.costaverage per event lower Cost per call llm.tokensaverage per event lower Input + output tokens per call llm.errorconversion lower Users who hit a failed call llm.abortedconversion lower Users who left before an answer finished (a slow model shows up here) llm.feedback_positiveconversion higher Users who rated an answer 👍 Call your model through the config
import { createClient, aiConfig, measureLLM, renderPrompt, trackFeedback } from "@liftcanary/node"; const liftcanary = createClient({ sdkKey: process.env.LIFTCANARY_SDK_KEY!, // Price per 1M tokens, so cost can be computed from token counts modelPrices: { "my-small-model": { input: 0.25, output: 1.25 }, "my-large-model": { input: 3, output: 15 } }, }); await liftcanary.ready(); export async function answer(req) { const user = liftcanary.createUser({ id: req.user.id, attributes: { plan: req.user.plan } }); const cfg = aiConfig(user, "support_bot", { model: "my-small-model", prompt: "You are a helpful assistant." }); const reply = await measureLLM( user, cfg, () => llm.chat({ model: cfg.model, system: renderPrompt(cfg.prompt, { language: "Turkish" }), ...cfg.params, messages: req.messages }), (r) => ({ inputTokens: r.usage.input_tokens, outputTokens: r.usage.output_tokens }), ); return reply; } // When the user rates the answer: trackFeedback(user, cfg, true);Start an experiment
Create an experiment on the flag with Current vs Challenger. Use a business goal as primary (resolved tickets, conversions), and latency, cost and errors as secondary goals or guardrails with a tolerance: “the challenger may be up to 10% slower”.
Streamed answers#
In a chat, what users feel is how soon the answer starts. Wrap the stream in measureLLMStream (measure_llm_stream in Python): it passes every chunk through unchanged and, when the stream ends, records the time to first token, the total time, tokens and cost. If your handler stops reading early (the user closed the page, a break), the call is recorded as aborted and the provider's stream is closed.
import { aiConfig, measureLLMStream } from "@liftcanary/node";
const cfg = aiConfig(user, "support_bot", { model: "my-small-model", prompt: "You are a helpful assistant." });
const stream = measureLLMStream(
user,
cfg,
() => openai.chat.completions.create({ model: cfg.model, messages, stream: true, stream_options: { include_usage: true } }),
{ usage: (c) => c.usage && { inputTokens: c.usage.prompt_tokens, outputTokens: c.usage.completion_tokens } },
);
for await (const chunk of stream) {
res.write(chunk.choices[0]?.delta?.content ?? "");
}usage reads token counts from any chunk that has them; later chunks override earlier ones, so OpenAI's final usage chunk and Anthropic's message_start / message_delta events both work. By default the first chunk counts as the first token; pass isToken (is_token) when a provider sends other events first, such as Anthropic's message_start.
Helpers#
| Function | What it does |
|---|---|
aiConfig(user, flagKey, fallback) | Model, prompt and params for this user, and logs the exposure. Missing or wrongly typed variables fall back to fallback. enabled: false means the flag is off for this user. |
renderPrompt(template, values) | Fills {{placeholders}}; unknown ones are left as they are. |
measureLLM(user, cfg, call, usage?) | Runs the call, measures latency, reads token counts from the result, and records how it ended: ok, error, or aborted when it was cancelled (an AbortError, or the OpenAI and Anthropic SDKs' APIUserAbortError). Errors are rethrown. |
measureLLMStream(user, cfg, stream, options?) | The same for a streamed answer, plus the time to first token; see Streamed answers. |
trackLLM(user, cfg, usage) | Records a call you measured yourself: latencyMs, timeToFirstTokenMs, inputTokens, outputTokens, cost, outcome. |
trackFeedback(user, cfg, positive) | Sends llm.feedback_positive or llm.feedback_negative. |
The Python SDK has the same helpers in snake case: ai_config(user, flag_key, model=…, prompt=…, params=…), render_prompt, measure_llm (and measure_llm_async), measure_llm_stream (and measure_llm_stream_async), track_llm and track_feedback. A cancelled task (asyncio.CancelledError, what web frameworks raise when the client goes away) is recorded as aborted.
Cost comes from modelPrices (kept in your code on purpose, because prices change and vary by contract) unless you pass cost yourself.
What is sent#
One llm.call event per call, whose properties hold flag, variation, model, outcome and the numbers it has: latency_ms, ttft_ms, input_tokens, output_tokens, tokens, cost. Results read each metric in the table above from it. Prompts and answers are never sent. Feedback is its own event.
Why “average per event”#
Latency or cost must be averaged per call, not summed per user: a variation whose users chat more would otherwise look more expensive. Ratio goals compute Σ value ÷ Σ calls per variation, with the variance taken over users, so the statistics stay valid.
Try it#
LIFTCANARY_SDK_KEY=pk_... npx tsx examples/ai-demo/simulate.tssends simulated traffic for two models. To see a guardrail stop a bad challenger, make it fail often:
LIFTCANARY_SDK_KEY=pk_... FLAG=summarizer USERS=4000 SMALL_ERROR_RATE=0.15 npx tsx examples/ai-demo/simulate.ts