LiftCanary

Guides

AI configs and LLM experiments

Test models, prompts and parameters on latency, cost and real outcomes.

A new model or prompt can look better in a playground and still be slower, more expensive or less helpful for real users. AI configs let you test model, prompt and parameter changes on real traffic, measured on latency, cost, errors, feedback and your business goals, with the same always-valid statistics as any experiment.

  1. Create an AI config flag

    Flags → New flag → AI config asks for the current model and prompt (and optionally a challenger model). It creates the variables model (text), prompt (text) and params (JSON), and the variations Off, Current and Challenger. Edit the challenger's values to whatever you want to test: another model, a rewritten prompt, a different temperature.

  2. Add the LLM goals

    Goals → Add LLM goals creates:

    MetricGoal typeBetter whenMeasures
    llm.latency_msaverage per eventlowerThe whole call, until the last token of a streamed answer
    llm.ttft_msaverage per eventlowerTime to first token: how long the user waits before a streamed answer starts
    llm.costaverage per eventlowerCost per call
    llm.tokensaverage per eventlowerInput + output tokens per call
    llm.errorconversionlowerUsers who hit a failed call
    llm.abortedconversionlowerUsers who left before an answer finished (a slow model shows up here)
    llm.feedback_positiveconversionhigherUsers who rated an answer 👍
  3. Call your model through the config

    import { createClient, aiConfig, measureLLM, renderPrompt, trackFeedback } from "@liftcanary/node";
    
    const liftcanary = createClient({
      sdkKey: process.env.LIFTCANARY_SDK_KEY!,
      // Price per 1M tokens, so cost can be computed from token counts
      modelPrices: { "my-small-model": { input: 0.25, output: 1.25 }, "my-large-model": { input: 3, output: 15 } },
    });
    await liftcanary.ready();
    
    export async function answer(req) {
      const user = liftcanary.createUser({ id: req.user.id, attributes: { plan: req.user.plan } });
      const cfg = aiConfig(user, "support_bot", { model: "my-small-model", prompt: "You are a helpful assistant." });
    
      const reply = await measureLLM(
        user,
        cfg,
        () => llm.chat({ model: cfg.model, system: renderPrompt(cfg.prompt, { language: "Turkish" }), ...cfg.params, messages: req.messages }),
        (r) => ({ inputTokens: r.usage.input_tokens, outputTokens: r.usage.output_tokens }),
      );
      return reply;
    }
    
    // When the user rates the answer:
    trackFeedback(user, cfg, true);
  4. Start an experiment

    Create an experiment on the flag with Current vs Challenger. Use a business goal as primary (resolved tickets, conversions), and latency, cost and errors as secondary goals or guardrails with a tolerance: “the challenger may be up to 10% slower”.

Streamed answers#

In a chat, what users feel is how soon the answer starts. Wrap the stream in measureLLMStream (measure_llm_stream in Python): it passes every chunk through unchanged and, when the stream ends, records the time to first token, the total time, tokens and cost. If your handler stops reading early (the user closed the page, a break), the call is recorded as aborted and the provider's stream is closed.

import { aiConfig, measureLLMStream } from "@liftcanary/node";

const cfg = aiConfig(user, "support_bot", { model: "my-small-model", prompt: "You are a helpful assistant." });
const stream = measureLLMStream(
  user,
  cfg,
  () => openai.chat.completions.create({ model: cfg.model, messages, stream: true, stream_options: { include_usage: true } }),
  { usage: (c) => c.usage && { inputTokens: c.usage.prompt_tokens, outputTokens: c.usage.completion_tokens } },
);
for await (const chunk of stream) {
  res.write(chunk.choices[0]?.delta?.content ?? "");
}

usage reads token counts from any chunk that has them; later chunks override earlier ones, so OpenAI's final usage chunk and Anthropic's message_start / message_delta events both work. By default the first chunk counts as the first token; pass isToken (is_token) when a provider sends other events first, such as Anthropic's message_start.

Helpers#

FunctionWhat it does
aiConfig(user, flagKey, fallback)Model, prompt and params for this user, and logs the exposure. Missing or wrongly typed variables fall back to fallback. enabled: false means the flag is off for this user.
renderPrompt(template, values)Fills {{placeholders}}; unknown ones are left as they are.
measureLLM(user, cfg, call, usage?)Runs the call, measures latency, reads token counts from the result, and records how it ended: ok, error, or aborted when it was cancelled (an AbortError, or the OpenAI and Anthropic SDKs' APIUserAbortError). Errors are rethrown.
measureLLMStream(user, cfg, stream, options?)The same for a streamed answer, plus the time to first token; see Streamed answers.
trackLLM(user, cfg, usage)Records a call you measured yourself: latencyMs, timeToFirstTokenMs, inputTokens, outputTokens, cost, outcome.
trackFeedback(user, cfg, positive)Sends llm.feedback_positive or llm.feedback_negative.

The Python SDK has the same helpers in snake case: ai_config(user, flag_key, model=…, prompt=…, params=…), render_prompt, measure_llm (and measure_llm_async), measure_llm_stream (and measure_llm_stream_async), track_llm and track_feedback. A cancelled task (asyncio.CancelledError, what web frameworks raise when the client goes away) is recorded as aborted.

Cost comes from modelPrices (kept in your code on purpose, because prices change and vary by contract) unless you pass cost yourself.

What is sent#

One llm.call event per call, whose properties hold flag, variation, model, outcome and the numbers it has: latency_ms, ttft_ms, input_tokens, output_tokens, tokens, cost. Results read each metric in the table above from it. Prompts and answers are never sent. Feedback is its own event.

Why “average per event”#

Latency or cost must be averaged per call, not summed per user: a variation whose users chat more would otherwise look more expensive. Ratio goals compute Σ value ÷ Σ calls per variation, with the variance taken over users, so the statistics stay valid.

Try it#

Terminal
LIFTCANARY_SDK_KEY=pk_... npx tsx examples/ai-demo/simulate.ts

sends simulated traffic for two models. To see a guardrail stop a bad challenger, make it fail often:

Terminal
LIFTCANARY_SDK_KEY=pk_... FLAG=summarizer USERS=4000 SMALL_ERROR_RATE=0.15 npx tsx examples/ai-demo/simulate.ts