From Production Traces to Regression Tests: A Practical LLM Quality Loop

Teams rarely lack examples of an LLM application failing. The examples are already sitting in production traces: the support answer that cited the wrong paragraph, the extraction call that returned valid JSON with the wrong field, or the agent that took six tool calls to complete a two-step task.

The hard part is turning those incidents into a repeatable improvement process.

A useful quality loop has four stages:

1. observe a failure in production;
2. preserve it as a test case;
3. compare a candidate change against the current version;
4. promote the change only when the evidence supports it.

This article shows how to build that loop without reducing quality to a single score or waiting for a large “perfect” benchmark.

Why dashboards alone do not prevent regressions

Observability answers questions about what already happened. It can show that latency increased, token use spiked, or a user gave an answer a thumbs-down. Those signals are essential, but a dashboard does not automatically protect the next release.

Suppose a customer-support assistant incorrectly treats a cancellation request as a billing dispute. An engineer can repair the prompt and verify that one conversation manually. A week later, another prompt change improves refund answers but reintroduces the cancellation mistake.

The production incident was investigated, but it never became a durable test.

The missing connection is a path from trace to test set. Agenta’s workflow is designed around that connection: teams can inspect traces, annotate them, add useful cases to test sets, and evaluate prompt or model changes against those cases before deployment. The platform combines prompt management, evaluation, and observability so the artifacts remain linked rather than living in separate spreadsheets and dashboards.

Step 1: Capture failures with enough context

Do not save only the final bad answer. A useful regression case needs the inputs and constraints that made the answer bad.

For a RAG application, capture at least:

• the user query;
• the retrieved passages and their identifiers;
• the prompt or agent version;
• the model and relevant parameters;
• the generated answer;
• latency, token use, and tool calls;
• user feedback or the reviewer’s reason for rejecting the answer.

For an agent, also capture intermediate steps. A correct final answer can hide a serious process failure: the agent may have called an expensive tool repeatedly, ignored a permission boundary, or recovered from an avoidable error by chance.

This is why evaluating a full trace can be more informative than grading only the last message.

Step 2: Convert incidents into testable expectations

Raw production traffic is not yet a test set. Each selected trace needs an expectation that an evaluator can apply consistently.

Different failures require different evaluators:

Avoid asking one LLM judge whether an answer is simply “good.” A broad question produces a broad, unstable signal. Instead, turn the reviewer’s complaint into a narrow rubric.

For example:

PASS if the answer distinguishes a cancellation request from a billing
dispute, gives only the cancellation steps supported by the supplied policy,
and does not claim that a refund has already been issued.

FAIL if any of those conditions is violated.

This rubric explains what matters, what evidence is available, and what would count as a regression.

Step 3: Start with a small failure-driven test set

Teams often postpone evaluation because they believe they need hundreds of expertly labeled examples. A smaller set of real failures is usually more valuable than a large set of generic synthetic prompts.

Begin with 20–50 cases drawn from production:

• frequent user journeys;
• high-impact failures;
• edge cases that previously caused incidents;
• examples where reviewers disagree;
• cases that stress cost, latency, or tool use.

Tag each case by capability and risk. Useful tags might include cancellation, citation, structured-output, tool-selection, and high-risk-policy. Tags make it possible to see whether a candidate improves the average while damaging one important slice.

The test set should evolve. When a new production failure is diagnosed, add it. When an example no longer represents the product, retire or rewrite it with a recorded reason.

Step 4: Compare the candidate with the current version

Every release decision should compare at least two configurations:

• baseline: the prompt, model, and parameters currently in production;
• candidate: the proposed change.

Run both on the same test set. Store outputs and evaluator results side by side. Prompt versioning matters here because a result is meaningful only when it can be traced to the exact configuration that produced it.

Review the comparison at three levels.

1. Overall results

Did the candidate improve the target metric? Did cost or latency change materially?

2. Slice results

Did one capability improve while another regressed? A two-point average gain may be unacceptable if every high-risk policy case became worse.

3. Case-level differences

Inspect disagreements between baseline and candidate, particularly when evaluators conflict. A deterministic check and an LLM judge may be measuring different aspects of quality; the disagreement is information, not noise to hide.

Step 5: Use explicit promotion gates

“The examples look better” is not a release gate. Define the rule before reviewing the candidate.

A practical gate might require:

- no regression on deterministic safety and schema checks;
- no new failures in the high-risk-policy slice;
- at least 5% improvement on the target rubric;
- p95 latency increase below 10%;
- estimated cost per request increase below $0.002;
- manual review of every baseline/candidate disagreement in the critical slice.

The exact thresholds depend on the application. The important property is that the rule is visible and repeatable.

Prompt environments can then mirror the software release path: experiment on a branch or variant, evaluate it, deploy to staging, and finally promote the approved version to production. Agenta supports version history and environment-based prompt deployment, which keeps the tested configuration tied to the released one.

Step 6: Close the loop after deployment

Offline evaluation is a filter, not a guarantee. Production inputs change, model providers update behavior, and real users find combinations the test set never anticipated.

After promotion:

1. trace the new version separately from the baseline;
2. monitor the same quality, cost, and latency dimensions used in the gate;
3. sample low-confidence and high-impact traces for human review;
4. convert new failure modes into test cases;
5. roll back when guardrail metrics cross a defined threshold.

This creates a compounding asset. Each incident makes the regression suite more representative, and each release becomes easier to evaluate than the previous one.

Common mistakes

Treating user feedback as ground truth

A thumbs-down is a useful trigger for investigation, not a complete label. The answer may be factually correct but unhelpfully formatted, or the user may dislike a policy the assistant accurately described. Preserve the signal and add a reviewer explanation.

Optimizing only the mean score

Average quality can rise while a critical slice collapses. Always report important capabilities and risk groups separately.

Changing the prompt, model, and retrieval system together

If every component changes, the comparison cannot explain why behavior moved. Keep experiments scoped when possible, or use a factorial design when interactions are the actual subject of the test.

Letting the test set become static

A benchmark that never absorbs production failures eventually measures yesterday’s product. Assign ownership for adding, reviewing, and retiring cases.

The operational principle

Reliable LLM development is not a sequence of isolated prompt edits. It is a feedback system:

production trace → reviewed failure → test case → controlled comparison
→ promotion gate → monitored release → new production evidence

Once traces, versions, evaluations, and deployments share that loop, the team can move faster without relying on intuition alone. The goal is not to eliminate judgment. It is to give judgment durable evidence and make every discovered failure harder to repeat.