Evaluating LLM applications in production

LLM quality is not a single score

Production LLM evaluation starts with a practical admission: quality depends on the workflow. A good answer for a support assistant, a coding assistant, a policy search tool, and an internal summarizer are not measured the same way. Some workflows need precise grounding. Some need conservative refusal behavior. Some need speed. Some need traceability. A single aggregate score hides those trade-offs and makes release decisions look simpler than they are.

The useful question is not "is the model good?" The useful question is "is this application good enough for this workflow, with these users, these data boundaries, and these failure costs?" That moves evaluation from model comparison into production engineering.

What usually goes wrong

The first failure is evaluating only happy-path examples. Teams ask a few expected questions, see fluent answers, and move on. Real users then ask ambiguous, incomplete, adversarial, stale, or permission-sensitive questions. The system behaves differently because the evaluation set did not represent the risk surface.

The second failure is mixing failure types. A grounded answer can still be unhelpful. A helpful answer can still violate policy. A safe refusal can still frustrate users if the system should have had enough context. Production evaluation has to separate these dimensions so the team knows what to fix.

Start with use-case-specific examples

A useful evaluation set starts with real tasks. What are users trying to decide, write, find, summarize, approve, or change? For each task, collect examples that represent normal work, difficult work, and unacceptable failure. The examples should include the input, expected behavior, relevant context, and the reason the case matters.

In enterprise settings, I like to tag examples by workflow, risk, data dependency, and expected behavior. A small, well-curated set is more valuable than a large set nobody understands. The team should know why each example exists.

I also separate examples that are release blockers from examples that are learning signals. A release blocker protects a boundary the product must not cross, such as exposing restricted context or giving an answer where the correct behavior is to refuse. A learning signal is still valuable, but it may feed prioritization rather than stop a release. This distinction keeps evaluation useful when teams are moving quickly.

Separate correctness, usefulness, and risk

Correctness asks whether the answer is factually and procedurally right. Usefulness asks whether the answer helps the user move forward. Risk asks whether the answer exposes data, violates policy, overstates confidence, or encourages a harmful action. These dimensions can disagree, so the evaluation framework should score or review them separately.

That separation keeps evaluation useful. Model behavior needs measurement, reviewers need room to mark ambiguous cases, source data has to prove grounding and permissions, and application code has to turn the policy into repeatable release checks.

Regression tests matter after every prompt or model change

LLM applications change even when the surrounding product looks stable. A prompt adjustment, a retrieval change, a model version change, a policy update, or a new tool can improve one class of answers and weaken another. Regression tests protect the workflows that matter most.

Regression coverage should include examples that previously failed, examples that protect high-value behavior, and examples that represent policy boundaries. The point is not to freeze the product. The point is to make change visible before users discover the regression.

Regression results should be reviewed like other engineering signals. If a model change improves summaries but weakens grounded answers, the team needs a product decision, not only a technical score. Sometimes the right answer is to split workflows, add routing, change instructions, or keep a previous model for a narrow path.

Human review is still part of evaluation

Automated checks can catch many problems, but they cannot replace human judgment for usefulness, tone, workflow fit, or nuanced risk. Human review should be structured. Reviewers need clear criteria, examples of unacceptable answers, and a way to record why an answer passed or failed.

The leadership implication is that evaluation is not owned only by an ML engineer. It is a team capability spanning product, domain experts, engineering, risk, and operations. Seniority shows up in making those responsibilities explicit.

Production monitoring closes the loop

Offline evaluation is necessary but incomplete. Production monitoring should show usage, latency, cost, refusal rates, missing-answer rates, user feedback, incident signals, and drift in input patterns. The team should regularly compare production behavior against the evaluation set and add new cases when real users expose gaps.

A useful feedback loop protects sensitive content. Teams can collect enough signal to improve the system without copying private prompts and answers into broad-access tools.

Monitoring also protects leadership from false confidence. Usage growth does not prove quality. Low complaint volume does not prove trust. A production review should connect telemetry, evaluation examples, support signals, and qualitative feedback so the team can decide what to improve next.