Evals and guardrails: shipping LLM features you can defend

Shipping an LLM feature without evaluations is basically pushing code to production blind. You might get away with it in a demo, but real users find the edge cases fast. If you cannot measure quality, you cannot defend the system when it inevitably misbehaves.

Building an eval dataset from scratch

Start by saving every production input and output once you go live. Before that, manually curate 100 to 200 representative examples covering your core use cases. This is not a benchmark; it is a regression suite. Expect to spend roughly 20 hours on initial curation.

Label inputs with the expected behaviour, not exact output strings. For a summarisation task, define criteria like accuracy, absence of hallucination, and format compliance. Exact match grading sets you up to fail because LLM outputs are inherently variable.

Grading methods that scale honestly

Rule-based grading works for format checks, JSON schema validation, and keyword inclusion. It catches about 30 percent of failures in typical text generation tasks. Use it as a fast, cheap gate, but do not pretend it covers semantic quality.

LLM-as-a-judge handles semantic assessment but introduces its own failure modes. GPT-4o agrees with human raters around 75 to 80 percent of the time on structured rubrics. Keep rubrics narrow, test the judge against your human-graded set, and track judge drift.

Human grading remains necessary for high-stakes or subjective outputs. Rotate reviewers, calculate inter-rater reliability, and use human scores to calibrate your automated graders. Budget for ongoing human review at roughly 5 percent of daily volume.

Guardrails as runtime enforcement

Evals tell you what broke yesterday. Guardrails stop it from breaking tomorrow. Separate input and output guardrails. Input guardrails catch prompt injections, out-of-scope queries, and PII before the model ever sees them.

Output guardrails verify schema, scan for prohibited content, and check factual claims against retrieved context. A typical setup adds 80 to 150 milliseconds of latency. That is the cost of not sending a fabricated legal citation to a client.

Structure guardrails as a pipeline, not a monolith. If the PII filter blocks an input, you do not need to run the injection classifier. Order checks from cheapest to most expensive. Short-circuit aggressively to keep p95 latency manageable.

What to measure and what to ignore

Track precision and recall of your guardrails separately. A guardrail that blocks 95 percent of bad outputs but also rejects 30 percent of good ones will destroy user trust. Aim for under 5 percent false positive rate on valid queries.

Log every override. If users can bypass a guardrail, record it. If your team disables a rule during an incident, record it. Overrides are your most valuable signal for improving the system. Ignoring them hides the real failure rate.

Do not obsess over aggregate pass rates. A 98 percent eval score means nothing if the 2 percent failures cluster in one critical user segment. Slice evals by customer tier, query type, and input length. Skewed failures are the ones that escalate.

The pragmatic takeaway

Invest in evals before you invest in prompt engineering. A solid eval suite turns prompt changes from scary guesses into measurable experiments. Pair it with runtime guardrails, and you have a system you can actually defend in a post-mortem. Start small, automate the boring checks, and keep humans in the loop for anything that matters.