LLM Evaluation: How Do You Know Your AI Application Is Actually Getting Better?
Reading five good-looking outputs isn't evaluation, it's a guess with extra steps. Real metrics, golden datasets, LLM-as-judge, and a CI/CD gate that actually works.
A prompt change ships. The engineer who wrote it reads five example outputs, they look noticeably better than before, and the change goes out. Two weeks later, support tickets start climbing for a category of question nobody happened to check in that five-example review. Nothing was wrong with the engineer's judgment. The problem is that reading a handful of outputs was never actually measuring whether the system works. It was measuring whether five specific outputs looked good to one specific person, on one specific day.
The one sentence to remember
"The response looks good" is a data point. "The system works" is a measured, repeatable property, checked the same way, against a representative set of cases, every single time something changes.
This is AI engineering, not AI content: the actual metrics that separate a good-looking answer from a correct one, the methodology that catches regressions before your users do, and a real CI/CD pipeline that gates a deploy on measured quality instead of a vibe check.
Why "It Looks Good" Doesn't Generalize
The three specific ways eyeballing fails
Sample size. Five outputs tell you almost nothing about behavior across the thousands of real, varied inputs your system actually sees. Confirmation bias. The person reviewing a change is looking for the improvement they intended, which makes them the worst-positioned person to notice a regression somewhere else entirely. No baseline. Reading today's outputs in isolation tells you nothing about whether they're better, worse, or just different from what the system produced yesterday, on the exact same inputs.
None of that makes eyeballing worthless. Reading real outputs is how you develop intuition for what "good" even means in your domain, and that intuition is exactly what a well-built eval encodes into something repeatable. It just can't be the entire mechanism deciding whether a change ships.
Answer Quality: Accuracy, Faithfulness, Relevance
These three sound similar and measure genuinely different failure modes.
| Metric | What it actually asks | Fails when |
|---|---|---|
| Accuracy | Is the answer factually correct? | The model states something false, regardless of whether it's grounded in a source |
| Faithfulness | Is the answer actually supported by the retrieved context it was given? | The model adds information that isn't in the source material it was handed, even if that information happens to be true |
| Relevance | Does the answer actually address what was asked? | The answer is accurate and faithful, but answers a different question than the one the user asked |
Faithfulness has a real, computable formula
A standard approach: an LLM judge breaks the generated answer into individual atomic claims, then checks each claim against the retrieved context. Faithfulness is the number of claims actually supported by that context, divided by the total number of claims made. This turns "does this sound grounded" into a number you can track over time and compare across versions, not just an impression from one read-through.
A response can score well on any one of these and still fail the other two. A faithful, relevant answer can still be inaccurate if the retrieved source itself was wrong. An accurate, faithful answer can still miss relevance if it correctly answers a question adjacent to, but not actually, what was asked. Measuring only one gives you a false sense of coverage.
Hallucination and Toxicity
Hallucination is closely related to faithfulness, but it's worth tracking as its own signal: specifically, content the model generates that isn't grounded in anything it was actually given, invented facts, fabricated citations, details that sound plausible and specific but trace back to nothing real. A low faithfulness score is often the measurable symptom; hallucination is the underlying failure it's detecting.
Toxicity is a different axis entirely, safety rather than correctness: does the output contain harmful, offensive, or inappropriate content, regardless of whether it's factually accurate. A perfectly accurate, perfectly faithful answer can still fail a toxicity check if it's delivered in a way that shouldn't reach a user. Both are typically scored with a dedicated classifier or judge pass separate from the quality metrics above, because optimizing for accuracy and faithfulness alone gives a system no signal about either.
Tool-Call Accuracy and Retrieval Quality
For agentic systems, two more dimensions matter as much as the answer text itself, because a technically fluent final answer can still sit downstream of a real failure.
Tool-Call Accuracy
Did the agent choose the right tool, with the right arguments, in the right circumstances? This is a distinct measurement from whether the final answer reads well, and it's the same distinction explored in depth in AI Observability: a wrong tool choice and a right tool with wrong arguments are two different bugs, and an eval that only scores the final answer text will miss both, only ever seeing their downstream effect.
Retrieval Quality
For RAG-backed systems, this splits into two measurable pieces that fail independently:
| Signal | What it measures | What a drop means |
|---|---|---|
| Context precision | Are the retrieved chunks actually relevant to the query? | The retriever is pulling in noise alongside, or instead of, what actually matters |
| Context recall | Does the retrieved context contain what's actually needed to answer correctly? | The right document exists in your index but never made it into what the model saw |
A diagnostic that saves real debugging time
When faithfulness drops while context relevance holds steady, the problem is in generation, the model isn't using good context well. When context relevance drops while faithfulness on what was retrieved holds steady, the problem is in retrieval, the model is being faithful to context that was never the right context to begin with. Tracking both separately tells you which half of the system to actually go fix.
Regression Testing and Golden Datasets
A golden dataset is the fixed, curated set of test cases every version of your system gets measured against, the same way a software regression suite runs the same tests against every build. Borrowing that discipline directly from traditional software testing is what turns evaluation from a one-time check into a standing safeguard.
Each case needs more than just an input and an expected answer
A well-built golden case includes the input, an expected answer or the properties a correct answer must have, and for RAG systems, the specific documents the retriever should have returned. That last field is what lets you score retrieval and generation separately instead of only ever seeing their combined result.
Cover the cases that have actually broken before, not just the happy path
A golden dataset that only contains easy, obvious questions will pass every version and catch nothing. The highest-value cases are the ones drawn from real production traffic that previously produced a wrong or borderline answer.
Compare against a fixed baseline, not last week's baseline
Regression testing means comparing the current version against a stable, versioned baseline, so a slow, creeping decline across several small changes still gets caught, instead of each individual change looking like a negligible difference from the one right before it.
LLM-as-a-Judge
Human review doesn't scale to thousands of test cases run on every change, so a second model is typically used to grade the first one's output against a defined rubric. This works, but it needs to be built deliberately, not improvised.
Decide exactly what's being evaluated before writing a single test case
Answer quality, tool selection, retrieval, or some specific combination, stated explicitly, so the judge's rubric actually measures what you care about instead of a vague, generic sense of "good."
Source real test cases, don't only synthesize them
An eval set built entirely from invented examples tends to test what you expect to go wrong, not what actually goes wrong. Pulling real cases from production transcripts, alongside synthesized edge cases for known-risky scenarios, covers both.
Split your data, and don't grade on what you tuned against
Hold out a genuine test split you never look at while iterating. Tuning a prompt against the same cases you use to declare success measures how well you memorized the eval set, not how well the system generalizes to a real user's next question.
Treat the judge itself as something that needs calibration
An LLM judge is a model, and models have their own biases and blind spots. Measured cost and a runnable, repeatable script matter here too, an eval nobody can actually re-run cheaply and consistently quietly stops being used.
Human Evaluation
LLM-as-judge doesn't replace human review, it needs to be checked by it
Periodically score a sample of the same cases with real human reviewers and compare their judgments against the LLM judge's scores. Where they disagree consistently, that's either a rubric that needs sharpening or a genuine blind spot in the judge model. Skipping this step means trusting an automated grader that's never actually been graded itself.
Human evaluation is also where the judgments an automated rubric genuinely can't capture belong: tone, domain-specific correctness that requires real expertise, and edge cases where "correct" is a matter of professional judgment, not a fact that's checkable against a source document.
Continuous Evaluation
Evaluation doesn't stop at deploy time. The practice covered in AI Observability, scoring a sample of live production traffic and attaching that score directly to the trace it came from, is what catches drift a pre-deploy eval suite structurally cannot: a retrieval index that's grown stale, a shift in the kinds of questions real users are actually asking, a slow decline that never shows up as a single, obvious regression.
Pre-deploy evaluation answers "did this specific change make things worse." Continuous evaluation answers "is the system, as a whole, still working as well today as it was last month," which is a different question with a different failure mode, and both need their own standing process.
A Real CI/CD Pipeline for AI Quality
Two speeds are usually the right architecture
Running the full golden dataset with a full LLM-as-judge pass on every single push can get slow and expensive fast. A common, practical pattern: a fast, smaller-sample gate that blocks every pull request, and the full, comprehensive sweep running nightly against the main branch, catching anything the lighter gate's smaller sample missed before it compounds.
The quality threshold has to be a real, pre-agreed number
"Deploy if it looks about as good as before" isn't a threshold, it's the same eyeballing problem this whole article is arguing against, just moved one step later in the pipeline. Define the actual bar, faithfulness above a specific score, tool-call accuracy above a specific rate, zero tolerance on toxicity, before you're looking at a specific change's results and feeling pressure to ship it anyway.
The Bottom Line
"The response looks good" and "the system works" feel like the same claim, and treating them as interchangeable is how a genuinely reasonable-looking change quietly ships a regression nobody catches until a customer does. The fix isn't more careful reading. It's the same discipline traditional software has run for decades, a real test suite, a stable baseline, a defined threshold, applied to output that happens to be probabilistic instead of deterministic.
The test to run against your own system
Before your next prompt or model change ships, ask: is this being measured against a golden dataset, with a pre-agreed threshold, comparing against a fixed baseline, or is someone reading a few outputs and deciding they look better? If it's the second one, that's not a process gap to fix eventually. That's the actual reason the next regression will reach a real user before it reaches you.
A good-looking answer is where evaluation starts. It was never where it was supposed to end.
Written by
Chetan Yamger
Cloud Engineer · AI Automation Architect · Modern Workplace Consultant
Cloud Engineer, AI Automation Architect, and Modern Workplace Consultant based in Amsterdam, Netherlands. Specializing in scalable, secure enterprise solutions with Microsoft Azure, Intune, PowerShell, and AI-driven automation using ChatGPT, Gemini, and modern LLM technologies.
Stay in the loop.
New articles, straight to you.
Deep-dive technical articles on Intune, PowerShell, and AI — no noise, no spam.
Discussion
Share your thoughts — your email stays private
Leave a comment