Prompt Evaluation: How Do You Know If Your Prompt Is Actually Good?
Reading one better-looking answer isn't evaluation. A real testing framework compares prompt versions against a fixed dataset, with real, repeatable metrics.
"I changed my prompt and the answer looks better" is not evaluation. It's one read-through, on one example, by one person who already expected it to look better. This post is the fix: a real testing framework for comparing prompt versions against each other, the same discipline this site's LLM evaluation post applies to whole AI systems, scoped down to something one person can run against a single prompt.
The one sentence to remember
A prompt version isn't "better." It's better at a measured rate, against a fixed set of cases, compared to a specific prior version. Drop any part of that sentence and you're back to reading one output and guessing.
The Testing Framework
The order matters more than it looks
Expected results are written down before looking at what the new prompt produces. Deciding "the right answer" after seeing the output is how confirmation bias gets baked into the eval itself, you'll unconsciously grade in favor of the version you were hoping would win.
Building a Test Dataset
This doesn't need to be a data engineering project. Five to fifteen real, varied cases, written down once and reused for every future version, beats zero. The goal isn't scale, it's a fixed thing to measure against, so "better" stops being a feeling and starts being a comparison against the same yardstick every time.
Pull real cases, don't invent all of them
Take actual inputs your prompt has handled before, especially any that produced a wrong, awkward, or inconsistent answer. A dataset built entirely from imagined "nice" examples tests what you expect to go wrong, not what actually does.
Write the expected result for each one, in advance
Not the exact wording, the properties a correct answer must have: which devices should be flagged, what format the output must take, what it must not claim. This is what makes scoring possible instead of subjective.
Keep the dataset itself version-controlled and unchanged
Every future prompt version gets tested against the same file. If the dataset quietly changes between tests, you're no longer comparing prompt versions, you're comparing prompt versions against different tests, which tells you nothing.
The Metrics
Accuracy
Did the output get the actual right answer, for each case in the dataset? This is the most direct metric and also the easiest to fool yourself on with a single example, since one lucky correct answer says nothing about the other fourteen cases in the set.
Consistency
Run the identical prompt against the identical input multiple times. Do you get the same answer, or does it drift? Some drift is expected, language models are probabilistic, but a prompt whose classification flips between runs on the exact same input is not a reliable prompt, no matter how good any one of its individual answers looks.
Lower temperature helps, it doesn't fix this
Setting temperature to 0 reduces run-to-run variance but doesn't guarantee identical output, especially across model versions or providers. Consistency is something you measure across several runs, not something you assume because a setting is turned down.
Relevance
Does the output actually answer what was asked, or does it wander into general advice, caveats nobody requested, or a slightly different question than the one in the prompt? A relevant answer stays scoped to exactly what the task asked for.
Format Compliance
Does the output match the exact structure you specified, every single time, not just on the run you happened to check? This is the same discipline covered in full here: a prompt that sometimes returns a clean list and sometimes wraps it in a paragraph has a format compliance problem, even if the content in both is equally correct.
Hallucination Rate
Across the dataset, how often does the output include a claim, a value, or a detail that wasn't actually present in or derivable from the input? For a prompt working from data you supplied yourself, this is easier to check than it sounds: any output containing information that traces back to nothing in your test input counts.
Edge Cases
Ordinary inputs tell you a prompt works on the happy path. A good test dataset deliberately includes the inputs where prompts actually break: a missing field, an unusual name with special characters, a boundary value, an input the prompt was never explicitly told how to handle. The Anatomy post's Constraints ingredient is what an edge case usually exposes: a constraint you needed and didn't write down.
Regression Testing
Comparing version 2 against version 1 answers "did this specific change help." Regression testing answers a longer-running question: does version 4 still correctly handle the edge case that broke version 1, or did a later change quietly reintroduce it? This only works if every past failure that got fixed stays in the dataset permanently, so a prompt can't silently regress on a case it already learned to handle.
A Worked Comparison
How this comparison is run
This is a worked, representative comparison in the same spirit as every other worked example in this series, not a literal multi-run statistical study. It's built to show what each metric actually catches, not to claim a precise measured percentage.
Take the BitLocker compliance prompt from the Anatomy post and test two versions against a 5-device dataset that includes two deliberate edge cases: LAPTOP-1004, whose BitLocker status came back as a null value instead of on or off, and O'BRIEN-LT-05, a device name containing an apostrophe.
Prompt V1: "Write a script that checks BitLocker status and flags devices that aren't encrypted."
Prompt V2: the full anatomy-structured version, role, context, task, constraints, an example output format, all six ingredients present.
| Metric | Prompt V1 | Prompt V2 |
|---|---|---|
| Accuracy | 3 of 5 devices classified correctly | 5 of 5 |
| Consistency | Flag wording changes between runs | Same [DEVICE] - [DATE] - [STATUS] - [ACTION] line every run |
| Relevance | Adds general BitLocker background nobody asked for | Stays scoped to the flagged-device list |
| Format compliance | A paragraph one run, a bullet list the next | Matches the specified format every time |
| Hallucination rate | Once stated a "risk score" that wasn't in the input data | None, constraints restrict it to the given fields |
| Edge cases | Silently drops LAPTOP-1004's null status, and misparses the apostrophe in O'BRIEN-LT-05 once | Flags the null-status device for investigation instead of dropping it, handles the apostrophe correctly every run |
| Regression testing | First version tested, no prior baseline | Passes every case in V1's original dataset, plus both edge cases added after they broke V1 |
Reading this table is a different exercise than reading two scripts and picking the one that looks more polished. It shows exactly where V1 fails and confirms V2 doesn't just look better, it measurably handles the cases V1 didn't.
Where This Leads
The same discipline, at a larger scale
Once a prompt is one piece of a larger system, RAG retrieval feeding it context, an agent choosing when to call it, multiple prompts chained together, this same framework needs to grow with it: golden datasets in the hundreds of cases, an LLM-as-judge for cases too numerous for manual review, and a real CI/CD gate blocking a deploy on a measured quality threshold. That full version is covered here, and it's worth reading once a single prompt's test dataset stops being enough for what you're actually building.
The Bottom Line
A prompt that "looks better" and a prompt that's measurably better feel like the same claim right up until the version that looked better ships a regression nobody catches until a real case hits it. Five to fifteen real test cases, written-down expected results, and the same fixed dataset run against every future version turns "looks better" into something you can actually show.
The test to run against your own prompts
Before your next prompt change ships, ask: is there a dataset this gets tested against, or is it one read-through and a feeling? If it's the second one, that gap is exactly where the next silent regression is going to come from.
Do you keep a real test dataset for your most important prompts, or does "better" still mean "read three examples and it felt right"? That gap is worth closing before a prompt handles something that actually matters. Drop a comment with how you'd build a first test set for the prompt you use most.
Written by
Chetan Yamger
Cloud Engineer · AI Automation Architect · Modern Workplace Consultant
Cloud Engineer, AI Automation Architect, and Modern Workplace Consultant based in Amsterdam, Netherlands. Specializing in scalable, secure enterprise solutions with Microsoft Azure, Intune, PowerShell, and AI-driven automation using ChatGPT, Gemini, and modern LLM technologies.
Stay in the loop.
New articles, straight to you.
Deep-dive technical articles on Intune, PowerShell, and AI — no noise, no spam.
Discussion
Share your thoughts — your email stays private
Leave a comment