The eval score was honest. Nothing broke. So what actually stops this at request time?
Reviewed by Senior AI Engineer / Architect (LLM Evaluation & Guardrails)
The eval score was honest. Nothing broke. So what actually stops this at request time?
A banking agent scores 9/10 on every evaluation case you wrote, including the entitlement cases. It ships. A real customer asks for their balance, and the agent reads out a stranger's.
1. Prompt evaluation is unit testing. It runs in your build, not on the call.
An eval harness generates test cases, runs the prompt under test, grades the output by model and by code, and averages to one number. That number gates a deploy. It never executes while a customer is mid-conversation, the same way a green test suite never runs again once the build it validated has shipped.
2. The system prompt is on the list of controls, and it enforces nothing.
A system prompt can ask a model never to invent a balance or disclose another customer's account. It cannot make that refusal happen, because it has no enforcement mechanism, only wording in the same channel an attacker writes in. Every other layer in the pipeline can return a hard no. The system prompt can only make a request the model is free to decline.
3. The LLM is never the authorisation boundary.
If access to an account is a sentence in the system prompt, there is no access control, only a polite request that can be argued with. Entitlement has to be a deterministic check against the account store, before the model is called, the same way you would never implement an access group check as free text in a flow note.
4. A grounding check is twenty lines of code, not a hope.
Every figure a model states has to appear in the context it was given. That check runs after the model replies, compares numbers against the source data, and blocks or reroutes the response if one doesn't match. 'Don't hallucinate' is an instruction. A grounding check is the thing that actually catches an invented balance.
5. Seven layers run at request time; only six of them can say no.
PII masking, injection detection, and entitlement run before the model sees the message. The system prompt runs in the middle and only asks. Grounding and output policy run after the model replies. Stack them in order and you get a trace that names exactly which layer caught which failure, instead of a single number that tells you nothing about what would have stopped it.
6. The two systems close a loop; neither replaces the other.
A blocked request in production, or a customer complaint about a wrong figure, is a real case. Add it to the evaluation dataset and the next prompt or model change gets scored against every attack that has actually worked. Runtime produces the evidence. Evaluation is what makes that evidence count before the next deploy.
Lesson
Prompt evaluation is unit testing. It runs in your build and decides what ships. It never runs while a customer is talking to your agent.
Book a Discovery Call
Verdict table
10 of 10 checked claims verified. Eval-harness and eval-statistics claims against Anthropic's evaluation documentation and research; the Converse API mechanics, sensitive-information filters and contextual grounding check against AWS Bedrock documentation; the prompt-injection taxonomy and the authorisation-boundary claim against the OWASP Top 10 for LLM Applications (2025). No claim required correction. Full verdict table in research.md.
| Claim | Source |
|---|---|
| An evaluation harness combines code-based, model-based and human-SME grading into a score that gates a release | https://platform.claude.com/docs/en/test-and-evaluate/develop-tests |
| Eval datasets should draw from real production failures, not only model-generated cases, because synthetic attacks have the generating model's own blind spots | https://arize.com/blog/ai-agent-evaluation-how-to-build-evals-you-can-trust/ |
| Direct prompt injection is malicious instructions typed by the user; indirect prompt injection arrives via documents or tool output the model later reads | https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-v2025.pdf |
| Authorization for an LLM-driven system must be enforced by downstream code, not by the model's own output or wording | https://github.com/OWASP/www-project-top-10-for-large-language-model-applications/blob/main/2_0_vulns/LLM06_ExcessiveAgency.md |
| A contextual grounding check flags any information in a model response not present in the supplied source as ungrounded, and can block low-confidence responses | https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-contextual-grounding-check.html |
| Sensitive information filters detect and block or anonymise 50+ PII entity types on input or output, independent of the model's own behaviour | https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-sensitive-filters.html |
The Bedrock Converse API (client.converse(modelId=..., messages=...)) is the unified chat-completion interface used across Claude and other Bedrock-hosted models |
https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-examples.html |
| Model evaluation scores carry sampling variance; small score deltas should be treated as noise and only larger drops as signal | https://www.anthropic.com/research/statistical-approach-to-model-evals |