AI Architecture Pattern 04: Evaluator Optimizer

The answer was valid. It was also wrong.

Pattern 03 gave us schema enforcement — every response field is correctly typed, enum values are constrained during generation, a schema violation surfaces immediately with the exact field and reason. That was the right problem to solve.

But it doesn’t catch this:

“Your refund of £49.99 will be processed within 3-5 business days and credited back to your card immediately.”

That answer is well-typed. The status field is a valid enum. The JSON is parseable. The refund policy says 5-10 business days. “Immediately” contradicts it. And nothing in Pattern 03 checks whether the answer is true — only whether it’s well-formed.

Mike waited. The refund didn’t arrive on Friday. He called support. He was angry. He filed a complaint.

The complaint wasn’t about format. It was about a confident, wrong answer that reached him without any check on its content.

[!NOTE] This post is part of a continuing 16-pattern series: a payment-support assistant at a fictional bank, starting simple and accumulating exactly the complexity a real one would, in the order a real one would need it. Read the series overview or clone the repo and run python -m launcher.main from the repo root to browse the whole series locally.

Why This Pattern Exists

The previous three patterns built a grounded, contract-enforced, schema-valid assistant. Each one fixed a real gap:

1. Pattern 01 — the model answers from its own training data (wrong). Fix: retrieve trusted policy and transaction context first.
2. Pattern 02 — a second team sends different field names, the prompt silently receives bad data. Fix: a validated schema contract per caller.
3. Pattern 03 — the model generates a free-text status that doesn’t match the routing queue’s enum. Fix: constrained decoding enforces the exact value during generation.

What’s left: the model could still produce a schema-valid, well-grounded-sounding answer that contradicts the policy it was given. Pattern 03 can’t catch it — it checks shape, not content.

Pattern 04’s job: add an independent model call that judges the draft against the policy context, catches the contradiction, and supplies a correction hint before the answer is delivered.

The Pipeline

Three Converse calls, not one:

Customer Question + Policy Context + Transaction
        │
        ▼
[1] Generate ─────────────────── Converse call #1
        │  draft answer
        ▼
[2] Judge ───────────────────── Converse call #2
        │  {verdict, confidence, reason}
        ▼
[3] Gate ─────────── PASS → deliver
        │ FAIL
        ▼
[4] Rewrite (with judge reason) ─ Converse call #3
        │
        ▼
[5] Deliver + Audit Trail

The key property: the judge is always a separate call. If the same call generated and judged the draft, it would defend its own output — a documented failure mode in LLM-as-a-judge research. Separation is what makes the judge independent.

The correction hint: the judge returns a reason string — “Draft states refund will arrive by Friday — contradicts the 5-10 business day policy.” That reason is injected into the rewrite call’s system prompt. A rewrite that knows what was wrong is meaningfully better than a blind retry. This is the key architectural property the inline judge has that Bedrock Guardrails contextual grounding doesn’t: Guardrails returns block or allow with no reason field.

What the Demo Shows

Four scenarios, each teaching a different outcome:

The demo’s left panel shows the generator’s original draft. The right panel shows the judge’s verdict, confidence score, reason, and the final answer Mike actually received. The “↺ Regenerated” pill appears when a rewrite happened.

Design Decision: Why Not Bedrock Model Evaluation?

The ROADMAP.md entry for this pattern said “Bedrock Model Evaluation (LLM-as-a-Judge)” — that’s what I started with. Step 2 research changed the answer.

Bedrock Model Evaluation is a managed batch service. You submit a CreateEvaluationJob, it runs offline, and the results land in S3. It’s excellent for auditing a corpus of historical responses — testing a new model version, weekly quality sampling, regulatory reporting. But it cannot block delivery in real time. An evaluation job can’t gate a single response that’s about to go to Mike right now.

The inline judge is plain application code: two sequential Converse calls in the same request lifecycle. No managed service required. Simpler, cheaper per-request, and actually capable of doing the job.

The right pick depends on what problem you’re solving. This pattern solves per-request quality gating with a targeted rewrite. That’s Option A.

The Audit Trail

Every response from run_evaluator_loop() includes a full audit trail:

{
  "answer": "...",         # final delivered answer
  "evaluation": {          # judge verdict
    "verdict": "fail",
    "confidence": 0.15,
    "reason": "Draft states refund will arrive 'by Friday'..."
  },
  "audit": [
    {"stage": "generate",  "output": "<original draft>"},
    {"stage": "evaluate",  "verdict": "fail", "confidence": 0.15, "reason": "..."},
    {"stage": "rewrite",   "output": "<corrected answer>"},
  ]
}

For a payment-support system, this is the compliance record: proof that the draft was caught before delivery, what specifically was wrong, and what was actually sent to Mike. In a dispute, that audit trail is evidence.

On AWS

Replace the local mock client with a real boto3.client("bedrock-runtime", region_name="us-east-1"). The converse() call shape is identical — the mock was built to mirror the real API exactly. Both the generator and judge use amazon.nova-micro-v1:0: $0.035/1M input + $0.14/1M output tokens.

A full loop — generate + judge + rewrite — uses roughly 800 tokens. That’s ~$0.00004 per request. At 10,000 requests a day, the judge costs about $0.40/day. For the protection it provides against wrong-but-confident answers in a financial context, that’s effectively free.

What I Learned

The biggest correction in building this pattern: Bedrock Model Evaluation is not the inline judge. The ROADMAP described it as the LLM-as-a-Judge service, which it technically is — but only in an offline batch context. Discovering that forced me to read the actual Converse API docs more carefully and arrive at the cleaner, simpler, more honest answer: two Converse calls, synchronous, no managed service needed.

The second correction: Bedrock Guardrails contextual grounding is explicitly documented as not supporting conversational QA. I checked before assuming, found the limitation documented clearly, and that’s why it’s Option C (not built) in the design comparison rather than a surprise during implementation.

More Than One Way to Build This

Is there an open-source tool that does what this pattern just built? The closest real match is promptfoo, a CI/testing tool that lets one Bedrock model grade another call’s output using a rubric — genuinely the same “one model judges another” idea, verified straight from promptfoo’s own docs. But it’s a test-suite construct, YAML config in a CI pipeline, not something a live request path calls synchronously.

Here’s the honest gap worth naming directly: no AWS-published resource — not a blog post, not a workshop, not a samples repo — describes this pattern’s actual shape: one synchronous, per-request, two-Bedrock-call generate-then-judge-then-rewrite loop. Everything AWS publishes in this space, including RAGAS integration, is batch or offline. So this design isn’t following an established AWS pattern — it’s a genuine, distinct architectural choice, worth saying plainly rather than implying prior art that doesn’t exist.

The Wider Evaluator Field: 9 Real Approaches, We Build 1

Option A/B/C above is the design decision — inline judge vs. the two managed alternatives. Underneath that sits a wider field of ways to structure the judge itself, worth an honest map even though this pattern only builds the simplest one.

Why single inline judge and not multi-criteria or chain-of-thought: this pattern’s job is teaching the core mechanic — an independent second call gates delivery — not building the most sophisticated judge possible. Multi-criteria and chain-of-thought are real, natural next steps once the basic gate exists, not competitors to it.

A documented risk worth surfacing plainly: self-reflection (#6) — asking the generator to grade its own output in the same call — is the cheapest option on this list, and also the one this pattern’s whole architecture exists to avoid. A model defending its own output shows measurable confirmation bias in the research; that’s exactly why the judge here is always a separate call, even using the same underlying model.

When to use this pattern

[!TIP] Use an inline judge when:

1. Compliance demands a per-request grounding check, not a periodic audit.
2. A real-time response is required — the check has to happen before delivery, not after.
3. A correction path is needed when the judge fails, not just a flag.
4. The cost of a confident, wrong answer is high enough to justify a second model call.

When not to use this pattern alone

[!WARNING] Do not use this pattern by itself when:

1. You need offline auditing across a corpus of historical responses, not a per-request gate — that’s a batch job (Bedrock Model Evaluation), not a synchronous judge on every request.
2. It’s single-turn RAG grounding over one retrieved document — Bedrock Guardrails’ contextual grounding check is simpler, though it doesn’t support conversational QA and has no reason field for a rewrite loop.
3. Sub-second latency matters more than a grounding check — two or three sequential model calls roughly doubles response time.

Who’s using this in the real world

Catching a confident, well-formed, wrong answer with a second independent model call is a real production technique, not a hypothetical.

Datadog’s own engineering blog describes a production LLM-as-a-judge system for detecting hallucinations in RAG applications — the same independent-scoring idea this pattern builds, running in production today. (datadoghq.com/blog/ai/llm-hallucination-detection)

Google DeepMind published FACTS Grounding, an official benchmark that checks whether a model’s answer is actually grounded in a source document — using multiple independent LLM judges, deliberately mixed model families so no judge just favors its own family’s output. (deepmind.google/blog/facts-grounding-a-new-benchmark-for-evaluating-the-factuality-of-large-language-models)

This isn’t just a house opinion, either. Peer-reviewed research measured the self-preference risk directly: Panickssery, Bowman & Feng found GPT-4 was 73.5% accurate at recognizing its own writing versus other models’ or humans’ — and, more importantly, that self-recognition ability correlates with self-preference bias. The better a model is at telling its own output apart, the more it favors that output when scoring, even when human judges rate it no better than the alternative. That’s the concrete, measured version of the reason this pattern never lets the generator grade its own draft. (Panickssery, Bowman & Feng, “LLM Evaluators Recognize and Favor Their Own Generations,” NeurIPS 2024, arXiv:2404.13076)

Further reading

• Research: Panickssery, Bowman & Feng — “LLM Evaluators Recognize and Favor Their Own Generations,” NeurIPS 2024 (arXiv:2404.13076) · Liu, Iter, Xu, Wang, Xu, Zhu — “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” (arXiv:2303.16634)
• AWS docs: Converse API · Bedrock Model Evaluation — judge model · Bedrock Guardrails — contextual grounding check · Amazon A2I — human review loops (closing 2026-07-30)
• Industry blogs: Google DeepMind — FACTS Grounding · Datadog — LLM hallucination detection · AWS ML blog — Bedrock Agents + RAGAS
• Tools: promptfoo — AWS Bedrock provider · evjang.com — Self-Reflection in LLMs (Jang, 2023)

What’s Next

Pattern 04 catches wrong answers before they reach the customer. It still only knows what it was given — if the policy context was stale or the transaction data was incomplete, both generator and judge are working from wrong information. Pattern 05: Tool Calling solves this: the model requests a real-time data lookup instead of reasoning over whatever context was passed in.

[!TIP] Star the GitHub repo and follow along with the series.

response = client.converse(
    modelId="amazon.nova-micro-v1:0",
    system=[{"text": (
        "You are a grounding judge. Evaluate if the draft contradicts the policy. "
        'Respond with ONLY valid JSON: {"verdict": "pass"|"fail", "confidence": <float>, "reason": "<string>"}'
    )}],
    messages=[{"role": "user", "content": [{"text": (
        f"Policy:\n{context}\n\nQuestion: {question}\n\nDraft:\n{draft}"
    )}]}],
    inferenceConfig={"maxTokens": 256, "temperature": 0.0},
)
judge_text = response["output"]["message"]["content"][0]["text"]
verdict = json.loads(judge_text)

[
  {"stage": "generate",  "output": "<draft text>"},
  {"stage": "evaluate",  "verdict": "fail", "confidence": 0.15, "reason": "..."},
  {"stage": "rewrite",   "output": "<corrected text>"},   # only if fail
]

Have questions or feedback? Drop a comment below or connect on LinkedIn.