Interview Guide

The model generated an answer from its training data — it never looked up the real payment record. The fix is RAG: retrieve the real transaction record and matching policy text before generating any answer, then verify the answer cites its sources. If it doesn’t cite, escalate to a human.

The flow after the fix:

1. Fetch the real payment record (with ownership check)
2. Find the matching policy text
3. Generate answer using only what was retrieved
4. Verify the answer cites those sources before returning it

If step 4 fails — escalate. Never let a model fill a gap with a guess.

This is a prompt injection attack — the user embedded instructions inside their question. The assistant passed user text directly to the model without an intent check first.

Layered defence — the intent guardrail runs BEFORE retrieval:

• Scan for: “ignore”, “override”, “pretend”, refund/cancel/escalation keywords
• If triggered → escalate immediately, no retrieval, no model call
• Only if clean → retrieve → generate → grounding check

Order matters. Check intent first (it’s cheap), then retrieve, then verify grounding. An injected hallucinated fact won’t survive the grounding check even if it slips past the intent filter.

This is broken object-level authorization (BOLA). The system fetched the transaction by ID without checking whether the requesting user owns it.

The correct pattern — ownership check is part of the query:

SELECT * FROM transactions
WHERE id = 'txn_9999'
  AND user_id = 'user_123'   ← both fields, one query

No rows → generic “not found”. Never fetch first and check after — that leaks information in the error path.

In grounded RAG for payment support, the model isn’t reasoning — it’s reading a retrieved paragraph and formatting it. That’s a reading task, not a reasoning task.

Nova Micro is ~170× cheaper than Claude Sonnet per token. For a high-volume support system answering short factual questions from retrieved chunks, that cost difference is enormous at scale.

Upgrade to Sonnet only if: the retrieved chunks have conflicting information requiring synthesis, the answer requires multi-step reasoning, or the retrieved content spans many documents that need reconciling.

The Knowledge Base wasn’t re-ingested after the policy update. The vector store still holds the old chunks — new files in S3 don’t automatically appear in the KB.

Fix: triggered sync, not manual: S3 event → Lambda → StartIngestionJob API call. New policy is live in the KB within minutes of upload, no manual step required.

Scheduled sync (EventBridge cron) works for low-frequency policy changes. Manual sync only for one-off updates in dev.

Small chunks (100 tokens) match individual sentences precisely but miss cross-paragraph answers. Large chunks (500 tokens) give context but add noise that confuses the model.

Best approach for policy documents: semantic + hierarchical chunking

• Parent chunk: entire policy section (for context)
• Child chunks: individual paragraphs (for precise matching)
• ~15% overlap between consecutive chunks catches cross-boundary sentences

Bedrock KB supports fixed-size, semantic, and hierarchical chunking — start with semantic for policy documents since paragraphs are the natural unit of a policy rule.

The demo uses a single FastAPI process with in-memory retrieval. At 10k rpm each layer needs a change:

• Compute: Lambda + API Gateway (auto-scales) or ECS Fargate with ALB
• Auth DB: Aurora Serverless v2 or DynamoDB instead of SQLite
• Retrieval: Bedrock KB stays the same — already serverless
• Model: Request provisioned throughput (on-demand has per-account RPM limits)
• Caching: ElastiCache (Redis) — cache identical question+transaction answers, TTL 5 min
• Observability: CloudWatch Logs + X-Ray to trace per-step latency

The Bedrock-specific constraint: at 10k rpm you need either a quota increase or Provisioned Throughput — a capacity commitment that guarantees throughput rate.

A grounding check verifies that the answer’s claims can be traced back to what was retrieved in the same request. It answers: “Is this claim in the retrieved context?”

A hallucination filter is a separate classifier that scores whether the model invented something — usually requiring a second LLM call to judge.

Grounding check: fast, cheap, deterministic — run it on every response. If the answer can’t cite the retrieved text, escalate. Hallucination scoring is a Pattern 04 concern once you have real traffic and reference answers to score against.

RAG and fine-tuning solve different problems. Use RAG when data changes frequently, you need citations, or knowledge must be up-to-date without retraining. Use fine-tuning when the task requires a specific style, domain vocabulary the base model misunderstands, or a stable curated dataset you can afford to retrain on.

The combined approach (production-grade): fine-tune on domain vocabulary and response style, then add RAG so the fine-tuned model reads real, current facts instead of guessing.

The key: fine-tuning can’t teach a model what happened to txn_4471 last night. Only retrieval can provide that.

“Instead of the AI guessing from memory, it looks up the real answer first — like a support agent checking the system before replying.”

For a product manager: the AI reads the actual payment record and relevant policy, then explains it. The citation tells you exactly where the answer came from — so if it’s wrong, you know which document to fix. A policy change takes effect immediately (update the doc in S3, re-ingest) — no retraining.

Avoid: “the AI reads your database” (it doesn’t query live, it reads pre-indexed chunks) and “the AI knows your data” (it knows only the chunks that matched the question).