Smarter Models Still Hallucinate: Evidence Gates for Production RAG Agents

A real e-commerce support incident made this distinction concrete. Given a district, a building name, and several weakly related passages about pickup services, the model completed the missing street address, floor, opening hours, pickup policy, and customer notification promise. Upgrading to a stronger model did not eliminate the behavior: with the factual gate disabled, the upgraded system passed only 3 of 9 targeted scenarios.

The effective fix was neither another model switch nor a blacklist for addresses, bank accounts, or brand relationships. It was a single-call Evidence Gate: the model must expose the propositions required by its answer, the direct evidence for those propositions, and anything still unresolved. The runtime then publishes only the branch selected by that structured gate.

The central lesson is simple:

Model upgrades raise the capability ceiling. Runtime evidence contracts set the factual floor.

1. Intelligence Is Not a Reliability Contract

“More intelligent” and “supported by evidence” are different properties.

Language models are exceptionally good at turning incomplete context into coherent prose. The closer the context is to a valid answer, the more natural the completion tends to sound. In high-stakes business settings, that strength can become a liability. An obviously absurd answer is easy to reject; an invented detail surrounded by a real district, a real building, and a familiar service process is much harder to detect.

A model upgrade may improve instruction following, multilingual fluency, and reasoning. It still cannot independently define three production guarantees:

1. Which sources count as factual evidence in this turn?
2. Does that evidence support the complete business proposition, rather than merely share its topic?
3. When evidence is incomplete, which part of the response—if any—is allowed to leave the runtime?

If the system does not define these guarantees, publication is effectively governed by the model’s confidence of expression. No level of model intelligence should be treated as a reliability contract.

2. The Case: Plausible Completeness Was the Failure

The user’s request was ordinary: after buying online, could the order be collected from a physical store?

The answer was extraordinarily complete. It provided a street, building number, floor, unit, opening hours, pickup policy, and a promise that the customer would be notified when the order was ready. That completeness created trust—and amplified the potential harm. A customer does not experience this as probabilistic text completion; they experience it as a statement made by the merchant.

The investigation decomposed the response into independently verifiable propositions and traced each one across the runtime instructions, business background, retrieved knowledge, tool output, and conversation history.

The most revealing runtime signal was the model’s own attribution: it labeled the source as “instructions.” The instruction merely said that basic information should follow the website and must not be altered. It did not contain the website’s content, and it did not establish any address or service policy.

This exposes a subtle but critical boundary:

• “The website is authoritative” is a source-governance rule.
• “The website explicitly states address X” is factual evidence.
• The first tells the agent where verification should occur; it cannot substitute for the verification itself.

The incident was dangerous not because the model knew nothing, but because it knew just enough to feel entitled to complete the rest.

3. How Partial Truth Becomes Complete Fabrication

The failure was not a single bad token. It was a reproducible chain of semantic substitutions.

Four mechanisms compounded in the same turn:

1. Proposition completion. The user needed an actionable answer, so the model expanded “district + building” into a postal-level address.
2. Relevance substituted for entailment. Retrieved text discussed pickup, but did not establish that this merchant supported this pickup process.
3. Cross-fragment relationship synthesis. Separate fragments mentioned a subject, location, or service concept; the model invented a relationship the sources never expressed together.
4. Self-certified evidence. The model named “instructions” as its source, while the runtime never checked whether those instructions supported the complete claim.

A response is not grounded merely because RAG returned content. Retrieval decides which material enters the context. A factual gate decides which propositions may leave the context and reach the user.

4. Why the Obvious Fixes Fail

4.1 Blacklist every sensitive fact type

Separate rules for addresses, payment accounts, brand relationships, opening hours, and business commitments quickly become an incident dictionary. Known failures are covered; the next unseen fact type passes through. Case-specific rules also consume prompt attention and turn a general reliability problem into a growing set of customer-specific exceptions.

The right abstraction is not “which words must never be generated?” It is “under what conditions may a factual proposition be published?”

4.2 Repeat “do not hallucinate”

A natural-language prohibition is still interpreted by the same probabilistic model. Unless the system distinguishes reference clues from sufficient evidence, the model may classify its completion as reasonable inference rather than fabrication.

A principle becomes dependable only when it is translated into an executable response protocol.

4.3 Upgrade the model

A stronger model can reduce some errors, but it cannot eradicate hallucination. In the targeted ablation, the upgraded model passed only 3 of 9 factual scenarios when the final Evidence Gate was disabled.

This does not mean model quality is irrelevant. It means model quality is insufficient: without a publication contract, even a stronger model may cross factual boundaries when given partial truth and distracting context.

4.4 Add a second reviewer

An independent reviewer can provide another defense layer, but it adds a serial model call. In an interactive support path, that usually creates a visible latency and cost increase. A reviewer shown the same ambiguous evidence may also reproduce the first model’s mistake.

The first intervention therefore moved evidence review inside the existing model call and assigned branch enforcement to the runtime. This preserved semantic judgment without paying for a second round trip.

5. The Design: A Single-Call Evidence Gate

The revised response contract requires six logical stages inside one model invocation.

The structured response separates semantic judgment from runtime control.

No address, account, merchant, or brand keyword participates in the decision. The model judges semantic support; the runtime validates the contract, selects the branch, and enforces failure behavior.

If parsing fails, the system retries or fails closed. The unchecked draft is never recovered as user-visible content. This is essential: if a malformed response can fall back to “return the best available text,” the Evidence Gate is only advice, not a publication boundary.

6. Complete-Proposition Grounding

Evidence sufficiency cannot stop at topical relevance. A business proposition normally contains a subject, relation, object, and material qualifiers. The evidence must support the relationship the user will actually rely on.

For example:

• “This building is in Central” does not support “this brand’s store is in unit X of this building.”
• “The knowledge base describes a pickup service” does not support “this merchant offers that service to this customer.”
• “The website should be treated as authoritative” does not support “the website has already provided the current answer.”
• Several individually true fragments cannot be combined into a business relationship that none of them explicitly states.

When a critical proposition lacks direct support, the model may query knowledge once if an appropriate tool is available. If the evidence remains insufficient, it states what cannot be confirmed and avoids inventing advice, replacement procedures, or future commitments.

A limitation response is not a failed answer. It is the user-facing expression of an evidence boundary. At the same time, evaluation must include positive cases with sufficient evidence; otherwise, the system can become safe but useless by refusing everything.

7. Validation Beyond the Incident Sentence

The regression matrix deliberately avoided overfitting to the original address question. It combined fact categories with different evidence states:

• addresses and contact details;
• payment and bank-account information;
• brand, ownership, and partnership relationships;
• service capabilities, fulfillment conditions, and business commitments;
• missing conversational context or instructions;
• missing, irrelevant, incorrect, or conflicting knowledge;
• positive cases in which complete evidence required a normal answer;
• multilingual behavior, language switching, workflow execution, and ordinary tool use.

The 3/9 ablation set and the final 60/60 matrix were different in size, so they should not be interpreted as a controlled effect-size comparison. They establish a narrower but important result: after the model upgrade, the system still produced repeatable factual failures without the gate. Component-level impact should be measured through paired ablations on the same dataset, sampling parameters, and repetition count.

8. Cost, Latency, and Operational Stability

The design does not add a second model invocation. Estimated per-turn overhead is:

• approximately 500–700 additional fixed input tokens;
• approximately 150–300 private output tokens for the draft, evidence, and gate;
• approximately 850–1,300 additional tokens in total, or 6%–10% of the average pre-production turn.

Across 49 pre-production turns, end-to-end latency was 10.42 seconds at p50, 17.67 seconds at p95, and 18.20 seconds at maximum. Pods had no restarts and the service produced no 5xx responses. In the initial production observation window, completed conversations measured 7.17 seconds at p50, 16.30 seconds at p95, and 18.07 seconds at maximum.

These figures describe the observed traffic windows rather than guaranteeing performance under every workload. They do show that the control was introduced without the step-function latency increase of a second serial model call and without changing the surrounding service architecture.

9. Relationship to Existing Methods

The design shares the “generate, then inspect” intuition of several factuality methods, but optimizes for a single invocation and an enforceable production boundary.

A single-call Evidence Gate is not a formal fact prover. The same model still produces the evidence record, while the runtime validates structure and branch selection rather than independently proving semantic entailment. Its value is pragmatic: it tightens the publication path at modest latency. Extremely high-risk workflows can add independent verification, human approval, or deterministic data interfaces on top of the same boundary.

10. Production Lessons

1. Treat every model as fallible. Model choice affects average quality; it should not change the system’s basic assumption about factual risk.
2. Make evidence an object, not a label. “Supported by instructions” is meaningless unless the relevant instruction directly supports the complete proposition.
3. Do not confuse relevance with entailment. Nearby truths can produce more convincing hallucinations than obviously unrelated content.
4. Define publication conditions, not incident keywords. Policies should state when a fact may be published, rather than enumerate fact types that must not be invented.
5. Let the model judge semantics and let the runtime enforce the contract. Each side owns a responsibility that can be observed and tested.
6. Fail closed without becoming useless. Unchecked drafts must not bypass the gate, while positive cases must detect excessive refusal.
7. Regress language and ordinary workflows. Increasing factual-safety weight can affect language adherence, procedures, tool use, and normal answers.
8. Complete the loop with observability. Gate decisions, unresolved claims, parse failures, latency, and token cost reveal whether safety is being purchased with unacceptable utility loss.

The durable boundary is larger than any single address incident:

An answer should not be publishable because it sounds plausible. It should be publishable only when trusted evidence from the current turn fully supports the material propositions the user will rely on.

References

• Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020.
• Gao et al. RARR: Researching and Revising What Language Models Say, Using Language Models, 2022.
• Madaan et al. Self-Refine: Iterative Refinement with Self-Feedback, 2023.
• Dhuliawala et al. Chain-of-Verification Reduces Hallucination in Large Language Models, 2023.
• Asai et al. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, 2023.
• Min et al. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, 2023.
• Gao et al. Enabling Large Language Models to Generate Text with Citations, 2023.
• Wei et al. Long-form Factuality in Large Language Models, 2024.