answer about a process that had changed eight months before the model’s training cutoff.
The engineer asking the question had no way to know it was wrong. That was the moment
the design question shifted from “how capable is this model” to “how do we make it safe.”
What a Language Model Cannot Do on Its Own
A large language model is trained on a fixed corpus up to a cutoff date. After that date,
it knows nothing that was not in its training data. For general knowledge this is a minor
limitation. For enterprise use — where the relevant facts are specific to your organisation,
your systems, your current state — it is a fundamental constraint.
Ask an ungrounded model about your current SLA attainment for a specific service and it
will synthesise a plausible answer from its training data about similar systems. The answer
will sound authoritative. It will not be yours. In a decision-support context — helping
an architect evaluate a change, an operations engineer diagnose an incident, or a manager
review a project status — a plausible wrong answer is worse than no answer.
The Retrieval Layer Is the Trust Mechanism
Retrieval-Augmented Generation inserts a search step between the user’s question and the
model’s response. The question is converted to an embedding. Azure AI Search runs a hybrid
search — BM25 keyword matching for exact terms combined with vector similarity for semantic
relevance — against an indexed corpus of your enterprise documents. The top results are
injected into the model’s context alongside the question. The model generates from what
it received, not from training memory.
The critical property is citability. The model does not generate from a black box — it
generates from specific retrieved chunks. Those chunks can be presented to the user
alongside the response. The user can follow a link to the source document, see the
timestamp, verify the claim. That verification path is what transforms AI from a
productivity tool into a trust-warranted decision-support system.
All enterprise AI assistants use RAG with Azure AI Search as the retrieval layer, refreshed from verified data stores on a defined schedule. Models generate only from retrieved context. Every response links to source documents. No model is used for enterprise reasoning without a retrieval grounding layer.
The Retrieval Configuration That Actually Matters
Chunk size is the first decision that separates working RAG from broken RAG. Chunks that
are too large dilute retrieval relevance — a 3,000-token chunk that happens to contain the
right sentence will score lower than a 300-token chunk that is entirely about the question.
Chunks that are too small lose surrounding context and produce responses that are
technically grounded but semantically incomplete.
In practice: 400–600 tokens per chunk with a 10% overlap for continuity. Domain-specific
documents — technical specifications, architecture decision records, runbooks — warrant
tighter chunks than narrative documents like project reports. Tune this with actual
test queries before indexing production content.
The semantic ranker is not optional for enterprise use. BM25 alone misses queries where
the user’s vocabulary differs from the document’s vocabulary. Vector search alone over-
retrieves on semantically similar but contextually irrelevant documents. The semantic
ranker re-scores the combined candidate set using a cross-encoder model. On every
domain-specific evaluation set I have tested, hybrid search with semantic ranking
outperforms either method alone by a margin that is meaningful in production.
Hallucination as a Governance Problem
Hallucination is not a model quality problem. It is a governance problem. A grounded model that generates outside its retrieved context because the groundedness check was not implemented is an engineering failure, not a model failure. The model does what models do. The architecture is responsible for containing it.
Every AI assistant response passes through a groundedness evaluation before delivery.
A secondary inference call checks whether the claims in the response are supported by the
retrieved chunks. Below a defined confidence threshold, the response is replaced: “I cannot
provide a confident answer from the available data. Please refer to the source documentation
directly.” That fallback is the correct answer when the data does not support a confident
response. It is not a failure mode. It is a feature.