Grounding agents in your own documents

Retrieval before generation

When a caller asks a question, the agent retrieves the relevant passages from your uploaded documents, help center, and policy pages, then answers from those passages only. The source is attached to the transcript so you can audit any answer after the fact.

The ordering is the whole mechanism. The model is not asked what it knows and then fact-checked; it is handed the relevant material and asked to answer from that. What it happens to have absorbed in training about refund policies in general never enters the picture, because it was never asked.

That auditability is not a nice-to-have either. The first time somebody disputes what your agent told them, the useful thing to have is not a recording of the answer — it is the passage the answer came from, so you can tell in ten seconds whether the agent misread your policy or your policy says something you didn't realise.

Retrieval quality is a content problem

Almost every grounding failure we investigate turns out not to be a retrieval bug. The passage was found correctly; the passage was just bad. Documentation written for humans leans on context the reader supplies — the page it sits on, the heading above it, the paragraph before — and a retrieved fragment arrives without any of that.

The patterns that cause trouble:

• Pronouns with no antecedent in the chunk. "This must be submitted within 14 days" is useless when "this" was named two headings ago.
• Conditions separated from the rule they qualify. The exception on the next page never gets retrieved alongside the rule it modifies.
• Two documents that disagree, usually because one is out of date and nobody deleted it. Retrieval will faithfully surface both.
• Tables converted to text, where the row and column headers stop being attached to the values.
• Internal shorthand a customer would never use, so the caller's phrasing never matches the document's.

Getting a knowledge base ready for an agent is mostly editing, not uploading. It is also the single highest-return work available, and it improves the pages for humans too.

The honest fallback

When retrieval comes back empty, the correct behaviour is not a plausible guess. The agent says it doesn't have that information and offers a handoff to someone who does.

Teams consistently underestimate how much trust this buys. Callers forgive "let me get someone who can answer that" far more readily than a confident wrong answer. The admission also gives them something actionable, which a wrong answer does not.

Designing the fallback badly undoes it, though. "I don't have that information" as a full stop is a dead end that leaves the caller exactly where they started. The fallback has to carry a next step: a transfer, a callback with a time, a note that a human will pick up with the context already attached. What makes the honesty tolerable is that something happens next.

Empty retrieval and wrong retrieval are different failures

It is worth separating two cases that get lumped together. Empty retrieval — nothing relevant found — is the easy one, because it is detectable and the fallback handles it.

The harder case is confident retrieval of the wrong passage. Something scored well, so the agent answers from it, and the answer is fluent, sourced and wrong. There is no signal at answer time that anything went astray. This is where relevance thresholds earn their keep: it is better to fall through to the honest fallback on a marginal match than to answer from a passage that merely resembles the question.

The practical test is to take the questions your agent answered with the lowest retrieval scores and read those answers by hand. That set is small, and it is where the wrong ones are.

What to expect when you connect a knowledge base

Two things happen fairly reliably in the first fortnight.

First, the agent surfaces contradictions in your own documentation that nobody had noticed, because no human had ever read those two pages within a minute of each other. This is genuinely useful and mildly embarrassing every time.

Second, you discover the questions your documentation never answered at all. Callers ask them constantly; your team has always answered from memory. Those answers live in people's heads and nowhere else, and the empty-retrieval log is the first written inventory of them you will ever have. Working through that list is the fastest way to improve both the agent and the help centre at once.