A Knowledge Assistant You Can Trust: Beyond Basic RAG

Connecting a language model to company documents is easy to demonstrate. Making the resulting assistant dependable is an operating-design problem.

A useful assistant must know more than how to retrieve similar text. It must respect permissions, distinguish current policy from obsolete material, handle conflicting sources, plan across several systems, cite the evidence behind its claims and refuse when the available evidence is insufficient.

The working artifact is a Knowledge Contract: a compact agreement between source owners, system designers and users about what knowledge can be used, what an answer must show and what happens when the evidence is weak.

Retrieval is necessary, but it is not governance

Basic retrieval-augmented generation usually follows one step: search a collection, place relevant passages in the model context, and ask for an answer.

That pattern works for narrow, well-curated collections. It becomes fragile when a business question requires:

• customer data from one system and policy from another;
• a current procedure rather than the most semantically similar document;
• different access rights for different employees;
• reconciliation of conflicting records;
• evidence that is scattered across several steps;
• a calculation or tool call after retrieval;
• a decision about whether enough evidence exists to answer.

Google Research describes this frontier as agentic RAG: the system plans, retrieves iteratively and reasons across sources instead of relying on one retrieve-and-generate pass. In its reported tests, the approach improved factuality by as much as 34% on selected datasets. That is promising evidence, not a universal guarantee. The operational controls around the retrieval process still determine whether the assistant can be trusted in a particular organisation.

Classify the question before designing the assistant

Not every knowledge request needs an agent.

Complexity should be earned by the question. A direct lookup does not need an autonomous planner. A multi-hop question should not be forced through a single similarity search.

The Knowledge Contract

The contract should exist before documents are embedded or connected.

Source rules

For every source, record:

• owner and system of record;
• intended audience and access policy;
• authority relative to other sources;
• effective date, review date and expiry rule;
• which document or policy it supersedes;
• ingestion and update frequency;
• permitted transformations and retention;
• expected citation granularity.

NIST's AI Risk Management Framework playbook recommends documenting data provenance, origins, transformations, dependencies and constraints, including the risks of third-party and obsolete data. For a knowledge assistant, these are runtime design inputs, not documentation added after launch.

Answer rules

Define what every answer must contain:

• a direct response to the question;
• citations close to the claims they support;
• the effective date or version when it matters;
• a visible distinction between sourced fact and system inference;
• unresolved source conflict;
• missing evidence and uncertainty;
• the next safe action when the system cannot answer.

Refusal rules

Refusal is a product behaviour, not a crash. Specify when the assistant should stop:

• no permitted authoritative source is available;
• the sources disagree and no precedence rule resolves the conflict;
• the evidence is stale for the decision;
• the question asks for a consequential judgement outside the assistant's authority;
• the request attempts to cross the user's access boundary;
• a required system or verification step is unavailable.

Preserve source identity

Normalising content can improve retrieval, but it should not erase where a claim came from.

Each retrievable unit should retain source ID, owner, title, version, effective date, access label, section and stable link. If the content was transformed, store the relationship between the original and the indexed form.

This allows the system to answer operational questions:

• Was this source permitted for this user?
• Was it effective when the answer was produced?
• Which source won when two documents disagreed?
• Can a reviewer open the exact supporting section?
• Which answers must be rechecked after the source changes?

A citation that points only to a long document title is not enough for verification. Prefer claim-level links to the smallest meaningful section.

Enforce access during retrieval

Do not retrieve broadly and ask the language model to hide unauthorised material. The access boundary should filter candidates before protected content enters the model context.

Use the authenticated user's identity and groups at query time. Apply document, record and, when necessary, field-level permissions. Keep the assistant's service identity narrow and log which sources were considered and selected.

Test negative cases explicitly: a user asks indirectly for another team's records, a permitted document links to a restricted attachment, or a retrieved summary contains information copied from an inaccessible source.

Check freshness and authority before relevance

Semantic similarity can rank an obsolete procedure above the current policy. The retrieval system therefore needs deterministic filters and precedence rules.

A practical order is:

1. verify identity and permission;
2. filter expired or inapplicable material;
3. select the authoritative source class;
4. retrieve relevant evidence;
5. detect conflict and missing coverage;
6. plan another retrieval step if required;
7. answer, escalate or refuse.

Authority and freshness are not properties a language model should infer from writing style. Store them as structured metadata owned by the business.

Make multi-step retrieval inspectable

For a complex request, keep a short evidence trace:

• the question as understood;
• subquestions created by the planner;
• sources queried for each subquestion;
• evidence accepted or rejected and why;
• calculations or tool results;
• the claims in the answer and their supporting source IDs.

The trace is for debugging and audit, not for exposing private chain-of-thought. It should record observable decisions and evidence without relying on hidden model reasoning.

NIST's 2026 work on agentic evaluation probes calls for machine-readable audit trails and tests of faithfulness, completeness and sufficiency. Those three dimensions translate directly into a knowledge assistant:

• faithfulness: does the answer match the cited evidence?
• completeness: are the material parts of the request addressed?
• sufficiency: is there enough evidence to justify the answer?

Evaluate answers and the retrieval path

Build a test set from real questions, not only documents.

Include:

• answerable direct lookups;
• multi-source and multi-hop questions;
• questions with no answer in the approved corpus;
• stale and superseded documents;
• conflicting sources with and without a precedence rule;
• users with different permissions asking the same question;
• multilingual phrasing and domain terminology;
• malicious instructions inside documents;
• questions where the safe result is escalation.

Measure:

• claim accuracy and citation correctness;
• evidence coverage;
• access-control violations;
• stale-source usage;
• conflict detection;
• justified refusal and unnecessary refusal;
• time to a verified answer;
• reviewer correction rate;
• cost per accepted answer.

Retrieval metrics such as recall are useful diagnostics, but the business outcome is a verifiable answer delivered to an authorised person.

Operate the knowledge, not only the model

Every production assistant needs two distinct owners:

• a system owner for retrieval, orchestration, evaluation and incidents;
• a knowledge owner for authority, freshness, access and source conflict.

One person may hold both roles in a small organisation, but both responsibilities must be explicit.

Set an operating rhythm:

• source owners review expiring material;
• access groups are recertified;
• changed documents trigger affected-answer tests;
• low-confidence and refused questions are sampled;
• reviewer corrections become evaluation cases;
• unused or low-value sources are removed;
• cost and latency are reviewed against accepted-answer value.

Without this rhythm, a knowledge assistant quietly becomes a fluent interface to yesterday's organisation.

A 30-day pilot

Week 1: define the contract

Choose one knowledge domain, one user group and 50–100 real questions. Name the authoritative sources, owner, permission model, freshness rule and refusal boundary.

Week 2: build the narrow path

Ingest only the approved sources. Preserve metadata and stable links. Add permission and expiry filters before retrieval. Start with direct lookup and introduce iterative retrieval only for question types that need it.

Week 3: evaluate

Test citation accuracy, unsupported claims, stale content, conflicts and access violations. Have domain reviewers score a sample using one rubric.

Week 4: release with ownership

Release to a limited group. Show citations and refusal clearly. Log corrections, questions without coverage and source problems. Hold the first knowledge operations review before expanding the corpus.

Common mistakes

Indexing every available document

More content can reduce reliability when ownership, authority and freshness are unknown. Begin with a small governed source set.

Treating a citation as proof

A link can be present while the claim contradicts the source. Evaluate the relationship between each material claim and its evidence.

Using one service account for everyone

This converts the assistant into a privilege-escalation layer. Retrieval must enforce the requesting user's permitted view.

Hiding conflict

Blending two incompatible policies into one smooth answer removes information the user needs. Show the conflict and route it to the knowledge owner.

Giving the assistant judgement authority

Evidence retrieval can improve a consequential decision without authorising the system to make it. Separate the knowledge package from the approval.

The practical rule

A dependable knowledge assistant does not merely answer quickly.

It can show:

1. which permitted and current sources were used;
2. how those sources support the material claims;
3. what remains uncertain or conflicting;
4. why it answered, escalated or refused;
5. who owns the knowledge and the correction loop.

That is the difference between a conversational search demo and an operational knowledge system.

Sources

• Google Research, Unlocking dependable responses with agentic RAG (opens in a new tab), 5 June 2026.
• NIST, AI RMF Playbook: Manage (opens in a new tab), reviewed 2 September 2026.
• NIST, Building Evaluation Probes for Agentic AI (opens in a new tab), reviewed 2 September 2026.
• Stanford HAI, 2026 AI Index Report: Responsible AI (opens in a new tab), reviewed 2 September 2026.

Continue in Methodfield

Use Value Stream Mapping to locate where knowledge enters and delays the workflow, FMEA to prioritise source and permission failures, Mistake Proofing to enforce access and freshness rules, and PDCA to operate the evaluation and correction loop.