A context window describes what a model can receive in one request or turn. It does not promise that every passage is used equally well. The availability of larger windows makes one design choice attractive: paste the complete contract, the whole customer history and every policy into the prompt. The system appears informed. Its inputs, latency and opportunity for confusion can all grow.
In Lost in the Middle (opens in a new tab), researchers found that the position of relevant material affected performance on the long-context retrieval and question-answering tasks they tested. RULER (opens in a new tab) argues for broader stress tests of usable long context. These studies do not establish a universal defect in all current models. They establish a practical distinction between a model's advertised capacity and the amount of context it can use reliably for your task.
Repeated context creates repeated work
An application that resends the same long history on every turn can produce a surprisingly large number of billed input-token appearances. Caching can change that equation, but a discount depends on eligible matching prefixes, model rules and actual cache hits. The OpenAI prompt caching guide (opens in a new tab) describes those conditions. Theoretical cache savings are not the same as measured savings.
Long prompts can even move a request into a different price tier. Google's listed Gemini 3.1 Pro Preview rates increase when a prompt exceeds 200,000 tokens. Google pricing (opens in a new tab). This is one reason to inspect tariff thresholds before treating extra context as a free safety margin.
Choose a context strategy deliberately
Full context is simple and useful when the material is modest and all of it matters. Retrieval supplies selected source passages and can cut input cost, but a missed passage may make the final answer confidently incomplete. Summaries or compact memory carry decisions forward, but may erase an important exception. Prompt caching rewards stable repeated material, subject to write, read and storage rules.
A sound pattern often combines short stable rules, retrieval from authoritative documents, citations that can be checked, and a compact record of prior decisions. Keep links to originals. A citation generated by the model is only helpful if the application verifies that the referenced passage exists.
Run the comparison on actual questions
Prepare representative cases with known required sources and answers. Compare full loading, retrieval and a hybrid on the same model. Measure source recall, answer quality, latency, tokens, review time and correction cost. Place important facts at different positions in large documents. A method that saves input but loses critical facts is not a bargain.
Context is a selection pipeline
Several decisions happen before the model sees a prompt. Which documents are authoritative? Which policy version applied when the event occurred? Which passages belong to this customer? How do we remove duplicates without dropping an exception? How do we keep source material from being confused with instructions? Even excellent generation cannot repair an incorrect source selection. Long-context quality therefore begins with input quality.
A retrieval-augmented system can fail in at least three distinct ways. Search can miss the required document. It can find the document but choose an irrelevant or outdated passage. Or the model can receive the right passage and still make an unsupported claim. If the team scores only the final answer, it cannot tell which component needs fixing. RAGAS (opens in a new tab) and ARES (opens in a new tab) propose separate assessment of retrieved context, faithfulness to the source and answer relevance. Automated judging speeds iteration, but it should be calibrated with expert-reviewed examples.
From a question to evidence
Separate documents by authority and date. An approved policy, a working instruction, an old discussion and a customer comment should not look interchangeable to the retrieval system. Choose a useful passage size. Tiny fragments can lose a condition; huge fragments add noise and cost. Preserve source identity, version, date and access rights for every retrieved passage. If an answer needs several documents together, test whether the system can retrieve that combination rather than just one convenient text.
Compression is useful when it removes repetition while retaining facts, exceptions and provenance. It is dangerous when “a refund is allowed under three conditions” becomes “a refund is allowed.” For consequential work, compact memory should retain a path to the original, and each decision should record the rule version applied.
When caching actually pays
Caching is not a general memory system. It avoids repeated processing of an eligible unchanged prefix. A unique identifier, current date or shifting tool order at the start of every prompt can undermine hits. Stable rules and schemas can be placed separately from case-specific content. But a cache hit is not worth mixing different customers’ data or using an outdated policy.
Estimate reuse frequency, stable-prefix size, write and read rates, storage duration and the likelihood of content changes. In an agent, do not keep the complete action log in active model context simply because the application needs an audit trail. Store the log in observability systems; give the model a verified compact state and the evidence needed for its next step.
When full context is the better choice
Retrieval is not mandatory for every long document. If there is one moderate contract and the question genuinely requires every clause, full loading may be simpler and safer. If a large document collection changes frequently and each question concerns only a small part, retrieval usually offers better control. If a decision needs conversation history and primary evidence, use compact state plus selected sources. The choice should follow completeness, verifiability and full cost rather than prompt length alone.
Running example: what the retailer's agent needs to know
A customer writes, “Where is my order, and can I change the address?” Loading the entire knowledge base would bring in warranty, returns and promotion instructions that do not answer the question. Yet a narrow search for “address” could miss the rule that once the carrier has accepted a parcel, changes require separate approval. The response needs three types of context: the current status of this order, the current address-change policy and any promise already made to this customer. Each has a different source and access rule.
The system first verifies the order reference and customer authorization. It asks the order system for status instead of guessing from correspondence. It retrieves the applicable policy version; for an action, it checks when the carrier handoff occurred. Finally it checks whether a previous employee promise conflicts with the rule. Only the required facts enter the model prompt, but their original sources and retrieval times remain attached.
If policy and status conflict, the system should not smooth the discrepancy into a confident reply. It should make the conflict visible and escalate. This is a case in which a smaller, well-structured context can improve reliability: the model sees an explicit contradiction rather than a long history in which the condition is buried. Reduction is safe only when evidence selection and coverage have been tested.
Measuring this design
Evaluation begins before generation. For each test request, a specialist lists the mandatory facts: status, rule, exception and action authority. Retrieval is graded on whether it supplied them. The generator is graded separately: did it invent a promise, express the rule accurately and admit missing information? An accepted-looking answer with the wrong source is still a defect because it cannot be audited.
Include multiple policy versions, similarly named documents, conflicting customer messages, long histories and bilingual requests. When policy changes, test both the new rule and regression cases from old common scenarios. Caches and compact memory make this especially important because they can preserve yesterday's information.
Access is another constraint. A large window can technically hold many customer records; it does not give a user permission to see them. Retrieval should filter by access before content reaches the model. Stable general instructions can be reused broadly, while caching customer information requires tenant isolation, retention rules and provider-specific review. Input-token savings never justify revealing another customer's order.
Context, then, is more than a text limit. It is a governed bundle of evidence with provenance, validity dates and an access boundary. Better structure makes model changes, answer review and cost control easier.
Two context budgets: capacity and attention
A window has a technical limit, but system design also needs a stricter attention budget: how much material must the model actually compare for this decision? Long text is not inherently bad. Unstructured long text is. System rules, verified facts, quotations, dialogue history and tool results should be distinguishable by role and provenance. Otherwise a customer message can be mistaken for an instruction, or a retired rule for current policy.
Pack context from the question and acceptance rule outward: mandatory instructions, a compact verified case state and selected evidence. Filter large tool responses down to necessary fields. Keep the full action log outside the active model prompt and pass a checked summary of current state. This is not a magic prompt order. It makes failures diagnosable: was a fact not retrieved, lost in compression or interpreted incorrectly?
Multimodal inputs demand further care. A PDF page, invoice image or audio recording is not equivalent to a fixed number of text words in either billing or recognition errors. Check extraction first: OCR may miss an amount, a table may lose column relationships, and transcription may get a name wrong. Changing the language model cannot restore a missing source fact. Keep separate evaluation cases for perception errors and reasoning errors.
A decision rule for full loading or retrieval
Ask five questions. Does the answer need almost every part of the document? How often does the material change? How many questions will be asked of the same source? How costly is missing one passage? Can citations be verified automatically? If one moderate document matters in full, full loading can be justified. If many questions each touch a small part of a large collection, retrieval with coverage checks usually scales better. If omissions are critical, use a hybrid: retrieve candidates, expand around the relevant sections, or escalate.
Do not optimize context apart from quality. A short prompt with poor evidence is cheap only until the first mistake. A long prompt in which the key fact cannot be found is expensive and unreliable. The target is the smallest sufficient evidence package. Sufficiency is established with cases that have known answers and revisited when the knowledge base changes.
The best context is sufficient, relevant and verifiable. Window size is a design option, not the objective.
Next: How to choose a model for a job.