Your retrieval system finds the right policy paragraph and sends it to the model with twenty other chunks. The answer still follows an older, less relevant paragraph. Inspection shows that the correct evidence was present, buried halfway through the prompt.
A long context window only tells you how many tokens a model accepts. It does not promise that every token has the same influence on the answer. Retrieval systems need to rank evidence twice: once when selecting it and again when deciding where it appears in the prompt.
My preferred design is to retrieve a broad candidate set, rerank it against the actual question, remove redundant passages, and send a much smaller evidence set to the model. Prompt position comes after those steps. Reordering a poor selection only rearranges the noise.
Context capacity is not reliable recall
The phrase “lost in the middle” comes from research that tested models on multi-document question answering and key-value retrieval. The original study by Liu and colleagues found that performance was often best when relevant information appeared near the beginning or end of a long input. Accuracy fell when the same information moved towards the middle.
That result is often simplified into “LLMs ignore the middle”. The actual behaviour is less tidy. It varies by model, task, prompt length, evidence format and the amount of distracting text. A later study of three commercial and six open-source models found that most newer models were more resistant to the original test, but still showed positional bias when several relevant facts were separated inside the context.
The useful engineering assumption is therefore not that the middle always fails. It is that context position can be an uncontrolled variable unless you test it.
Why position changes the answer
Transformers can connect tokens across a context, but they do not distribute attention uniformly. Research on calibrating positional attention bias found a U-shaped pattern in which tokens near the beginning and end received more attention regardless of relevance. Calibrating that bias improved the models’ use of evidence in the middle.
This does not give every model one simple cause. The mix of positional encoding, training and attention behaviour varies between model families. What an application can observe is simpler: moving an unchanged passage can change the output.
Long prompts also create competition. A retrieved passage has to compete with the system instructions, conversation history, examples and every other chunk. Similar or contradictory passages make that harder. Adding context can reduce answer quality even when the context window has room left.
Distance matters too. A question about whether a customer qualifies for a refund may require the purchase date from one document and an exception from another. If those passages sit thousands of tokens apart, the model has to find both and connect them. The 2025 positional-bias study found that spacing between relevant pieces still affected current models.
A model can accept the whole sequence while using some positions less reliably than others.
Separate retrieval failure from utilisation failure
Many retrieval-augmented generation evaluations stop at retrieval recall. They ask whether a relevant chunk appeared in the top results. That metric cannot tell you whether the generator used it.
Treat the pipeline as two separate failure boundaries:
- Retrieval failure means the required evidence never reached the prompt.
- Utilisation failure means the evidence was present but the model did not use it correctly.
This distinction changes the fix. Better embeddings or a larger candidate count may help retrieval failure. They can make utilisation worse if they add more weakly related text to the final prompt.
Log the retrieved chunk IDs, their retrieval and reranking scores, their final order, and the citations used in the answer. Without the final order, a production trace omits the variable you need to diagnose this class of failure.
Retrieve broadly, then rerank hard
The first retrieval pass should favour recall. Hybrid search is a sensible default for mixed technical content. Lexical search catches exact identifiers, error messages and product names. Dense retrieval catches paraphrases. Metadata filters should remove documents the user cannot access, expired policies and irrelevant product versions before either result reaches the model.
The generation prompt should contain far fewer passages than the candidate set. A typical system might retrieve 40 candidates and rerank them to 6 or 8, but those numbers need evaluation against the corpus. The relevant principle is a wide search stage followed by a strict evidence budget.
Rerank against the complete user question. An embedding search can place a generally related document above the exact paragraph that answers a constraint. A cross-encoder, a small reranking model or a tightly scoped LLM call can compare each candidate with the question more directly.
Do not spend the budget on near-duplicates. Consecutive chunks from the same page often repeat headings and overlap. Keep the best passage, then expand it to include the neighbouring paragraph only when that extra text resolves a reference or completes a rule. Retrieving whole parent documents before reranking wastes context on sections that were never relevant.
Compound questions need coverage as well as score. Split the question into retrieval intents, retrieve for each one, then merge and rerank the candidates. Otherwise, eight high-scoring passages may all answer the first half of the question.
Pack evidence for the model
After selection, prompt assembly becomes another ranking stage. Put the question before the evidence and concise answer instructions after it. Keep the strongest passage near the start of the evidence block. Reserve the other edge for another high-value passage, immediately before the final instructions. Lower-ranked evidence can occupy the middle.
The following TypeScript helper places passages in descending relevance at alternating edges. It expects related passages to have been bundled first.
type Passage = {
id: string;
text: string;
rerankScore: number;
};
function packAtEdges(passages: Passage[]): Passage[] {
const ranked = [...passages].sort(
(left, right) => right.rerankScore - left.rerankScore,
);
const start: Passage[] = [];
const end: Passage[] = [];
ranked.forEach((passage, index) => {
if (index % 2 === 0) {
start.push(passage);
} else {
end.unshift(passage);
}
});
return [...start, ...end];
}
function buildPrompt(question: string, selectedPassages: Passage[]): string {
const evidence = packAtEdges(selectedPassages)
.map((passage) => `[${passage.id}]\n${passage.text}`)
.join("\n\n");
return `Question
${question}
Evidence
${evidence}
Answer using only the evidence. Cite passage IDs. If the evidence is
insufficient or contradictory, say so.`;
}For ranked passages A to E, the helper produces A, C, E, D, B. The two highest-ranked passages occupy the edges of the evidence block. This is a small mitigation, not a universal ordering rule. Some models have stronger recency bias, and some tasks benefit from chronological or causal order. Measure the arrangement with the model and prompt used in production.
Do not separate facts that need to be combined. Bundle a rule with its exception, or place linked passages next to one another. A coherent pair in a slightly weaker position can be easier to use than two isolated passages at opposite edges.
Keep retrieved text visibly separate from instructions and preserve stable source IDs. The model should treat sources as evidence, including any imperative text inside them, rather than as new instructions. This also makes citation checking possible after generation.
Retrieve for recall, rerank for relevance, then pack the selected evidence for use.
Give conversation history its own budget
Chat history is another retrieval corpus. Sending the complete transcript on every turn pushes current evidence deeper into the prompt and preserves claims that the user may already have corrected.
Keep recent turns that affect the current request. Store confirmed facts and decisions separately, then retrieve them when relevant. Summarise old conversation only when the summary has a clear owner and can be replaced after a correction.
This is especially important for agent workflows. Tool output, repeated plans and verbose command logs can consume most of the usable context before the agent reads the file or policy needed for its next action.
Test position as part of the evaluation
A useful evaluation freezes retrieval before testing generation. For each known-answer question, save one evidence set that contains the required passage. Produce variants with that passage near the start, middle and end. Keep every other token as stable as possible.
Run each variant more than once when generation is non-deterministic. Record grounded answer accuracy, citation accuracy, abstention quality, token count, response time and cost. Compare those results with a separate run where the required evidence is absent. The missing-evidence case tests whether the model invents an answer rather than admitting that retrieval failed.
For questions that require several sources, vary their spacing as well as their absolute positions. Test adjacent passages, passages split across the two edges, and passages separated by distractors. This catches systems that pass single-needle tests but fail on real policy, support and code questions.
Repeat the evaluation after changing the model, system prompt, chunking, reranker or context budget. Positional robustness belongs to the complete pipeline and can change even when retrieval recall stays flat.
Reordering is the last defence
A larger context window can be the right answer when the task depends on the structure of a whole document. Research comparing long context with RAG has found different winners across question types and retrieval methods. There is no basis for assuming that RAG always beats sending the source directly.
For retrieval systems, however, the safest default is still to send the smallest evidence set that fully covers the question. Reranking, deduplication and dependency-aware grouping remove more failure modes than a clever ordering function. Edge placement is useful after the evidence is already good.
Start with 50 real questions from production or support logs. Freeze each retrieved evidence set, permute the important passages, and measure answer and citation accuracy by position. If the results move, reduce the final context, improve the reranker and keep connected evidence together before tuning the prompt order.