- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores how random/neutral, relevant but hard and distracting documents affect the performance of LLMs on long-context question answering tasks during inference time. To evaluate this task the authors propose a 2 stage framework. For stage1, they compare the effects of neutral, relevant but hard and misleading documents for a question-answering task across multiple model and context length. For stage2, they compare against high and low reasoning modes of the LLMs and test whether providing hints can help the low reasoning model achieve better performance. Experiments show that the misleading documents have the capability to pull other wrong answers towards the misleading answer.
The paper is well written and easy to follow.
Critical strengths
- The paper is clearly written and mostly easy to follow. The motivation is also clear.
- The authors experiment with 6 LLMs on the MuSiQue dataset and the 2WikiMHQA dataset. The experiments are mostly well designed.
- The findings showing that wrong answers tend to shift towards the misleading document is a strength of this paper.
Critical weaknesses
- The biggest weakness is the design of experiments and claims on Stage2 of the experiment. Using direct hints from the high reasoning model and providing them to the low reasoning model is somewhat questionable. There is a chance that this would provide answer specific hints and is a reason why the gains on this setting is substantially higher.
- The use of misleading information and distracting prompts/documents have been studied previously. Hence the novelty of the work is limited.
Questions for authors
- Is it possible to setup some experiments to show that during Stage2, the low reasoning model is actually better at reasoning?
- Can the authors experiment and report the performance of the LLMs when the positioning of the misleading documents are at the start, middle and end?
- Would it matter how far the misleading document is from the gold-position document?
- Can the authors clearly differentiate their contributions from [1] and [2]?
- In the manuscript page 5 - line 225, there is a mention of Stage 3 in the experiment pipeline; can you clarify what is stage 3 here?
References:
-
Bhuiya, Neeladri, Viktor Schlegel, and Stefan Winkler. "Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
-
Lee, Seongyun, et al. "Lost in the noise: How reasoning models fail with contextual distractors." arXiv preprint arXiv:2601.07226 (2026).
2 comments
3. Distance between misleading and supporting documents.
We found no consistent pattern of lower accuracy when the misleading passage was closer to the supporting documents. Qwen3.5-27B High matches the designated wrong answer more often in near layouts (Table R3).
For each of the 87 questions, we create four versions of the same context, keeping all text and total length fixed. The supporting documents stay together as one block. Positions below refer to the centers of this block and the misleading passage.
- Near: supporting documents at about 20% and the misleading passage at 30%, or supporting documents at 80% and the misleading passage at 70%.
- Far: supporting documents at about 20% and the misleading passage at 70%, or supporting documents at 80% and the misleading passage at 30%.
We average the results of the two layouts in each group. Their mean center-to-center distances are approximately 2.5K tokens for Near and 12.5K for Far. Both groups use the same positions equally often and include the misleading passage both before and after the supporting documents.
Table R3. Near/far accuracy and paired differences. Target match means exact normalized equality with the designated wrong answer.
| Model | Thinking | n | Near accuracy (%) | Far accuracy (%) | Near − far accuracy (pp) [95% CI] | Near − far target match (pp) [95% CI] |
|---|---|---|---|---|---|---|
| Qwen3.5-27B | Off | 87 | 44.25 | 45.40 | -1.15 [-4.60, 1.74] | 2.87 [-0.57, 6.32] |
| Qwen3.5-27B | High | 86 | 42.44 | 46.51 | -4.07 [-8.72, 0.58] | 4.65 [0.58, 8.72] |
| Qwen3.5-35B-A3B | Off | 87 | 33.33 | 32.76 | 0.57 [-4.02, 5.17] | -0.57 [-4.60, 3.45] |
| Qwen3.5-35B-A3B | High | 87 | 36.21 | 33.33 | 2.87 [-1.15, 6.90] | 0.00 [-4.02, 4.02] |
| DeepSeek V4.1 Flash | Off | 87 | 44.83 | 42.53 | 2.30 [-0.57, 5.75] | -1.15 [-4.02, 1.16] |
| DeepSeek V4.1 Flash | High | 70 | 55.71 | 55.71 | 0.00 [-5.00, 5.00] | 3.57 [-0.71, 7.86] |
Table notes: n counts questions with valid answers at every layout. DeepSeek High has fewer such questions because some runs reached the output limit before answering. Scoring and statistical details are in Appendix I (Tables 42–44).
The supplementary experiments show no common position or distance pattern in accuracy across the tested models, whereas several background-quality and evidence-use findings recur in the main experiments. Together, these results illustrate the value of systematic comparisons for distinguishing recurring patterns from findings that vary across settings.
4. Differences from Bhuiya et al. [1] and Lee et al. [2].
Our aim is to understand how background quality affects answers when all supporting evidence is available, and what higher reasoning contributes on those tasks. Table 1 compares the experimental settings.
| Existing findings | What our comparisons add |
|---|---|
| Plausible but incorrect reasoning paths can impair multi-hop QA even when the supporting paragraphs are present. Bhuiya et al. (2024) [1]. | The accuracy loss from related background to background containing a short misleading passage was larger than the loss from sampled to related background in nine of twelve model–setting combinations (Section 4.1, Figure 2). |
| Under hard-negative distraction, longer sampled reasoning trajectories were associated with lower accuracy. Lee et al. (2026), Appendix B.7 [2]. | We probe what higher reasoning offers by passing document selections and evidence-use guidance to the same model in low mode. Even when the document IDs exactly match the correct supporting documents, five models answer more accurately with guidance than with IDs alone (Table 3). These comparisons support interpreting reasoning gains through both finding useful evidence and organizing and using it to answer. |
These measurements of input quality and evidence use in long-context QA provide a reference for future research on data construction and training.
5. Meaning of “Stage 3.”
“Stage 3” tests models using only the documents that support the correct answer, without added background passages (Section 4.2, Table 28). We now call it the “gold-only reference” in the revised main text.
References
- Bhuiya, Neeladri, Viktor Schlegel, and Stefan Winkler. "Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
- Lee, Seongyun, Yongrae Jo, Minju Seo, Moontae Lee, and Minjoon Seo. "Lost in the Noise: How Reasoning Models Fail with Contextual Distractors." arXiv preprint arXiv:2601.07226. 2026.
Thank you for recognizing the motivation and experimental design.
1. Stage-2 hints and whether low mode becomes better at reasoning.
Stage 2 asks what higher reasoning offers in long-context QA. We use information transfer within the same model as a diagnostic tool. The model parameters, low-reasoning setting, and full context stay fixed. We interpret the gains as more accurate answers with guidance, rather than an increase in the model's underlying reasoning ability.
On cases where high mode succeeds and low mode fails, we compare teacher IDs, annotated gold IDs, and teacher IDs plus a hint. The original successful high-mode response supplies the teacher IDs, indicating where to look. A separate high-mode call uses those fixed IDs to generate guidance on organizing and using the evidence. The generation prompt asks for guidance without stating the final answer, and we reject hints containing the canonical answer or the original high-mode answer after text normalization (Appendix B.2). The example in Section 3.3 illustrates this information transfer.
On matched cases from questions accepted by both auditors, with accepted hints and valid answers to all three prompts, teacher+hint improves accuracy over teacher IDs alone by 25.1–39.3 percentage points in five models (Figure 3). Gains persist when the IDs exactly match the gold-document set (post-hoc analysis, Table 3). These results suggest considering both finding useful evidence and using it to answer when interpreting within-model reasoning gains.
2. Position of the misleading passage.
We found no common position ranking across the tested models. Accuracy point estimates are highest late for both Qwen models and early for DeepSeek, in each thinking setting (Tables R1–R2).
We use all 87 MuSiQue questions accepted by both auditors with gold evidence originally at the front. In approximately 25K-token inputs, we hold the gold prefix fixed and move the misleading passage to boundaries near 10%, 50%, or 90%. Contents, construction-tokenizer length, and other documents' order stay fixed; labels follow the new order.
Both experiments test the two Qwen models and DeepSeek V4.1-Flash-1017. Off disables thinking; High requests Qwen's 4,096-token thinking budget or DeepSeek's max reasoning effort with an 8,192-token total output cap.
Table R1. Accuracy by misleading-passage position.
Table R2. All paired position differences in accuracy, using the cohorts in Table R1.