- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies how the arrangement of supporting documents affects multi-hop question answering. It finds that grouping supporting documents generally helps, while fine-tuning on scattered evidence improves accuracy and robustness across layouts. Conversely, training with supporting documents consistently placed first induces substantial brittleness.
Critical strengths
-
Relevant problem with practical implications: The paper addresses an important challenge in multi-hop question answering and demonstrates that the arrangement of training documents can significantly affect model accuracy and robustness.
-
Systematic experimental analysis: The paper carefully studies the effects of document arrangement and shows that training on scattered evidence improves robustness across different layouts. The findings are further supported by experiments across multiple models, datasets, and settings.
Critical weaknesses
- The paper compares scattered training primarily against a fixed gold-first arrangement. However, it is unclear whether the improvements come specifically from scattering supporting documents or simply from varying their positions during training. Including baselines with randomly positioned contiguous supporting documents or mixed training layouts would strengthen the conclusions.
- The experiments assume access to gold supporting documents, which may not be available in realistic retrieval settings. Additionally, the relatively small evaluation samples and the use of neutral filler to construct longer contexts raise questions about how well the findings generalize to more realistic long-context scenarios.
Questions for authors
- How does scattered training compare with randomly positioned contiguous blocks or mixed-layout training?
4 comments
Random-Block Training Baseline (1/2)
This response is also shared with Reviewer Logo
We agree that comparing scattered training with fixed gold-first training does not fully separate the benefits of positional randomization from those of dispersing the supporting documents. To distinguish these effects, we added a random-block training baseline and evaluated it alongside our two existing regimes on Llama-3.1-8B-Instruct and Qwen3-8B (this experiment also addresses Reviewer kVdJ's Q1).
Training regimes. In our setup, gold documents are the paragraphs containing the supporting facts needed to answer a multi-hop question, and distractors are the remaining context paragraphs. We compare three training arrangements (seeded per example and fixed across epochs):
- Gold-first: All gold documents are placed together as a contiguous block at the very beginning of the context, in forward reasoning-chain order (following MuSiQue's annotated hop sequence).
- Random-block: All gold documents remain together as a contiguous block in forward reasoning-chain order, but the block is inserted at a uniformly sampled position among the distractors (including the beginning and end).
- Scattered: Gold documents are placed at randomly sampled slots across the context, and their internal order is randomized as well.
We kept the training examples, document contents, distractor order, answer targets, random seed, LoRA hyperparameters, and training budget (2 epochs, 604 steps) identical across all three regimes. To avoid dropping different over-length examples across arrangements or tokenizers, we applied a shared 4,096-token filter and retrained all models on the exact same 4,828 training and 294 dev examples from MuSiQue train (disjoint from evaluation). Thus, the numbers below come from strictly matched reruns across all three regimes.
Evaluation and metrics. We evaluated each fine-tuned model on the same 150 MuSiQue validation questions (50 per hop count for 2, 3, and 4 hops) across all seven test arrangements: contiguous gold blocks at the front, middle, or end (each in forward or reverse chain order), plus scattered gold (averaged over seeds 0, 1, and 2). We used greedy decoding with a 512-token budget; all 8,100 predictions terminated normally without truncation. We report:
- Pooled em_soft (%): Formatting-tolerant containment match (whether the normalized gold answer appears in the prediction), averaged equally across all seven test arrangements.
- Pooled judge accuracy (%): Semantic correctness graded by a blinded
Llama-3.3-70B-Instructjudge (given only the question, gold answer, and prediction), averaged equally across all seven test arrangements. - Front-minus-scattered gap (pp [percentage points]): Within the same trained model,
em_softwhen the forward gold block is at the front minusem_softwhen gold documents are scattered. A gap near 0 pp means the model performs equally well whether supporting evidence is grouped at the front or dispersed. - Combined rows: Equal average of the two backbone models.
| Backbone | Training regime | Pooled em_soft (%) | Pooled judge accuracy (%) | Front-minus-scattered gap (pp) |
|---|---|---|---|---|
| Llama-3.1-8B-Instruct | Gold-first | 42.3 | 47.8 | +40.2 |
| Llama-3.1-8B-Instruct | Random-block | 66.3 | 73.5 | +15.1 |
| Llama-3.1-8B-Instruct | Scattered | 69.0 | 77.3 | +3.8 |
| Qwen3-8B | Gold-first | 36.3 | 41.6 | +35.3 |
| Qwen3-8B | Random-block | 56.7 | 63.5 | +16.2 |
| Qwen3-8B | Scattered | 59.3 | 68.9 | -2.2 |
| Combined | Gold-first | 39.3 | 44.7 | +37.8 |
| Combined | Random-block | 61.5 | 68.5 | +15.7 |
| Combined | Scattered | 64.1 | 73.1 | +0.8 |
(To be continued..)
Random-Block Training Baseline (2/2)
Key takeaways:
Randomizing block position accounts for most of the average accuracy gain. Random-block improves combined pooled em_soft over gold-first by 22.3 pp (95% CI [18.4, 26.1]), recovering about 90% of the scattered regime's gain.
However, random-block training still lets the model rely on contiguous, forward-ordered evidence. Scattered training instead requires combining supporting facts from randomly sampled positions and orders, without a guaranteed gold block. It gives higher pooled em_soft and judge accuracy on both backbones, though random-block remains better on forward-contiguous test arrangements.
The clearer difference is placement robustness, learned through training. Random-block retains front-minus-scattered gaps of 15.1 pp [9.1, 21.1] for Llama and 16.2 pp [9.8, 22.7] for Qwen. Scattered training reduces these gaps by 11.3 pp [5.8, 17.1] and 18.4 pp [11.8, 25.3], respectively, with supervision and optimization otherwise unchanged.
We thank the reviewer again for the constructive feedback. We believe that the new experimental results along with the previous response help in clarifying the major concerns you raised in your review. We would request the reviewer to consider adjusting the score to support this paper.
Thank you for running the additional experiments. The random-block comparison is helpful and makes the training results more informative. Among the three training settings you tested, scattered training appears to be the best choice when the test layout is unpredictable.
Given the paper’s emphasis on the benefits of grouping evidence, I would also appreciate seeing how much random-block training outperformed scattered training when the supporting documents were grouped in reasoning order. Reporting the accuracy of both methods on those layouts would clarify the trade-off.
I will keep my score at 3, which I believe is appropriate under this year’s rating scale.
Authors' Response
Thank you for your time and efforts. We are gald that you find our problem relevant and experiments systematic. Please find below our response to your concerns.
The paper compares scattered training primarily against a fixed gold-first arrangement...
We are currently conducting this experiment and will share the results soon.
The experiments assume access to gold supporting documents, which may not be available...
Thank you for raising these concerns. In this work, we primarily focused on multi-hop question answering task that requires evidence across different documents. We thus assume the standard multi-hop QA setup where all the required gold documents are present. This allows us to systematically isolate effects of different document arrangements. A more realistic setup could be Retrieval based QA that may omit necessary evidence, requiring the model to recognize that the question is unanswerable from the context and abstain. This is an interesting, separate setting that we plan to investigate in future work.
Although our samples are modest, we evaluate the same questions across arrangements and report standard errors and paired McNemar tests (Appendix C, Table 12; Table 5; Appendix H, Table 14). Cross-dataset and model-family evaluations provide complementary evidence, although larger samples would improve precision.
Finally, constructing long-context multi-hop datasets with verified evidence is challenging. Neutral filler lets us isolate context length while preserving evidence and answerability. Separately, our BM25 hard-distractor study tests topically confusable passages: scattered-trained readers retain placement gaps of at most four percentage points (Table 5). Together, these studies support robustness to increased length and distractor difficulty separately; their combined effect remains an open question.