acceptodds
Under review as a conference paper at ICLR 2027

Gather, Don't Scatter: Trainable Placement Robustness for Multi-Hop Reasoning in Multi-Document Long-Context LLMs

Abstract

Language models are increasingly given multiple documents in context to answer questions that require external, specialized, or up-to-date information. However, their answers can depend on how those documents are arranged. We examine this dependence in multi-hop question answering, where every document, gold or distractor, is placed directly in the context and each gold document supplies one step of the reasoning. We separately examine three aspects of document ar- rangement: the order of gold documents in the reasoning chain, their position in the context, and whether they are grouped together or scattered among distractors. Across two 8B base models, whether the gold documents are together or scattered matters most. Keeping them together helps consistently wherever the block sits, while chain order and exact block position have smaller effects. We then vary only the arrangement of the fine-tuning data. Training on examples with scattered gold nearly eliminates placement gaps and improves pooled accuracy by 20–24 percentage points, whereas training on a fixed gold-at-the-front layout induces se- vere brittleness when the gold moves. These findings hold across three datasets, with contexts up to 32k tokens, under harder retrieved distractors, and in long- form generation with citations. The main result also holds for a 27B model from a third family. Diagnostic probes further show that many remaining failures occur even when the individual supporting facts are accessible, identifying multi-hop composition as an important residual bottleneck.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Order Dependence, LoRA Fine-Tuning, Short and Long-Form Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kVdJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how the arrangement of supporting documents affects multi-hop question answering. It finds that grouping supporting documents generally helps, while fine-tuning on scattered evidence improves accuracy and robustness across layouts. Conversely, training with supporting documents consistently placed first induces substantial brittleness.

Critical strengths
  1. Relevant problem with practical implications: The paper addresses an important challenge in multi-hop question answering and demonstrates that the arrangement of training documents can significantly affect model accuracy and robustness.

  2. Systematic experimental analysis: The paper carefully studies the effects of document arrangement and shows that training on scattered evidence improves robustness across different layouts. The findings are further supported by experiments across multiple models, datasets, and settings.

Critical weaknesses
  1. The paper compares scattered training primarily against a fixed gold-first arrangement. However, it is unclear whether the improvements come specifically from scattering supporting documents or simply from varying their positions during training. Including baselines with randomly positioned contiguous supporting documents or mixed training layouts would strengthen the conclusions.
  2. The experiments assume access to gold supporting documents, which may not be available in realistic retrieval settings. Additionally, the relatively small evaluation samples and the use of neutral filler to construct longer contexts raise questions about how well the findings generalize to more realistic long-context scenarios.
Questions for authors
  1. How does scattered training compare with randomly positioned contiguous blocks or mixed-layout training?
4 comments
AuthorsOpenReview ↗

Authors' Response

Thank you for your time and efforts. We are gald that you find our problem relevant and experiments systematic. Please find below our response to your concerns.

The paper compares scattered training primarily against a fixed gold-first arrangement...

We are currently conducting this experiment and will share the results soon.

The experiments assume access to gold supporting documents, which may not be available...

Thank you for raising these concerns. In this work, we primarily focused on multi-hop question answering task that requires evidence across different documents. We thus assume the standard multi-hop QA setup where all the required gold documents are present. This allows us to systematically isolate effects of different document arrangements. A more realistic setup could be Retrieval based QA that may omit necessary evidence, requiring the model to recognize that the question is unanswerable from the context and abstain. This is an interesting, separate setting that we plan to investigate in future work.

Although our samples are modest, we evaluate the same questions across arrangements and report standard errors and paired McNemar tests (Appendix C, Table 12; Table 5; Appendix H, Table 14). Cross-dataset and model-family evaluations provide complementary evidence, although larger samples would improve precision.

Finally, constructing long-context multi-hop datasets with verified evidence is challenging. Neutral filler lets us isolate context length while preserving evidence and answerability. Separately, our BM25 hard-distractor study tests topically confusable passages: scattered-trained readers retain placement gaps of at most four percentage points (Table 5). Together, these studies support robustness to increased length and distractor difficulty separately; their combined effect remains an open question.

AuthorsOpenReview ↗

Random-Block Training Baseline (1/2)

This response is also shared with Reviewer Logo

We agree that comparing scattered training with fixed gold-first training does not fully separate the benefits of positional randomization from those of dispersing the supporting documents. To distinguish these effects, we added a random-block training baseline and evaluated it alongside our two existing regimes on Llama-3.1-8B-Instruct and Qwen3-8B (this experiment also addresses Reviewer kVdJ's Q1).

Training regimes. In our setup, gold documents are the paragraphs containing the supporting facts needed to answer a multi-hop question, and distractors are the remaining context paragraphs. We compare three training arrangements (seeded per example and fixed across epochs):

  • Gold-first: All gold documents are placed together as a contiguous block at the very beginning of the context, in forward reasoning-chain order (following MuSiQue's annotated hop sequence).
  • Random-block: All gold documents remain together as a contiguous block in forward reasoning-chain order, but the block is inserted at a uniformly sampled position among the distractors (including the beginning and end).
  • Scattered: Gold documents are placed at randomly sampled slots across the context, and their internal order is randomized as well.

We kept the training examples, document contents, distractor order, answer targets, random seed, LoRA hyperparameters, and training budget (2 epochs, 604 steps) identical across all three regimes. To avoid dropping different over-length examples across arrangements or tokenizers, we applied a shared 4,096-token filter and retrained all models on the exact same 4,828 training and 294 dev examples from MuSiQue train (disjoint from evaluation). Thus, the numbers below come from strictly matched reruns across all three regimes.

Evaluation and metrics. We evaluated each fine-tuned model on the same 150 MuSiQue validation questions (50 per hop count for 2, 3, and 4 hops) across all seven test arrangements: contiguous gold blocks at the front, middle, or end (each in forward or reverse chain order), plus scattered gold (averaged over seeds 0, 1, and 2). We used greedy decoding with a 512-token budget; all 8,100 predictions terminated normally without truncation. We report:

  • Pooled em_soft (%): Formatting-tolerant containment match (whether the normalized gold answer appears in the prediction), averaged equally across all seven test arrangements.
  • Pooled judge accuracy (%): Semantic correctness graded by a blinded Llama-3.3-70B-Instruct judge (given only the question, gold answer, and prediction), averaged equally across all seven test arrangements.
  • Front-minus-scattered gap (pp [percentage points]): Within the same trained model, em_soft when the forward gold block is at the front minus em_soft when gold documents are scattered. A gap near 0 pp means the model performs equally well whether supporting evidence is grouped at the front or dispersed.
  • Combined rows: Equal average of the two backbone models.
BackboneTraining regimePooled em_soft (%)Pooled judge accuracy (%)Front-minus-scattered gap (pp)
Llama-3.1-8B-InstructGold-first42.347.8+40.2
Llama-3.1-8B-InstructRandom-block66.373.5+15.1
Llama-3.1-8B-InstructScattered69.077.3+3.8
Qwen3-8BGold-first36.341.6+35.3
Qwen3-8BRandom-block56.763.5+16.2
Qwen3-8BScattered59.368.9-2.2
CombinedGold-first39.344.7+37.8
CombinedRandom-block61.568.5+15.7
CombinedScattered64.173.1+0.8

(To be continued..)

AuthorsOpenReview ↗

Random-Block Training Baseline (2/2)

Key takeaways:

Randomizing block position accounts for most of the average accuracy gain. Random-block improves combined pooled em_soft over gold-first by 22.3 pp (95% CI [18.4, 26.1]), recovering about 90% of the scattered regime's gain.

However, random-block training still lets the model rely on contiguous, forward-ordered evidence. Scattered training instead requires combining supporting facts from randomly sampled positions and orders, without a guaranteed gold block. It gives higher pooled em_soft and judge accuracy on both backbones, though random-block remains better on forward-contiguous test arrangements.

The clearer difference is placement robustness, learned through training. Random-block retains front-minus-scattered gaps of 15.1 pp [9.1, 21.1] for Llama and 16.2 pp [9.8, 22.7] for Qwen. Scattered training reduces these gaps by 11.3 pp [5.8, 17.1] and 18.4 pp [11.8, 25.3], respectively, with supervision and optimization otherwise unchanged.


We thank the reviewer again for the constructive feedback. We believe that the new experimental results along with the previous response help in clarifying the major concerns you raised in your review. We would request the reviewer to consider adjusting the score to support this paper.

Thank you for running the additional experiments. The random-block comparison is helpful and makes the training results more informative. Among the three training settings you tested, scattered training appears to be the best choice when the test layout is unpredictable.

Given the paper’s emphasis on the benefits of grouping evidence, I would also appreciate seeing how much random-block training outperformed scattered training when the supporting documents were grouped in reasoning order. Reporting the accuracy of both methods on those layouts would clarify the trade-off.

I will keep my score at 3, which I believe is appropriate under this year’s rating scale.

Reviewer_LogoOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This article examines how the arrangement of evidence affects multi-hop question answering in long contexts by large language models, with a focus on the reasoning sequence, document position, and distribution of evidence. The authors found that whether the supporting documents are concentrated or dispersed has a greater impact on the results than the document order or position. Fine-tuning with dispersed evidence can significantly improve position robustness and accuracy. These findings were evaluated on multiple datasets, model series, and long context settings.

Critical strengths
  • Carefully designed experimental plan. This paper separately investigated the sequence, position and dispersion of evidence, which is helpful in identifying the main factors influencing multi-hop reasoning.
  • Simple and widely validated method. Just changing the arrangement of the training data can significantly improve robustness, and shows consistent trends in different datasets, model scales and challenging evaluation settings.
Critical weaknesses
  • Insufficient training control. The comparison mainly focuses on distributed and gold-first layouts, but there is no benchmark like randomly placing consecutive evidence blocks. Therefore, it is not clear whether the performance improvement is due to the dispersion of evidence or more general randomization of positions.
  • Inconsistent long-context configuration. Appendix A reports that the maximum inference context of the 8B parameter model is 8,192 tokens, while the results shown in the paper are 16k and 32k tokens.
Questions for authors
  • Can the authors include a training baseline that uses randomly positioned continuous chunks of evidence to distinguish the benefits of dispersion from the advantages of general positional randomization?
  • Also, given the 8192-token reasoning limit reported in Appendix A, how were the 16k/32k experiments conducted? Were different configurations or truncation strategies used?
2 comments
AuthorsOpenReview ↗

Authors' Response (1/2)

Thank you for your thoughtful review and constructive feedback. We are glad that you found our presentation clear, our experimental design careful in isolating sequence, position, and dispersion, and our training approach simple and broadly validated across datasets, model scales, and long-context settings. Please find below our response to your two questions and concerns.


1. Random-Block Training Baseline

Can the authors include a training baseline that uses randomly positioned continuous chunks of evidence to distinguish the benefits of dispersion from the advantages of general positional randomization?

We agree that comparing scattered training with fixed gold-first training does not fully separate the benefits of positional randomization from those of dispersing the supporting documents. To distinguish these effects, we added a random-block training baseline and evaluated it alongside our two existing regimes on Llama-3.1-8B-Instruct and Qwen3-8B (this experiment also addresses Reviewer kVdJ's Q1).

Training regimes. In our setup, gold documents are the paragraphs containing the supporting facts needed to answer a multi-hop question, and distractors are the remaining context paragraphs. We compare three training arrangements (seeded per example and fixed across epochs):

  • Gold-first: All gold documents are placed together as a contiguous block at the very beginning of the context, in forward reasoning-chain order (following MuSiQue's annotated hop sequence).
  • Random-block: All gold documents remain together as a contiguous block in forward reasoning-chain order, but the block is inserted at a uniformly sampled position among the distractors (including the beginning and end).
  • Scattered: Gold documents are placed at randomly sampled slots across the context, and their internal order is randomized as well.

We kept the training examples, document contents, distractor order, answer targets, random seed, LoRA hyperparameters, and training budget (2 epochs, 604 steps) identical across all three regimes. To avoid dropping different over-length examples across arrangements or tokenizers, we applied a shared 4,096-token filter and retrained all models on the exact same 4,828 training and 294 dev examples from MuSiQue train (disjoint from evaluation). Thus, the numbers below come from strictly matched reruns across all three regimes.

Evaluation and metrics. We evaluated each fine-tuned model on the same 150 MuSiQue validation questions (50 per hop count for 2, 3, and 4 hops) across all seven test arrangements: contiguous gold blocks at the front, middle, or end (each in forward or reverse chain order), plus scattered gold (averaged over seeds 0, 1, and 2). We used greedy decoding with a 512-token budget; all 8,100 predictions terminated normally without truncation. We report:

  • Pooled em_soft (%): Formatting-tolerant containment match (whether the normalized gold answer appears in the prediction), averaged equally across all seven test arrangements.
  • Pooled judge accuracy (%): Semantic correctness graded by a blinded Llama-3.3-70B-Instruct judge (given only the question, gold answer, and prediction), averaged equally across all seven test arrangements.
  • Front-minus-scattered gap (pp [percentage points]): Within the same trained model, em_soft when the forward gold block is at the front minus em_soft when gold documents are scattered. A gap near 0 pp means the model performs equally well whether supporting evidence is grouped at the front or dispersed.
  • Combined rows: Equal average of the two backbone models.
BackboneTraining regimePooled em_soft (%)Pooled judge accuracy (%)Front-minus-scattered gap (pp)
Llama-3.1-8B-InstructGold-first42.347.8+40.2
Llama-3.1-8B-InstructRandom-block66.373.5+15.1
Llama-3.1-8B-InstructScattered69.077.3+3.8
Qwen3-8BGold-first36.341.6+35.3
Qwen3-8BRandom-block56.763.5+16.2
Qwen3-8BScattered59.368.9-2.2
CombinedGold-first39.344.7+37.8
CombinedRandom-block61.568.5+15.7
CombinedScattered64.173.1+0.8

(To be continued...)

AuthorsOpenReview ↗

Authors' Response (2/2)

Key takeaways:

Randomizing block position accounts for most of the average accuracy gain. Random-block improves combined pooled em_soft over gold-first by 22.3 pp (95% CI [18.4, 26.1]), recovering about 90% of the scattered regime's gain.

However, random-block training still lets the model rely on contiguous, forward-ordered evidence. Scattered training instead requires combining supporting facts from randomly sampled positions and orders, without a guaranteed gold block. It gives higher pooled em_soft and judge accuracy on both backbones, though random-block remains better on forward-contiguous test arrangements.

The clearer difference is placement robustness, learned through training. Random-block retains front-minus-scattered gaps of 15.1 pp [9.1, 21.1] for Llama and 16.2 pp [9.8, 22.7] for Qwen. Scattered training reduces these gaps by 11.3 pp [5.8, 17.1] and 18.4 pp [11.8, 25.3], respectively, with supervision and optimization otherwise unchanged.


2. Long-Context (16k / 32k) Inference Configuration

Also, given the 8192-token reasoning limit reported in Appendix A, how were the 16k/32k experiments conducted? Were different configurations or truncation strategies used?

Thank you for catching this oversight in our Appendix A description—no truncation was used in any of the 16k or 32k experiments. Both 8B models natively support 32k+ contexts (Llama-3.1-8B-Instruct supports 128k tokens and Qwen3-8B supports 32,768 tokens natively without RoPE extrapolation).

The 8,192-token limit stated in Appendix A was the vLLM --max-model-len setting used only for the standard-length (~20-document, <4k-token) experiments, where capping KV-cache reservation at 8,192 tokens allowed higher request concurrency (max_workers=32). For all context-length sweep experiments (both short-answer in Section 4.3 and controlled long-form in Section 4.4), we relaunched the vLLM servers with --max-model-len 32768 and reduced request concurrency (max_workers=8) so the larger KV cache fit comfortably in GPU memory.

Specifically, for the 4k, 8k, 16k, and 32k sweep points:

  • We grew the document context to targets of 4,096, 8,192, 16,384, and 30,720 tokens (using neutral filler passages measured by the shared Llama-3.1-8B-Instruct reference tokenizer so both models saw the exact same document sets).
  • Target document lengths were capped at 30,720 tokens for the "32k" condition specifically so that the full prompt plus the generated answer always fit strictly inside the 32,768-token server window without any prompt truncation (with generation budget adjusted from 512 to 128 tokens on short-answer Qwen3-8B and 384 tokens on long-form at 30,720 to accommodate Qwen's slightly denser tokenization).

We will update Appendix A in the revision to explicitly state --max-model-len 32768 for the length-sweep experiments alongside the 8,192 setting used for the standard-length runs.

Reviewer_GjtJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how the arrangement of supporting documents affects multi-hop question answering when all documents are provided in the model’s context. The authors vary the order of the gold documents in the reasoning chain, the position of a contiguous gold-document block, and whether the gold documents are gathered or scattered among distractors. They find that gathering the gold documents generally helps the base models more than changing chain order or block position. They then fine-tune models using either scattered gold documents or a fixed gold-at-the-front layout. Scattered training improves accuracy and reduces performance differences across layouts, while gold-first training produces a strong preference for the layout seen during training. The paper also reports results across datasets, context lengths, distractor types, long-form answers, and a larger model

Critical strengths

The paper separates several aspects of document arrangement that could otherwise be conflated. Holding the document content fixed while changing its arrangement makes the placement comparisons easier to interpret. The experiments also test whether the observed behavior persists under longer contexts and harder distractors.

Critical weaknesses

It is unclear what new insight the paper provides beyond existing work. It is already known that the order and placement of relevant context can affect LLM performance [1,2]. The finding that gathering supporting documents helps multi-hop QA is more specific, but the paper should explain more clearly what this teaches us that was not already known from studies of position sensitivity and dispersed evidence. The training results also seem largely expected. Fine-tuning a model on document-based QA can improve its performance on document-based QA, so the accuracy gain over the base model does not by itself establish the value of the proposed training arrangement. Similarly, training only on examples with gold documents at the front creates a positional bias. A drop in performance when the gold documents appear elsewhere is an expected consequence of the train-test distribution gap.

[1] Yu et al., Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context, NAACL 2025 findings
[2] Lee et al., CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation, NAACL 2025 short

Questions for authors
  1. What is the main new insight about document arrangement beyond the previously established effects of evidence position and order?
  2. What conclusion should we draw from the gold-first result beyond the expected effect of a train–test layout mismatch?
2 comments
AuthorsOpenReview ↗

Authors' Response

Thank you for taking time in reviewing our work. Please find below our response to your concerns.

What is the main new insight about document arrangement beyond the previously established effects of evidence position and order?

Let us clarify. Our paper compares gathering versus scattering supporting documents, forward versus reverse reasoning-chain order, and gold-block position while holding document content fixed. We find that gathering helps base models more consistently than chain order or exact block position. We further show that scattered training substantially reduces arrangement sensitivity, whereas gold-first training increases dependence on the trained layout, and test whether this robustness holds under longer contexts and harder distractors.

We acknowledge that Yu et al. [1] study supporting-document permutations, permutations of supporting documents together with distractors, and gold-block position. CORD [2] studies training with perturbed retrieval rankings. However, neither explicitly compares forward versus reverse order defined by the annotated reasoning chain, nor separately reports a gathered-versus-scattered comparison. Our distinct insight is that keeping supporting evidence together helps more consistently than preserving reasoning-chain order or choosing a particular block position in our evaluated settings. Our training experiments further show that this preference can be substantially reduced.

[1] Yu et al., Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context, NAACL 2025 findings [2] Lee et al., CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation, NAACL 2025 short

What conclusion should we draw from the gold-first result beyond the expected effect of a train–test layout mismatch?

We agree that degradation under layout mismatch is expected. Gold-first serves as a matched control: with identical QA supervision, changing only the training layout produces substantially different placement robustness. We therefore present it as supporting evidence for the importance of training arrangement, rather than an independently novel finding.

AuthorsOpenReview ↗

Further Clarification on the Related Works

We extend Table 1 of the paper below to include the two cited studies and clarify the distinctions from our work. Our study explicitly isolates gathering versus scattering supporting evidence and tests whether a change in training layout can improve placement robustness, including under longer contexts and harder distractors.

MethodMulti-hopIsolates gath./scat.Minimal fix (data only)Causal (remove+induce)Robust 32k+hardnegLong-form + cites
Lost in the Middle [3]✗✗✗✗✗✗
Found in the Middle [4]✗✗✗✗✗✗
PINE [5]✗✗✗✗✗✗
Set-LLM [6]✗✗✗✗✗✗
FILM [7]✗✗✗✗✗✗
Never Lost in the Middle [8]✓✗✗✗✗✗
Long-Context LLMs Meet RAG [9]✓✗✗✗✓✗
CoRe (Yu et al., 2025)✓✗✗✗✗✗
CORD (Lee et al., 2025)✓✗✓✗✗✗
Ours✓✓✓✓✓✓

Hope this clarifies further on the position and novelty of our work with respect to the existing literature.

[1] Yu et al., Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context, NAACL 2025 findings

[2] Lee et al., CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation, NAACL 2025 short

[3] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.

[4] Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., & Pfister, T. (2024). Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 14982–14995).

[5] Wang, Z., Zhang, H., Li, X., Huang, K.-H., Han, C., Ji, S., Kakade, S. M., Peng, H., & Ji, H. (2025). Eliminating position bias of language models: A mechanistic approach. In International Conference on Learning Representations.

[6] Egressy, B., & Stühmer, J. (2025). Set-LLM: A permutation-invariant LLM. In Advances in Neural Information Processing Systems.

[7] An, S., Ma, Z., Lin, Z., Zheng, N., Lou, J.-G., & Chen, W. (2024). Make your LLM fully utilize the context. In Advances in Neural Information Processing Systems.

[8] He, J., Pan, K., Dong, X., Song, Z., Liu, Y., Sun, Q., Liang, Y., Wang, H., Zhang, E., & Zhang, J. (2024). Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.

[9] Jin, B., Yoon, J., Han, J., & Arık, S. Ö. (2025). Long-context LLMs meet RAG: Overcoming challenges for long inputs in RAG. In International Conference on Learning Representations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.