- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether question decomposition and passage grouping improve evidence selection under a limited reading budget. Using MuSiQue and 2WikiMultihopQA, it compares policies that choose paragraph bodies from visible candidate titles, with budgets of one, two or four reads. Its main controlled experiments start from identical evidence states and vary structure, selector strength and omissions from the inferred groups. Structure improves BM25 but adds little to the strong model selector in the main evaluation. Trajectory analyses suggest that the strong selector more often recovers support omitted from the structure.
Critical strengths
-
The same state comparisons make the policy effects easier to interpret. The question, evidence prefix, candidate pool and remaining budget are fixed, and injected omissions remove support from the structure while leaving it available in the pool. This separates errors in grouping from failures of candidate retrieval and allows the authors to test how different selectors respond to the same defect.
-
The main result has substantial experimental support within the tested setting. In the final sealed evaluation of 1,668 questions from 386 components, structure improves BM25 by 0.048 F1 [0.030, 0.069] and changes the strong selector by −0.010 [−0.018, −0.003] on the 2,898 primary states. Fresh source confirmation, within-session repetitions and a 100-passage retrieved pool variant strengthen this finding. Charging decomposition and guidance calls also makes the quality cost comparisons useful for deciding whether the additional structure is worth using.
Critical weaknesses
- The benefit for a weak selector is established most clearly for BM25, while evidence for smaller language model selectors remains limited. The exploratory Llama 3.1 8B experiment reports a structure gain of 0.033 F1 [0.013, 0.053], but this result does not receive the same confirmatory evaluation as the main BM25 comparison. The paper therefore leaves open a practical question: can decomposition consistently improve a cheaper language-model selector enough to offer a competitive quality cost trade-off against direct selection by a stronger model? The reported cost comparisons address the tested policies, but do not establish this benefit for smaller model selectors. This limits the deployment implications without undermining the demonstrated BM25 result.
Questions for authors
-
Clarification: In the sealed evaluation, how often does the structure-guided policy actually select a passage from a subquestion group before falling back to the full candidate pool? Could the small average effect partly reflect limited use of the structure?
-
Clarification: For the exploratory Llama 3.1 8B experiment, do the recorded results and call costs support a competitive quality cost trade-off for structure assisted selection compared with BM25 and direct selection by the stronger model? Which conclusions about smaller model deployment, if any, are supported by the existing experiment?
2 comments
Thanks.
Thank you for the precise review.
Q1. How often is the structure used? On the sealed split the structure directs 1.12 of the 2.74 remaining reads (41%). In 0.64 of them the group holds a single candidate, so the structure alone decides the read. The "one step in five" in Section 2 came from two-hop development states and is corrected.
Limited use explains part of BM25's small average gain. Where the structure steers reading to the evidence, it lifts BM25 by 0.195 F1, against 0.048 overall. It does not explain the difference between the selectors. Both spend the same number of reads in the structure, and on the same states the model selector changes by −0.015 [−0.037, +0.003] (Section 3.1, Table 11).
W1 and Q2. Smaller selector. In the exploratory run with Llama 3.1 8B (Table 13), structure adds +0.033 F1 [+0.013, +0.053] to the 8B selector. With the decomposition charged, the 8B selector with structure scores 0.335 F1 at USD 0.89 per 1,000 continuations and is dominated by deepseek-flash selecting directly (0.414 at USD 0.68). It reaches the frontier only if the structure is free (about USD 0.42). The run therefore does not support the claim that decomposition makes a small selector competitive.
Two settings could change this. A pipeline in which the small model also writes the decomposition is untested (Section 8). A larger budget was tested for BM25. At a fixed budget of eight reads, BM25 with structure costs 0.40 times as much as the model selector (0.470 against 0.531 F1), but the model selector with four reads is still cheaper and better (0.508 at USD 0.70).