acceptodds
Under review as a conference paper at ICLR 2027

The Decision Value of Inferred Structure in Budgeted Evidence Acquisition

Abstract

Model-inferred structures guide which evidence a retrieval-augmented system reads under a limited budget. When these structures omit useful evidence, the effect on answer quality may depend on whether later reading can recover it. We investigate this relationship through controlled comparisons on MuSiQue, starting from identical evidence states and independently varying structure and passage selection. We introduce structural defects by removing unread supporting passages from the structure while keeping them in the candidate pool. On affected states in a confirmation experiment using new source components, the same defect reduces answer F1 by with a lexical selector and with a strong model selector. This asymmetry persists with a second model endpoint and with 100-passage retrieved pools. On a sealed split of 1,668 questions, opened once for the final evaluation, the same defect reduces F1 by with the lexical selector and with the model selector, and structure changes the model selector's F1 by (within ). The model selector more often finds the omitted evidence after returning to the full candidate pool. Structure improves the lexical selector but adds little to the strong model selector (, bounded within ), and none of the tested reliability-based controllers reliably outperforms using the strong selector throughout. Across MuSiQue and 2WikiMultihopQA, structure-free policies also remain on the quality–cost frontier at every tested budget. These findings identify fallback recovery as a key factor in the cost of structural defects and show that the benefit of explicit structure depends on the selector used.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, multi-hop question answering, budgeted evidence acquisition, question decomposition, routing and gating, counterfactual evaluation, cost-quality trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SVdmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether question decomposition and passage grouping improve evidence selection under a limited reading budget. Using MuSiQue and 2WikiMultihopQA, it compares policies that choose paragraph bodies from visible candidate titles, with budgets of one, two or four reads. Its main controlled experiments start from identical evidence states and vary structure, selector strength and omissions from the inferred groups. Structure improves BM25 but adds little to the strong model selector in the main evaluation. Trajectory analyses suggest that the strong selector more often recovers support omitted from the structure.

Critical strengths
  1. The same state comparisons make the policy effects easier to interpret. The question, evidence prefix, candidate pool and remaining budget are fixed, and injected omissions remove support from the structure while leaving it available in the pool. This separates errors in grouping from failures of candidate retrieval and allows the authors to test how different selectors respond to the same defect.

  2. The main result has substantial experimental support within the tested setting. In the final sealed evaluation of 1,668 questions from 386 components, structure improves BM25 by 0.048 F1 [0.030, 0.069] and changes the strong selector by −0.010 [−0.018, −0.003] on the 2,898 primary states. Fresh source confirmation, within-session repetitions and a 100-passage retrieved pool variant strengthen this finding. Charging decomposition and guidance calls also makes the quality cost comparisons useful for deciding whether the additional structure is worth using.

Critical weaknesses
  1. The benefit for a weak selector is established most clearly for BM25, while evidence for smaller language model selectors remains limited. The exploratory Llama 3.1 8B experiment reports a structure gain of 0.033 F1 [0.013, 0.053], but this result does not receive the same confirmatory evaluation as the main BM25 comparison. The paper therefore leaves open a practical question: can decomposition consistently improve a cheaper language-model selector enough to offer a competitive quality cost trade-off against direct selection by a stronger model? The reported cost comparisons address the tested policies, but do not establish this benefit for smaller model selectors. This limits the deployment implications without undermining the demonstrated BM25 result.
Questions for authors
  1. Clarification: In the sealed evaluation, how often does the structure-guided policy actually select a passage from a subquestion group before falling back to the full candidate pool? Could the small average effect partly reflect limited use of the structure?

  2. Clarification: For the exploratory Llama 3.1 8B experiment, do the recorded results and call costs support a competitive quality cost trade-off for structure assisted selection compared with BM25 and direct selection by the stronger model? Which conclusions about smaller model deployment, if any, are supported by the existing experiment?

2 comments

Thank you for the precise review.

Q1. How often is the structure used? On the sealed split the structure directs 1.12 of the 2.74 remaining reads (41%). In 0.64 of them the group holds a single candidate, so the structure alone decides the read. The "one step in five" in Section 2 came from two-hop development states and is corrected.

Limited use explains part of BM25's small average gain. Where the structure steers reading to the evidence, it lifts BM25 by 0.195 F1, against 0.048 overall. It does not explain the difference between the selectors. Both spend the same number of reads in the structure, and on the same states the model selector changes by −0.015 [−0.037, +0.003] (Section 3.1, Table 11).

W1 and Q2. Smaller selector. In the exploratory run with Llama 3.1 8B (Table 13), structure adds +0.033 F1 [+0.013, +0.053] to the 8B selector. With the decomposition charged, the 8B selector with structure scores 0.335 F1 at USD 0.89 per 1,000 continuations and is dominated by deepseek-flash selecting directly (0.414 at USD 0.68). It reaches the frontier only if the structure is free (about USD 0.42). The run therefore does not support the claim that decomposition makes a small selector competitive.

Two settings could change this. A pipeline in which the small model also writes the decomposition is untested (Section 8). A larger budget was tested for BM25. At a fixed budget of eight reads, BM25 with structure costs 0.40 times as much as the model selector (0.470 against 0.531 F1), but the model selector with four reads is still cheaper and better (0.508 at USD 0.70).

Thanks.

Reviewer_E1gcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is about some structure in Retrieval Augmented question answering. It is not clear what a structure, the motivation nor the proposed approach.

The paper is hard to read let alone understand. I suggest the authors re-write it and resubmit, in this way it will be possible for reviewers to give any comments on it (goals, motivation, approach, etc.).

Critical strengths

N/A

Critical weaknesses

N/A

Questions for authors

N/A

2 comments

Thank you for your time. We are sorry the paper was hard to follow, and we have rewritten it. In brief:

To answer a multi-hop question under a small reading budget, many systems ask an LLM to split the question into sub-questions and to assign the retrieved passages to them. We call this the structure. Structures are often wrong, and the usual remedy is to follow them only when they look reliable. We test the assumption behind that remedy by measuring what an error costs. From the same reading state, we hide a needed passage from the structure and continue with BM25 or with the model choosing the next passage.

The revised abstract present this directly. If a particular section is still unclear, we would be glad to know which.

Reviewer_E1gcOpenReview ↗

Acknoledge authors' response

Thank you for your response. I maintain my score as think that the paper may benefit from deep re-writing. I also suggest that the authors re-write the paper and try using consistent terminology with previous work (e.g., sequence of sub-questions, question decomposition vs 'structure').

Reviewer_J51KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the role of model-inferred structure (decomposing a question into sub-questions and assigning candidate passages to each) in multi-hop QA when only a few passages can be read. The authors demonstrate that the cost of errors in the structure depends mainly on whether the fallback selector can recover the missing passages, and not on the accuracy of the structure itself. They show that structure helps a weak selector (BM25) but adds little on top of a strong model-based selector, and that gating controllers based on structure reliability do not beat simply using the strong selector.

Critical strengths
  • Uses same-state counterfactuals to isolate the reading decision from the rest of the pipeline.
  • Pre-specified decision rules, a sealed held-out split and replication across data, models and retrieved pools.
  • Charges the decomposition call in the cost comparison, which makes the comparison with structure-free selectors fair.
Critical weaknesses
  • Limited novelty: It is expected that a model-based selector recovers missing passages better than BM25. Also, the structure guides only ~20% of reading steps (Appendix E), so its limited value on top of the strong selector is not surprising.
  • Narrow setting: The experiments use a simple coverage rule, small pools of titled Wikipedia paragraphs and budgets of at most 4. It is not clear if the findings extend to untitled document chunks, larger budgets or other question types such as temporal/aggregation questions.
Questions for authors
  • What happens when the structure is used for more of the reading, for example with a larger reading budget?
  • The strong selector may pick passages because it already knows the Wikipedia titles. Can the authors test this by hiding the titles and showing a short description instead?
1 comment

Thank you for the careful review. We ran both experiments you suggested (Section 6, Tables 15 and 16). The hosted model has been updated since the submission, so each experiment includes its own reference condition in the same session.

Q1. More structure, larger budget. Under the coverage rule a larger budget lengthens only the fallback. We therefore added an exhaustive rule, which reads every assigned passage, and ran both rules at four and eight reads on 181 new questions. The exhaustive rule raises the structure's share of reads from 27% to 59%, and the model selector gets worse (−0.035 [−0.064, −0.009]). At eight reads, structure changes the model selector by −0.003 and the recovery gap is +0.195. One prediction of ours failed, and the paper reports it: the exhaustive rule did not detectably raise the model selector's loss to the omission.

Q2. Titles. We replaced every title with a one-sentence description that may not name the passage's subject, on 233 new questions. The model selector's advantage does not come from titles. The recovery gap (BM25's loss to the omission minus the model selector's) is +0.377 [+0.275, +0.477] without titles and +0.173 with them. Hiding titles costs BM25 0.21 F1 and the model selector 0.06. Structure still adds little to the model selector (+0.014 [−0.009, +0.040]). A description removes the title but not world knowledge, so the test shows that recovery does not need titles. It also covers untitled chunks.

W1a. "Expected." We agree that the model is the better selector, and the introduction now says so. Three results do not follow from that.

  1. With the intact structure both selectors reach the needed passage equally often. The whole difference arises in the fallback, where the model selector reads the omitted passage in 81% of cases and BM25 in 44%.
  2. A gate that is given the gold label "a needed passage was dropped" is no better than a random gate, on three models (Table 2).
  3. Once the decomposition call is charged, most of the cost advantage of structure disappears (Section 5).

W1b. "20% of steps." That figure came from two-hop development states. On the sealed split it is 41%, and Section 2 is corrected. Limited use cannot explain the difference between the selectors, because both use the same structure for the same number of reads. Where the structure steers reading to the evidence, it lifts BM25 by 0.195 F1 and leaves the model selector unchanged (−0.015). When it directs more reads (Q1), the model selector does worse.

W2. Setting.

  • Rule. Besides coverage, the paper tests rewriting, alternation, a hybrid and now the exhaustive rule. None beats model-named selection.
  • Pool size. Pools of 100 retrieved passages give the same result (Section 6, Table 1).
  • Untitled chunks and larger budgets. See Q2 and Q1.
  • Question types. Temporal and aggregation questions are untested, and Section 8 says so. Where the needed passages cannot be inferred from what was read, we expect the fallback to fail and structure to matter.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.