acceptodds
Under review as a conference paper at ICLR 2027

The Value of Evidence Selection Depends on Reasoning Effort

Abstract

Evidence selection reduces input length but also changes the information available for reasoning. Its accuracy and cost benefits may therefore depend on the reader’s reasoning effort. We test this dependence in multi-hop retrieval-augmented question answering with the model, prompt, retrieved passages, and selected contexts fixed. On held-out 2Wiki questions, a learned selector improves exact-match accuracy by 4.4 percentage points at minimal effort but reduces it by 3.0 points at medium effort. Despite the same 67.3% prompt-token reduction, reader-cost savings fall from 60.0% to 8.7%. To examine the role of evidence content, we restore omitted support in an annotation-guided intervention across three benchmarks. Relative to adding a similar-length non-support passage, restoration improves medium-effort accuracy by 9.3 points and reduces mean reported reasoning tokens by 42.2%. We extend this investigation to information computed within a source-only retrieval pipeline. Grounded execution reuses intermediate answers already extracted to guide search; exposing rather than withholding these answers, with acquired passages fixed, improves minimal-effort accuracy by 5.3 points on a three-benchmark development panel without another model call. Reuse also lowers medium-effort total API cost, including planning and extraction, while its accuracy effect is unresolved. These effort-dependent benefits motivate joint configuration selection. We evaluate joint calibration of existing evidence policies and reader effort on held-out questions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Reasoning Effort, Evidence Selection, Context Compression, Test-Time Compute, Inference Cost, Multi-Hop Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_auZBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper shows that the effectiveness of evidence selection in RAG depends on the reader’s reasoning effort.

Critical strengths

N/A

Critical weaknesses
  • The paper is hard to follow; after reading the abstract and introduction sections, I cannot figure out the main points/claims/results presented in the paper.
  • The authors use saturated benchmarks (back to 2018) and a small GPT-5-mini, which does not reflect the current development of frontier models.
  • The evaluation setup is limited.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer auZB (1/2)

Thank you for the feedback on clarity, the benchmarks and reader, and the scope of the evaluation. We revised the abstract and introduction so that the main claim and results appear earlier, and we extended the evaluation.

W1. Clarity of the main claim. The main claim is that the value of evidence selection depends on the reader's reasoning effort. We call this dependence the interaction. The paper shows three results.

  1. With the passages and the selected contexts fixed, evidence selection is worth more at low than at high effort, for three readers on 2Wiki. On held-out 2Wiki with gpt-5-mini, the same selected contexts gain 4.4 exact-match (EM) points over all five retrieved passages at minimal effort and lose 3.0 points at medium effort. Two open-weight readers, Qwen3-32B and Qwen3-8B, also benefit less from selection at higher effort on 2Wiki. For gpt-5-mini on held-out 2Wiki, the cost saving from selection also falls with effort, from 60.0% to 8.7%, because at higher effort most of the bill is output and the selected contexts lead to more reasoning.
  2. Controlled changes to the evidence connect this result to its content. On the 294 held-out 2Wiki questions whose passages contain all the supporting evidence, removing only the other passages improves gpt-5-mini accuracy by 7.5 points at minimal effort and has no clear effect at medium effort (+1.4, with an interval that includes zero). Restoring a missing supporting passage, compared with adding a passage of similar length that does not support the answer, improves accuracy at both efforts and cuts medium-effort reasoning by 42%. Showing the final reader the intermediate answers that the retrieval pipeline already extracted to guide search improves minimal-effort accuracy by 5.3 points, and by 7.0 points on 330 separate questions.
  3. On held-out questions, choosing the evidence policy and the effort together does better than choosing the policy by accuracy at a single effort, when cost receives low or high weight.

In the revision, the first paragraph of the introduction ends with the main finding, the three contributions carry these labels (interaction, interventions, calibration), and we simplified dense sentences in the abstract and introduction.

AuthorsOpenReview ↗

Response to Reviewer auZB (2/2)

W2. Older benchmarks and gpt-5-mini. These benchmarks are not saturated in our setting, where the reader sees only five retrieved passages: the best accuracy of any policy and effort is 66.4% EM for gpt-5-mini on held-out 2Wiki (Table 1) and 50.3% for Qwen3-32B over 1,500 questions (Table 37). We use them because they annotate which passages support each answer, which the removal and restoration interventions need. Every comparison is also paired on the same questions, so memorization that is common to all policies and efforts cancels out of the interaction.

We chose gpt-5-mini because it exposes discrete effort settings and reports reasoning tokens, which the accuracy and cost analysis needs. To reduce the dependence on one model, the revision adds two open-weight reasoning models as readers, and the dependence on effort holds with both on 2Wiki, without the reversal (Table 2). We agree that a larger frontier reader is a natural next step, and Section 6 lists broader readers among the limitations.

W3. The evaluation setup is limited. The submitted paper tested this dependence with one reader. The revision adds two readers whose effort is set differently (a thinking budget and a thinking switch), selection by the reader's own model in place of the released selector, 250 more questions each for HotpotQA and MuSiQue with Qwen3-32B, the removal intervention, and a confirmation of answer reuse on separate questions. The dependence on effort is now tested with three readers on 2Wiki and two readers on all three benchmarks (Table 2). The benefit of selection falls with effort for all three readers on 2Wiki, and for Qwen3-32B also on MuSiQue; on HotpotQA the differences in Table 2 are positive, but their 95% intervals include zero. Details are in our replies to Reviewers yvd7 and vzoj.

We hope the revised abstract and introduction make the main claim clear, and we would be grateful if you would take them into account in your assessment. We are glad to answer any further questions.

Thank you for pointing out that the original introduction made the main finding hard to identify. The question is whether passages chosen for a reader at one reasoning setting remain a good choice when that reader is given more reasoning effort. On 2Wiki, the same passage selection helped GPT-5-mini at minimal effort and hurt it at medium effort. In our first reply, we described testing a frontier reader as a next step. We have now run that comparison: on the same questions and passages, the interaction was 0.2 points for GPT-6-astra (Astra), against 7.4 for GPT-5-mini (Section 5.2). We also extended the evaluation beyond Wikipedia. On 600 held-out MultiHop-RAG news questions, giving Astra the first three retrieved passages cost 62% less than asking it to choose three. Exact-match accuracy was 84.3% and 83.3%, respectively (difference +1.0 points [−0.7, 2.7], Section 5.6). A selector should therefore be compared with spending the same budget on the reader.

In response to the three concerns you raised, we revised the introduction to state the main finding more directly, tested a frontier reader, and extended the evaluation beyond Wikipedia to news articles. We hope these changes address your concerns and would be happy to answer any remaining questions about the results or their interpretation.

Reviewer_vzojOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the value of evidence selection in RAG depends on the reader's reasoning effort, so evidence policy and effort should be treated as one joint decision rather than tuned separately. To isolate this interaction, it crosses four fixed policies over a five-passage pool (Full5, Top2, Top3, and the released UtilityQwen1.7B selector) with three API effort levels of one reader (gpt-5-mini), holding the selected contexts fixed across efforts and counting both input and generated tokens in cost. The main finding is that effort reverses the policy ranking: the learned selector beats Full5 at minimal effort but loses at medium effort, and its cost saving mostly disappears at higher effort despite an unchanged input reduction, which the authors attribute to extra reasoning on the selected contexts. Two interventions connect this interaction to evidence content. Restoring an omitted gold-support passage (vs. a length-matched non-support passage) helps more at minimal than at medium effort and shortens medium-effort reasoning, while reusing intermediate answers already computed for retrieval improves accuracy at minimal effort but mainly lowers cost at medium effort. Because no single policy is best at every effort, the paper finally selects policy and effort jointly over the 12 actions; this beats single-effort selection at the lowest and highest cost weights but not at an intermediate one.

Critical strengths
  1. Controlled design and transparent reporting

Policy outputs are computed once and reused across efforts; prompt, scorer and token cap are fixed; the primary contrast and reversal criterion were set before generation; and every cell, including null results (Top2, HotpotQA, Qwen confirmation), is reported with intervals. The interaction can be read directly from the tables, and the negative findings are credible.

  1. Evidence completeness changes generation length

With passage count matched exactly and length approximately, restoring the omitted support raises EM and cus medium-effort reasoning (-165 tokens vs. the control; 360.6 --> 221.8 vs. the Utility context itself), with unchanged answer length, and the direction holds for three readers. This is the paper's cleanest evidence that context content, not just length, drives output cost.

Critical weaknesses
  1. The reversal holds in one setting only.

It is resolved only on held-out 2Wiki with one reader. It is unresolved on the three-dataset sample, absent on HotpotQA, and unresolved against Top2.

  1. The reversal may come from a weak selector, not from selection in general

Utility returns an empty context for 42/500 questions and one passage for 278, so two-hop support is often missing. The result may only show that missing support hurts a reasoning reader more.

  1. Most of the lost cost saving is arithmetic, not extra reasoning

At medium effort, output tokens dominate the bill, so cutting input saves little anyway. With Table 7 and list prices (Eq. 8), Utility would save only 16% even with zero extra reasoning.

Questions for authors

Please see the weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer vzoj (1/2)

Thank you for the careful review and for three precise points: where the reversal holds, whether it comes from the selector, and what drives the lost cost saving. Each led to changes in the paper.

W1. The reversal holds in one setting. We agree that the reversal is specific, and we see that the submission did not separate it clearly from the main claim. The claim is the interaction: the benefit of a fixed evidence policy changes with reasoning effort. The ranking reversal is the strongest form of this interaction, where the benefit also changes sign, and it appears with gpt-5-mini on held-out 2Wiki. The revision makes this distinction explicit in the introduction, Section 3.2, and the limitations.

Part of the evidence for this claim was already in the submission, but we did not bring it forward clearly; the revision does (Section 5.2, Table 2; Appendices C.3 and C.5). With gpt-5-mini, the minimal-minus-medium interaction (EM points, paired 95% intervals) is:

Setting (submitted paper)Interaction
Utility − Full5, held-out 2Wiki (500 questions)+7.4 [3.4, 11.4]
Utility − Full5, three-benchmark sample (750 questions)+5.6 [1.3, 9.8]
Utility − Full5, 2Wiki development sample with HippoRAG2 retrieval (1,000 questions)+9.0 [5.7, 12.3]
BRIEF-Pro − raw passages, three benchmarks (450 questions)+8.9 [4.3, 13.5]

On the three-benchmark sample the two endpoint effects are not both resolved, so there is no strict reversal, but the interaction is resolved. On held-out 2Wiki, Utility − Top2 is unresolved (+1.8 [−1.2, 4.8]; Table 10) because Top2 − Full5 has nearly the same interaction (+5.6 [2.0, 9.1]): the dependence is not specific to the learned selector.

The revision extends the test to two open-weight readers with different effort controls. Qwen3-32B reads with no thinking or a 512-/2,048-token thinking budget on 500 questions per benchmark, with the released Utility and with Utility-32B, which runs the Utility prompt with the reader itself. The table gives selector − Full5 interactions, no thinking minus the 2,048-token budget.

DatasetUtilityReader-matched Utility-32B
2Wiki+5.8 [2.4, 9.2]+5.2 [2.2, 8.3]
MuSiQue+7.0 [2.1, 11.7]+8.4 [3.3, 13.6]
HotpotQA+3.2 [0.0, 6.7]+2.2 [−0.8, 5.2]
Pooled (1,500)+5.3 [3.0, 7.5]+5.3 [3.1, 7.5]

Qwen3-8B, with its thinking switch, gives +7.2 [3.2, 11.3] on 2Wiki (Appendix C.7), and repeating the gpt-5-mini reads on held-out 2Wiki gives +6.0 [2.0, 10.0], with an unresolved minimal-effort gain; on the same new reads, NativeK shows the full reversal (W2; Appendix C.8). The interaction is therefore resolved for all three readers on 2Wiki, and for Qwen3-32B also on MuSiQue and pooled over 1,500 questions (Table 2; Appendix C.6). On HotpotQA, all interactions in Table 2 are positive (+0.8 to +3.2) but none is resolved.

AuthorsOpenReview ↗

Response to Reviewer vzoj (2/2)

W2. The reversal may come from the selector rather than from selection. The counts of 42 empty and 278 single-passage contexts are correct. Because each context is held fixed and only the reader's effort changes, the experiment measures how the value of a given context depends on effort, whatever the quality of the selector that produced it. Your concern is then whether this dependence is specific to Utility's contexts, which we tested in several ways.

The same dependence appears when the reader's own model chooses the passages. Utility-32B runs the Utility prompt with Qwen3-32B, the reader, and selects two passages more often than Utility (751 versus 401 of 1,500 questions). It gives higher accuracy than Utility at every effort, yet its advantage over Full5 still falls with effort, +5.3 [3.1, 7.5] over 1,500 questions. NativeK lets gpt-5-mini choose which passages to keep, with Utility's count for each question, and it reproduces the 2Wiki reversal: +4.8 [1.6, 7.8] at minimal effort and −2.8 [−5.2, −0.4] at medium effort (Appendix C.8). It tests which passages are kept; Utility-32B, which keeps two passages more often, and the subsets below test how many. Top2, which always keeps two passages, and the BRIEF-Pro compressor (+8.9 [4.3, 13.5]) show it as well. Utility's own interaction remains without its 42 empty contexts (+6.6 [2.6, 10.7]) and among its 180 contexts with at least two passages (+8.9 [2.2, 15.6]; Appendix C.9).

For gpt-5-mini, missing support does not explain it either. On the 294 held-out 2Wiki questions whose passages contain all supporting evidence, removing only the non-support passages improves accuracy by 7.5 points at minimal effort and has no resolved effect at medium effort (interaction +6.1 [1.7, 10.6]). Starting from the Utility context with its omitted support restored, removing that support costs 5.8 points at both efforts (293 questions; interaction 0.0 [−3.4, 3.4]; Appendix D.6). Over 1,147 questions, restoring an omitted supporting passage, relative to a similar-length non-support passage, also helps more at minimal effort (+14.1) than at medium effort (+9.3; difference +4.8 [1.8, 7.9], Appendix D.5). For gpt-5-mini, missing support therefore does not hurt the reasoning reader more; on the 294 support-complete questions, what changes with effort is the benefit of removing passages that do not support the answer, though removal also shortens the input. Full5 already reaches 85.0% EM on them at medium effort, which leaves less room for improvement (Appendix D.6). For Qwen3-32B, your explanation holds in part: a descriptive regression over all selected contexts attributes 3.1 of Utility's 5.8-point interaction to omitted support and 2.9 to removed non-support passages (Appendix D.7).

W3. Output share and extra reasoning. You are right that output dominates the medium-effort bill, and the abstract, introduction, and Section 5.3 now give both parts of the saving (Appendix F, Table 84). The split also shows that the output share is itself set by effort: input is 89% of the Full5 list-price bill at minimal effort and 24% at medium effort. The same 67.3% input cut is therefore worth 60.1% at minimal effort but 16.1% at medium effort, before any change in generation. Of the 51.4-point fall in saving between these efforts, 44.1 points come from this shift.

Within those 16.1 points, extra reasoning still matters. At medium effort, selected contexts make gpt-5-mini generate 33 more reasoning tokens per question (Table 3), which removes 7.1 points [2.1, 12.3], nearly half of the 16.1-point saving, and leaves an observed saving of 8.9% at list prices (8.7% as billed).

Share of the Full5 bill, list pricesMinimalMedium
Saving from the input cut, generation unchanged60.116.1
Change from extra reasoning0.0−7.1
Change from visible output+0.20.0
Observed saving60.48.9

The rows add up to the observed saving up to rounding.

Both parts of the saving depend on effort, so the cost value of the same context changes with effort, as its accuracy value does.

Thank you again for the accounting point, which made the cost analysis more precise. We hope the new readers, selectors, and interventions address your concerns, and we would welcome any further questions.

Thank you again for your point that output's share of the bill limits the saving from shorter inputs. We tested it with a second model family. On held-out 2Wiki questions, selection reduced Gemini 4 Carbon’s input tokens by 65.5% at both effort levels. At high effort, the input reduction alone would have saved 2.0% if output length had stayed the same. The observed saving was 2.9% (failed requests excluded), compared with 61.4% at low effort. Section 5.3 and Appendix J.3 report the comparison. We also addressed all of your other points in our earlier replies: the scope of the reversal and selection by the reader itself (Section 5.2). We would be happy to clarify any of these points.

The new evidence supports the reframed claim, so I am keeping my rating at 3.

Reviewer_yvd7OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper authors presented evidence selection which remains valuable as inference-time reasoning effort changes, with performance varying across effort levels. Experiments conducted on 2WikiMultiHopQA showed UtilityQwen1.7B improved exact match by 4.4P at minimal effort and reduces it by 3P at medium effort, while cost savings diminish at higher effort. Cost accounting is thoughtful. The paper separates input tokens, reasoning tokens, visible output, and billed cost. Equation 3 clearly explains why an input reduction can overstate the savings.

Critical strengths

Paper adresses important research by showing that evidence-selection policies can change relative effectiveness depending on reasoning effort; Controlled experimental design, careful statistical analysis, and detailed cost accounting strengthen the reliability of the findings. The support-restoration experiment helps distinguish the value of specific evidence from the effect of adding more text, while the candid discussion of limitations acknowledges the single-reader setup and unresolved results on MuSiQue and HotpotQA.

Critical weaknesses

The main weakness is limited generalizability, as the headline reversal is demonstrated using a single reader and dataset. Besides results on HotpotQA and MuSiQue remain inconclusive. n addition, selector’s mismatch with the reader, provider-defined reasoning effort, and limited evidence for answer reuse raise questions about whether the findings reflect general effects or specific experimental choices.

Morerover, restoration analysis and joint calibration also have methodological limitations, as well as conditional eligibility, and unclear cost trade-offs. In summary, the paper does not establish a clear mechanism explaining why reasoning effort changes the effectiveness of evidence selection.

Questions for authors

Can the policy reversal be replicated across multiple readers and datasets using a reader-matched selector?

How robust are the findings to different providers’ definitions of reasoning effort and alternative baseline choices?

Better to clarify the statistical evidence for answer reuse and the restoration effect across effort levels?

How sensitive is joint calibration to its cost weights and penalty parameters, and can the authors identify a mechanism explaining the observed interaction between evidence selection and reasoning effort?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yvd7 (1/2)

Thank you for the detailed review. Below, W1–W6 are the weaknesses in the order of the review (W1 one reader and one dataset, W2 selector–reader mismatch, W3 provider-defined effort, W4 answer reuse, W5 restoration, calibration, eligibility, and cost trade-offs, W6 mechanism), and Q1–Q4 are the questions. We call an effect resolved when its paired 95% interval excludes zero.

W1, W2, Q1. Replication across readers and datasets, with a reader-matched selector. The reversal holds for gpt-5-mini on held-out 2Wiki (+4.4 EM at minimal effort, −3.0 at medium, interaction +7.4 [3.4, 11.4]), and it appears again with selection by the reader: gpt-5-mini itself selects as many passages as Utility returns for each question (NativeK, Appendix C.8): +4.8 [1.6, 7.8] at minimal effort and −2.8 [−5.2, −0.4] at medium effort. Repeating the Utility reads in the same run gives an interaction of +6.0 [2.0, 10.0], with an unresolved minimal-effort gain.

The interaction holds in more settings than the reversal. With gpt-5-mini, it is also resolved on the 750-question three-benchmark sample (+5.6 [1.3, 9.8]). With Qwen3-32B as reader, it is resolved on 2Wiki, MuSiQue, and pooled, both for the released Utility and for Utility-32B, which runs the Utility prompt with Qwen3-32B itself, on 500 questions per benchmark, with no thinking or a 512- or 2,048-token thinking budget. The table gives selector − Full5 interactions, no thinking minus the 2,048-token budget (EM points, paired 95% intervals; Table 2, Appendix C.6).

DatasetUtilityReader-matched Utility-32B
2Wiki+5.8 [2.4, 9.2]+5.2 [2.2, 8.3]
MuSiQue+7.0 [2.1, 11.7]+8.4 [3.3, 13.6]
HotpotQA+3.2 [0.0, 6.7]+2.2 [−0.8, 5.2]
Pooled (1,500)+5.3 [3.0, 7.5]+5.3 [3.1, 7.5]

Utility-32B gives higher accuracy than Utility at every effort, yet its advantage over Full5 still falls with effort. MuSiQue, inconclusive with gpt-5-mini on 250 questions, is resolved for both selectors on 500; HotpotQA is positive but not resolved. With Qwen3-8B as both selector and reader, on 768 further HotpotQA and MuSiQue questions kept apart from development, the interactions are +7.9 [2.5, 13.7] on MuSiQue and +5.3 [0.0, 10.5] on HotpotQA, with Bonferroni 97.5% intervals for the two tests. MuSiQue is resolved; the HotpotQA interval reaches zero (unadjusted [0.8, 9.7]; Appendix C.8, Table 40).

With the Qwen3 readers and their own selectors, no reversal is resolved. The closest cases are Utility-32B on 2Wiki (+4.0 [1.8, 6.3] with no thinking, −1.2 [−3.2, 0.6] at the 2,048-token budget) and NativeK with Qwen3-8B on MuSiQue (+4.9 [0.6, 9.2] with thinking off, −3.0 [−6.6, 0.5] with thinking on).

We agree that the reversal itself is specific to gpt-5-mini on 2Wiki. It is the strongest case of the paper's claim, which is the interaction: with the same contexts, prompt, reader, and scoring, changing only the reasoning effort changes how much a fixed evidence policy helps. We revised the introduction and Section 3.2 to make this clearer.

W3, Q2. Other definitions of effort and other baselines. The two Qwen3 readers have no provider effort setting. Qwen3-32B uses thinking budgets that we set (1.4% of reads at the 2,048-token budget reach it; Appendix C.6), and Qwen3-8B uses its thinking switch, which gives an interaction of +7.2 [3.2, 11.3] on 2Wiki (Appendix C.7). Both added readers run locally, so a second API provider remains untested (Section 6). The interaction is nevertheless resolved on 2Wiki under three effort definitions: provider effort levels, thinking budgets, and a thinking switch. Across baselines, it appears for the rank-based Top2 (+5.6 [2.0, 9.1] on held-out 2Wiki), the BRIEF-Pro compressor (+8.9 [4.3, 13.5]), Utility-32B, and NativeK, and with a second retriever, HippoRAG2 (+9.0 [5.7, 12.3] on a 1,000-question 2Wiki development sample). It is not resolved for Top3, which keeps three passages (+1.6 [−1.6, 5.0]; Appendices C.2 and C.8). With Top3 or Top2 as the reference instead of Full5, Utility's interaction on held-out 2Wiki is +5.8 [2.0, 9.4] and +1.8 [−1.2, 4.8]; the latter is not resolved because Top2 shows nearly the same interaction as Utility (Table 10).

W4, W5, Q3. Statistical evidence for restoration and answer reuse. Appendix D.5 and Appendix D.8 now report, at each effort, the EM effect with its paired interval, the numbers of answers fixed and broken, and the effort interaction:

InterventionMinimalMediumMinimal − medium
Restoration (1,147 questions)+14.1 [11.3, 16.8]+9.3 [7.3, 11.5]+4.8 [1.8, 7.9]
Answer reuse, development (360)+5.3 [1.7, 14.4]+0.8 [−1.9, 2.4]+4.4 [1.2, 14.0]
Answer reuse, confirmation (330)+7.0 [3.0, 11.0]+0.3 [−1.8, 2.5]+6.7 [2.4, 10.9]
AuthorsOpenReview ↗

Response to Reviewer yvd7 (2/2)

W4, W5, Q3 (continued). Restoration fixes 220 answers and breaks 58 at minimal effort, and fixes 135 and breaks 28 at medium effort (McNemar p = 2e−23 and 5e−18, treating questions as independent). It is positive on each benchmark at both efforts, and at medium effort it cuts reasoning by a mean of 165 tokens. Eligibility requires that the five passages contain all support, that Utility omits exactly one supporting passage, and that a non-support passage of similar length is available as the control. The estimate is therefore the value of that passage relative to the control, not the quality of a selector.

The answer-reuse confirmation addresses the question of general effects. Its 330 questions share no passage with the development panel, questions that share a passage, title, or question text are linked and resampled together, and the largest such group has four questions (one development group held 93 of the 120 2Wiki questions), and its starting retriever differs from development. The minimal-effort gain, its primary endpoint, replicates at +7.0 [3.0, 11.0], and medium-effort API cost, with planning and extraction included, still decreases by 1.6% [0.5, 2.9] at list prices (Appendix D.8). The gain is resolved on 2Wiki (+12.9 [5.1, 21.0]) and MuSiQue (+9.8 [2.3, 17.4]) but not on HotpotQA (+0.7 [−5.2, 6.6]; Table 66).

W5, Q4. Calibration sensitivity and cost trade-offs. We swept 401 cost weights and five selector penalties for both calibration sets (Appendix E.2). Every setting shows the same three regimes: gains over single-effort selection at low and high cost weights, and a loss at intermediate weights. Calibrating on one half of held-out 2Wiki reduces the mean loss at λ = 250 on the other half from 4.2 to 0.8 points, so the loss comes mainly from calibration transfer.

Two further controls qualify the results (Appendix E.2, Table 75). When calibration is refitted inside each bootstrap draw, the intervals of the low- and high-weight gains include zero, and so does the interval of the λ = 250 loss. When the single-effort rules choose the policy with the same cost-weighted objective instead of accuracy, they select the same configuration as joint calibration at 90% to 99% of the cost weights. The gain therefore comes from choosing the policy with cost in view. On cost trade-offs, Section 3.3 now states that one EM point is worth $10/λ per 1,000 questions, and Section 5.3 separates the cost saving into the part set by the output share and the part lost to extra reasoning.

W6, Q4. Where the interaction comes from. Selection does two things: it removes passages not annotated as support, and it can omit passages that are. A new intervention separates them on the 294 held-out 2Wiki questions whose five passages contain all supporting evidence. SupportOnly keeps only the supporting passages, in their original order. With new gpt-5-mini reads, it improves EM by 7.5 points at minimal effort and has no resolved effect at medium effort (interaction +6.1 [1.7, 10.6]; Section 5.4, Appendix D.6). Starting from the Utility context with its omitted support restored, removing that support costs 5.8 points at both efforts (293 questions; interaction 0.0 [−3.4, 3.4]) and adds 72 reasoning tokens at medium effort.

For gpt-5-mini, the effort dependence on these questions therefore comes from removing passages that do not support the answer, while missing support costs accuracy at both efforts. Full5 already reaches 85.0% EM there at medium effort, which leaves less room for improvement (Appendix D.6). Removal also shortens the input and can move the supporting passages forward. Position alone does not explain the result: on the 248 questions where they already come first, the interaction is +7.7 [2.8, 12.5]. These contrasts do not separate the effect of content from that of length (Appendix D.6). For Qwen3-32B the split differs: a descriptive regression over all selected contexts attributes 3.1 of Utility's 5.8-point interaction to omitted support and 2.9 to removed non-support passages (Appendix D.7). These interventions show how the reader responds to specified changes in the evidence; they do not identify its internal mechanism (Section 6).

Thank you again for these questions; they led to the added readers, the removal intervention, and the calibration controls. We would be grateful if you would reconsider your assessment in light of these results, and we are happy to clarify anything further.

Thank you for the detailed responses and conducting additional experiments. The extended evaluation across readers and effort controls, reader-matched selection, and stronger statistical evidence substantially address some of my concerns. I also appreciate the clearer distinction between the broadly supported interaction and the reversal observed in a specific setting. However, some limitations remain, particularly the unresolved HotpotQA results, limited frontier-model coverage, and weaker evidence for an independent benefit from joint calibration. I will keep my score.

Thank you for your reply and for naming the three remaining concerns.

1. Does the 2Wiki result extend to newer readers (GPT-6-astra and Gemini 4 Carbon)?

On the 500 held-out 2Wiki questions, GPT-5-mini (Mini) and GPT-6-astra (Astra) received the same passages from the released selector (Utility) and the same five-passage reference contexts (Full5). Increasing reasoning effort reduced the benefit of selection over Full5 by 7.4 percentage points in exact-match accuracy (EM) for Mini and by 0.2 points for Astra. The decrease was 7.2 points larger for Mini (paired 95% interval [2.6, 11.7]). Each reader used its own effort settings (Section 5.2, Appendix J.1, Table 106).

For Gemini 4 Carbon, the lower-minus-higher effort interaction was −0.8 EM points [−3.9, 2.4], even though increasing effort raised Full5 accuracy from 71.6% to 77.8%. Utility reduced input tokens by 65.5% at both settings. Savings on successful reader responses (failed API attempts excluded) fell from 61.4% at low effort to an estimated 2.9% at high effort (95% interval [−6.3%, 11.4%]). Section 5.3 and Appendix J.3 give the cost breakdown and sensitivity to capped outputs.

2. How large is the interaction on HotpotQA?

Both Qwen readers were tested on the same 1,800 questions held out from earlier experiments. Astra was tested on 1,200 of those questions. For each question, Utility set the number of passages and each Qwen reader chose which ones to keep. Astra chose three passages. The two gain columns give the accuracy gain over Full5 in EM points. The interaction is the lower-effort gain minus the higher-effort gain.

ReaderLower-effort gainHigher-effort gainInteraction (98.75% interval, Bonferroni over four comparisons)
Qwen3-8B+0.9+0.1+0.8 [−1.4, 3.2]
Qwen3.8-27B−0.3−1.5+1.2 [−0.9, 3.2]
GPT-6-astra+0.3−0.5+0.8 [−0.9, 2.5]

All three intervals include zero, with upper bounds of 2.5 to 3.2 points. At both efforts, each Qwen reader scored 9–10 EM points higher when it selected the passages than when it read the same number in retrieval order (Section 5.2, Appendix K).

3. What does joint calibration add?

When the single-effort baselines used the same cost objective as joint calibration, they chose the same configuration at 90–99% of the cost weights. We agree that this leaves limited evidence of an independent gain from joint calibration (Appendix E.4).

We then asked whether the same budget was better spent on passage selection or on the final reader. We tuned reading effort for Full5 and for the first three passages in retrieval order (Top3). We allowed native selection (the reader selects three passages; Native4 in Section 5.6) to tune both selection and reading effort, and counted both calls.

Original 2Wiki and HotpotQA questions. On the 500 held-out 2Wiki questions, at the lowest budget tested with Mini, Full5 gained 5.2 [2.8, 7.5] EM points over calibrated native selection, and Top3 gained 4.3 [1.9, 6.8]. Both were within 0.1% of the total cost of native selection. At two higher budgets Top3 was cheaper and less accurate. On the original HotpotQA sample, the intervals for both policies’ accuracy differences from native selection included zero at all four budgets (Appendix L.1).

Held-out news questions. On MultiHop-RAG, we calibrated on 200 questions and evaluated on 600. The splits shared a retrieval corpus and had disjoint annotated support articles. The table compares each policy against calibrated native selection at the lowest budget, counting selection and reading at uncached prices.

ReaderPolicyEM difference (paired 95% interval)Cost change
GPT-5-miniFull5−1.1 [−3.9, 1.3]−1.1%
GPT-5-miniTop3−1.5 [−3.8, 0.8]−2.2%
GPT-6-astraFull5+1.0 [−0.2, 2.7]−43.6%
GPT-6-astraTop3+1.0 [−0.7, 2.7]−62.3%

All four alternatives met the prespecified cost criterion of less than 5% additional cost. None established the prespecified accuracy gain of more than 2 EM points, using one-sided 98.75% bounds. For Astra, Full5, Top3 and native selection all chose low reader effort (Appendix M, pp. 76–77, Tables 125–127).

Our earlier replies addressed the other points in your review. We would be grateful if you would reconsider your assessment in light of these comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.