- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies Pay to Reveal, a sequential retrieval setting in which a controller can inspect a fixed catalog of document titles but must spend a limited acquisition budget to obtain question-conditioned, length-capped excerpts from document bodies. It proposes two policies: BM25-RF, which expands the retrieval query with previously acquired excerpts, and Residual-CMG, which combines static title-BM25 relevance with the residual signal from a support-document classifier, defined as the difference between the classifier's scores in the acquired state and those in the empty state. The paper also derives a finite-horizon support-recall bound under an explicit expected-progress assumption. Experiments cover 22,246 retrieval questions and 3,000 downstream answer questions across 2WikiMultiHopQA, MuSiQue, and HotpotQA FullWiki. BM25-RF improves both acquisition and downstream answer quality on the first two datasets but degrades on HotpotQA. Retaining the static BM25 prior improves the learned policies' support-document average precision relative to a classifier-only control.
Critical strengths
-
The paper specifies the information-access protocol precisely enough to support replication and evaluates it at substantial scale: 22,246 acquisition questions and 3,000 downstream answer questions, with paired bootstrap confidence intervals and Holm-corrected tests where reported. This is a substantial empirical investment for a sequential-retrieval study.
-
It provides credible evidence that simple lexical feedback (BM25-RF) improves both acquisition and downstream F1 on 2Wiki and MuSiQue—including raising all-support coverage from 0.3432 to 0.6104 on 2Wiki—and it reports the HotpotQA degradation transparently rather than suppressing it.
Critical weaknesses
-
The residual‑state subtraction component in Residual‑CMG lacks independent empirical validation. As shown in Table 11, Prior+Full‑CMG accounts for 89 %‑99 % of the AP improvement from Direct‑CMG to Residual‑CMG. Residual‑CMG only brings marginal AP increments (0.0171, 0.0002, 0.0049). Critically, there are no paired statistical significance tests for these small gaps, nor corresponding downstream answer‑quality comparisons. Therefore it remains unclear whether the residual correction provides tangible practical benefits beyond simply retaining static title‑BM25 prior.
-
Given the linear logistic classifier setup, subtracting empty‑state logit cancels out all classifier features invariant to acquired evidence. The residual term is purely an internal state‑dependent contrast computed inside the pre‑trained classifier, rather than a genuinely separate, independent ranking signal. This blurs the conceptual novelty of Residual‑CMG as a new scoring formulation.
Questions for authors
-
Classifier training uses local candidate pools and local BM25 normalization, but inference evaluation uses full‑catalog normalization and retrieval‑generated candidate sets. What empirical evidence separates genuine algorithmic gains from artifacts compensating this train‑evaluation mismatch?
-
Is there any evaluated experimental setting where both progress coefficient γ and per‑step selection‑regret produce non‑vacuous, numerically informative bounds for Theorem 1? If no such setting exists, should Theorem 1 be re‑framed in the manuscript as a conditional accounting framework rather than an approximate performance guarantee for BM25‑RF / Residual‑CMG?
-
Subtracting the empty‑state logit cancels static features in this linear classifier, making the residual merely an internal contrast rather than an independent ranking signal. Can targeted ablations separate gains from the state‑dependent contrast operation versus combining classifier outputs with the static title‑BM25 prior?
Response to Reviewer kqMr
Q1. Does the residual gain survive training with retrieval candidates and catalog normalization?
The original answer-F1 advantage on 2Wiki and HotpotQA reversed under this training condition. At the same λ = 0.1, simple fusion (Prior+Full-CMG) led Residual-CMG by 0.0856 F1 on 2Wiki and 0.0697 on HotpotQA. On MuSiQue the residual led by 0.0084, with a paired 95% interval of [−0.0003, +0.0173] (Table 17, p. 23).
We varied candidate source and BM25 normalization using the same training questions, histories, state-weighting rule, feature definitions and procedure for selecting regularization. Each fit selected its own C. The table gives support AP differences (Residual minus Prior+Full); paired 95% intervals appear in Table 16 (p. 22).
On 2Wiki, switching to retrieval candidates favored simple fusion under either normalization. On HotpotQA, its lead was largest with catalog normalization and retrieval candidates. On MuSiQue, retrieval candidates favored the residual on AP. The original fit used different histories, so only comparisons among the four retrained models isolate the two training choices.
Q2. Does Theorem 1 give a numerically informative guarantee for either policy?
No. None of the evaluated settings supplies both an informative progress coefficient γ and a bound on per-step selection regret. We adopted the reviewer’s suggested framing: Section 4.3 presents Theorem 1 as a conditional lower bound on expected support-document recall. Equations (7) and (8) show how a given rate of progress and the accumulated losses from action selection determine that bound (p. 5, proof in Appendix A.1, p. 13).
The branching experiment measures final recall after continuing from eight alternative next reads (Table 25, p. 25). Estimating the theorem’s selection regret would require conditional one-step gains over the full catalog. The branch outcomes also supply no estimate of γ.
Q3. What does empty-state subtraction add beyond static score fusion?
You are right about the algebra. In this linear classifier, subtracting the same candidate’s empty-evidence logit at the same step cancels the intercept and every feature contribution that stays fixed. The residual measures how the evidence changes the score inside one classifier. Static title relevance enters separately through the BM25 prior.
With b(e) the normalized static title-BM25 score and f the classifier logit, Equation 5 (p. 5) defines the residual score. The three scoring rules are
sdirect(e)sfull(e)sres(e)=f(e,Ht),=b(e)+λf(e,Ht),=b(e)+λ[f(e,Ht)−f(e,∅t)].Direct versus Prior+Full tests the difference between using the classifier alone and adding its λ-scaled logit to BM25. Prior+Full versus Residual holds the classifier, prior and λ fixed and changes only the subtraction in the ranking rule. At the submitted λ = 0.1, subtraction improved answer F1 on 2Wiki and HotpotQA. The table reports question-paired results on 1,000 questions per dataset with six reads and the same Qwen3-4B-Instruct-2507 reader (Table 15, p. 22).
For the support-AP increments you identified, the Holm-adjusted p values were 0.0030 for +0.0171 on 2Wiki, 0.8661 for +0.0002 on MuSiQue and 0.0030 for +0.0049 on HotpotQA. Table 15 gives the paired intervals for AP, F1 and exact match. AP and F1 each use a separate 15-test Holm correction.
The answer advantage depended on the weight. When each rule selected λ by AP@6 on 500 held-out training questions per dataset, the paired F1 differences (Residual minus Prior+Full) were −0.0054 on 2Wiki, approximately zero on MuSiQue and +0.0015 on HotpotQA. All three intervals included zero (Tables 17 and 18, p. 23).