- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether accuracy improvements in search agents reflect better use of retrieved evidence. It introduces counterfactual evidence accounting, which removes passages containing accepted-answer aliases and reruns the complete search policy to distinguish D-failing from D-surviving correctness.
Critical strengths
- The paper examines multiple search-agent families and includes closed-book comparisons, evidence substitution, residual-witness checks, and replay sensitivity analyses.
Critical weaknesses
W1. Incomplete evidence deletion limits the interpretation of D-survival. Table 1 (p. 6) reports that both primary judges identify residual direct-answer witnesses in 28.5% of D-surviving cases. Consequently, D-surviving correctness cannot be directly interpreted as evidence independence, and the reported 92% concentration of scaling gains depends on the intervention's ability to remove relevant evidence.
W2. Scaling comparisons are confounded by execution differences. Section 4 (p. 5) states that the 3B checkpoint uses a different tensor-parallel configuration from the 7B and 14B checkpoints. Appendix C.5 (p. 18) further reports an 8.57-point accuracy shift when execution configuration changes, raising concerns about attributing the observed differences primarily to model scale.
W3. Self-ablation reduces evidence-survival risk at a substantial correctness cost. Table 3 (p. 8) shows that self-ablation reduces D-surviving risk from 8.14% to 2.44%, but lowers coverage to 80.78% and total correct yield by 8.26 percentage points. This raises questions about its practical utility as a deployment-time diagnostic, particularly when correctness and coverage are also important.
Questions for authors
Q1. Can the authors evaluate stronger semantic evidence-removal interventions, beyond answer-alias matching, to determine how much D-surviving correctness remains when paraphrased and indirect evidence is also removed?
Q2. How robust are the scaling conclusions under fully matched execution configurations and repeated independent rollouts, rather than question-level bootstrap uncertainty from a single paired execution?
3 comments
Response to Reviewer Z6Sm (2/2)
(W2, Q2) Configuration and execution variability. We agree that the tensor-parallel difference and single-execution intervals were limitations. The 8.57-pp example used inconsistent parsers: with the same documented parser the difference is 0.06 pp on the same 1,692 questions, with unchanged D-surviving counts, and all 3,738 primary-ladder labels are unchanged. We withdraw it as an estimate of the configuration effect (Appendix C.5). Five matched TP2 RTX5090 blocks yield accuracy/surviving/failing gains of +10.79/+10.91/−0.13 pp on the fixed 623 questions, and five matched TP1 H100 executions +10.43/+11.08/−0.64, with failing growth negative in all five H100 executions. All-700 analyses and adverse strict-parser bounds retain predominantly surviving growth. Table 1 summarizes the gains; Figure 7 in Appendix L and Appendix N report execution variability separately from question uncertainty. These repetitions establish repeatability for fixed checkpoints and configurations, without isolating parameter count. About 25.5% of questions are ordinary-correct and deletion-incorrect, versus about 2–4% under ordinary replay; joint correctness across replays also grows with scale, so the additional effect of deletion on growth remains uncertain (Tables 35–36).
(W3) Correctness cost and diagnostic purpose. We agree that withholding correct answers is a substantial cost. In the historical evaluation, lowering survival risk from 8.14% to 2.44% at 80.78% coverage withholds 214 correct answers and needs another rollout. In a new support-or-abstain task (98 development and 350 held-out application questions, mandatory initial retrieval), self-ablation retains 71 correct answers supported by both judges versus 86 for containment and 82 for sufficiency, withholding 30 correct answers; containment has higher coverage and valid yield with lower joint risk, and adding self-ablation to either baseline shows no detected validity gain at matched coverage (§7.3; Table 2; Appendix Q). We therefore present self-ablation as a diagnostic rather than a deployment gate: on 951 ordinary-correct questions, its D-survival/full-support AUROCs are .843/.525 versus .562/.812 for sufficiency, and over five matched replays its D-survival AUROC is .848 versus .560 for ordinary replay (Appendices H.2, H.4).
Across matched repetitions, the alias-defined result recurs. Stronger removal and stopping controls clarify its scope. Together, these results support reporting accuracy, evidence support, deletion response and release costs under stated interventions and response policies.
We are happy to address any remaining questions.
Thank you again for your review. We wanted to follow up on our responses and revised manuscript. Section 5.3 and Table 1 summarize stronger removal and repeated, matched execution; Appendix C.5 explains the parser correction to the earlier configuration comparison. Section 7.3 reports the supported-answer evaluation.
We have addressed every point raised in your review in our responses and revised manuscript through additional experiments, analyses, and clarifications.
If any concerns remain about these analyses or their interpretation, we would appreciate the opportunity to address them before discussion closes.
Response to Reviewer Z6Sm (1/2)
We thank the reviewer for the precise review and the questions about removal completeness, execution variability and the cost of self-ablation.
(W1, Q1) Stronger semantic removal. We agree that D-surviving denotes correctness under the specified intervention, not evidence independence; the revision states this (§1; §3.2). Three paired blocks on a separately selected, dataset-stratified 300-question panel target direct, paraphrased and jointly derivable support at each search and refill, using observed history (§5.3; Appendices L–M). Ordinary growth is 8.33 percentage points (pp). Semantic surviving/failing growth is +11.00/−2.67 pp; their difference is +13.67 [6.33,21.22], positive in every block (approximate question-and-block intervals). Stronger removal lowers surviving correctness at both sizes (7.00→4.89% at 3B; 16.67→15.89% at 14B). The surviving-growth difference from local count control is +1.67 [−0.89,4.44]; the control matches removal counts, while relevance, displayed tokens and later searches can differ.
Two blinded models audit every alias, semantic and control trajectory. At 14B, the share with detected support or an uncertainty finding falls from 19.44–20.56% under alias deletion to 8.56–9.78% under semantic removal, across all questions; differences from count control remain below zero at both sizes, including missing-judgment bounds and question/block uncertainty. Bounds retain all 58 unresolved judgments among 10,800; surviving growth without support-or-uncertainty findings is 9.78–10.00 pp. Shared judge errors remain uncalibrated, so missed evidence remains possible.
Combining semantic and source removal. On 125 previously examined, annotation-eligible questions, Union adds annotated-title removal to full-pool semantic removal; on identical inputs it excludes everything the semantic rule excludes; later queries can differ, and no annotated source is displayed (Figure 6; Appendix S, Tables 75–77). It reduces surviving growth from 5.60 to 0.80 pp, a paired −4.80 [−9.60,−0.80]; the difference in edited-world accuracy gains between Union and semantic removal, and the surviving-growth difference from count control, have intervals including zero. At 14B, 203/375 Union trajectories reach the four-search limit without answering. One final-answer turn given to every trajectory recovers 26 correct answers without new retrieval. A separate experiment also recovers correct answers under neutral wording; the 14B Union-specific effect of explicit final-answer wording is −1.33 [−5.33,1.60] pp (Appendices S, U). On 100 further questions selected before evaluation, one execution block gives Union surviving/failing growth of 0/9 pp (4/8 after continuation), and all four prespecified comparisons have Bonferroni-adjusted 98.75% intervals including zero (Appendix V).
(Continued in the next comment.)