acceptodds
Under review as a conference paper at ICLR 2027

Search-Agent Scaling Gains Survive Answer-Witness Deletion

Abstract

Search agents are evaluated by answer accuracy and by whether retrieved evidence supports their answers, but neither reveals how an answer responds when that evidence is removed. We introduce counterfactual evidence accounting: at every retrieval call, passages matched as containing an accepted-answer alias are deleted and replaced by lower-ranked candidates, and the complete search policy is rerun. Paired rollouts partition ordinary accuracy exactly into correct answers that become incorrect after this deletion (D-failing) and those that remain correct (D-surviving). On 623 matched questions, the released 14B Search-R1 checkpoint is 10.3 percentage points more accurate than the 3B checkpoint, and 9.5 of those points are D-surviving; the analysis detects no increase in D-failing correctness. Five released Qwen2.5 search-agent recipes, prompted Qwen checkpoints without search-agent reinforcement learning, and an SSRL Llama comparison show the same concentration, which persists under residual-witness, parsing, and exposure sensitivity analyses. With matched weights, 7.9 of the 9.5 points overlap with closed-book correctness. Deleting passages that match the agent's own answer yields a label-free diagnostic, self-ablation: with one additional rollout, it lowers D-surviving correctness among released answers from 8.1% to 2.4% at 81% coverage, at a cost of 8.3 points of total correct yield. Search-agent evaluations should report how accuracy responds to targeted changes in retrieved evidence, alongside accuracy and support.

Primary area:
datasets and benchmarks
Keywords:
search agents, counterfactual evaluation, retrieval-augmented generation, evidence dependence, model scaling, selective prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z6SmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether accuracy improvements in search agents reflect better use of retrieved evidence. It introduces counterfactual evidence accounting, which removes passages containing accepted-answer aliases and reruns the complete search policy to distinguish D-failing from D-surviving correctness.

Critical strengths
  • The paper examines multiple search-agent families and includes closed-book comparisons, evidence substitution, residual-witness checks, and replay sensitivity analyses.
Critical weaknesses

W1. Incomplete evidence deletion limits the interpretation of D-survival. Table 1 (p. 6) reports that both primary judges identify residual direct-answer witnesses in 28.5% of D-surviving cases. Consequently, D-surviving correctness cannot be directly interpreted as evidence independence, and the reported 92% concentration of scaling gains depends on the intervention's ability to remove relevant evidence.

W2. Scaling comparisons are confounded by execution differences. Section 4 (p. 5) states that the 3B checkpoint uses a different tensor-parallel configuration from the 7B and 14B checkpoints. Appendix C.5 (p. 18) further reports an 8.57-point accuracy shift when execution configuration changes, raising concerns about attributing the observed differences primarily to model scale.

W3. Self-ablation reduces evidence-survival risk at a substantial correctness cost. Table 3 (p. 8) shows that self-ablation reduces D-surviving risk from 8.14% to 2.44%, but lowers coverage to 80.78% and total correct yield by 8.26 percentage points. This raises questions about its practical utility as a deployment-time diagnostic, particularly when correctness and coverage are also important.

Questions for authors

Q1. Can the authors evaluate stronger semantic evidence-removal interventions, beyond answer-alias matching, to determine how much D-surviving correctness remains when paraphrased and indirect evidence is also removed?

Q2. How robust are the scaling conclusions under fully matched execution configurations and repeated independent rollouts, rather than question-level bootstrap uncertainty from a single paired execution?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Z6Sm (1/2)

We thank the reviewer for the precise review and the questions about removal completeness, execution variability and the cost of self-ablation.

(W1, Q1) Stronger semantic removal. We agree that D-surviving denotes correctness under the specified intervention, not evidence independence; the revision states this (§1; §3.2). Three paired blocks on a separately selected, dataset-stratified 300-question panel target direct, paraphrased and jointly derivable support at each search and refill, using observed history (§5.3; Appendices L–M). Ordinary growth is 8.33 percentage points (pp). Semantic surviving/failing growth is +11.00/−2.67 pp; their difference is +13.67 [6.33,21.22], positive in every block (approximate question-and-block intervals). Stronger removal lowers surviving correctness at both sizes (7.00→4.89% at 3B; 16.67→15.89% at 14B). The surviving-growth difference from local count control is +1.67 [−0.89,4.44]; the control matches removal counts, while relevance, displayed tokens and later searches can differ.

Two blinded models audit every alias, semantic and control trajectory. At 14B, the share with detected support or an uncertainty finding falls from 19.44–20.56% under alias deletion to 8.56–9.78% under semantic removal, across all questions; differences from count control remain below zero at both sizes, including missing-judgment bounds and question/block uncertainty. Bounds retain all 58 unresolved judgments among 10,800; surviving growth without support-or-uncertainty findings is 9.78–10.00 pp. Shared judge errors remain uncalibrated, so missed evidence remains possible.

Combining semantic and source removal. On 125 previously examined, annotation-eligible questions, Union adds annotated-title removal to full-pool semantic removal; on identical inputs it excludes everything the semantic rule excludes; later queries can differ, and no annotated source is displayed (Figure 6; Appendix S, Tables 75–77). It reduces surviving growth from 5.60 to 0.80 pp, a paired −4.80 [−9.60,−0.80]; the difference in edited-world accuracy gains between Union and semantic removal, and the surviving-growth difference from count control, have intervals including zero. At 14B, 203/375 Union trajectories reach the four-search limit without answering. One final-answer turn given to every trajectory recovers 26 correct answers without new retrieval. A separate experiment also recovers correct answers under neutral wording; the 14B Union-specific effect of explicit final-answer wording is −1.33 [−5.33,1.60] pp (Appendices S, U). On 100 further questions selected before evaluation, one execution block gives Union surviving/failing growth of 0/9 pp (4/8 after continuation), and all four prespecified comparisons have Bonferroni-adjusted 98.75% intervals including zero (Appendix V).

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer Z6Sm (2/2)

(W2, Q2) Configuration and execution variability. We agree that the tensor-parallel difference and single-execution intervals were limitations. The 8.57-pp example used inconsistent parsers: with the same documented parser the difference is 0.06 pp on the same 1,692 questions, with unchanged D-surviving counts, and all 3,738 primary-ladder labels are unchanged. We withdraw it as an estimate of the configuration effect (Appendix C.5). Five matched TP2 RTX5090 blocks yield accuracy/surviving/failing gains of +10.79/+10.91/−0.13 pp on the fixed 623 questions, and five matched TP1 H100 executions +10.43/+11.08/−0.64, with failing growth negative in all five H100 executions. All-700 analyses and adverse strict-parser bounds retain predominantly surviving growth. Table 1 summarizes the gains; Figure 7 in Appendix L and Appendix N report execution variability separately from question uncertainty. These repetitions establish repeatability for fixed checkpoints and configurations, without isolating parameter count. About 25.5% of questions are ordinary-correct and deletion-incorrect, versus about 2–4% under ordinary replay; joint correctness across replays also grows with scale, so the additional effect of deletion on growth remains uncertain (Tables 35–36).

(W3) Correctness cost and diagnostic purpose. We agree that withholding correct answers is a substantial cost. In the historical evaluation, lowering survival risk from 8.14% to 2.44% at 80.78% coverage withholds 214 correct answers and needs another rollout. In a new support-or-abstain task (98 development and 350 held-out application questions, mandatory initial retrieval), self-ablation retains 71 correct answers supported by both judges versus 86 for containment and 82 for sufficiency, withholding 30 correct answers; containment has higher coverage and valid yield with lower joint risk, and adding self-ablation to either baseline shows no detected validity gain at matched coverage (§7.3; Table 2; Appendix Q). We therefore present self-ablation as a diagnostic rather than a deployment gate: on 951 ordinary-correct questions, its D-survival/full-support AUROCs are .843/.525 versus .562/.812 for sufficiency, and over five matched replays its D-survival AUROC is .848 versus .560 for ordinary replay (Appendices H.2, H.4).

Across matched repetitions, the alias-defined result recurs. Stronger removal and stopping controls clarify its scope. Together, these results support reporting accuracy, evidence support, deletion response and release costs under stated interventions and response policies.

We are happy to address any remaining questions.

Thank you again for your review. We wanted to follow up on our responses and revised manuscript. Section 5.3 and Table 1 summarize stronger removal and repeated, matched execution; Appendix C.5 explains the parser correction to the earlier configuration comparison. Section 7.3 reports the supported-answer evaluation.

We have addressed every point raised in your review in our responses and revised manuscript through additional experiments, analyses, and clarifications.

If any concerns remain about these analyses or their interpretation, we would appreciate the opportunity to address them before discussion closes.

Reviewer_fuXPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how a search agent’s accuracy changes when passages containing accepted answer aliases are removed. This lets it distinguish correct answers that become incorrect after alias deletion (D-failing) from those that remain correct (D-surviving). Across LLMs of the same family but different sizes, most of the net accuracy gains come from D-surviving correctness. The paper also proposes self-ablation, which removes the passages that match the agent’s own answer.

Critical strengths

The paper addresses a relevant limitation of common search-agent evaluations: the fact that retrieved evidence supports an answer does not imply that it caused the agent’s final answer. Their approach to testing this distinction by running a full new search trajectory is a useful methodological choice because it lets the agent adapt its queries and retrieve other information.

The empirical results are intriguing and could yield impactful insights. A clear pattern is indeed present in the experiments: most accuracy gains are in the form of D-surviving correctness as model size increases.

Critical weaknesses

My main concern is that the intervention intended to remove the evidence towards the answer still leaves substantial evidence behind. In E.1, both judges identify direct evidence in 28.5% of audited D-surviving cases and judge the answer still derivable in 34.8%. This suggests that larger models, with stronger reasoning, could increase D-surviving accuracy by finding or figuring out the remaining evidence. An item-level transition analysis across model sizes would make the reported effect much clearer and more interpretable: the fact that gains come in the form of D-surviving correctness might be because the model is able to answer new questions (including D-failing ones), but some of the previously D-failing ones become D-surviving because the larger model picks up more of the residual information.

The authors appropriately acknowledge this and note that it does not invalidate the decomposition, but it limits what the result allows us to interpret and what we can attribute this difference to. The adverse reassignment analysis in E.2 and the closed-book overlap are useful in that regard, but do not fully establish what would happen if the residual evidence were actually fully removed.

Execution variability is not fully characterized. Confidence intervals cover question variability rather than repeated-execution variability; although the independent 14B run helps in this direction, it does not address the stability of the scale comparison. I don’t think this is critical, given the consistency across the reported comparisons, but considering the variability in H.4, I would like to see this specifically accounted for in the main results in Figure 1.

The practical usefulness of selective release is unclear. Beyond the paper providing a useful diagnostic for deeper understanding and attribution, it is unclear why a correct answer that survives deletion should be considered undesirable, except in settings with explicit evidence-dependence requirements. It also seems drastic that the survival-risk metric treats such answers as undesirable but does not penalize incorrect answers, so reducing this metric does not necessarily improve reliability. In fact, self-ablation withholds 214 correct answers (8.26 percentage points of total correct yield), and when incorrect answers are also considered invalid, a simple string-containment rule outperforms self-ablation. I therefore find self-ablation more convincing as an evaluation tool than as a practical one, and I invite the authors to incorporate a setting in which the LLM is instructed to provide the correct answer only when supported by evidence, or abstain otherwise.

Questions for authors
  1. Can you rerun the scale comparison on an independently selected subset with more complete evidence deletion (e.g., as validated by the judges)?
  2. Could you provide an item-level transition analysis across model sizes? That is, how many previously D-failing correct answers become D-surviving at larger scales, and how many newly solved questions enter each category?
  3. How stable is the main scaling pattern across repeated executions?
  4. Can you evaluate self-ablation in a task where the agent is explicitly required to answer only when supported by retrieved evidence, and compare its correctness/coverage trade-off against baselines?

The first two points would have the greatest impact on my assessment.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fuXP (1/2)

We thank the reviewer for the constructive review and for identifying the analyses that matter most. The transition analysis clarifies the headline: most newly solved questions are D-failing, while transitions among previously correct questions account for most of the net D-surviving increase. Repeated executions reproduce this pattern.

(W1, Q2) Individual-question transitions. Of 93 newly solved questions, 67 become D-failing and 26 D-surviving. Losses are 24 failing and five surviving; among questions correct at both sizes, 46 move from failing to surviving and eight reverse. The surviving increase is 59 = (26−5) + (46−8): 21 net new survivors and 38 net category changes (§5.1; Appendix D, Figure 4). The failing increase is five because these changes offset 43 net new failing solutions. Thus the larger checkpoint acquires many deletion-sensitive solutions despite near-zero net failing growth.

Five H100 repetitions reproduce this structure: mean new failing/surviving counts are 63.2/31.0, with 49.0 failing-to-surviving and 6.6 reverse changes (Appendix N.2). For the 46 historical failing-to-surviving questions, matched-weight closed-book correctness rises from 11 (3B) to 31 (14B); both residual-witness judges find support in 17 of the 14B deletion traces. Seven have detected support without closed-book correctness and 21 show the reverse (§6; Table 5). In the H100 runs, 14B answers 70.6% of failing-to-surviving questions without retrieval, versus 31.9% of questions failing at both sizes. These measure availability; the information used in the ordinary answer remains uncertain.

(W1–W2, Q1) Stronger removal and remaining evidence. Three paired blocks on a separately selected, dataset-stratified 300-question panel target direct, paraphrased and jointly derivable support at each search and refill, using observed history (§5.3; Appendices L–M). Ordinary growth is 8.33 percentage points (pp); semantic surviving/failing growth is +11.00/−2.67 pp, a difference of +13.67 [6.33,21.22], positive in every block (approximate intervals resampling questions and blocks).

Two blinded models audit all alias, semantic and count-control trajectories. At 14B, the share with detected support or uncertainty falls from 19.44–20.56% under alias deletion to 8.56–9.78% under semantic removal, and also falls relative to count control at both sizes (ranges bound all 58 unresolved judgments among 10,800). Unflagged trajectories contribute 9.78–10.00 of the 11.00 surviving-growth points. Shared judge errors and missed evidence remain possible. With identical contexts, the benefit of question-aligned residual text is not clearly larger at 14B (Figure 9; Appendix P).

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer fuXP (2/2)

Source removal and additional questions. On 125 previously examined, annotation-eligible questions, adding annotated-title removal to full-pool semantic removal (Union) reduces surviving growth from 5.60 to 0.80 pp, a paired −4.80 [−9.60,−0.80]; the difference in edited-world accuracy gains between Union and semantic removal, and the surviving-growth difference from count control, have intervals including zero (Figure 6; Appendix S).

Across the three blocks, Union's net surviving increase is 18 observations smaller than under semantic removal: 14 from changes among questions ordinary-correct at both sizes and four from newly solved questions. This post-hoc accounting connects the stronger-removal result to the historical transition analysis (Appendix S, p. 68).

At 14B, 203/375 Union trajectories reach the four-search limit without answering; one final-answer turn applied to every trajectory recovers 26 correct answers without new retrieval. A separate experiment also recovers correct answers under neutral wording (Appendices S, U). On 100 questions selected before evaluation and excluding those recorded in earlier experiments, Union surviving/failing growth is 0/9 pp (4/8 after the turn); all four prespecified comparisons of accuracy gains have Bonferroni-adjusted intervals including zero, so this single execution leaves effects in either direction plausible (Appendix V).

(W3, Q3) Execution stability. Five matched TP2 RTX5090 blocks and five matched TP1 H100 executions reproduce predominantly surviving growth under alias deletion on the fixed 623 questions: surviving/failing gains are +10.91/−0.13 and +11.08/−0.64 pp. Table 1 summarizes the gains; Figure 7 in Appendix L and Appendix N report execution variability separately from question uncertainty. Under ordinary replay, most accuracy growth also comes from answers correct in both runs; the difference in surviving growth under deletion versus ordinary replay remains uncertain.

(W4, Q4) Supported-answer utility. We agree that survival risk ignores wrong answers, so the new evaluation scores joint wrong-or-unsupported risk. We ran the setting the reviewer proposes: after 98 development questions, a fixed support-or-abstain protocol evaluates 350 held-out application questions with mandatory initial retrieval; release rules filter the same 279 answers (§7.3; Table 2; Appendix Q). Self-ablation retains 71 correct answers supported by both judges versus containment's 86 and sufficiency's 82, requires another rollout, and withholds 30 correct answers. Containment has higher coverage and valid-answer yield with lower joint risk, and adding self-ablation to either baseline shows no detected validity gain at matched coverage. One execution and correlated model judgments limit generality. Self-ablation remains informative as a diagnostic: it predicts deletion response better than ordinary replay (AUROC .848 vs .560; Appendix H.4).

We are happy to address any remaining questions.

Thank you again for suggesting the transition analysis. Section 5.1 now separates newly solved questions from changes among previously correct questions, and Appendix S (p. 68) connects these transitions to the stronger-removal results. Our response also covers the stronger-removal results on an independent panel (§5.3), and the repeated executions and supported-answer evaluation you requested.

We have addressed every point raised in your review in our responses and revised manuscript through additional experiments, analyses, and clarifications.

If there are points you would like us to clarify or discuss further, please let us know while the discussion remains open.

Thank you for the response and the transition analysis.

  • It clarifies the result: most newly solved questions are D-failing, and the net D-surviving growth comes mainly from previously correct questions flipping from failing to surviving, which fits rising closed-book correctness.
  • Execution stability and the support-or-abstain evaluation are well addressed, and self-ablation is now framed as a diagnostic.
  • However, adding source removal reduces surviving growth from 5.6 to 0.8 pp on the 125 annotated questions, and on the fresh 100 questions the surviving-vs-failing difference is not statistically significant. The core claim, therefore, is not established under the most complete removal, which limits its interpretation.

I maintain my score.

Reviewer_5e8dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reruns search agents while deleting retrieved passages matching accepted-answer aliases at every retrieval call. It partitions ordinary correctness into D-failing and D-surviving outcomes. Across released checkpoints, most measured scaling gains fall in the D-surviving component. It also studies closed-book overlap, self-ablation, selective release, and attempted interventions.

Critical strengths
  • Applying deletion throughout a complete policy rerun provides a useful behavioral measurement of interactive retrieval systems. On the matched Search-R1 comparison, 9.47 of the 10.27-point accuracy gain survives the specified deletion, a finding that ordinary accuracy does not reveal.
  • The study carefully limits its interpretation and provides extensive sensitivity analyses, including residual witnesses, parsing, exposure, closed-book overlap, and replay placebos. It also reports self-ablation's loss of correct-answer yield, making the diagnostic's limitations assessable.
Critical weaknesses
  • The deletion result remains specific to the operational intervention. Both primary judges find direct witnesses in 28.5% of D-surviving cases and answer-derivable evidence in 34.8%. The adverse reassignment supports persistence of the pattern but does not rerun agents after stronger evidence removal. The evidence therefore supports survival under alias-based deletion, with limited validation under more complete removal of answer-supporting information.
  • The practical value of selective release needs a clearer application. Table 3 reduces D-surviving-correct output by withholding 214 correct answers and losing 8.26 points of total correct yield. For the combined criterion that counts wrong and D-surviving-correct answers as invalid, a free string-containment gate performs better. A concrete requirement for deletion-sensitive answers is needed to justify this operating point beyond its diagnostic use.
Questions for authors
  1. Does the scaling pattern persist after semantically identified residual witnesses are removed, with a count-matched control throughout the full rerun and repeated O/D runs under consistent execution settings?
  2. What concrete application requires correct answers to change under witness deletion? Under that application's objective, what benefit remains after accounting for correctness, coverage, and additional rollout cost?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 5e8d (1/2)

We thank the reviewer for the careful review and the precise questions about stronger removal and practical value.

The evaluation distinguishes new solutions from changed deletion responses on already solved questions. Of 93 newly solved questions, 67 are D-failing, yet transitions among previously correct questions contribute 38 of the 59 additional D-surviving answers (§5.1). Reporting both prevents a large surviving share of net growth from being read as evidence that most newly solved questions survive deletion.

(W1, Q1) Stronger removal throughout retrieval. We agree that D-surviving denotes correctness under the specified intervention, not evidence independence; the revision states the claim this way (§1; §3.2). Three paired 3B/14B blocks on a separately selected, dataset-stratified 300-question panel target direct, paraphrased and jointly derivable support at each search and refill, using observed history (§5.3; Appendices L–M). Ordinary growth is 8.33 percentage points (pp). Semantic surviving/failing growth is +11.00/−2.67 pp; their difference is +13.67 [6.33,21.22], positive in every block (approximate 95% intervals resampling questions and blocks jointly). Stronger removal lowers surviving correctness at both sizes but preserves its growth across sizes. Surviving growth differs from a count control by +1.67 [−0.89,4.44]; that control matches removal counts within its own search, while relevance and later searches can differ.

Two models, blinded to size, condition and agent answer, audit every alias, semantic and control trajectory. At 14B, the share with detected support or uncertainty falls from 19.44–20.56% under alias deletion to 8.56–9.78% under semantic removal (ranges bound all 58 unresolved judgments among 10,800), and also falls relative to count control at both sizes. Unflagged trajectories contribute 9.78–10.00 of the 11.00 surviving-growth points. Shared judge errors are not bounded, so missed evidence remains possible.

Combining semantic and source removal. On 125 previously examined questions with annotated supporting sources, adding source-title removal to full-pool semantic removal (Union) reduces surviving growth from 5.60 to 0.80 pp, a paired −4.80 [−9.60,−0.80]; the difference in edited-world accuracy gains between Union and semantic removal, and the surviving-growth difference from count control, have intervals including zero (Figure 6; Appendix S). At 14B, 203/375 Union trajectories reach the four-search limit without answering; one final-answer turn applied to every trajectory recovers 26 correct answers without new retrieval. A separate experiment also recovers correct answers under neutral wording (Appendices S, U). The surviving-minus-failing growth interval includes zero under each rule before continuation, and for Union also after this turn. On 100 further questions selected before evaluation, one execution block gives Union surviving/failing growth of 0/9 pp (4/8 after the turn), and all four prespecified comparisons of accuracy gains are uncertain (Appendix V).

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer 5e8d (2/2)

(Q1) Repeated matched execution. Five two-GPU tensor-parallel (TP2) RTX5090 blocks and five single-GPU (TP1) H100 executions, each with identical settings across sizes, reproduce predominantly surviving growth under alias deletion on the fixed 623 questions: surviving/failing gains are +10.91/−0.13 and +11.08/−0.64 pp (Table 1; Figure 7; Appendices L, N). In the H100 setting, a full-rerun control that randomly removes as many candidates as alias deletion at every retrieval call turns 3.2% of 14B questions from correct to incorrect, versus 25.6% under alias deletion; it matches candidate counts, not displayed changes (Appendix N.3). Ordinary replay also concentrates growth in answers correct in both runs; the difference in surviving growth under deletion versus ordinary replay remains uncertain (§5.3).

(W2, Q2) Application and practical benefit. We agree that correct, supported answers can legitimately survive withdrawal. We evaluated a support-or-abstain setting, and self-ablation is not the better release rule there. With 98 development and 350 held-out application questions and mandatory initial retrieval, all rules filter the same 279 answers (Table 2; Appendix Q). Self-ablation retains 71 correct answers supported by both judges, versus 86 for containment and 82 for sufficiency. It withholds 30 correct answers, needs an extra rollout, and has higher joint risk than containment; adding it to either baseline shows no detected validity gain at matched coverage. Sufficiency has lower loss than self-ablation at all three prespecified abstention penalties. In the historical analysis, self-ablation lowered survival risk from 8.14% to 2.44% at 80.78% coverage but withheld 214 correct answers (Appendix H.3).

The revision therefore presents self-ablation as a diagnostic: it estimates deletion response without reference answers, e.g., on queries where gold deletion cannot be run. On the same 951 ordinary-correct questions, its D-survival/full-support AUROCs are .843/.525, versus .562/.812 for sufficiency. Across five matched replays, its D-survival AUROC is .848 versus .560 for ordinary replay, a paired gap of .289 [.250,.328]. Shared alias matching and the retrospective correct-only sample limit generality (§§7.1–7.2; Appendix H.4).

We are happy to address any remaining questions.

Thank you again for your review. We wanted to follow up on our responses and the revised manuscript posted on November 15. The stronger-removal comparisons and matched repetitions are summarized in §5.3 and Table 1; the supported-answer evaluation is in §7.3 and Table 2.

We have addressed every point raised in your review in our responses and revised manuscript through additional experiments, analyses, and clarifications.

If any concerns remain after these additions, please let us know. We would welcome the opportunity to clarify them before discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.