- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies memory selection for a reader that must reconsider an answer after evidence is withdrawn. A revision basis is a set of retained records that still justifies the warranted answer after withdrawal. The paper shows that memories that preserve a correct, supported current answer can lose every revision basis (up to 42.2% on eQASC), and that two histories can produce the same memory but require different answers after the same withdrawal, so memory alone cannot identify the warranted answer. Paired interventions link reader behavior to argument completeness and to dependence on the withdrawn evidence. The paper then proposes survival scoring, which selects records by whether complete arguments survive a prespecified withdrawal family under an executable checker, and Pair-rescue, which admits record pairs whose joint gain is positive when single additions have none. Evaluations cover ProofWriter, a CLUTRR-derived kinship task, held-out GO-Plus ontology queries, and natural-text sources (eQASC, RuleTaker, FEVEROUS, SciFact, QASPER), with Qwen3.8-27B and Gemma-4-31B-it as the main readers.
Critical strengths
-
The paired interventions are clean. At matched record counts and similar lengths, replacing one necessary revision fact lowers revised-answer accuracy by 66.0 [61.8, 70.2] pp, and targeted repair beats sham repair by 64.7 [60.5, 68.8] pp. This causally separates argument completeness from record count and relevance, and it is the main evidence for the paper's central claim.
-
The diagnosis is well quantified and grounded formally. On eQASC, 42.2% of correct answers with complete current proofs (269 of 638 question–reader pairs) lose every revision basis, and the construction in Section 2 shows that the same memory can require different answers after the same withdrawal. The statistics use paired cluster-bootstrap CIs, prespecified Holm families and a two-point non-inferiority margin for current accuracy.
Critical weaknesses
-
Retention gains mostly do not become grounded answers, except in a setting built to favor the method. On ProofWriter, Robust over Verified gains +25.1 pp in retention but only +4.3 [2.7, 6.2] pp in grounded review (G_ref), and Pair-rescue over Robust gains only +0.6 [0.1, 1.2] pp in G_exec. In the three-reader comparison, of the 336 pairs where only Robust retains a basis, 12.5% are grounded and 73.8% abstain. The largest gain is on GO-Plus (+42.5 pp over Verified), but there every candidate pool is constructed to contain a complete justification that survives the withdrawal, which favors the method.
-
The natural-text results are weak and mixed, and the method needs information that is rarely available at selection time. With model-proposed groups, revised-answer accuracy improves on RuleTaker but changes slightly or turns negative on FEVEROUS and SciFact. On QASPER, with support annotations withheld, estimated survival gives lower grounded-review rates than review-aware selection. Survival scoring relies on an executable checker or annotated arguments at selection time; on natural text this requires extra model calls to propose and check alternatives.
Questions for authors
-
(Clarification) On GO-Plus, how often does the natural candidate pool contain a surviving justification without the construction that guarantees one? The answer determines whether the largest gain transfers.
-
(Clarification) The separate experiment that shows the reader a verified basis adds 28.5 pp of grounded review. Did you combine this step with Robust or Pair-rescue selection, and what is the joint gain over Verified? This would show whether the retention gains can be turned into grounded answers.
-
(Clarification) For natural text without an executable checker or annotations, what is the model-call cost of proposing and checking alternatives, and is there a setting other than the controlled 300 cases where estimated survival beats review-aware selection?
As the discussion period is now more than halfway through, we would like to follow up on our response and revised manuscript. We have revised the paper following your suggestions and would appreciate it if you could read the revised main text in full alongside our response. If these revisions address your concerns, we would be grateful if you could reconsider your assessment and score.
Please let us know if anything remains unclear. Thank you for your time and feedback.