acceptodds
Under review as a conference paper at ICLR 2027

When Memory Is Enough to Decide but Not Enough to Reconsider: Memory Selection for Argument Survival

Abstract

Language models answer from selected or compressed memories, commonly evaluated by whether they preserve the current answer. Evidence redundant for that answer can become essential after a source is withdrawn. We study this gap across five evidence and reasoning benchmarks using budgeted memories and several open-weight language models. Memories that preserve a correct, supported initial answer can lose every evaluated revision basis, a set of records sufficient to justify the answer after withdrawal, even while the full history remains sufficient; this loss reaches 42.2% of such answers. Two histories can share a memory yet warrant different answers after the same withdrawal, so memory alone cannot identify the warranted answer. Paired interventions link model responses to argument completeness and dependence on withdrawn evidence. Replacing one necessary premise reduces revised-answer accuracy by 66.0 percentage points at matched record counts; abstention increases when alternative arguments share the withdrawn premise. These findings motivate argument survival as a selection objective across a prespecified family of withdrawals. Two records can each have zero marginal gain under this objective yet positive joint gain by completing a surviving argument. Pair-rescue implements simple joint admission to capture this value. Executable reasoning tasks and held-out GO-Plus ontology queries with surviving alternatives in the candidate pool show gains in retention and grounded review (correct revised answers with sufficient valid citations) at equal budgets, with current accuracy within a two-point non-inferiority margin. In a separate experiment, identifying and showing the model a verified basis already in memory adds 28.5 points of grounded review. A memory can be enough to decide yet not enough to reconsider.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
language model memory, memory selection, evidence selection, argument survival, revision basis, evidence revision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_US5sOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies memory selection for a reader that must reconsider an answer after evidence is withdrawn. A revision basis is a set of retained records that still justifies the warranted answer after withdrawal. The paper shows that memories that preserve a correct, supported current answer can lose every revision basis (up to 42.2% on eQASC), and that two histories can produce the same memory but require different answers after the same withdrawal, so memory alone cannot identify the warranted answer. Paired interventions link reader behavior to argument completeness and to dependence on the withdrawn evidence. The paper then proposes survival scoring, which selects records by whether complete arguments survive a prespecified withdrawal family under an executable checker, and Pair-rescue, which admits record pairs whose joint gain is positive when single additions have none. Evaluations cover ProofWriter, a CLUTRR-derived kinship task, held-out GO-Plus ontology queries, and natural-text sources (eQASC, RuleTaker, FEVEROUS, SciFact, QASPER), with Qwen3.8-27B and Gemma-4-31B-it as the main readers.

Critical strengths
  1. The paired interventions are clean. At matched record counts and similar lengths, replacing one necessary revision fact lowers revised-answer accuracy by 66.0 [61.8, 70.2] pp, and targeted repair beats sham repair by 64.7 [60.5, 68.8] pp. This causally separates argument completeness from record count and relevance, and it is the main evidence for the paper's central claim.

  2. The diagnosis is well quantified and grounded formally. On eQASC, 42.2% of correct answers with complete current proofs (269 of 638 question–reader pairs) lose every revision basis, and the construction in Section 2 shows that the same memory can require different answers after the same withdrawal. The statistics use paired cluster-bootstrap CIs, prespecified Holm families and a two-point non-inferiority margin for current accuracy.

Critical weaknesses
  1. Retention gains mostly do not become grounded answers, except in a setting built to favor the method. On ProofWriter, Robust over Verified gains +25.1 pp in retention but only +4.3 [2.7, 6.2] pp in grounded review (G_ref), and Pair-rescue over Robust gains only +0.6 [0.1, 1.2] pp in G_exec. In the three-reader comparison, of the 336 pairs where only Robust retains a basis, 12.5% are grounded and 73.8% abstain. The largest gain is on GO-Plus (+42.5 pp over Verified), but there every candidate pool is constructed to contain a complete justification that survives the withdrawal, which favors the method.

  2. The natural-text results are weak and mixed, and the method needs information that is rarely available at selection time. With model-proposed groups, revised-answer accuracy improves on RuleTaker but changes slightly or turns negative on FEVEROUS and SciFact. On QASPER, with support annotations withheld, estimated survival gives lower grounded-review rates than review-aware selection. Survival scoring relies on an executable checker or annotated arguments at selection time; on natural text this requires extra model calls to propose and check alternatives.

Questions for authors
  1. (Clarification) On GO-Plus, how often does the natural candidate pool contain a surviving justification without the construction that guarantees one? The answer determines whether the largest gain transfers.

  2. (Clarification) The separate experiment that shows the reader a verified basis adds 28.5 pp of grounded review. Did you combine this step with Robust or Pair-rescue selection, and what is the joint gain over Verified? This would show whether the retention gains can be turned into grounded answers.

  3. (Clarification) For natural text without an executable checker or annotations, what is the model-call cost of proposing and checking alternatives, and is there a setting other than the controlled 300 cases where estimated survival beats review-aware selection?

Reviewer_NpkeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies an important limitation of memory-augmented language models: a compressed memory may preserve enough evidence to answer a question correctly but discard alternative evidence needed to justify or revise that answer after a source is withdrawn.

Critical strengths

The paper distinguishes preserving an answer from preserving the evidence necessary to reconsider it. This is highly relevant to long-term memory and retrieval-augmented reasoning The indistinguishable-history construction in Equation (2) shows that two different histories can produce identical retained memories but require different post-withdrawal answers. This provides a rigorous information-theoretic motivation for the problem.

Critical weaknesses

The proposed survival objective requires evaluating whether a retained memory can derive an answer after a possible withdrawal. This is feasible in ProofWriter and GO-Plus, but most real-world LLM memory systems operate over unstructured conversations

The GO-Plus experiment reports an impressive increase in grounded review from 20.3% to 62.8%. However, the evaluation explicitly selects queries whose candidate pools already contain a complete surviving justification.

Questions for authors

How does Pair-rescue perform when candidate pools do not contain a complete surviving argument, or when the existence of such an argument is unknown?

Reviewer_hE7DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The goal of the paper is to study how the responses of the LLM change when they are relying on outdated memories. Memories get outdated as a result of information getting retracted. The paper motivates the problem by noting that around 10K papers are retracted every year. While the problem seems quite important, the paper is very difficult to follow. Despite several reads of the first two pages, this reviewer could not understand the precise problem and the key constructs. I have provided detailed feedback under the weaknesses section. I believe the readability is far below the level required for a top venue like ICLR.

Critical strengths
  1. How LLMs should change their output in the face of an evolving world is an important and timely problem.
Critical weaknesses
  1. The writing of the paper is too robotic at certain points, and clearly below the level expected at ICLR. Terse, sometimes disjointed, statements make it difficult to follow the arguments. For instance, the sentence at line 40 takes multiple readings to follow. The authors could have split the sentence into two. One saying that 10K papers per year are withdrawn. And second that there is a website, called Crossref, that provides API-based querying to check for retracted papers.
  2. More on writing: Figure 1 is very difficult to understand. The intro para where the figure is discussed mentions it very briefly. The caption of the figure is also rather short. The authors should describe what history H, revision basis B, etc, are. And how retraction of one item renders the conclusions obsolete. Line 43 describes several important items in quick succession and without going into much detail. It is unclear how b1 and b2 are linked to each other. Similarly, the key sentence on line 46, connecting these items is very difficult to parse. Unpacking each item would be greatly helpful.
  3. It is unclear what “Preserving the current proof can thus conceal the loss of every surviving alternative” means. What are the surviving alternatives? What is the loss of an alternative?
  4. Jumping into concrete numbers (42.2%) when the problem has not been sufficiently well-motivated adds more confusion. Does it mean the problem is prevalent across a large variety of LLMs? Is this number representative of a large class of models? What is “LongLLMLingua” and how does this experiment connect to the previous statements?
  5. The paper uses terms without defining them. Having a clear definition of these terms is critical for understanding. Examples are: “full history answer” in line 78, “recovery” in line 79, “paired interventions” in line 80 and “written explanation” in line 81.
  6. Similarly, it is unclear what “marginal gain” and “joint gain” are in paragraph starting line 86.
  7. Section 2 jumps straight into terms like history and token budget. I suggest first describing the problem setup, e.g., the users, the LLMs they interact with, the claims that are retracted around them, and then moving to terminology.
Questions for authors

Please see questions in the weaknesses section.

1 comment

As the discussion period is now more than halfway through, we would like to follow up on our response and revised manuscript. We have revised the paper following your suggestions and would appreciate it if you could read the revised main text in full alongside our response. If these revisions address your concerns, we would be grateful if you could reconsider your assessment and score.

Please let us know if anything remains unclear. Thank you for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.