acceptodds
Withdrawn from ICLR 2027

Correcting Errors Is Not Enough:Exact Audits of Answer Preservation on Shared Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory-augmented large language models (LLMs) need to use helpful memory while preserving correct answers when memory is unreliable. We call the latter ability correct-answer preservation. Comparing this ability across model config- urations is difficult because different configurations may answer different tasks correctly without memory, while missing responses and incomplete task alignment create further uncertainty. We propose a shared-task audit that compares corruption rates on tasks that every configuration in a declared group answers correctly at baseline. When task alignment or outcomes are incomplete, we derive sharp finite- set bounds over all task assignments and missing-result completions consistent with the retained summaries. We also prove an exact solver that needs to enumerate missing-correct counts only for the two target configurations while preserving the same outer bounds as full search. The framework further identifies when additional overlap information is sufficient to resolve an otherwise uncertain comparison and when task-level labels are required. On reported MEMIMPACTBENCH counts, pair-specific shared-task support determines 14 of 15 harmful-memory directions, while one all-six shared set determines eight misleading-memory and seven stale- memory directions. We also validate the audit on public row-level StrategyQA conflict records spanning four model configurations and 12 incorrect-prior con- ditions: all 144 count-based intervals contain the matched-task truth, and all 43 strict direction certificates match the directly counted sign. These results show that rates computed on different task subsets do not necessarily support the same cross-model ordering. The shared-task audit states which preservation differences are supported by the available evidence and which require additional information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM memory, answer preservation, shared-task evaluation, partial identification, model auditing, missing data, memory reliability, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a1jtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to compare correct-answer preservation across memory-augmented LLM configurations. A corruption rate is normally computed on the tasks each configuration answers correctly without memory, so two configurations are usually evaluated on different denominators, and the sign of their difference need not match the sign on a common task set. The authors define a shared-task target, the corruption-rate difference on the intersection of baseline-correct sets for a declared group, and derive sharp finite-set bounds on it when only per-configuration counts are retained and some pairs are missing. They show the bounds are monotone in the shared-set size, which yields an exact reduced search that only enumerates missing-correct counts for the two target configurations. They also give an evidence query that reports whether a verified overlap lower bound would settle a direction or whether task-level labels are needed. The empirical part applies the audit to reported MemImpactBench counts (14 of 15 pair-specific harmful-memory directions determined, far fewer on the all-six set), validates it on public StrategyQA conflict rows (all 144 intervals contain the matched truth and all 43 certificates have the correct sign), and adds synthetic evidence-refinement, stress, and solver-scaling studies.

Critical strengths
  • The paper isolates a real and commonly overlooked flaw in how memory and context-conflict benchmarks compare models. Corruption rates conditioned on each model’s own baseline-correct set are routinely placed side by side as if they measured the same thing, and the StrategyQA replay shows this is not just a theoretical worry, since own-support signs disagree with the shared-task sign in 14 of 133 nonzero cases, and it gives the work a clear and well-motivated contribution to evaluation practice.
Critical weaknesses
  • The technical novelty is insufficient for a main-track ICLR paper. The authors themselves state that the basic set-overlap bound and partial identification are not new, and that the multiway overlap inequality is standard. What remains is the monotonicity of the bounds in overlap size and the two-target reduction, and both are technically elementary. Monotonicity follows from a few lines of algebra on the per-configuration corruption ranges, and the reduction is a short corollary of it. The computational benefit is also modest in practice, since a generic MILP already solves the hardest reported case (J = 12, u = 20) in under half a second. The contribution therefore rests mainly on applying standard bounds to one particular contrast.
  • (i) The conclusions depend on the declared comparison group, and the paper offers no principle for choosing it. The authors acknowledge that changing G changes the estimand, but they leave users without guidance when scopes disagree. Kimi minus Claude under misleading memory is tightly determined pair-specifically ([14.32, 14.62]) yet unresolved on the all-six set ([−8.98, 30.59]), and the paper does not say which result should be reported. (ii) Neither scope yields a stable ranking either. Pair-specific results each use a different task set and cannot be combined into a leaderboard, while a group-wide shared set changes whenever a configuration is added, so existing pairwise conclusions can flip as a benchmark grows. (iii) Finally, the method is least informative exactly where cross-model comparison is most needed. The shared set shrinks quickly with group size, and in the stress suite the count-only audit determines only 0.10% of directions at J = 9 even with no hidden pairs. Since leaderboards with many models are the main use case for such comparisons, this sharply limits the practical value.
Questions for authors
  • I found the title a little confusing, and I would appreciate it if the authors could clarify the intended message. “Correcting errors is not enough” reads to me like an empirical finding about memory systems, and it led me to expect an analysis showing that preservation matters as much as rescue. However, the main text studies the trade-off between rescue and corruption only briefly, and the core contribution appears to be a measurement method for bounding corruption differences from summary data.
  • What principle should a practitioner use to choose the comparison group G, and which result should be reported when pair-specific and group-wide scopes disagree? Can the framework produce any ranking that remains stable when configurations are added or removed?
Reviewer_3thqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies correct-answer preservation: whether a model retains an initially correct answer after receiving potentially unreliable external memory. Its starting point is that different configurations may answer different questions correctly without memory. Consequently, comparing their individual corruption rates does not necessarily establish which configuration preserves answers better on the same questions. The authors therefore define a comparison restricted to questions that every configuration in a chosen group answered correctly at baseline. The main technical contribution concerns situations where task matching or outcomes are incomplete. The paper derives tight bounds on the possible corruption-rate difference, provides an efficient exact calculation that only enumerates missing-correct counts for the two configurations being compared, and identifies what additional evidence could resolve an uncertain comparison.

Critical strengths

The paper identifies a meaningful distinction between two evaluation questions. A model’s corruption rate on its own initially correct questions and its corruption rate on questions that other configurations also answered correctly are not interchangeable. The paper makes this distinction explicit without claiming that the individual-model rates are themselves incorrectly calculated. This is a useful clarification for interpreting cross-configuration comparisons.

Critical weaknesses
  1. I would like a more complete account of the task sources, the construction of the memory interventions, and the correctness-scoring procedure. Appendix I explains that the original task-level alignment records are unavailable for this reconstruction. Although the retained summaries can support the count-based audit, this limits my ability to assess the underlying experiment and understand which parts another researcher could reproduce. The concern is about the empirical evidence and its documentation, rather than the validity of bounds conditional on the reported counts.
  2. preserving correct answers is a reasonable evaluation goal. However, I remain uncertain about the practical situations in which researchers would have the required summary counts but lack the task-level information needed for a direct comparison. The paper states that complete task-level labels reduce the problem to direct counting. A more concrete account of who needs the proposed audit, why the records are unavailable, and what decision the audit enables would strengthen the motivation.
  3. In the public StrategyQA validation, the count-based method establishes a direction in 35 of 72 pair-specific comparisons, but only eight of 72 comparisons using the all-four shared set. By contrast, the MemImpactBench analysis establishes 14 of 15 pair-specific directions under each harmful-memory condition. Thus, it would be too broad to conclude that the method usually leaves comparisons unresolved; the concern is particularly relevant to some larger-group comparisons. Reporting insufficient information is appropriate and is not a correctness failure. Nevertheless, I would like clearer practical guidance on when the bounds are likely to be informative enough to support a decision. The evidence-recovery experiments help address this question, but the connection between those experiments and realistic evaluation workflows could be developed further.
  4. The authors explicitly acknowledge that baseline-to-memory transitions are descriptive and that repeated no-memory draws would be needed to separate ordinary generation variability. However, the reported comparisons do not establish how stable the shared baseline-correct sets or preservation differences would be across repeated runs. This limits the interpretation of the empirical findings as persistent properties of the configurations, even when the bounds are exact for the recorded experiment.
Questions for authors

MemImpactBench construction and reproducibility: Could you provide a clearer account of the task sources, memory-generation procedures, and correctness scoring? Which task definitions, prompts, recorded outputs, and configuration metadata remain available, and which parts of the original experiment can be independently reproduced? Intended practical use: What is a concrete evaluation scenario in which a researcher has the required transition counts but cannot obtain the matched task records? In that scenario, what decision does the proposed audit enable beyond reporting each configuration’s overall accuracy and individual corruption rate?

Reviewer_tgg9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The following paper proposes a methodology to audit a model's correct answer preservation under a memory. The paper proposes to perform an audit on shared tasks where both models without memory (at baseline) answered correctly. The paper also defines sharp bounds from the counts and proposes techniques to handle missing answers. On MEMIMPACTBENCH shared-task setting, the proposed method identifies 14 of 15 harmful-memory directions, while the single all six shared set settles only only 8 of 15 (misleading) and 7 of 15 (stale). Furthermore the paper experimented with Public StrategyQA and studied the speed of the solver.

Critical strengths
  • The problem statement is formulated clearly: It shows that the choice of the comparison groups matter and that the idea can be used beyond memory.
  • The paper handled edge cases well: E.g. missing answers, empty shared sets,
Critical weaknesses
  • The paper presents numbers (e.g. lines 80 - 81) and definitions (on Figure 1) in the first 1 - 3.5 pages without explaining how those numbers were exactly calculated (line 80) and without defining the Evidence refinement categories (E1 - E4). It requires reading in the appendix and page 5 to understand the beginning of the paper. It would be great to have more explanations in the beginning of the paper. It would be good to define K = 384 in the main text. The readers can't check any calculations without it. Demonstrate a working example in the introduction. The luck of examples make it difficult to understand the paper and the assumptions made under the hood.

  • The novelty of the work is not clear. The bound estimations eq 4, 5 and set-overlap bounds don’t seem to be novel

  • Some confusions in the notation: Confusion Using A, B for the models and B_j for baseline sets. L and K: Both letters are names for the bounds, and both get reused for counts in the same sections that compute those bounds.

Questions for authors

Is there an exact citation to MemImpactBench ? Is this something adopted and modified by the authors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.