- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to compare correct-answer preservation across memory-augmented LLM configurations. A corruption rate is normally computed on the tasks each configuration answers correctly without memory, so two configurations are usually evaluated on different denominators, and the sign of their difference need not match the sign on a common task set. The authors define a shared-task target, the corruption-rate difference on the intersection of baseline-correct sets for a declared group, and derive sharp finite-set bounds on it when only per-configuration counts are retained and some pairs are missing. They show the bounds are monotone in the shared-set size, which yields an exact reduced search that only enumerates missing-correct counts for the two target configurations. They also give an evidence query that reports whether a verified overlap lower bound would settle a direction or whether task-level labels are needed. The empirical part applies the audit to reported MemImpactBench counts (14 of 15 pair-specific harmful-memory directions determined, far fewer on the all-six set), validates it on public StrategyQA conflict rows (all 144 intervals contain the matched truth and all 43 certificates have the correct sign), and adds synthetic evidence-refinement, stress, and solver-scaling studies.
Critical strengths
- The paper isolates a real and commonly overlooked flaw in how memory and context-conflict benchmarks compare models. Corruption rates conditioned on each model’s own baseline-correct set are routinely placed side by side as if they measured the same thing, and the StrategyQA replay shows this is not just a theoretical worry, since own-support signs disagree with the shared-task sign in 14 of 133 nonzero cases, and it gives the work a clear and well-motivated contribution to evaluation practice.
Critical weaknesses
- The technical novelty is insufficient for a main-track ICLR paper. The authors themselves state that the basic set-overlap bound and partial identification are not new, and that the multiway overlap inequality is standard. What remains is the monotonicity of the bounds in overlap size and the two-target reduction, and both are technically elementary. Monotonicity follows from a few lines of algebra on the per-configuration corruption ranges, and the reduction is a short corollary of it. The computational benefit is also modest in practice, since a generic MILP already solves the hardest reported case (J = 12, u = 20) in under half a second. The contribution therefore rests mainly on applying standard bounds to one particular contrast.
- (i) The conclusions depend on the declared comparison group, and the paper offers no principle for choosing it. The authors acknowledge that changing G changes the estimand, but they leave users without guidance when scopes disagree. Kimi minus Claude under misleading memory is tightly determined pair-specifically ([14.32, 14.62]) yet unresolved on the all-six set ([−8.98, 30.59]), and the paper does not say which result should be reported. (ii) Neither scope yields a stable ranking either. Pair-specific results each use a different task set and cannot be combined into a leaderboard, while a group-wide shared set changes whenever a configuration is added, so existing pairwise conclusions can flip as a benchmark grows. (iii) Finally, the method is least informative exactly where cross-model comparison is most needed. The shared set shrinks quickly with group size, and in the stress suite the count-only audit determines only 0.10% of directions at J = 9 even with no hidden pairs. Since leaderboards with many models are the main use case for such comparisons, this sharply limits the practical value.
Questions for authors
- I found the title a little confusing, and I would appreciate it if the authors could clarify the intended message. “Correcting errors is not enough” reads to me like an empirical finding about memory systems, and it led me to expect an analysis showing that preservation matters as much as rescue. However, the main text studies the trade-off between rescue and corruption only briefly, and the core contribution appears to be a measurement method for bounding corruption differences from summary data.
- What principle should a practitioner use to choose the comparison group G, and which result should be reported when pair-specific and group-wide scopes disagree? Can the framework produce any ranking that remains stable when configurations are added or removed?