acceptodds
Under review as a conference paper at ICLR 2027

From Bottleneck Labels to Repair Surfaces: Factorial Diagnosis of Agent Memory

Abstract

An oracle repair may recover many tasks from a native memory-agent pipeline, yet tell us little about where the original failure occurred if another repair recovers the same tasks. We examine this ambiguity by crossing repairs to upstream state, retrieval, and read context. In the primary setting, upstream-state and read-context repairs each add about 91 pp from the native cell, but no more than +0.45 pp once the other is active; they correct 199 of the same 206 native errors. Across all seven complete surfaces, the two repairs have negative accuracy interactions. Retrieval repair has smaller native gains across eleven settings, and paired intervals do not resolve a change in its gain after upstream repair. The same trajectories yield nominal p < .05 retrieval-background contrasts with individual judges, but none with adjudicated labels. Stage-level interpretation therefore depends on both the repair background and the scoring rule.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, memory-augmented agents, failure diagnosis, counterfactual intervention, factorial design, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VPAkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the reliability of diagnosing failures in agent memory systems through oracle interventions. The authors evaluate three interventions that correct upstream information, retrieval, or the solver's read context, and compare their effects individually and in combination. Experiments are conducted on 221 MemoryArena tasks across seven language models and multiple memory backends. Results show that correcting upstream information and directly providing the solver with gold context recover largely overlapping failures, while retrieval repair provides much smaller gains. The authors also find that different LLM judges can lead to inconsistent diagnostic conclusions.

Critical strengths
  1. The paper presents a well-controlled empirical study with good coverage across models and memory backends.
  2. The experimental analysis is thorough. The authors examine task-level error overlap, provide statistical significance tests, and investigate the sensitivity of the findings to different evaluation methods.
Critical weaknesses
  1. The paper is very difficult to read. The presentation introduces many abbreviations and mathematical formulations that make the core idea less accessible than necessary. I would suggest simplifying the presentation and using more concrete examples. I think the abstract/intro/related work writing is currently below the ICLR standard as well.
  2. The novelty and significance of the findings seem limited. The paper mainly presents an empirical comparison of different oracle interventions. The central finding that correcting earlier answers and directly supplying the same information to the solver recover overlapping failures is somewhat expected. While the empirical evidence is useful, it is unclear what new insights the proposed framework provides.
  3. The evaluation scope limits the generalizability of the findings. All experiments are conducted on one benchmark family with relatively small models, and the native accuracy is extremely low (1–9%). It would be helpful to evaluate stronger agents and more realistic memory tasks where failures are distributed across memory formation, retrieval, and utilization.
Questions for authors

Please refer to the weaknesses.

Reviewer_NodNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how we can locate failures in memory augmented LLM agents. It crosses three oracle repairs and finds that the upstream and read-conetxt repairs fix almost the same error, while retrieval effets are small and sensitivit to how ansers are judged.

Critical strengths
  1. The paired design is careful. The rescued/broken transition counts make clear where gians come from.
  2. Judge errors can differ by intervention cell, and so bias a constrat, is a useful caution for anyone using LLM judges in ablations.
Critical weaknesses
  1. I think the biggest concern is that the main result is just close to true by construction. S and U both inject the same gold prior answers, just at different points. the paper itself also says the overlap shows "substitutable information routes", not anything about write or read failures. So I guess the headline does not tell us much about where memory agents fail. It mostly shows that later subtasks fial because earlier answers are wrong right?
  2. Another thing is that the large negatie interaction seems mostly just a ceiling effect. Each repair alone reaches about 97%, so any interation on the accuracy scale has to be near -90pp.
  3. i also found the judge analysis weak. Currently, the judges are mistral 7b and llama 3.1 8b, which are very old and known to be unreliable.
  4. Also the 18.6% of the adjudicated labels come from these same judges agreeing, and the manual part uses a ginale annotator with no agreement check. So the referecne is not independent of what it is used to evaluate.
  5. And the scope is pretty narrow. There is only one benchmark, 221 reused tasks, and models of 9B or smaller iwth 1 to 9% native accuracy. It is unclear whetehr any of this holds for agents that actually work.
Questions for authors

See weakness

Reviewer_pGkJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether gains from individual oracle repairs reliably identify bottlenecks in memory-augmented agents. It evaluates combinations of three interventions—correcting upstream answers before storage, reselecting existing memory chunks, and supplying gold prior answers in the read context—and uses conditional gains and factorial interactions to characterize their effects. Experiments cover 221 paired tasks from MemoryArena’s progressive-search environment across seven complete factorial settings and four additional partial settings. Upstream-state and read-context repairs each yield large improvements but provide limited additional benefit when combined, while retrieval repairs produce smaller gains. Rescoring the same trajectories with different judges also changes the estimated retrieval contrasts and their statistical significance. The paper advocates reporting repair effects across intervention backgrounds and auditing cell-dependent scoring disagreements rather than assigning bottleneck labels from isolated repair gains.

Critical strengths

The paper makes a useful distinction between recovering performance and locating a failure. Testing all eight repair combinations on the same tasks makes it possible to see what each intervention adds once another is already active. Reporting corrected and newly broken tasks also helps interpret small net gains: in the primary setting, retrieval repair fixes 15 errors but breaks 7 previously correct answers. This is more informative than reporting accuracy changes alone.

The scoring analysis is a useful part of the study. By rescoring fixed trajectories, the authors show that the estimated retrieval contrasts depend on how correctness is judged. The concentration of judge disagreements in the native and retrieval-only cells explains why an aggregate agreement rate would be insufficient. This gives a concrete reason to audit scoring separately across intervention conditions.

Critical weaknesses
  1. The main empirical result is largely constrained by ceiling effects. In the primary setting, upstream-state and read-context repairs reach 97.29% and 97.74% accuracy from a 6.79% baseline. Since joint accuracy cannot exceed 100%, their interaction must be at most 100 - 97.29 - 97.74 + 6.79 = -88.24 pp, already close to the reported -90.50 pp. Similarly, correcting 201 and 202 of the same 206 native errors forces an overlap of at least 197 tasks; the observed overlap is 199. These observations are valid, but their magnitude provides limited evidence beyond the individual repair success rates. Both interventions also supply gold prior answers, making substantial redundancy unsurprising. Although the paper acknowledges limited headroom, it does not establish what additional diagnostic information the surface provides beyond these constraints. Partial-repair experiments or controlled failures with known locations would make the contribution substantially more convincing.

  2. The scoring analysis lacks an independent reference for establishing judge errors. The final labels accept dual-judge agreements for 18.6% of records and use one human adjudicator for the remaining 8.0%, after deterministic extraction resolves the rest. The same judges are then evaluated against these labels. Shared mistakes in the accepted-agreement subset are therefore invisible, and the reported false-positive and false-negative counts measure disagreement with this particular protocol rather than independently verified errors. This does not invalidate the finding that scoring choices change the estimates, but it limits the stronger interpretation that adjudication corrects misleading retrieval conclusions. Moreover, counting significant contrasts under one scoring rule and nonsignificant contrasts under another does not directly test the difference between scoring rules. A stratified, independent human audit and paired confidence intervals for the changes in contrasts would provide the missing evidence.

Questions for authors

as weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.