acceptodds
Under review as a conference paper at ICLR 2027

What Does Your Agent Remember? Auditing Storage, Reachability, and Behavioral Influence

Abstract

Information stored in an agent's memory may no longer support its later answers. We operationalize an audit that links durable records, active context, and behavior through native interventions and matched controls, returning explicit indeterminate verdicts when evidence or controls are unavailable. Across six agent-model configurations and 3,600 declared core executions, native Pi compaction reduces target adoption relative to restart by 20.8 points with DeepSeek V4 Flash and 15.0 points with GLM-5.3 Flash. Every success-to-failure pair in those contrasts still contains the complete source history in pre-probe durable records. Among the 25 GLM-5.3 Flash cases that succeed after restart and fail after compaction, seven, all recommendation tasks, still carry the literal target in active context. The procedure distinguishes preservation, inclusion, and use without treating any single surface as evidence of all three.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent memory, large language model agents, memory auditing, context compaction, behavioral evaluation, causal controls

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kfFyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to audit what a stateful agent remembers by separating durable storage, active context, and behavior. It proposes a case-linked auditing procedure with native interventions, matched controls, and explicit indeterminate outcomes, then applies it across six agent-model configurations. The main empirical finding is that native compaction can reduce target adoption even when the full source history remains in durable records.

Critical strengths
  1. The paper draws a genuinely useful boundary between storage, inclusion, and use, and it operationalizes that boundary carefully. Section 2, Table 1, and Section 3.3 make the evidence ladder, control logic, and indeterminate outcomes explicit rather than quietly collapsing missing evidence into negatives; this is consequential because many memory evaluations in practice blur exactly these distinctions.

  2. The core empirical demonstration is stronger than a simple before/after accuracy table. Table 4 shows paired restart-versus-compaction drops on Pi, while Section 4.1 and Table 5 add case-linked evidence that all 480 Pi restart/compaction runs still preserve the full source history in durable records, including every success-to-failure pair. That combination is the main reason I lean positive: the paper does not just report degradation, it localizes where along the pathway the separation occurs.

Critical weaknesses
  1. The behavioral labels underlying the headline adoption numbers are only partially validated, which limits how strongly I can read the empirical magnitudes. Section 3.1 states that the accepted target projections are machine-derived rather than adjudicated human gold, Section 3.3 reports only a targeted human check, and Section 6 explicitly says that “absolute answer and causal accuracy remains provisional.” Table 15 further shows unresolved disagreement even in the reviewed subset. This matters because the main quantitative claims in Table 4 and Table 6 are adoption-rate differences relative to those references; if the target itself is noisy or underspecified for some cases, the exact size and task-family interpretation of the compaction effect become less secure. A convincing resolution would be a stratified adjudication of at least the restart/compaction discordant pairs, or a sensitivity analysis restricted to cases whose references are human-verified.

  2. The empirical scope is narrower than the paper’s framing initially suggests, especially for claims about agent memory systems more broadly. The main population is entirely based on 120 Memora adaptations (Section 3.1), native compaction is unavailable for OpenCode (Table 2, Table 3), no tested configuration exposes all four audit views jointly (Section 4.3, Table 7), and the only external follow-up is a small 24-case LongMemEval oracle sample on Pi alone, which Section 4.5 explicitly says “does not establish generality.” This does not invalidate the paper’s within-scope findings, but it does cap their significance: the stable compaction story is really a harness- and configuration-specific phenomenon plus a reusable audit procedure, not yet a broadly established property of agent memory. The clean fix is mostly in positioning, namely narrowing the headline empirical takeaway and making the contribution read more as an auditing methodology demonstrated on a few systems than as a general statement about compaction in agent memory.

Questions for authors
  1. [Clarification] For the paired contrasts in Table 4 and Table 6, how sensitive are the reported effect sizes to restricting the analysis to cases whose target projections were supported in the targeted human review or to a small adjudicated discordant subset? If the sign and rough magnitude persist on a vetted subset, my confidence in the empirical results would increase.

  2. [Clarification] In Section 4.3 and Table 7, no configuration provides jointly determinate storage and active views. Is this absence fundamentally imposed by the tested system interfaces, or mainly by the current frozen extraction adapters? The answer affects how strongly I interpret the “multi-view disagreement” result as an intrinsic property of audits versus an instrumentation limitation.

  3. [Clarification] For the Pi reasoning failures discussed in Section 4.1, especially the GLM cases in Table 5 where the literal target remains present after compaction but is not adopted, do you have any frozen per-case evidence that the surrounding request structure changed in a consistent way? A concrete explanation here would help separate “content retained but unused” from “content present but behavior redirected by other prompt changes.”

Reviewer_3RFLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes auditing agent memory by separating what is stored, what reaches the model, what appears in the answer, and what causally influences behavior. The main result is that compaction can make stored information less likely to influence the model's later answers.

Critical strengths
  • The distinction between storage, availability to the model, behavior, and causal influence is useful and important. Moreover, the paper is careful not to treat any single observable (e.g., durable storage, active context, the final answer) as sufficient evidence of how memory is retained or used.

  • The compaction experiment shows why it matters to distinguish between what is stored and what affects the model’s answer. Even if the complete source history remains in durable storage, after compaction the model is less likely to reflect that information in its answer.

  • The matched no-memory and altered-memory controls are another strength. The authors do not assume that a correct answer means the model used memory; instead, they use controls to test whether the stored information actually influenced the answer.

Critical weaknesses
  • The full audit requires evidence about what remains in durable storage, what reaches the model at inference, what appears in the answer, and what actually influences the answer. The paper does not demonstrate all of these within a single configuration. In particular, Pi provides active-context and causal evidence but not storage-to-causal comparisons, whereas OpenCode and Codex provide storage and causal evidence but no active-context evidence.

  • The human validation seems to be fairly limited. The targeted check covers only 48 judgments, and disagreements about the reference answers are unresolved. Since many conclusions depend on whether the target is judged present or adopted, a larger independent human validation would make the results more convincing.

Questions for authors

Can the complete audit (storage, active context, answer behavior, and causal influence) be demonstrated within a single agent–model configuration?

Would the conclusions remain the same under a larger independent human validation of the reference answers and model responses?

Reviewer_5VZSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an audit procedure for agent memory that separately traces durable storage, active inclusion in the model's context, and behavioral influence. It uses evidence levels E1–E5 and four three-valued audit views (storage-only, active-only, answer-only, matched causal) that return indeterminate verdicts when evidence or controls are unavailable. The procedure is applied to 120 Memora-derived cases across six harness–model configurations (Pi, OpenCode and Codex), with 3,600 core executions under no-memory, live-context, restart, native-compaction and altered-memory conditions plus 960 pathway-matched control executions. Main results show that native Pi compaction lowers target adoption relative to restart by 20.8 and 15.0 points while durable records retain the full source history, and characterize disagreement among audit views. Validation uses case-clustered bootstrap intervals, 30-case repeats, a wording ablation, a 24-case LongMemEval oracle follow-up and a small human review.

Critical strengths
  1. Separating retention, active inclusion and behavioral influence, with an explicit indeterminate verdict, is a useful audit discipline (Sec. 2.1, Table 1).
  2. The controls are strong. No-memory and pathway-matched altered-memory controls run through restart and compaction, and failed controls stay indeterminate (Table 2, Sec. 4.2).
  3. Case-linked evidence separates reachability from use. All 480 Pi durable records keep the full source, and 35 GLM markers stay verbatim in summaries yet are not adopted (Sec. 4.1–4.2).
  4. Accounting is transparent: case-clustered bootstrap intervals, repeat runs (Table 9), and failed or unsupported cells reported explicitly (Table 3).
Critical weaknesses
  1. Very hard to read. Dense, idiosyncratic terminology and constant caveats obscure the results, and key definitions are deferred to App. I–K.
  2. The headline effect is expected and narrow. Compaction lowers Pi adoption by 20.8 and 15.0 points (Table 4) while logs keep the source. The more novel retained-but-unused finding rests on 7 cases (Table 5) and does not replicate for Codex (Sec. 4.4).
  3. The view-disagreement results are largely structural. Storage vs. active has 0 jointly determinate pairs, and answer vs. causal agrees by construction (Table 7).
  4. Ground truth is weak. References are machine-derived. Human review covers 24 pairs, with reference κ = −0.029 and response κ = .631 (App. L).
  5. The design is confounded and novelty is unclear. Harness and model are not crossed, the cases come from one source (Memora), and the overlap with Osmond et al. (2026) is not delineated (Sec. 2.3, App. B–C).
Questions for authors
  1. Excluding pairs that agree by construction or have no joint determinacy, what fraction of comparisons disagree in a way a practitioner would act on?
  2. For the 7 present to present cases and the 35 retained-but-unadopted markers, is non-adoption genuine non-use or a scorer artifact (paraphrase, hedging)?
  3. What exactly is new relative to Osmond et al. (2026) and Osmond (2026)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.