- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to audit what a stateful agent remembers by separating durable storage, active context, and behavior. It proposes a case-linked auditing procedure with native interventions, matched controls, and explicit indeterminate outcomes, then applies it across six agent-model configurations. The main empirical finding is that native compaction can reduce target adoption even when the full source history remains in durable records.
Critical strengths
-
The paper draws a genuinely useful boundary between storage, inclusion, and use, and it operationalizes that boundary carefully. Section 2, Table 1, and Section 3.3 make the evidence ladder, control logic, and indeterminate outcomes explicit rather than quietly collapsing missing evidence into negatives; this is consequential because many memory evaluations in practice blur exactly these distinctions.
-
The core empirical demonstration is stronger than a simple before/after accuracy table. Table 4 shows paired restart-versus-compaction drops on Pi, while Section 4.1 and Table 5 add case-linked evidence that all 480 Pi restart/compaction runs still preserve the full source history in durable records, including every success-to-failure pair. That combination is the main reason I lean positive: the paper does not just report degradation, it localizes where along the pathway the separation occurs.
Critical weaknesses
-
The behavioral labels underlying the headline adoption numbers are only partially validated, which limits how strongly I can read the empirical magnitudes. Section 3.1 states that the accepted target projections are machine-derived rather than adjudicated human gold, Section 3.3 reports only a targeted human check, and Section 6 explicitly says that “absolute answer and causal accuracy remains provisional.” Table 15 further shows unresolved disagreement even in the reviewed subset. This matters because the main quantitative claims in Table 4 and Table 6 are adoption-rate differences relative to those references; if the target itself is noisy or underspecified for some cases, the exact size and task-family interpretation of the compaction effect become less secure. A convincing resolution would be a stratified adjudication of at least the restart/compaction discordant pairs, or a sensitivity analysis restricted to cases whose references are human-verified.
-
The empirical scope is narrower than the paper’s framing initially suggests, especially for claims about agent memory systems more broadly. The main population is entirely based on 120 Memora adaptations (Section 3.1), native compaction is unavailable for OpenCode (Table 2, Table 3), no tested configuration exposes all four audit views jointly (Section 4.3, Table 7), and the only external follow-up is a small 24-case LongMemEval oracle sample on Pi alone, which Section 4.5 explicitly says “does not establish generality.” This does not invalidate the paper’s within-scope findings, but it does cap their significance: the stable compaction story is really a harness- and configuration-specific phenomenon plus a reusable audit procedure, not yet a broadly established property of agent memory. The clean fix is mostly in positioning, namely narrowing the headline empirical takeaway and making the contribution read more as an auditing methodology demonstrated on a few systems than as a general statement about compaction in agent memory.
Questions for authors
-
[Clarification] For the paired contrasts in Table 4 and Table 6, how sensitive are the reported effect sizes to restricting the analysis to cases whose target projections were supported in the targeted human review or to a small adjudicated discordant subset? If the sign and rough magnitude persist on a vetted subset, my confidence in the empirical results would increase.
-
[Clarification] In Section 4.3 and Table 7, no configuration provides jointly determinate storage and active views. Is this absence fundamentally imposed by the tested system interfaces, or mainly by the current frozen extraction adapters? The answer affects how strongly I interpret the “multi-view disagreement” result as an intrinsic property of audits versus an instrumentation limitation.
-
[Clarification] For the Pi reasoning failures discussed in Section 4.1, especially the GLM cases in Table 5 where the literal target remains present after compaction but is not adopted, do you have any frozen per-case evidence that the surrounding request structure changed in a consistent way? A concrete explanation here would help separate “content retained but unused” from “content present but behavior redirected by other prompt changes.”