- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an executable specification for auditing memory transitions in language agents. Given a restricted scenario schema, a deterministic router generates operation labels, successor states, interpretations of displaced beliefs, observation scope, and obligations. The paper states compilability and axiomatic characterization results and introduces MEMLEDGER, comprising 900 synthetic bilingual sessions and 40,500 transitions.
The evaluation compares different views of seeded ledger violations, tests the downstream effect of displaced-belief interpretation, examines three memory frameworks, and investigates operation extraction and probe-induced changes. While the paper identifies useful distinctions in memory semantics, I find the manuscript insufficiently developed and the practical validation inadequate for acceptance.
Critical strengths
The distinction between a belief that was previously correct and one that was incorrect from the outset is meaningful: an actual change and a correction can produce the same current value while requiring different answers about the past. Section 5.2 provides a useful mechanism check. Changing only the disposition sentence improves history answers from 2/36 to 33/36, while the control answers remain correct in every arm. This supports the relevance of the distinction, although the experiment is limited to twelve fixtures and does not establish the effectiveness of the complete auditing framework.
Critical weaknesses
- The presentation lacks the clarification and editorial care needed to assess the technical contribution. The manuscript introduces formal objects, routing rules, contracts, axioms, complexity claims, and adapter lemmas in rapid succession, without sufficient explanation or worked examples connecting them. Important definitions remain ambiguous, including “undefined comparison” in A6 and the change set minimized by A2. These ambiguities affect the characterization claim rather than merely the writing style. Figure 2 also uses very small text and crowded labels, making the central empirical comparison difficult to inspect. Together, these problems indicate that the manuscript requires substantial explanatory and editorial revision.
- The empirical evidence does not establish reliable practical auditing. Section 5.1 modifies an intermediate ledger entry and successor snapshot without replaying subsequent events. The reported 0.8% final-value recall therefore characterizes this particular corruption protocol and does not establish how frequently final-answer evaluation misses faults during actual execution. The compiled judge's perfect detection is guaranteed by construction. On deployed frameworks, both automatic operation extractors perform below the 80.1% always-NO-OP baseline, and visibility and obligations cannot be observed through the available disclosures. The benchmark also uses one schema skeleton across seven vocabularies; agreement among author annotators under a shared handbook validates adherence to that specification rather than applicability to natural dialogue. These limitations leave a substantial gap between the proposed specification and an effective auditing framework. Combined with the presentation problems, they motivate my clear-rejection recommendation.
- I am also seriously concerned about the lack of editorial care in the manuscript. The writing and figure presentation give me the impression that substantial portions may have been generated using AI and retained without adequate human review or careful polishing. I cannot verify the extent of AI involvement; my criticism concerns the submitted material: dense terminology, insufficiently explained connections between claims, ambiguous definitions, and crowded figure labels. These problems substantially impair readability and make the submission appear insufficiently checked and refined for a conference paper.
Questions for authors
- Formal clarification: What precisely constitutes an “undefined comparison” under A6? For conflicting reports from incomparable sources, does R4 prescribe dissent without clarification, or does A6 require a clarification obligation? Please demonstrate how this case satisfies both the router and the axioms.
- Formal clarification: What changes are included in A2's change set? For stronger-source repetition of an existing value, how is mandatory support refinement compatible with minimality when the existing literal and support already satisfy success, consistency, and support closure?
- Assessment-changing evidence: Does the supplementary analysis evaluate faults propagated through subsequent execution, or detection before later writes can erase their effects? If so, how does final-value recall change? This would clarify how strongly the headline result depends on the mutation protocol.
- Practical scope clarification: Which contracts have been audited reliably end to end on native framework disclosures? Please distinguish demonstrated capabilities from those requiring additional disclosure interfaces or improved operation extraction.
4 comments
[Q2, continued] Corrected Case 1b. By A3 the repeated value is accepted iff . If it is accepted, success requires the support edge of to be , which the identity does not achieve. Replacing the edge achieves it, and no proper subset of that change does: the empty change fails success, removing the old edge alone leaves unsupported, and adding the new edge alone is not a state, since a literal carries one support edge. So refinement is the minimal change, and A2 forces it. If the input is not accepted, the identity satisfies A1 and A2 forces the identity. This is retain () or no-op (), which differ only in the label.
Nothing else moves. The six counterexample operators of the independence proof go through against the restated clauses; for example, last-writer-wins accepts every input, so it meets A1 and A2 and fails only A3. No label changes either: the same-value branch has always been checked at run time against its own contract on every compiled transition (refine requires a strictly stronger source, no agent's support is downgraded, and a weaker repetition leaves the state unchanged without dissent; App. G).
You rated the soundness concerns as not easy to address. For the two formal points, the complete repair is the restatement above: four axioms and one proof case. The router, the contracts and all 40,500 labels are exactly as submitted, and Proposition 1 holds as stated.
[Q3] "Does the supplementary analysis evaluate faults propagated through subsequent execution, or detection before later writes can erase their effects? If so, how does final-value recall change?"
Short answer: both questions have an answer. Detection before later writes is in Figure 2. For propagation, nine of the twelve classes are exactly invariant to replay, so final-value recall stays at 0 of 900 for them and cannot exceed 25% overall under any replay outcome.
(a) Nine classes are invariant to replay. Nine mutations (ghost update, revise without was-wrong, swapped disposition, scope broadcast, broken trigger, dissent overwritten, deletion not ledgered, clarify missing, time reorder) edit only fields of the ledger entry, or state components that no routing rule reads: history intervals with their disposition marks, and the pending-clarification list. The antecedents of rules R1–R8 read the current belief value, its source and support, the slot type, the marker, the trigger status and the clock. Replaying the later events from the corrupted snapshot therefore reproduces the clean twin's belief values at every later step. This follows directly from the released router and injector code. Final-value recall on these 900 pairs is zero with or without replay, because a current-value question does not read the corrupted fields.
(b) This bounds the dependence on the protocol. Even if replay made every fault in the remaining three classes visible at session end, final-value recall would be at most 300 of 1,200, that is 25%. Three quarters of the seeded violations are invisible to a final-value question whichever protocol is used.
(c) Detection before later writes. The question battery is asked at the corrupted step. It sees 100/100 wrong-value faults and 94/100 missing updates; the final-value question sees 4/100 and 6/100. So for these classes the faults are visible when they happen and are overwritten by the end of the session, which is the case for auditing transitions and not only final answers.
(d) The same pattern on a deployed framework. In Sec. 5.2 the fault sits in LangMem's own store. The current-value question is answered correctly in 12 of 12 fixtures both with the fault and with the repair, while history answers go from 2/36 to 33/36.
We will revise the abstract sentence to read: "On a balanced suite of 1,200 seeded record corruptions, a question about the final value sees under one percent, the best single question per class asked at the corrupted step sees three quarters, and one large language model given the complete record in a single pass sees under ten percent." We will add points (a)–(c) to Sec. 5.1 and App. G, and the final version will include the replay run for the three value-bearing classes.
[Q4] "Which contracts have been audited reliably end to end on native framework disclosures? Please distinguish demonstrated capabilities from those requiring additional disclosure interfaces or improved operation extraction."
all in all:the disposition contract is demonstrated end to end on LangMem's own store, and current value, history and provenance are read through probes on three frameworks. Operation-level audits need better extraction; visibility and obligation contracts need a disclosure interface.**
| Status | Capability | Evidence |
|---|---|---|
| Demonstrated | Compile gold for any schema in the fragment; independent contract check; replay determinism | Thm. 1; 40,500 transitions |
| Demonstrated | Detect all twelve violation classes on a ledger, with no false alarm on clean twins | Sec. 5.1 (the reference check) |
| Demonstrated on a native store | Disposition contract, from fault to downstream answers | Sec. 5.2: LangMem, 2/36 to 33/36, controls 24/24 |
| Demonstrated on native stores | Fork-and-probe protocol with probe-free control arms; parent-state isolation held in every run | Sec. 5.4: LangMem, Mem0, 100 sessions |
| Measured on three frameworks | Current value, history and provenance through forked probes against the compiled reference | Table 2 |
| Readable from native disclosures | Realized operation (207 of 249) and disposition (192 of 234), by human annotators | Sec. 5.3 |
| Requires improved extraction | Fully automatic operation-level audit | Sec. 5.3 |
| Requires a disclosure interface | Visibility and obligation contracts | Sec. 5.3 |
We will place this table at the start of Sec. 5.
[W2] "The empirical evidence does not establish reliable practical auditing."
answer shortly: we respectfully see this differently. The paper does not claim a finished audit product for deployed systems. It claims a specification, a benchmark and diagnostic evidence, and the observations in your paragraph are either design properties or findings that only compiled gold makes measurable.
| Your observation | What it shows |
|---|---|
| The 0.8% figure characterizes the corruption protocol | It is exact under replay for 900 of 1,200 pairs and bounded at 25% overall (Q3) |
| The compiled judge is perfect by construction | Correct: it is the reference, reported as a completeness check. The finding of Sec. 5.1 concerns the views that existing evaluations use: final value, question battery, LLM judge |
| Both extractors are below the always-no-op baseline | A finding that supports the thesis. Measured against compiled gold, two LLMs cannot read operations off framework disclosures, and they answer definitely on all 54 items the annotators marked unknown. Operation labels produced by an LLM cannot serve as gold, which is the reason to compile them |
| Visibility and obligations cannot be observed | Also a finding. Of all label fields, exactly these two are absent from the disclosures of all three frameworks. That is a concrete interface requirement for framework developers |
| One skeleton across seven vocabularies | The guarantees are proved for every schema in the fragment, and all 40,500 transitions pass an independent check. We will add a second, structurally different skeleton in the final version |
| Annotator agreement validates adherence to the handbook | That is its purpose: it checks that the rendered text carries the compiled label. Fit to natural dialogue is not claimed |
What the evidence does establish. (i) Gold labels for every memory transition can be compiled with guarantees. (ii) The views used by current evaluations miss most transition-level violations, and this is not explained by the fault protocol. (iii) On a deployed framework, a single field of the label changes downstream answers from 2/36 to 33/36, which you describe as a useful mechanism check and Reviewer HreN as "direct evidence that the proposed field affects downstream behaviour". (iv) The paper measures precisely what deployed frameworks must expose before they can be audited automatically. We believe these four results stand on their own, and we will state the scope in these terms at the start of Sec. 5.
[fUL2-W1] "The paper introduces formal objects, routing rules, contracts, axioms, complexity claims, and adapter lemmas in rapid succession, without sufficient explanation or worked examples connecting them ... Figure 2 also uses very small text and crowded labels."
Short answer: we agree on Secs. 2–3 and on Figure 2, and we will make four concrete changes in the revision. Reviewer HreN rated the presentation "clear and easy to follow", so we read your comment as aimed chiefly at Secs. 2–3, where the formal objects pile up.
- One worked example will connect the three layers. We will move the six-event scenario of Table 3 from App. G into Sec. 2 and state, for every event, which rule fires, what the contract checks, and which axiom forces the outcome. One event is written out this way below.
- Explicit definitions. We will define the change set, the accepted input and the undefined comparison as in our answers to Q1 and Q2, and call the source order a strict total order.
- Figure 2 will be redrawn at full text width with larger type, with the per-class counts printed in the cells.
- Scope first. The capability table from our answer to Q4 will open Sec. 5.
The worked example, for one event. Three people are in the conversation: the user, the assistant and Bob. At step 3 the user tells everyone that the project is due on June 15. At step 4 the user tells the assistant in private: "My mistake, it is June 8."
| Layer | What it says about step 4 |
|---|---|
| Routing rule | The new statement contradicts an earlier one from the same speaker and carries an explicit correction ("my mistake"). The router therefore treats it as a correction of an error, and the operation is revise. |
| Contract (what an auditor checks) | The assistant's current deadline is June 8. June 15 stays in the history and is marked was-wrong: it was never true. It is not marked true-until, which would mean it was true and then changed. Only the user and the assistant are updated, because Bob did not hear the correction. |
| Axioms (why no other outcome is allowed) | A3: an explicit self-correction by the same speaker is accepted. A4: a correction of an error is recorded differently from a change in the world. A5: an agent who did not hear an event does not change. |
So the assistant now answers "June 8" for the deadline and "June 15" for "what did you believe at step 3?", and Bob still believes June 15. Every row of the table in Sec. 2 will be read in this way.
On editorial care and AI use.
We take this concern seriously. Our use of generative tools is disclosed in the AI use statement after Sec. 7, and we take full responsibility for the content of the paper. On the substance, your reading was careful enough to find two imprecise statements in the axioms, and we have answered both in this thread with exact restatements and a corrected proof case. The other items you list (the density of Secs. 2–3, the missing connection between rules, contracts and axioms, the type size in Figure 2) are addressed by the four changes above.
In sum.
- Q1, Q2: both formal points are corrected in this thread. The router, the contracts and all labels are unchanged, and Proposition 1 holds as stated.
- Q3: the result is exactly replay-invariant for 900 of 1,200 pairs and bounded at 25% under any replay outcome.
- Q4, W2: demonstrated capabilities are separated from those that need better extraction or a disclosure interface, and that separation is one of the paper's findings.
- W1: four concrete changes.
Your two soundness concerns (the formal ambiguities and the dependence on the mutation protocol) are the ones you marked as hard to address, and we believe both are now resolved with exact statements. We would be grateful if you would reconsider your soundness and overall ratings in that light. We are happy to elaborate on any point.
Thank you for the detailed review, and in particular for reading the formal part closely. Q1 and Q2 found two places where the axioms were worded more loosely than the implementation, and we are grateful for both. They are corrected below with exact statements. We are also glad that you find the disposition distinction meaningful and Sec. 5.2 "a useful mechanism check". We answer every point; the reply is in four parts because of length. We take your four questions first, then W2 and W1.
[Q1] "What precisely constitutes an 'undefined comparison' under A6? For conflicting reports from incomparable sources, does R4 prescribe dissent without clarification, or does A6 require a clarification obligation?"
to sum up:the comparison is undefined exactly when the two assertions come from the same source and carry no correction marker. Incomparable distinct sources do not exist in the fragment, so R4 and A6 never apply to the same event, and the router satisfies both.**
The comparison in A3 is between σ, the source of the input, and τ, the source supporting the entry it conflicts with. When σ=τ the source order is silent, since it is irreflexive, and only the marker convention can order two assertions by one source. With no marker, nothing orders them. That is the undefined comparison, and it is the only instance in the fragment.
The text says "strict partial order" and then rejects equally ranked distinct sources at schema compile time. That wording is inconsistent. What the fragment requires, and what the released schema loader enforces, is a unique rank for every declared source, that is, a strict total order. We will correct the text accordingly. The cases are then disjoint:
The printed A3 ended with "otherwise nothing changes and the dissent is recorded", which overlapped A6. The restated A3 and A6 below remove the overlap. All 900 sessions use four sources with distinct ranks, so no compiled label is affected.
[Q2] "What changes are included in A2's change set? For stronger-source repetition of an existing value, how is mandatory support refinement compatible with minimality ...?"
in short:your argument is correct against the printed text, and the corrected statement makes refinement the minimal change.** Case 1b of the proof says "A1 is satisfied by the identity". For a strictly stronger source that sentence is wrong. The cause is that we stated success for literals, while the state, the router and the contracts all work with supported literals. We will restate the four axioms as follows:
The dissent record, the pending candidates, the obligation and the ledger entry are appended outputs fixed by A3 and A6. They are not in Δ.