acceptodds
Under review as a conference paper at ICLR 2027

Executable Gold Semantics: Compiling Gold Labels for Auditing Memory Transitions in Language Agents

Abstract

Memory systems let language agents carry information across long interactions, and they are evaluated by whether the agent eventually answers questions correctly. Whether each individual memory operation was right is rarely measured, because per-operation labels are held to be impractical to obtain. The one operation-level benchmark generates its memory points with a language model and verifies them by hand, and no benchmark records how a displaced belief should be read, who observed a change, or what the agent still owes. We show that once a scenario schema is fixed, these labels can be compiled instead of annotated. A small router of belief-change operators computes, for every event, the correct operation, the successor memory, whether the displaced belief was once true or wrong all along, who observed the change, and the open obligations. We prove that the router is well defined and runs in polynomial time, that every compiled trace satisfies contracts stated over states alone, and that six axioms characterize it, so that dropping any one yields an operator with a recognizable design flaw. The same semantics generates gold with paired counterfactuals, judges systems against it, and defines a training signal and a runtime guard. We release MEMLEDGER, the compiler and a benchmark of 900 sessions in two languages, and use it to ask what each evaluation view can see. On a balanced suite of 1,200 seeded contract violations, questions about final values detect under one percent, the best single question per violation class detects three quarters, and one large language model given the complete record in a single pass detects under ten percent. Changing only how a displaced belief is interpreted inside a real memory framework moves its answers about the past from 2 to 33 correct out of 36. Two author annotators, working without access to the router or to each other's labels, agreed with the compiled labels on 1,199 and 1,177 of 1,200 items, and with each other on 1,178.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
agent memory, language agents, belief revision, dynamic epistemic logic, executable semantics, operation-level evaluation, synthetic benchmarks, process supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fUL2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an executable specification for auditing memory transitions in language agents. Given a restricted scenario schema, a deterministic router generates operation labels, successor states, interpretations of displaced beliefs, observation scope, and obligations. The paper states compilability and axiomatic characterization results and introduces MEMLEDGER, comprising 900 synthetic bilingual sessions and 40,500 transitions.

The evaluation compares different views of seeded ledger violations, tests the downstream effect of displaced-belief interpretation, examines three memory frameworks, and investigates operation extraction and probe-induced changes. While the paper identifies useful distinctions in memory semantics, I find the manuscript insufficiently developed and the practical validation inadequate for acceptance.

Critical strengths

The distinction between a belief that was previously correct and one that was incorrect from the outset is meaningful: an actual change and a correction can produce the same current value while requiring different answers about the past. Section 5.2 provides a useful mechanism check. Changing only the disposition sentence improves history answers from 2/36 to 33/36, while the control answers remain correct in every arm. This supports the relevance of the distinction, although the experiment is limited to twelve fixtures and does not establish the effectiveness of the complete auditing framework.

Critical weaknesses
  1. The presentation lacks the clarification and editorial care needed to assess the technical contribution. The manuscript introduces formal objects, routing rules, contracts, axioms, complexity claims, and adapter lemmas in rapid succession, without sufficient explanation or worked examples connecting them. Important definitions remain ambiguous, including “undefined comparison” in A6 and the change set minimized by A2. These ambiguities affect the characterization claim rather than merely the writing style. Figure 2 also uses very small text and crowded labels, making the central empirical comparison difficult to inspect. Together, these problems indicate that the manuscript requires substantial explanatory and editorial revision.
  2. The empirical evidence does not establish reliable practical auditing. Section 5.1 modifies an intermediate ledger entry and successor snapshot without replaying subsequent events. The reported 0.8% final-value recall therefore characterizes this particular corruption protocol and does not establish how frequently final-answer evaluation misses faults during actual execution. The compiled judge's perfect detection is guaranteed by construction. On deployed frameworks, both automatic operation extractors perform below the 80.1% always-NO-OP baseline, and visibility and obligations cannot be observed through the available disclosures. The benchmark also uses one schema skeleton across seven vocabularies; agreement among author annotators under a shared handbook validates adherence to that specification rather than applicability to natural dialogue. These limitations leave a substantial gap between the proposed specification and an effective auditing framework. Combined with the presentation problems, they motivate my clear-rejection recommendation.
  3. I am also seriously concerned about the lack of editorial care in the manuscript. The writing and figure presentation give me the impression that substantial portions may have been generated using AI and retained without adequate human review or careful polishing. I cannot verify the extent of AI involvement; my criticism concerns the submitted material: dense terminology, insufficiently explained connections between claims, ambiguous definitions, and crowded figure labels. These problems substantially impair readability and make the submission appear insufficiently checked and refined for a conference paper.
Questions for authors
  1. Formal clarification: What precisely constitutes an “undefined comparison” under A6? For conflicting reports from incomparable sources, does R4 prescribe dissent without clarification, or does A6 require a clarification obligation? Please demonstrate how this case satisfies both the router and the axioms.
  2. Formal clarification: What changes are included in A2's change set? For stronger-source repetition of an existing value, how is mandatory support refinement compatible with minimality when the existing literal and support already satisfy success, consistency, and support closure?
  3. Assessment-changing evidence: Does the supplementary analysis evaluate faults propagated through subsequent execution, or detection before later writes can erase their effects? If so, how does final-value recall change? This would clarify how strongly the headline result depends on the mutation protocol.
  4. Practical scope clarification: Which contracts have been audited reliably end to end on native framework disclosures? Please distinguish demonstrated capabilities from those requiring additional disclosure interfaces or improved operation extraction.
4 comments

Thank you for the detailed review, and in particular for reading the formal part closely. Q1 and Q2 found two places where the axioms were worded more loosely than the implementation, and we are grateful for both. They are corrected below with exact statements. We are also glad that you find the disposition distinction meaningful and Sec. 5.2 "a useful mechanism check". We answer every point; the reply is in four parts because of length. We take your four questions first, then W2 and W1.

[Q1] "What precisely constitutes an 'undefined comparison' under A6? For conflicting reports from incomparable sources, does R4 prescribe dissent without clarification, or does A6 require a clarification obligation?"

to sum up:the comparison is undefined exactly when the two assertions come from the same source and carry no correction marker. Incomparable distinct sources do not exist in the fragment, so R4 and A6 never apply to the same event, and the router satisfies both.**

The comparison in A3 is between σ\sigma, the source of the input, and τ\tau, the source supporting the entry it conflicts with. When σ=τ\sigma=\tau the source order is silent, since it is irreflexive, and only the marker convention can order two assertions by one source. With no marker, nothing orders them. That is the undefined comparison, and it is the only instance in the fragment.

The text says "strict partial order" and then rejects equally ranked distinct sources at schema compile time. That wording is inconsistent. What the fragment requires, and what the released schema loader enforces, is a unique rank for every declared source, that is, a strict total order. We will correct the text accordingly. The cases are then disjoint:

CaseRuleAxiomOutcome
σ≻τ\sigma\succ\tauR3A3revise
τ≻σ\tau\succ\sigmaR4A3no-op, dissent recorded, no clarification
σ=τ\sigma=\tau, no marker, fact slotR5A6clarify, no dissent

The printed A3 ended with "otherwise nothing changes and the dissent is recorded", which overlapped A6. The restated A3 and A6 below remove the overlap. All 900 sessions use four sources with distinct ranks, so no compiled label is affected.

[Q2] "What changes are included in A2's change set? For stronger-source repetition of an existing value, how is mandatory support refinement compatible with minimality ...?"

in short:your argument is correct against the printed text, and the corrected statement makes refinement the minimal change.** Case 1b of the proof says "A1 is satisfied by the identity". For a strictly stronger source that sentence is wrong. The cause is that we stated success for literals, while the state, the router and the contracts all work with supported literals. We will restate the four axioms as follows:

A1 Success and well-formedness. If the operator accepts an epistemic input ℓ\ell carried by event ee for agent aa, then ℓ∈Ba′\ell\in B_a' and the support edge of ℓ\ell in Ba′B_a' is ee (success). Ontic postconditions hold. The successor is consistent, support-closed and ledger-complete.

A2 Minimal change. Δ(s,s′)\Delta(s,s') is the set of per-agent differences between ss and s′s' in three kinds of item: base literals (first and second order); support edges (one per literal, naming the event and source of its strongest accepted input); and history intervals with their disposition marks. For the operator's acceptance decision at (s,e)(s,e), no state whose difference from ss is a proper subset of Δ(s,T(s,e))\Delta(s,T(s,e)) satisfies the four clauses of A1.

A3 Source monotonicity. An input on an empty slot is accepted. If σ≻τ\sigma\succ\tau the input is accepted; if τ≻σ\tau\succ\sigma nothing changes and the dissent is recorded. A marked self-correction counts as σ≻τ\sigma\succ\tau against the same source's earlier assertion. An input that repeats the current value is accepted iff σ≻τ\sigma\succ\tau.

A6 Abstention. If neither σ≻τ\sigma\succ\tau nor τ≻σ\tau\succ\sigma (in the fragment: σ=τ\sigma=\tau, no marker, fact slot), nothing changes, both candidates are recorded and a clarification obligation is produced.

The dissent record, the pending candidates, the obligation and the ledger entry are appended outputs fixed by A3 and A6. They are not in Δ\Delta.

[Q2, continued] Corrected Case 1b. By A3 the repeated value is accepted iff σ≻τ\sigma\succ\tau. If it is accepted, success requires the support edge of ℓ\ell to be ee, which the identity does not achieve. Replacing the edge achieves it, and no proper subset of that change does: the empty change fails success, removing the old edge alone leaves ℓ\ell unsupported, and adding the new edge alone is not a state, since a literal carries one support edge. So refinement is the minimal change, and A2 forces it. If the input is not accepted, the identity satisfies A1 and A2 forces the identity. This is retain (σ=τ\sigma=\tau) or no-op (τ≻σ\tau\succ\sigma), which differ only in the label.

Nothing else moves. The six counterexample operators of the independence proof go through against the restated clauses; for example, last-writer-wins accepts every input, so it meets A1 and A2 and fails only A3. No label changes either: the same-value branch has always been checked at run time against its own contract on every compiled transition (refine requires a strictly stronger source, no agent's support is downgraded, and a weaker repetition leaves the state unchanged without dissent; App. G).

You rated the soundness concerns as not easy to address. For the two formal points, the complete repair is the restatement above: four axioms and one proof case. The router, the contracts and all 40,500 labels are exactly as submitted, and Proposition 1 holds as stated.

[Q3] "Does the supplementary analysis evaluate faults propagated through subsequent execution, or detection before later writes can erase their effects? If so, how does final-value recall change?"

Short answer: both questions have an answer. Detection before later writes is in Figure 2. For propagation, nine of the twelve classes are exactly invariant to replay, so final-value recall stays at 0 of 900 for them and cannot exceed 25% overall under any replay outcome.

(a) Nine classes are invariant to replay. Nine mutations (ghost update, revise without was-wrong, swapped disposition, scope broadcast, broken trigger, dissent overwritten, deletion not ledgered, clarify missing, time reorder) edit only fields of the ledger entry, or state components that no routing rule reads: history intervals with their disposition marks, and the pending-clarification list. The antecedents of rules R1–R8 read the current belief value, its source and support, the slot type, the marker, the trigger status and the clock. Replaying the later events from the corrupted snapshot therefore reproduces the clean twin's belief values at every later step. This follows directly from the released router and injector code. Final-value recall on these 900 pairs is zero with or without replay, because a current-value question does not read the corrupted fields.

(b) This bounds the dependence on the protocol. Even if replay made every fault in the remaining three classes visible at session end, final-value recall would be at most 300 of 1,200, that is 25%. Three quarters of the seeded violations are invisible to a final-value question whichever protocol is used.

(c) Detection before later writes. The question battery is asked at the corrupted step. It sees 100/100 wrong-value faults and 94/100 missing updates; the final-value question sees 4/100 and 6/100. So for these classes the faults are visible when they happen and are overwritten by the end of the session, which is the case for auditing transitions and not only final answers.

(d) The same pattern on a deployed framework. In Sec. 5.2 the fault sits in LangMem's own store. The current-value question is answered correctly in 12 of 12 fixtures both with the fault and with the repair, while history answers go from 2/36 to 33/36.

We will revise the abstract sentence to read: "On a balanced suite of 1,200 seeded record corruptions, a question about the final value sees under one percent, the best single question per class asked at the corrupted step sees three quarters, and one large language model given the complete record in a single pass sees under ten percent." We will add points (a)–(c) to Sec. 5.1 and App. G, and the final version will include the replay run for the three value-bearing classes.

[Q4] "Which contracts have been audited reliably end to end on native framework disclosures? Please distinguish demonstrated capabilities from those requiring additional disclosure interfaces or improved operation extraction."

all in all:the disposition contract is demonstrated end to end on LangMem's own store, and current value, history and provenance are read through probes on three frameworks. Operation-level audits need better extraction; visibility and obligation contracts need a disclosure interface.**

StatusCapabilityEvidence
DemonstratedCompile gold for any schema in the fragment; independent contract check; replay determinismThm. 1; 40,500 transitions
DemonstratedDetect all twelve violation classes on a ledger, with no false alarm on clean twinsSec. 5.1 (the reference check)
Demonstrated on a native storeDisposition contract, from fault to downstream answersSec. 5.2: LangMem, 2/36 to 33/36, controls 24/24
Demonstrated on native storesFork-and-probe protocol with probe-free control arms; parent-state isolation held in every runSec. 5.4: LangMem, Mem0, 100 sessions
Measured on three frameworksCurrent value, history and provenance through forked probes against the compiled referenceTable 2
Readable from native disclosuresRealized operation (207 of 249) and disposition (192 of 234), by human annotatorsSec. 5.3
Requires improved extractionFully automatic operation-level auditSec. 5.3
Requires a disclosure interfaceVisibility and obligation contractsSec. 5.3

We will place this table at the start of Sec. 5.

[W2] "The empirical evidence does not establish reliable practical auditing."

answer shortly: we respectfully see this differently. The paper does not claim a finished audit product for deployed systems. It claims a specification, a benchmark and diagnostic evidence, and the observations in your paragraph are either design properties or findings that only compiled gold makes measurable.

Your observationWhat it shows
The 0.8% figure characterizes the corruption protocolIt is exact under replay for 900 of 1,200 pairs and bounded at 25% overall (Q3)
The compiled judge is perfect by constructionCorrect: it is the reference, reported as a completeness check. The finding of Sec. 5.1 concerns the views that existing evaluations use: final value, question battery, LLM judge
Both extractors are below the always-no-op baselineA finding that supports the thesis. Measured against compiled gold, two LLMs cannot read operations off framework disclosures, and they answer definitely on all 54 items the annotators marked unknown. Operation labels produced by an LLM cannot serve as gold, which is the reason to compile them
Visibility and obligations cannot be observedAlso a finding. Of all label fields, exactly these two are absent from the disclosures of all three frameworks. That is a concrete interface requirement for framework developers
One skeleton across seven vocabulariesThe guarantees are proved for every schema in the fragment, and all 40,500 transitions pass an independent check. We will add a second, structurally different skeleton in the final version
Annotator agreement validates adherence to the handbookThat is its purpose: it checks that the rendered text carries the compiled label. Fit to natural dialogue is not claimed

What the evidence does establish. (i) Gold labels for every memory transition can be compiled with guarantees. (ii) The views used by current evaluations miss most transition-level violations, and this is not explained by the fault protocol. (iii) On a deployed framework, a single field of the label changes downstream answers from 2/36 to 33/36, which you describe as a useful mechanism check and Reviewer HreN as "direct evidence that the proposed field affects downstream behaviour". (iv) The paper measures precisely what deployed frameworks must expose before they can be audited automatically. We believe these four results stand on their own, and we will state the scope in these terms at the start of Sec. 5.

[fUL2-W1] "The paper introduces formal objects, routing rules, contracts, axioms, complexity claims, and adapter lemmas in rapid succession, without sufficient explanation or worked examples connecting them ... Figure 2 also uses very small text and crowded labels."

Short answer: we agree on Secs. 2–3 and on Figure 2, and we will make four concrete changes in the revision. Reviewer HreN rated the presentation "clear and easy to follow", so we read your comment as aimed chiefly at Secs. 2–3, where the formal objects pile up.

  1. One worked example will connect the three layers. We will move the six-event scenario of Table 3 from App. G into Sec. 2 and state, for every event, which rule fires, what the contract checks, and which axiom forces the outcome. One event is written out this way below.
  2. Explicit definitions. We will define the change set, the accepted input and the undefined comparison as in our answers to Q1 and Q2, and call the source order a strict total order.
  3. Figure 2 will be redrawn at full text width with larger type, with the per-class counts printed in the cells.
  4. Scope first. The capability table from our answer to Q4 will open Sec. 5.

The worked example, for one event. Three people are in the conversation: the user, the assistant and Bob. At step 3 the user tells everyone that the project is due on June 15. At step 4 the user tells the assistant in private: "My mistake, it is June 8."

LayerWhat it says about step 4
Routing ruleThe new statement contradicts an earlier one from the same speaker and carries an explicit correction ("my mistake"). The router therefore treats it as a correction of an error, and the operation is revise.
Contract (what an auditor checks)The assistant's current deadline is June 8. June 15 stays in the history and is marked was-wrong: it was never true. It is not marked true-until, which would mean it was true and then changed. Only the user and the assistant are updated, because Bob did not hear the correction.
Axioms (why no other outcome is allowed)A3: an explicit self-correction by the same speaker is accepted. A4: a correction of an error is recorded differently from a change in the world. A5: an agent who did not hear an event does not change.

So the assistant now answers "June 8" for the deadline and "June 15" for "what did you believe at step 3?", and Bob still believes June 15. Every row of the table in Sec. 2 will be read in this way.

On editorial care and AI use.

We take this concern seriously. Our use of generative tools is disclosed in the AI use statement after Sec. 7, and we take full responsibility for the content of the paper. On the substance, your reading was careful enough to find two imprecise statements in the axioms, and we have answered both in this thread with exact restatements and a corrected proof case. The other items you list (the density of Secs. 2–3, the missing connection between rules, contracts and axioms, the type size in Figure 2) are addressed by the four changes above.

In sum.

  • Q1, Q2: both formal points are corrected in this thread. The router, the contracts and all labels are unchanged, and Proposition 1 holds as stated.
  • Q3: the result is exactly replay-invariant for 900 of 1,200 pairs and bounded at 25% under any replay outcome.
  • Q4, W2: demonstrated capabilities are separated from those that need better extraction or a disclosure interface, and that separation is one of the paper's findings.
  • W1: four concrete changes.

Your two soundness concerns (the formal ambiguities and the dependence on the mutation protocol) are the ones you marked as hard to address, and we believe both are now resolved with exact statements. We would be grateful if you would reconsider your soundness and overall ratings in that light. We are happy to elaborate on any point.

Reviewer_HreNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an executable specification that compiles reference labels for individual memory transitions in language agents. Given a scenario schema, it determines the operation, successor state, interpretation of displaced beliefs, visibility, and outstanding obligations. The key distinction is between information that was previously true and information that was incorrect from the outset. The paper introduces MEMLEDGER, comprising 900 bilingual sessions and 40,500 transitions, and states compilability and axiomatic characterization guarantees. Validation combines injected violations, framework experiments, and human agreement checks. In a controlled LangMem intervention, correcting only the interpretation of displaced information improves historical answers from 2/36 to 33/36, while control answers remain unchanged.

Critical strengths

A useful specification of transition-level correctness. The framework goes beyond whether the current value is correct by recording how prior information should be interpreted, who observed a change, and what remains to be done. Counterfactual examples with identical final values but different correct historical answers make the benefit of this distinction concrete.

A well-controlled intervention supports the importance of historical interpretation. The LangMem experiment changes only the sentence describing the displaced value’s status. Historical answers improve substantially, current-value and unrelated controls remain correct, and repeating the faulty condition reproduces the original answers. This provides direct evidence that the proposed field affects downstream behaviour.

Critical weaknesses

The relationship between the routing rules and the axiomatic characterization needs clarification. R4 assigns conflicting evidence from a different, non-dominating source to a no-op with dissent recorded, whereas A6 requires a clarification obligation when the source comparison is undefined. Since the source ordering is partial, it is unclear how these provisions interact for incomparable sources. This matters because obligations are part of the compiled gold labels: an unchanged belief alone does not fully specify the expected transition.

Questions for authors

For conflicting evidence from two incomparable sources, should the router record dissent, emit a clarification obligation, or do both? Please clarify what “undefined” means in A6 and explain how this case is handled consistently by R4, A3, and A6. Does the clarification affect the compiled labels or only their description in the manuscript?

2 comments

Thank you for the positive and careful review. We are glad that the specification of transition-level correctness is useful to you and that you find the LangMem intervention well controlled. Your question isolated a real inconsistency in our wording, and we are grateful for it.

1."For conflicting evidence from two incomparable sources, should the router record dissent, emit a clarification obligation, or do both?"*

inside the fragment the case cannot occur, because the source order is in fact total. If incomparable sources were admitted, the answer would be a clarification obligation only, never both.**

The paper calls the source order a "strict partial order" and, in the same sentence, rejects equally ranked distinct sources at schema compile time. These two phrases do not fit together. What the fragment requires, and what the released schema loader enforces, is a unique rank for every declared source, that is, a strict total order. A schema with two sources of equal rank is rejected before any session is compiled. We will change the text to "strict total order". R4's "different, non-dominating source" therefore means a source that the current support strictly outranks.

2."Please clarify what 'undefined' means in A6." The comparison in A3 is between σ\sigma, the source of the input, and τ\tau, the source supporting the entry it conflicts with. It is undefined exactly when σ=τ\sigma=\tau and no correction marker is present. The source order is irreflexive, so it cannot rank a source against itself, and the marker convention, the only other thing that can order two assertions by one source, is silent. On a fact slot this routes to clarify (R5, third branch). With a marker, the later assertion ranks above the earlier one and R3 applies.

3."Explain how this case is handled consistently by R4, A3, and A6." With the order total on distinct sources, three cases partition every epistemic conflict:

CaseRuleAxiomOutcome
σ≻τ\sigma\succ\tauR3A3revise
τ≻σ\tau\succ\sigmaR4A3no-op, dissent recorded
σ=τ\sigma=\tau, no marker, fact slotR5A6clarify

The printed A3 said "otherwise nothing changes and the dissent is recorded", and that "otherwise" swallowed the third case. This is the overlap you noticed. We will restate the two axioms as follows:

A3 Source monotonicity. An input on an empty slot is accepted. If σ≻τ\sigma\succ\tau the input is accepted; if τ≻σ\tau\succ\sigma nothing changes and the dissent is recorded. A marked self-correction counts as σ≻τ\sigma\succ\tau against the same source's earlier assertion. An input that repeats the current value is accepted iff σ≻τ\sigma\succ\tau.

A6 Abstention. If neither σ≻τ\sigma\succ\tau nor τ≻σ\tau\succ\sigma (in the fragment: σ=τ\sigma=\tau, no marker, fact slot), nothing changes, both candidates are recorded and a clarification obligation is produced.

"An unchanged belief alone does not fully specify the expected transition." We agree, and the compiled ledger already keeps the two no-change outcomes apart:

R4R5, third branch
operationno-opclarify
statusdissentambiguous
obligationrecord_dissent for the slot and the rejected valueclarify for the slot, naming both candidates
pending candidatesnoneboth recorded

"Does the clarification affect the compiled labels or only their description in the manuscript?" Only the description. All 900 sessions use the four sources third_party ≺\prec assistant_inference ≺\prec user_direct ≺\prec system_record, with distinct ranks. None of the 40,500 transitions involves incomparable sources, and every one of them already passed the contract check for its branch. No label changes.

If incomparable sources were admitted. The restated axioms give a single answer: a clarification obligation with both candidates recorded and no dissent record, because neither source outranks the other. Recording dissent would keep whichever report arrived first, which makes the outcome depend on arrival order without any evidential reason. The restated A6 already covers this case, so extending the fragment needs one new router rule and one new case in the proof of Proposition 1.

A related imprecision in A2 (the definition of the change set) was raised by Reviewer fUL2. It is corrected in our reply to them and likewise changes no label. With both points fixed, the characterization holds exactly as stated. We hope this resolves your soundness concern and that you will consider it in your final score. We are happy to elaborate on any point.

Reviewer_HreNOpenReview ↗

Response to rebuttal

Thank you for your clear response, which resolves my question. Since the source order is in fact strictly total, as the schema loader enforces, R4 and A6 can no longer apply to the same event, and the “undefined comparison” in A6 reduces to a same-source conflict on a fact slot with no correction marker. Also, the restated A3/A6 and the table separating NO-OP (dissent) from CLARIFY make the expected transition explicit. I appreciate the confirmation that no compiled label changes. Please include in the revision the “strict total order” wording, the restated A3 and A6, the R4 vs. R5 table, and the explicit source ranking used in MEMLEDGER. With these changes, my soundness concern is addressed, and I keep my rating at 3 (weak accept).

Reviewer_HFZrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an executable semantics for compiling gold labels for memory transitions in language agents, covering operations, historical validity, visibility, and obligations. It presents MEMLEDGER, a synthetic benchmark, and shows that final-value evaluation can miss process-level errors, while a controlled intervention demonstrates that historical interpretation affects downstream answers. The main contribution is a principled framework for diagnostic evaluation; its broader applicability remains limited by restrictive schema assumptions and unreliable operation extraction from deployed systems.

Critical strengths
  • A meaningful evaluation problem. The paper clearly demonstrates that correct final answers do not guarantee correct memory transitions. Distinguishing beliefs that were previously true from beliefs that were wrong from the outset is particularly compelling, since identical current states can require different historical answers.

  • A principled diagnostic framework. Compiling labels from explicit semantics before rendering natural language provides a controlled alternative to LLM-generated annotations. Counterfactual pairs isolate specific semantic factors, while executable contracts make the evaluation criteria inspectable and reproducible.

Critical weaknesses
  • The theoretical novelty needs sharper positioning. Several axioms closely encode the router’s design choices, including source priority, event typing, and clarification behavior. The characterization establishes consistency with these conventions, but does not establish their broader appropriateness. The contribution beyond integrating established theories could be articulated more clearly.

  • The headline detection results depend strongly on the experimental construction. Injected mutations do not trigger replay of subsequent events, limiting their impact on final-state answers, while the compiled judge detects them by construction. These results demonstrate differences in observability more convincingly than practical detection advantages under realistic failures.

Questions for authors
  • How robust are the detection results to a more realistic fault model? Would the conclusions hold if faults altered actual transitions and subsequent events were replayed, or if error frequencies differed from the balanced mutation suite?

  • Can the framework generalize beyond the current schema and support reliable end-to-end auditing? Evidence from a structurally different schema and an independently evaluated system with sufficient state disclosure would substantially strengthen the contribution.

  • What explains the LLM judge’s low detection rate? Could shorter windows or stepwise auditing separate difficulties in locating relevant events from difficulties in reasoning about the contracts?

3 comments

Thank you for the constructive review. We are glad that you find the evaluation problem meaningful, the past-true versus never-true distinction "particularly compelling", and compiled labels "a controlled alternative to LLM-generated annotations". Your two weaknesses are specific, and we believe both are resolved below. The reply is in three parts because of length.

"Several axioms closely encode the router's design choices ... The contribution beyond integrating established theories could be articulated more clearly."

Shortly speaking: the contribution is not that the axioms are surprising. It is that gold labels for memory transitions can be compiled at all, and that the compiler is uniquely determined once a deployment declares three conventions. We claim three things as new.

(1) The paradigm: gold labels are compiled, not annotated. Memory benchmarks grade final answers, and the one operation-level benchmark we know of generates its memory points with an LLM and verifies them by hand. Here the label of every transition is the output of an executable semantics, fixed before any text exists. This is what makes counterfactual pairs, contract checks and exact replay possible, and it is the property you list as a strength.

(2) The disposition as a recorded field. The update/revision distinction is decades old, but no memory benchmark records it. Two sessions can end in the same state and still require different answers about the past. Sec. 5.2 shows on a deployed framework that this single field moves history answers from 2/36 to 33/36.

(3) The composition, which none of AGM, KM or DEL provides. Each of those theories characterizes one kind of change under a single input. None dispatches a mixed stream (ontic writes, epistemic inputs, announcements to a sub-audience, triggers, deletions) into one per-step record. Lemma 1 gives a condition under which product update with factual change stays exactly inside per-agent literal bases at O(n∣A∣2∣Alt∣)O(n|A|^2|\mathrm{Alt}|) per event. Lemmas 2–3 show that revision followed by a support sweep stays inside Hansson's postulates and is removal-minimal. Without these, per-transition gold would be either intractable or ambiguous.

Why a characterization relative to declared conventions is the right result. Which source to trust is a property of the deployment: a medical assistant and a scheduling assistant should not share a source order, so an axiom that fixed one would be wrong for most applications. The schema therefore declares exactly three things: the source order, the marker convention, and the typing of unmarked same-source change on preference and commitment slots as drift. Proposition 1 shows that nothing else is a choice. The removal set, the sweep set, the disposition, which agents change and when to abstain are all forced. A deployment that trusts its sources differently declares a different order and gets a different compiler with the same guarantees. The introduction now states this.

The axioms are also a taxonomy of real designs. The independence proof shows that dropping each axiom yields a store design in actual use: last-writer-wins (A3), broadcast to every agent (A5), treating a change in the world as a correction (A4), over-deletion (A2). So the characterization says which property a given memory store gives up, and Sec. 5.2 shows the consequence of giving up one of them on a deployed framework. Reviewer HreN describes the result as "a useful specification of transition-level correctness".

For the W2, Q1 "Injected mutations do not trigger replay of subsequent events ... Would the conclusions hold if faults altered actual transitions and subsequent events were replayed, or if error frequencies differed from the balanced mutation suite?"

In short: yes. For nine of the twelve fault classes the result is exactly the same under replay, which caps final-value recall at 25% under any replay outcome, and at 6% under any class mix in the current protocol.

(a) Nine classes are invariant to replay. Nine mutations (ghost update, revise without was-wrong, swapped disposition, scope broadcast, broken trigger, dissent overwritten, deletion not ledgered, clarify missing, time reorder) edit only fields of the ledger entry, or state components that no routing rule reads: history intervals with their disposition marks, and the pending-clarification list. The antecedents of rules R1–R8 read the current belief value, its source and support, the slot type, the marker, the trigger status and the clock. Replaying the later events from the corrupted snapshot therefore gives the same belief values as the clean twin at every later step, and a final-value question returns the same answer with or without replay. This follows directly from the released router and injector code. For these 900 pairs the zero comes from what a current-value question reads. It does not come from the protocol.

(b) This bounds the headline. Even if replay made every fault in the remaining three classes visible at session end, final-value recall would be at most 300 of 1,200, that is 25%. Three quarters of the seeded violations are invisible to a final-value question under any fault model of this kind.

(c) For the value-bearing classes, Figure 2 already reports detection before later writes. The question battery is asked at the corrupted step. It sees 100/100 wrong-value faults and 94/100 missing updates; the final-value question sees 4/100 and 6/100. So these faults are real and visible when they happen, and they are overwritten by the end of the session. An end-of-session question is too late, and a per-transition audit is not.

(d) Class frequencies. Recall is reported per class (Figure 2), so recall under any class mix ww is ∑cwcrc\sum_c w_c r_c. Final-value recall is at most 6% for every possible mix, since no class exceeds 6/100. The conclusion does not depend on the balanced design.

(e) The same pattern on a deployed framework. In Sec. 5.2 the fault sits in LangMem's own store. The current-value question is answered correctly in 12 of 12 fixtures both with the fault and with the repair, while history answers go from 2/36 to 33/36. A current-value question cannot tell the two arms apart.

On the compiled judge. You are right that it detects the violations by construction: it is the reference, and Sec. 5.1 reports its 1,200/1,200 with no false alarm as a completeness check. The comparison that carries the finding is among the views that existing evaluations use: a final-value question, a five-type question battery, and a large LLM reading the complete record, which sees under ten percent.

[ next Q2] "Can the framework generalize beyond the current schema and support reliable end-to-end auditing?"

yes on generalization, by construction. On end-to-end auditing, one contract is demonstrated on a deployed framework's own store, and the paper measures exactly what the others need.**

Beyond the schema. Theorem 1 and Proposition 1 are proved for every schema in the fragment and do not refer to our skeleton. The compiler accepts any schema in the grammar of App. G, and nothing in the router, the contract checker, the injector or the probes refers to a domain. The released corpus already spans seven domains and two languages, and all 40,500 transitions pass an independent contract check. A new schema is one declarative file, and the final version adds a second, structurally different skeleton.

End-to-end auditing.

CapabilityEvidence
Disposition contract on a native framework store, from fault to downstream answersDemonstrated end to end on LangMem: 2/36 to 33/36, controls 24/24 (Sec. 5.2)
Current value, history and provenance read through forked probes against the compiled referenceMeasured on Mem0, LangMem and Graphiti (Table 2)
Realized operation and disposition read from native disclosuresHuman annotators recover the operation for 207 of 249 transitions and the disposition for 192 of 234 (Sec. 5.3)
Fully automatic operation-level auditNeeds better extraction than current LLMs provide (Sec. 5.3)
Visibility and obligation contractsNeed a disclosure interface that none of the three frameworks exposes (Sec. 5.3)

The third row matters: the information needed for an operation-level audit is present in what the frameworks disclose, and people can read it. The last two rows are findings of the paper. They name the two things a framework must provide to be auditable, and they could not have been measured without compiled gold as the yardstick.

[Q3] "What explains the LLM judge's low detection rate? Could shorter windows or stepwise auditing separate difficulties in locating relevant events from difficulties in reasoning about the contracts?"

the existing evidence points to both, and it shows that the low rate is not an artifact of one model or of input length.**

  • Two models, two input lengths, the same discrimination. Besides the complete-view judge (balanced accuracy 54.8%), App. H reports a second model on a compressed view without successor states. It reaches 50.1% recall at a 40.3% false-alarm rate, a balanced accuracy of 54.9%. Shortening the input moved the operating point and left the separation where it was.
  • Locating is hard even when no reasoning is needed. Wrong-value mutations place an out-of-vocabulary token in the successor state, and the complete-view judge found 31 of 100.
  • Reasoning about a single transition is hard too. In Sec. 5.3 two LLMs judge one transition at a time from a short before/after disclosure, and neither beats the always-no-op baseline.

Together these say that an LLM reading the record is not a substitute for executable contracts, which is the point of the comparison. Your stepwise design is the clean way to separate the two factors. Both judges give one verdict per record in a single pass, and the final version adds the stepwise variant.

In sum. W1: the paper now states what is new (compiled gold, the disposition as a recorded field, the composition result) and why a characterization relative to declared conventions is the right one. W2/Q1: the result is exactly replay-invariant for 900 of 1,200 pairs, bounded at 25% under any replay outcome, and independent of the class mix. Q2 and Q3 are answered with evidence already in the paper. We hope this resolves your two weaknesses, and we would be grateful if you would consider raising your score. We are happy to elaborate on any point.

We will revise the abstract sentence to read: "On a balanced suite of 1,200 seeded record corruptions, …" We will add points (a)–(d) to Sec. 5.1 and App. G, and the final version will include the replay run for the three value-bearing classes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.