acceptodds
Under review as a conference paper at ICLR 2027

Evaluating LLM Agent Behaviors and memory Under Evolving Constraints

Abstract

LLM agents are increasingly deployed as long-running assistants and companions equipped with memory systems. Yet, as real-world constraints inevitably change over time, no existing benchmark systematically evaluates whether an agent can maintain a correct picture of current rules across hundreds of sessions without falling back on stale information. We introduce RevokeBench, a benchmark of 1,202 tasks designed to evaluate agent compliance with dynamically evolving instructions over extensive histories. RevokeBench leverages a novel two-layer graph representation to track an evolving body of rules as unambiguous current states, enabling deterministic task annotation without relying on an LLM-as-a-judge framework. We construct 100 evaluation episodes across 6 realistic agent settings, each featuring massive context records of 300k–1M tokens and 10–15 graded tasks, drawn from an open pool of over 20,000 tasks. We evaluate 8 backbone models, further pairing 2 selected models with 4 memory frameworks. Our results show that state violation rates range from 31% to 83% across flash-tier models, and frontier models do not uniformly outperform their flash-tier counterparts. Across all four memory frameworks evaluated spanning middleware and self-evolving memory, agents incur more violations and complete fewer tasks than a simple heuristic baseline. While our introduced Rule-Tracking Memory baseline improves performance, reliable constraint tracking in long-horizon deployments remains an open challenge. We release the full dataset, task generator, solver, evaluation framework, and baselines to support future research.

Primary area:
datasets and benchmarks
Keywords:
agent memory, graph generator, benchmark

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_cFnmOpenReview ↗
Rating
4
Confidence
1(−2)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This project introduces RevokeBench, a benchmark for evaluating whether long-running LLM agents can correctly track evolving constraints and act according to the currently valid state. The benchmark contains long multi-session histories with rule additions, revocations, conflicts, authority changes, and tool-use tasks. The authors find that several existing memory frameworks underperform a simple compact-summary baseline, and propose Rule-Tracking Memory (RTM) as a structured state-tracking baseline.

I find the problem important and the observation that stale but historically valid memories can lead to invalid actions interesting. However, I currently lean toward Weak Reject because the novelty relative to recent state-updating memory benchmarks is not sufficiently established, the necessity of the proposed formalization is unclear, and some experimental conclusions require stronger evidence.

Critical strengths
  1. The distinction between remembering historical information and maintaining what is currently valid for action is important for long-running agents.

  2. Evaluating actual tool actions rather than only QA responses is valuable.

  3. The full-context experiment is a useful control, suggesting that many failures come from memory construction/retrieval rather than purely from model reasoning.

  4. The finding that revoked permissions can remain actionable through memory retrieval is interesting and practically relevant.

Critical weaknesses
  1. Insufficient positioning against related state-updating benchmarks. Recent benchmarks such as AMA-Bench already explicitly evaluate state updating over long agent trajectories and also use graph-based memory representations. RevokeBench appears to differ through its emphasis on evolving authorization, authority hierarchy, rule interaction, and action-level compliance, but this distinction should be much more clearly articulated. The current novelty framing around long-horizon evolving-state tracking feels somewhat overstated.

  2. The necessity of the two-layer formalism is unclear. The paper argues that defeated rules should remain stored because they may become active again later. I agree with this requirement, but it does not seem to require the proposed grounded-argumentation and fixed-point machinery. A simpler persistent rule/event graph could retain all historical rules and dependencies, maintain active/inactive labels, and expose only the latest materialized state to the agent. When a rule changes, affected nodes could simply be recomputed. Since the benchmark itself is generated from a finite set of predefined transition patterns, the paper should demonstrate which cases genuinely require the proposed formalism rather than a simpler state-transition/dependency graph.

  3. Experimental robustness and baseline fairness need improvement. The main experiments are single runs, while the appendix reports substantial variation across the limited repeated runs. In addition, some memory baselines appear constrained by their configurations; for example, Dynamic Cheatsheet frequently reaches its memory cap. RTM is also relatively benchmark-aware, as it uses structured benchmark fields and the same rule solver used by the grader. These factors make the broad conclusion that existing memory frameworks systematically fail less convincing.

  4. Presentation. Several figures are visually dense and overly infographic-like, with small text and unnecessary decorative elements. Simpler scientific diagrams would make the benchmark and method much easier to understand.

Questions for authors
  1. How does RevokeBench differ fundamentally from AMA-Bench and other recent benchmarks that already study state updating over long agent trajectories?

  2. Can the authors provide a concrete RevokeBench example that cannot be correctly represented using a persistent event/rule graph with dependency edges, active/inactive labels, and a materialized current-state view?

  3. What fraction of tasks actually require the proposed multi-rule conflict/fixed-point reasoning, rather than simple state transitions?

  4. Given that RTM uses the benchmark's structured representation and solver, how should its gains be interpreted as evidence for a general memory method rather than a benchmark-specific diagnostic baseline?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cFnm (1/3)

We thank the reviewer for the insightful review, and appreciate the acknowledgement on our contributions. To your questions, we have the following answers.

Q1. Positioning on AMA-Bench and others

Thank you for pointing this. We now cite AMA-Bench. Briefly the difference lies in what state evolves and what is graded. (1) What evolves. AMA-Bench's State Updating category tracks environment state (inventories, counters, container contents) within a single-task log, and conversational benchmarks (LongMemEval, etc.) track user facts where the newest value wins. RevokeBench's state is normative: which instructions are in force, decided by issuer authority, priority, conditions, group membership, withdrawal and reinstatement, so the newest statement is often not the operative one. In 68% of tasks, at least one option's correct status was set by an event that does not mention that option. (2) What is graded. AMA-Bench asks a memory system questions about a frozen trajectory produced by another agent and scores free-text answers with an LLM judge. RevokeBench grades the agent's own action at the moment it must act, against a solver over the rule history, with no judge, and separates acting on stale state (violation) from not acting (abstention). (3) Closest work. StateMemBench models retractions and scoped exceptions, but in conversations of at most ~15K tokens, graded on answers, without issuer hierarchy or reinstatement, etc. To our best knowledge, none combines normative state that evolves under defeasible semantics over hundreds of sessions with judge-free, action-level grading. AMA-Bench's finding that similarity-based retrieval is lossy on agent logs is complementary to ours.

Q2. Why not one persistent graph with active/inactive labels?

Such a graph can represent RevokeBench, but only if its labels are recomputed from the whole rule history at every step, which is exactly our two-layer construction. The intuitive alternative, deleting or deactivating the defeated rule when a conflict arises, is what the construction was designed to avoid. Considering an extended example of workplace decision-makings: the team lead approves vendor A (s2, π=1); compliance bans A during low-budget periods (s5, π=2), defeating the approval; the budget recovers (s7); it tightens again (s8); an executive endorses A, raising the lead's rule to π=3 (s9). The correct status of A is approved, banned, approved, banned, approved. A store that deletes the defeated approval at s5 answers approved, banned, no rule, banned, banned: when the ban lapses there is nothing left to restore, and the endorsement has no rule to raise. Keeping every rule in a persistent layer and recomputing the defeat relation at each step (the grounded extension, which is unique and independent of deletion order) restores the approval in both cases. Whole-set recomputation also avoids over-deleting conclusions that keep an alternative derivation (the situation Delete-and-Rederive handles). On the benchmark itself, replaying the ground-truth event logs with perfect extraction, a delete-on-conflict store mislabels at least one option in 284 of 1,202 tasks (24%), and in 118 tasks (10%) it no longer approves any option that is in fact approved.

Q3. How many tasks need conflict resolution.

We measured this on the ground-truth logs with a ladder of resolvers that are given the true rule base at each task (no memory loss), recording the simplest one that gets every offered option right:

Simplest sufficient resolverTasks
Per-option state transition (latest ruling naming the option)10.0%
+ conditions, group membership and list regimes (latest applicable ruling)11.6%
+ priority between conflicting rulings, one pass76.5%
Full derivation layer (fixed point)1.9%

So 90% of tasks cannot be answered by per-option state transitions, and 78% require resolving conflicting rules by priority rather than recency. Most of these involve a standing list regime (only certified or listed items) that outranks later approvals of individual items; in 5% of tasks, an earlier ruling by a higher-ranked speaker outranks a later one. Iterated fixed-point reasoning, where a rule's condition is itself decided by a conflict, is needed in only 23 tasks (1.9%). With strict priorities, the grounded extension reduces to "the highest applicable priority wins" for a single literal; we use the full construction because it gives every state the generator can produce a unique, order-independent ground truth, not because most tasks exercise deep chains. These resolvers see the true rule base; an agent must first reconstruct it from 344k–935k tokens of history.

AuthorsOpenReview ↗

Response to Reviewer cFnm (2/3)

Q4. Intepreting RTM

Our intent is that RTM is not being proposed for a general memory method on such class of problems yet, and we will present it as a privileged-structure reference baseline for baselining future works. RTM borrows the grader's rule language and solver, a rank-over-recency priority convention, the entity list as an extraction hint, and each task's options. It never sees events or labels, and the backbone extracts every ruling from the raw transcript. Its role is diagnostic, and three findings carry beyond RTM. (1) Having the grader's logic does not solve the benchmark: RTM scores +0.191 against +0.789 with full context, and offline its verdicts match the ground truth on 0.723 of task–option pairs, against 1.000 by construction on the true rules. The remaining error is extraction from long dialogue, the general memory problem. (2) Update semantics matter independently of RTM: on the same extracted statements, with perfect extraction a delete-on-conflict store(the policy Mem0 applies to contradicted memories) mislabels some option on 24% of tasks. It suggests current methods might have methodological flaws against RevokeBench class problems. (3) Structure is needed: given the true rules, conditions plus one-pass priority are fully right on 98.1% of tasks. We read this, as a hypothesis, that it might suggest that evolving rule state suits LLM extraction plus explicit resolution, as in solver-backed reasoning (PAL, Logic-LM) and in policy engines that resolve conflicts by declared precedence (XACML, Cedar). So RTM might carries some methodological values on evolving state problems in agent memory that RevokeBench introduces if the longer-to-full-context could be well handled to provision the raw information that a reliable structural representation like RTM requires.

On AI-use statement

We do have an AI-use statement which was placed right after references per the guideline. We can move it to be right after conclusion and/or page limits so it reads a bit better.

On presentation

Thanks you for your feedback. We will be sure to refine and polish the figures read more scientifically informative.

On experimental robustness and baseline fairness need improvement.

Thank you for the feedback, we agree with that. Therefore we have run multiple runs on stage1 and updated configurations on stage2 for a more aligned provisioning. The results are the following:

We repeated Stage 1 three times on all 1,202 tasks, using compact notes hard-capped at 4k tokens. This revised configuration replaces the earlier prompt-only cap. Values are mean ± standard deviation, Exam intervals are 95% t confidence intervals for the mean across three runs (df = 2), conditional on the fixed evaluation set. They do not quantify uncertainty from sampling new histories.

BackboneViol. ↓Comp. ↑Abst.Exam ↑Exam 95% CI
Qwen3.8-Flash0.308 ± 0.0330.385 ± 0.0090.307 ± 0.038+0.038 ± 0.014[0.002, 0.073]
DeepSeek-V4-Flash0.294 ± 0.0400.491 ± 0.0520.216 ± 0.021+0.175 ± 0.119[−0.121, 0.472]
GLM-5.3-Flash0.470 ± 0.0250.390 ± 0.0180.141 ± 0.011−0.140 ± 0.050[−0.264, −0.017]
Qwen3.7-Flash0.546 ± 0.0050.244 ± 0.0120.209 ± 0.007−0.383 ± 0.029[−0.456, −0.310]
Llama-4-Maverick0.842 ± 0.0230.088 ± 0.0140.071 ± 0.014−0.786 ± 0.038[−0.881, −0.690]
Spark 1.30.073 ± 0.0100.903 ± 0.0090.024 ± 0.002+0.829 ± 0.023[0.773, 0.885]
GPT-5.50.119 ± 0.0190.767 ± 0.0340.114 ± 0.018+0.633 ± 0.072[0.455, 0.811]
Claude Opus 5.50.367 ± 0.0270.462 ± 0.0460.171 ± 0.020+0.048 ± 0.091[−0.178, 0.274]
AuthorsOpenReview ↗

Response to Reviewer cFnm (3/3)

(continued)

For the repeated Stage 2 comparison, we used:

  • mem0: v1.0.11 with automatic UPDATE/DELETE enabled (infer=True). search(limit=100, threshold=None, rerank=False).

  • MemOS: v2.0.36, text_mem backend general_text; top_k=50, max_turns_window lowered from 15 to fit the token allocation, mem_reader.config.llm.config.max_tokens=4096; enable_activation_memory, enable_parametric_memory, enable_preference_memory, enable_mem_scheduler and PRO_MODE left at False.

  • Dynamic Cheatsheet: approach_name=DynamicCheatsheet_Cumulative, max_tokens=2048 (the curator call uses 2 * max_tokens = 4,096). We removed the additional runner-imposed sheet-size truncation.The cumulative variant regenerates the full sheet at each update, the newly generated sheet remains bounded by this output budget.

  • Memp: policy={"build": "direct", "retrieve": "query", "update": "vanilla"}, retrieve_num=50; MAX_TOKENS=4096 and TEMPERATURE=0 in llm_api.py.

The following results are three-run means over the same 1,202 Stage 2 tasks, including full-context references: 3,606 task records per condition. Interval definitions match Q1.

BackboneMemoryViol. ↓Comp. ↑Abst.Exam ↑Exam 95% CI
Qwen3.8-FlashFull context0.055 ± 0.0030.845 ± 0.0060.100 ± 0.006+0.789 ± 0.013[0.757, 0.822]
No memory0.582 ± 0.0060.157 ± 0.0020.261 ± 0.005-0.456 ± 0.005[-0.468, -0.444]
compact-8k0.313 ± 0.0060.393 ± 0.0020.293 ± 0.004+0.020 ± 0.008[0.001, 0.039]
RTM0.344 ± 0.0090.557 ± 0.0070.098 ± 0.002+0.191 ± 0.025[0.129, 0.254]
mem00.453 ± 0.0040.436 ± 0.0060.111 ± 0.003−0.078 ± 0.009[−0.100, −0.056]
MemOS0.457 ± 0.0070.438 ± 0.0040.105 ± 0.008−0.057 ± 0.009[−0.079, −0.035]
Memp0.503 ± 0.0050.376 ± 0.0030.121 ± 0.002−0.174 ± 0.006[−0.189, −0.159]
Dynamic Cheatsheet0.428 ± 0.0100.340 ± 0.0010.232 ± 0.010−0.132 ± 0.008[−0.152, −0.112]
Claude Opus 5.5Full context0.009 ± 0.0010.774 ± 0.0010.216 ± 0.002+0.763 ± 0.001[0.760, 0.766]
No memory0.571 ± 0.0020.168 ± 0.0040.261 ± 0.005-0.432 ± 0.006[-0.446, -0.417]
compact-8k0.353 ± 0.0020.479 ± 0.0020.167 ± 0.002+0.106 ± 0.005[0.094, 0.119]
RTM0.312 ± 0.0040.518 ± 0.0040.170 ± 0.001+0.256 ± 0.018[0.212, 0.300]
mem00.382 ± 0.0040.507 ± 0.0060.111 ± 0.003+0.064 ± 0.009[0.042, 0.086]
MemOS0.331 ± 0.0040.446 ± 0.0020.223 ± 0.004+0.082 ± 0.003[0.075, 0.089]
Memp0.416 ± 0.0020.378 ± 0.0080.206 ± 0.008−0.083 ± 0.006[−0.098, −0.068]
Dynamic Cheatsheet0.394 ± 0.0050.358 ± 0.0020.248 ± 0.006−0.067 ± 0.004[−0.077, −0.057]

Some revised framework results improve, but the central observed comparison remains.

Reviewer_95KaOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RevokeBench, a benchmark testing whether LLM agents act on the currently valid rules across very long, multi-session histories (344k–935k tokens), rather than on stale constraints. The core idea: operational rules change over time (approve → prohibit → reinstate), updates arrive as passing remarks with varying speaker authority, and "knowing a rule is not the same as applying it" — so the benchmark grades tool-use decisions (pick one licensed option via an act tool), not QA recall.

The technical foundation is a two-layer representation: a persistent rule base (rules with integer priorities, six update ops: ADD/SUPERSEDE/CONDITION/CONFLICT/SUPPORT/RETRACT) and a derivation layer that recomputes consequences per session via Dung's grounded extension over a priority-ordered defeat graph plus forward chaining. Because defeat never erases stored rules, a later SUPPORT can restore an earlier permission. This enables deterministic, LLM-judge-free grading: each action asserts literals, a contradiction in the forward-chained closure is a Violation; a non-violating pick of a licensed option is a Completion; everything else is Abstention. Results are summarised by an exam score S ∈ [−1,1] (+1/0/−1, difficulty-weighted; always-abstain = 0).

RevokeBench is 100 episodes across 6 settings, 1,202 graded tasks drawn from a pool of 21,996, with a deterministic generator/solver/renderer. Evaluation: Stage 1 scores 8 backbones under a shared compact-8k memory; Stage 2 pairs 2 backbones (Qwen3.8-Flash, Claude Opus 5.5) with 4 memory frameworks (mem0, MemOS, Memp, Dynamic Cheatsheet) plus no-memory and full-context controls. Headline findings: flash-tier violation rates 31–83%; frontier models don't uniformly beat flash; all four memory frameworks score worse than a plain compact-8k summary on both backbones; and a proposed Rule-Tracking Memory (RTM) baseline tops the bounded-memory conditions (+0.152 / +0.140 exam over compact-8k), though still far below full context.

Critical strengths

1. Deterministic, LLM-judge-free grading of actions over revised state — the right design, executed rigorously. The central contribution is methodological: by separating stored rules from recomputed consequences and grading via a formal solver, the benchmark removes the LLM-judge noise that undermines most long-horizon memory evaluations, and it grades what actually matters (the tool decision) rather than whether the agent can state a rule. The rigour is unusually deep: every evaluated state is re-derived grader-side independently of the generator (shared engine aside), all 54,282 session states and 1,202 task-augmented states converged and were consistent, and every label was re-derived under a broader propagation reading that agreed on all 6,940 (task, option) pairs. The over-deletion guard (check 3) specifically targets the failure mode incremental view-maintenance is prone to. For a benchmark paper, this verification discipline is the core asset and it is strong.

2. A clear, actionable negative result with a cause, not just a leaderboard. The finding that mem0, MemOS, Memp, and Dynamic Cheatsheet all score below compact-8k on both backbones (every one negative exam; Table 2) is the kind of result that redirects a subfield. Crucially the paper doesn't leave it as a number: the compact-8k → full-context gap (e.g. Qwen violation 0.309 → 0.054) shows the models can apply the rules when the right text is present, so the failure is in what memory keeps and surfaces. The RTM diagnostic sharpens this further — RTM records 88% of prohibitions and 94% of supersessions, 87% of items trace to a rule event, yet its derived verdict matches ground truth on only 73% of options, and errors concentrate where one rule governs many entities (62% under a group ban). That is a precise, mechanism-level explanation of where constraint-tracking breaks, which is more valuable than the headline rates.

Critical weaknesses

1. The headline conclusions rest on single runs, and the paper's own provenance appendix shows this is not safe at the resolution of several key claims. Appendix E states plainly that these are single runs, that temperature 0 is non-deterministic at these providers, and that 40–70% of decisions differed between the few repeat pairs, and it concedes that "differences below about 0.1 in violation rate between adjacent rows are not resolved; compact-8k and RTM intervals overlap." But the abstract and §5.4 present RTM as the best bounded-memory method (+0.152 / +0.140 exam) as a headline contribution — and on Qwen3.8-Flash the RTM-vs-compact-8k cluster-bootstrap intervals overlap (Table 10: RTM [+0.008, +0.380] vs compact-8k [−0.202, +0.247]). The framework-loses-to-summary result is large enough (frameworks at −0.3 to −0.6 vs compact-8k at +0.03/+0.11) to survive this noise, and I weight it accordingly; but the RTM claim, several cross-tier comparisons in Stage 1 (e.g. "Claude Opus 5.5 falls below DeepSeek-V4-Flash"), and any ranking separated by <0.1 are under-powered. For a benchmark whose value is the reliability of its rankings, multi-run means with variance on at least the two Stage-2 backbones are close to mandatory, and their absence is the single biggest issue.

2. The benchmark is entirely synthetic and template-generated, and its difficulty is partly an artifact of construction choices rather than of realistic rule evolution. Every history is built from 16 patterns over template corpora with ~85% filler, and much of the difficulty comes from deliberately adversarial construction: fixed option sets so stale picks bite, rank-0 "near-miss" lines phrased exactly like decisions, requester nudges toward violating options in 757/1,202 tasks, and cycle counts raised purely to stress memory compression (§C.2 notes raising cycles "does not make any single decision harder to derive; it raises the compression a memory system faces"). This is internally coherent and well-controlled, but it means violation rates measure behaviour on a distribution designed to trap summarisation, not an estimate of real deployment rates — which the paper acknowledges ("should not be interpreted as estimates of action prevalence"). Two consequences: (a) the frameworks (mem0 etc.) are being evaluated on exactly the stale-trap axis they were never optimised for, under budgets the authors set (Table 8), so "framework < summary" may partly reflect configuration rather than an intrinsic limitation — the paper would be stronger with a budget-sensitivity sweep for at least one framework; and (b) with no human-authored or real-log validation episodes, there's no evidence the measured difficulty transfers to non-synthetic histories.

Secondary:

  • RTM is both the proposed method and privileged by construction. RTM runs the same solver that produces the benchmark's labels (§5.4) and reads structured blind-item fields (option sets, parameters, entity names; Appendix E). The paper is honest that RTM is "a baseline," and the point it makes — explicit rule-state tracking helps — is fair. But presenting a method that shares the grader's inference engine as the best bounded-memory result invites a fairness question the paper should address head-on: how much of RTM's edge is the solver versus the memory design, and would a framework given the same structured inputs close the gap?
  • Grading fragility revealed by the substring-match bug. That an earlier matcher scored refusals as actions and inflated one condition's violation rate from 0.048 to 0.462 (Appendix E) is reported transparently, which is good — but it shows the deterministic grader's action-resolution layer is sensitive to argument phrasing, and the 80-char / 60%-prefix heuristics are themselves tunable choices that could shift borderline numbers. Reporting unresolved-argument counts per condition (as the paper says it does) mitigates but doesn't eliminate this.
Questions for authors
  1. Can you provide ≥3 runs with means and either s.d. or bootstrap CIs for the Stage-2 table on both backbones? Specifically, does RTM remain the top bounded-memory method with non-overlapping intervals vs compact-8k, and do the Stage-1 cross-tier orderings (e.g. Opus 5.5 vs DeepSeek-V4-Flash) survive repetition? Given the 40–70% decision-flip rate you report, this is the pivotal question.

  2. The "frameworks < compact-8k" result is central. For at least one framework (say mem0 or MemOS), can you show a budget/top-k/threshold sweep demonstrating the deficit is not an artifact of the chosen memory and retrieval budgets in Table 8 (e.g. mem0 at top-k 12 with "0 updates/deletes" suggests its update path may not have been exercised)?

  3. How much of RTM's advantage comes from sharing the grader's solver versus from explicit rule-state extraction? A control where a competing framework is given the same structured blind-item fields RTM consumes, or an RTM variant whose verdicts come from the backbone rather than the solver, would isolate this.

  4. Is there any human-authored or real-log episode (even a small held-out set) on which the measured difficulty ordering reproduces, to support that the synthetic traps reflect a real deployment failure rather than a constructed one?

  5. How many actions are unresolved per condition under the final matcher, and how sensitive are the headline violation rates to the 80-char / 60%-prefix thresholds? A small ablation on these would reassure that rankings aren't matcher-dependent.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 95Ka (1/3)

We thank the reviewer for the insightful review. We hereby address your questions by either clarifying or providing additional results.

Q1. multiple runs on both stages

We repeated the Stage 1 evaluation three times, with compact-8k with notes hard-capped at 4k, for each backbone on all 1,202 tasks. We report mean ± standard deviation for violation, completion, abstention, and Exam, together with 95% t confidence intervals for mean Exam across runs (n = 3, df = 2). These intervals characterize run-to-run uncertainty on the fixed evaluation set.

BackboneViol. ↓Comp. ↑Abst.Exam ↑Exam 95% CIFar viol. ↓Trap viol. ↓Weighted viol. ↓
Flash-tier models
Qwen3.8-Flash0.308 ± 0.0330.385 ± 0.0090.307 ± 0.038+0.038 ± 0.014[+0.002, +0.073]0.2960.3480.326
DeepSeek-V4-Flash0.294 ± 0.0400.491 ± 0.0520.216 ± 0.021+0.175 ± 0.119[−0.121, +0.472]0.2370.1930.304
GLM-5.3-Flash0.470 ± 0.0250.390 ± 0.0180.141 ± 0.011−0.140 ± 0.050[−0.264, −0.017]0.3920.3530.498
Qwen3.7-Flash0.546 ± 0.0050.244 ± 0.0120.209 ± 0.007−0.383 ± 0.029[−0.456, −0.310]0.5780.5560.585
Llama-4-Maverick0.842 ± 0.0230.088 ± 0.0140.071 ± 0.014−0.786 ± 0.038[−0.881, −0.690]0.8500.8680.851
Frontier models
Spark 1.30.073 ± 0.0100.903 ± 0.0090.024 ± 0.002+0.829 ± 0.023[+0.773, +0.885]0.0770.0670.077
GPT-5.50.119 ± 0.0190.767 ± 0.0340.114 ± 0.018+0.633 ± 0.072[+0.455, +0.811]0.1290.0940.128
Claude Opus 5.50.367 ± 0.0270.462 ± 0.0460.171 ± 0.020+0.048 ± 0.091[−0.178, +0.274]0.3510.3620.385

The repeated evaluations continue to reveal substantial behavioral revocation failures for several backbones.

For the repeated Stage 2 comparison, we used:

  • mem0: v1.0.11 with automatic UPDATE/DELETE enabled (infer=True). search(limit=100, threshold=None, rerank=False).

  • MemOS: v2.0.36, text_mem backend general_text; top_k=50, max_turns_window lowered from 15 to fit the token allocation, mem_reader.config.llm.config.max_tokens=4096; enable_activation_memory, enable_parametric_memory, enable_preference_memory, enable_mem_scheduler and PRO_MODE left at False.

  • Dynamic Cheatsheet: approach_name=DynamicCheatsheet_Cumulative, max_tokens=2048 (the curator call uses 2 * max_tokens = 4,096). We removed the additional runner-imposed sheet-size truncation.The cumulative variant regenerates the full sheet at each update, the newly generated sheet remains bounded by this output budget.

  • Memp: policy={"build": "direct", "retrieve": "query", "update": "vanilla"}, retrieve_num=50; MAX_TOKENS=4096 and TEMPERATURE=0 in llm_api.py.

The following results are three-run means over the same 1,202 Stage 2 tasks, including full-context references: 3,606 task records per condition. Interval definitions match Q1.

BackboneMemoryViol. ↓Comp. ↑Abst.Exam ↑Exam 95% CI
Qwen3.8-FlashFull context0.055 ± 0.0030.845 ± 0.0060.100 ± 0.006+0.789 ± 0.013[0.757, 0.822]
No memory0.582 ± 0.0060.157 ± 0.0020.261 ± 0.005-0.456 ± 0.005[-0.468, -0.444]
compact-8k0.313 ± 0.0060.393 ± 0.0020.293 ± 0.004+0.020 ± 0.008[0.001, 0.039]
RTM0.344 ± 0.0090.557 ± 0.0070.098 ± 0.002+0.191 ± 0.025[0.129, 0.254]
mem00.453 ± 0.0040.436 ± 0.0060.111 ± 0.003−0.078 ± 0.009[−0.100, −0.056]
MemOS0.457 ± 0.0070.438 ± 0.0040.105 ± 0.008−0.057 ± 0.009[−0.079, −0.035]
Memp0.503 ± 0.0050.376 ± 0.0030.121 ± 0.002−0.174 ± 0.006[−0.189, −0.159]
Dynamic Cheatsheet0.428 ± 0.0100.340 ± 0.0010.232 ± 0.010−0.132 ± 0.008[−0.152, −0.112]
Claude Opus 5.5Full context0.009 ± 0.0010.774 ± 0.0010.216 ± 0.002+0.763 ± 0.001[0.760, 0.766]
No memory0.571 ± 0.0020.168 ± 0.0040.261 ± 0.005-0.432 ± 0.006[-0.446, -0.417]
compact-8k0.353 ± 0.0020.479 ± 0.0020.167 ± 0.002+0.106 ± 0.005[0.094, 0.119]
RTM0.312 ± 0.0040.518 ± 0.0040.170 ± 0.001+0.256 ± 0.018[0.212, 0.300]
mem00.382 ± 0.0040.507 ± 0.0060.111 ± 0.003+0.064 ± 0.009[0.042, 0.086]
MemOS0.331 ± 0.0040.446 ± 0.0020.223 ± 0.004+0.082 ± 0.003[0.075, 0.089]
Memp0.416 ± 0.0020.378 ± 0.0080.206 ± 0.008−0.083 ± 0.006[−0.098, −0.068]
Dynamic Cheatsheet0.394 ± 0.0050.358 ± 0.0020.248 ± 0.006−0.067 ± 0.004[−0.077, −0.057]
AuthorsOpenReview ↗

Response to Reviewer 95Ka (2/3)

(continued)

Though we observe modest improvements in some framework results relative to the originally reported settings, the central finding remains unchanged. All four evaluated memory frameworks have higher mean violation rates, lower completion, and lower Exam scores than both compact-8k and RTM on both backbones. The repeated evaluations under these revised configurations continue to show that the tested memory frameworks underperform these baselines on behavioral revocation.

Tuning the frameworks' configurations changes how much they keep, but not what they are built to keep. They are designed to remember what is relevant and reuse what worked: they retrieve by similarity, so a revocation that names a group or condition rather than the option is easily missed in favour of the original approval. RevokeBench instead asks which instruction is currently in force, which requires tracking authority, scope, time and revocation as explicit state, the gap the benchmark is designed to expose.

Q2. Memory Framework Provisioning Thanks for pointing this out. We have adjusted configurations on framework to ensure a closer closer provisioning and conducted the multiple run involving memory frameworks. Please refer to the stage 2 multiple run results from the answer to Q1 (above). Specifically, we iterate our updated configurations as follow:

  • mem0: v1.0.11 with automatic UPDATE/DELETE enabled. (latest mem0 2.0.x deprecated UPDATE/DELETE and moved to a solely ADDITIVE architecture)
  • Dynamic Cheatsheet: sheet-size cap removed.
  • Memp: retrieval top-k increased to 10.
  • compact-8k: notes hard-capped at 4k tokens.

Q3. RTM solver-rule control

Thanks for pointing this out. We agree that separation matters. RTM is intended as a reference baseline for RevokeBench, not as a general method. We therefore verify the latter of which the reviewer has suggested. Two existing measurements bound the solver's share. (i) Offline, with zero model calls, we keep RTM's extracted statements fixed and replace the two-layer solver with per-option latest-wins bookkeeping: agreement with the ground-truth status of each offered option falls from 0.723 to 0.582. (ii) Natural-language memories that leave the decision to the model score well below RTM on the same panel (exam: compact-8k +0.002, memOS −0.329, RTM +0.191). Since neither keeps an explicit rule table, the revision adds both conditions we describe on the same panel: (a) RTM without structured blind-item fields (no entity list, no option or parameter fields. Both are parsed from the task text), and (b) a natural-language rule table maintained by the same backbone from the raw sessions (option, current status, who set it, when, under what condition), with no solver, no rule language and no structured fields. The agent reads the table and decides itself. Results: (a) exam +0.142, (b) exam -0.121. The gap between (b) and compact-8k measures explicit rule state whereas the gap between (a) and (b) measures the solver. The results suggests that most of RTM’s lead comes from formalism. Which may be explained by a structured representation might be a good tool on representing complex memory current states comparing to natural language counterparts which could be prolonged and by nature token costy.

Q4. Human-authored Held-out

To provide human authored held-out by which reflects more real-world scenario cues, we additionally constructed RevokeBench-H, a held-out set of four new histories with 15 tasks each (60 tasks total), using generator seeds disjoint from the release. Four human writers, without AI tools, rewrote 1,947 lines carrying or imitating rules, working from descriptions of what those lines must convey. The event log and solver labels were preserved. Session titles and non-rule-bearing filler remained generated. This set contains no real records. Histories contain approximately 30k–60k tokens, this is a check of human-written surface realization, not a replication of the main release's 344k–935k-token horizon or an independently human-designed task distribution.

An annotator who had not seen the line descriptions annotated authority, item, effect, and conditions against the intended events, disagreements were rewritten and checked again. Separately, four people each answered 15 tasks with the full history and text search, and all 60 answers matched the solver.

The Qwen3.8-Flash Exam score evaluations on RevokeBench-H are:

ConditionExam
Full context+0.816
RTM+0.650
compact-8k+0.516
mem0 v1.0.11−0.107
MemOS−0.178
Memp−0.058
Dynamic Cheatsheet−0.147

These supplementary descriptive results retain the framework-versus-baseline pattern under human-rewritten rule wording. We will include the construction and validation protocol, writer guide, annotation scheme, and the --split heldout evaluation option in the revision and release.

AuthorsOpenReview ↗

Response to Reviewer 95Ka (3/3)

Q5. Unresolved arguments and Matcher sensitivity.

Thanks for pointing this out, we have zero unresolved arguments discovered after fixing the matching bug, and we've further audited among the multiple run experiments. Zero unresolved arguments was discovered and few harness error was detected and specified, here're the details below.

Here, an unresolved argument is an act call whose argument does not name exactly one entity. For example, a refusal listing every option or chain-of-thought text placed in the argument slot. The grader does not extract an entity name from such text and scores the task as an abstention. Before grading, the runner provides every backbone with the same JSON fallback procedure for an unusable tool call.

The coverage audit below is for Qwen3.8-Flash, Stage 2, across three runs:

ConditionDirect actNo act (read loop / text)HTTP errorNetwork errorEmpty replyUnresolved argumentActed via JSON fallbackOff-option act
Full context360300000n/r3
compact-8k3564221152020
RTM35968002000
mem0358114325010
MemOS359610000000
Memp359312000010
Dynamic Cheatsheet358123100010

Direct acts exclude the separately reported JSON-fallback and off-option acts, n/r means not reported. The unresolved-argument count is zero in every condition across all evaluated backbones, so no reported score depends on resolving an ambiguous argument. Under the evaluation protocol, HTTP errors, network errors, and empty replies remain in the grading denominator and count as abstentions.

Matcher sensitivity. We re-graded every trace we hold under 60 matcher variants (length cap 20–200). 99.7% of arguments are the option's exact name and never reach either threshold, the rest are short phrases naming exactly one option, so violation rates are unchanged and all rankings are identical across variants.

Clarification: Data provenance

The data used in this study originated from AI-generated meeting minutes produced by a video conference product during the provision of video conferencing services. Participants were informed of, and explicitly consented to, recording and transcription before these activities began. Research use and controlled sharing with the research institution were subject to a separate, voluntary opt-in process, acceptance of general terms of service was not treated as research authorization. Declining research participation or discontinuing future data contributions did not affect access to regular meeting services. Only meetings for which all participants had explicitly consented to the relevant uses were included. The consent materials described the research purposes, data scope, recipients, retention periods, foreseeable privacy risks, and withdrawal procedures, and specified whether model training and commercial use were permitted.

Before external disclosure, real names, locations, and dates were replaced, and other direct and indirect identifying information was removed or generalized using human annotation and algorithmic random allocations, followed by audit and risk assessment. Research institutions accessed the data only for the purposes and under the security conditions specified in a data use agreement. Original recordings and identity-linkage keys were not provided with the dataset. Public release required additional explicit consent, including disclosure that de-identification does not eliminate all re-identification risk and that copies held by third parties may not be fully recoverable after release. The dataset documentation identified the records as AI-generated summaries rather than verbatim transcripts.

Large language models (LLMs) were used to process and synthesize the data described above into entries containing deterministic agentic tasks drawn from the relevant finite task sets. This process also involved light to moderate rewriting and editing.

An authorized review body at the relevant institution determined that this study did not constitute human subjects research.

Therefore the data is not 100% synthetic, it is exactly like what the reviewer has outlined that "de-identified real-world records adapted into fictional corpora, with a schema that forbids real entities (enforced by review)." We appreciate the clarity.

The data collection, the research opt-in, and the sharing of de-identified data with the research institution were reviewed and approved by an institutional review board engaged by the commercial source platform, before the study began. The approval also covers public release of the benchmark. We withhold the name of the board and the protocol number during review to preserve anonymity. We will include the details in the camera-ready version.

Reviewer_NtxSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RevokeBench, a benchmark for whether a long-running LLM agent acts on the rules that are in force at the moment it acts, in a setting where rules get approved, prohibited, conditioned, overridden by higher-ranked speakers, and reinstated over hundreds of sessions.

The core contribution is a two-layer representation. A persistent layer holds prioritized rules that are modified only by explicit updates (ADD, SUPERSEDE, CONDITION, CONFLICT, SUPPORT, RETRACT); a derivation layer is recomputed at every session by alternating Dung grounded semantics over a priority defeat graph with forward chaining. Defeat removes a conclusion but never deletes a rule, so a rule can later become effective again, which is exactly what a flat "current state" summary cannot represent. The same solver both labels tasks and grades actions, so no LLM judge is used. An action counts as a violation if adding it to the closure yields a contradiction, and as a completion only if it selects a licensed option.

The release contains 100 synthetic episodes over six settings (clinical ward, care circle, design system, software release, game narrative, meeting minutes), each 344k–935k estimated tokens with about 85% combinatorial filler. It has 1,202 graded single-choice act-tool tasks drawn from a regenerable pool of 21,996: 848 are stale-memory traps, 971 have a recall span beyond 200 sessions, and 757 include a rank-0 requester nudging toward a violating option. Histories are built from 16 rule-update patterns overlaid with a speaker hierarchy, a certification regime, near-miss noise and referential lift lines. Seven acceptance checks verify each episode, and the authors audited all 54,282 session states and all 1,202 task-augmented states for convergence and consistency.

Evaluation has two stages. Stage 1 runs five flash-tier and three frontier backbones under a shared compact-8k summarization baseline. Stage 2 takes one backbone per tier (Qwen3.8-Flash, Claude Opus 5.5) and adds four memory frameworks (mem0, MemOS, Memp, Dynamic Cheatsheet) plus no-memory, full-context and the authors' Rule-Tracking Memory (RTM). Flash-tier violation rates run 0.31–0.83, and Claude Opus 5.5 scores below DeepSeek-V4-Flash. All four frameworks score below compact-8k and below zero on both backbones. RTM (LLM-extracted rules plus the benchmark's solver) gains about 0.14–0.15 exam score over compact-8k, roughly a fifth of the remaining gap to full context.

Critical strengths
  1. The benchmark fills a real gap, and does so with a rigorous, judge-free design that is hard to game.** Existing memory benchmarks mostly ask whether an agent can recall a rule; this one grades the action taken under the rules in force, which matches the actual deployment risk of a long-running assistant. The two-layer formalism is the right abstraction: because defeat does not delete rules, the benchmark can represent reinstatement, derived prohibitions, alias inheritance, moving thresholds and authority hierarchies, all of which a summarization-based memory cannot express. The engineering checks are unusually thorough — convergence on all 54,282 session states, an over-deletion guard against lost alternative derivations, a recoverability check that every victory in a defeat edge is readable from the history, doubly enforced lexical invariants so filler cannot be mistaken for a rule, and ~20% seed rejection before release. Deterministic grading, a seeded generator and a regenerable task pool also make the benchmark robust to contamination and saturation. This is the primary reason I lean toward acceptance.
  2. The empirical headline is practically valuable and mechanistically diagnosed rather than merely reported.** That all four off-the-shelf memory frameworks fall below a plain running-summary baseline, and on Qwen3.8 barely above no memory at all, is a result many practitioners would not expect, and the paper does not stop at the number: it identifies a concrete failure chain in execution logs (a revoked permission is retrieved later and used as current authorization). Because RTM's state can be compared with ground truth option by option, the authors can also localize the residual gap to extraction accuracy (87% of recorded items match a rule-writing event, 88% of prohibitions, yet only 73% verdict agreement) rather than to update coverage. That decomposition is genuinely useful guidance for anyone building agent memory.
Critical weaknesses
  1. The negative result on memory frameworks is not established as a property of the frameworks, because the baselines are under-provisioned by the paper's own provenance tables. mem0 performs zero updates and zero deletes and always returns 12 hits because its threshold never binds; Dynamic Cheatsheet is truncated at its 12,000-character cap on 91.1% of curator calls and the authors did not run a larger cap; Memp retrieves top-1 at $0.056 per episode against $2.714 for compact-8k (Table 9), i.e. about 2% of the comparator's spend; and every memory-system call disables reasoning, with the memory-condition runner described as uncommitted. Comparing systems whose compute differs by up to 50x, with memory revision effectively turned off and no tuning attempted, cannot support the general claim that "external memory does not improve on compact summaries". This matters because it is the paper's headline empirical claim and the motivation for RTM.
  2. Statistical reliability and the interpretation of RTM's gain. (a) Every condition is a single run, and the authors report that 40–70% of decisions differed in the few repeat pairs that exist. Their own 95% intervals — compact-8k [−0.202, +0.247] versus RTM [+0.008, +0.380] — overlap, and they state that violation-rate differences below about 0.1 are unresolved; so both the RTM gain (0.152 / 0.140) and the tier claim (Claude Opus 5.5 +0.107 vs DeepSeek-V4-Flash +0.150) are not statistically distinguished. (b) On Qwen3.8 the headline gain is mostly a coverage effect rather than better judgement: RTM cuts abstention from 29.8% to 10.0% and adds 15.6 points of completion while adding 4.1 points of violations. The composition of "act more often" and "judge better" differs between the two backbones (on Claude, violations fall 3.7 points with abstention unchanged), so a single exam-score delta hides two quite different behavioural changes. (c) RTM runs the solver that produces the labels and reads structured blind-item fields (option sets, task parameters, entity names), so part of its advantage shows that knowing the benchmark's ontology helps, not that the design generalizes. Together these weaken the strength of the positive result as well as the negative one.
Questions for authors
  1. Baseline provisioning. Can you re-run mem0 with its normal update/delete pipeline enabled, Dynamic Cheatsheet with a cap large enough that curator outputs are not truncated, and Memp with top-k > 1, all at a token or dollar budget comparable to compact-8k and with reasoning enabled for memory calls? Do the frameworks still fall below compact-8k on both backbones? If the conclusion survives, it becomes much stronger.
  2. Variance. Can you report at least three runs, or paired per-task tests with bootstrap intervals, for the Stage 2 comparisons and for the key Stage 1 rows? Specifically, is RTM vs compact-8k, and Claude Opus 5.5 vs DeepSeek-V4-Flash, significant under a paired test on the same 100 episodes?
  3. RTM attribution. How does RTM perform if it may not use the benchmark's solver and structured blind-item fields, e.g. when it maintains an explicit rule table in natural language and decides itself? This would separate "explicit rule state" from "the grader's solver".
  4. Stage 1. Can you report full-context results for all eight backbones, so that rule-application ability can be separated from note-writing ability under compact-8k? Without this, the tier comparison confounds the two.
  5. Validity. Have humans attempted a sample of tasks given the rendered history or a fair condensed view, to confirm that the intended verdict is recoverable and unambiguous — especially for the hierarchy, derived and certification cases, and for referential lifts such as "our ruling from session s is withdrawn"?
  6. Data provenance. What exactly were the "de-identified real-world records collected during the routine operation of a commercial platform" (Section 3, footnote 2), and what consent or terms-of-use basis permitted adapting them and releasing the corpora?
3 comments
AuthorsOpenReview ↗

Response to Reviewer NtxS (1/3)

We thank the reviewer for the helpful review, we appreciate the feedback and would love to provide additional information to address your questions.

Q1. Baseline Provisioning in stage 2

We have reconfigured relevant configurations on selected baselines and memory frameworks in order to achieve more fair/closer provisions. For the repeated Stage 2 comparison, we used:

  • mem0: v1.0.11 with automatic UPDATE/DELETE enabled (infer=True). search(limit=100, threshold=None, rerank=False).

  • MemOS: v2.0.36, text_mem backend general_text; top_k=50, max_turns_window lowered from 15 to fit the token allocation, mem_reader.config.llm.config.max_tokens=4096; enable_activation_memory, enable_parametric_memory, enable_preference_memory, enable_mem_scheduler and PRO_MODE left at False.

  • Dynamic Cheatsheet: approach_name=DynamicCheatsheet_Cumulative, max_tokens=2048 (the curator call uses 2 * max_tokens = 4,096). We removed the additional runner-imposed sheet-size truncation.The cumulative variant regenerates the full sheet at each update, the newly generated sheet remains bounded by this output budget.

  • Memp: policy={"build": "direct", "retrieve": "query", "update": "vanilla"}, retrieve_num=50; MAX_TOKENS=4096 and TEMPERATURE=0 in llm_api.py.

The following results are three-run means over the same 1,202 Stage 2 tasks, including full-context references: 3,606 task records per condition. Interval definitions match Q1.

BackboneMemoryViol. ↓Comp. ↑Abst.Exam ↑Exam 95% CI
Qwen3.8-FlashFull context0.055 ± 0.0030.845 ± 0.0060.100 ± 0.006+0.789 ± 0.013[0.757, 0.822]
No memory0.582 ± 0.0060.157 ± 0.0020.261 ± 0.005-0.456 ± 0.005[-0.468, -0.444]
compact-8k0.313 ± 0.0060.393 ± 0.0020.293 ± 0.004+0.020 ± 0.008[0.001, 0.039]
RTM0.344 ± 0.0090.557 ± 0.0070.098 ± 0.002+0.191 ± 0.025[0.129, 0.254]
mem00.453 ± 0.0040.436 ± 0.0060.111 ± 0.003−0.078 ± 0.009[−0.100, −0.056]
MemOS0.457 ± 0.0070.438 ± 0.0040.105 ± 0.008−0.057 ± 0.009[−0.079, −0.035]
Memp0.503 ± 0.0050.376 ± 0.0030.121 ± 0.002−0.174 ± 0.006[−0.189, −0.159]
Dynamic Cheatsheet0.428 ± 0.0100.340 ± 0.0010.232 ± 0.010−0.132 ± 0.008[−0.152, −0.112]
Claude Opus 5.5Full context0.009 ± 0.0010.774 ± 0.0010.216 ± 0.002+0.763 ± 0.001[0.760, 0.766]
No memory0.571 ± 0.0020.168 ± 0.0040.261 ± 0.005-0.432 ± 0.006[-0.446, -0.417]
compact-8k0.353 ± 0.0020.479 ± 0.0020.167 ± 0.002+0.106 ± 0.005[0.094, 0.119]
RTM0.312 ± 0.0040.518 ± 0.0040.170 ± 0.001+0.256 ± 0.018[0.212, 0.300]
mem00.382 ± 0.0040.507 ± 0.0060.111 ± 0.003+0.064 ± 0.009[0.042, 0.086]
MemOS0.331 ± 0.0040.446 ± 0.0020.223 ± 0.004+0.082 ± 0.003[0.075, 0.089]
Memp0.416 ± 0.0020.378 ± 0.0080.206 ± 0.008−0.083 ± 0.006[−0.098, −0.068]
Dynamic Cheatsheet0.394 ± 0.0050.358 ± 0.0020.248 ± 0.006−0.067 ± 0.004[−0.077, −0.057]

We observe that memory framework's mean Exam scores remain below compact-8k and RTM on both backbones. Comparisons on individual metrics are mixed: for example, on Opus, MemOS has a lower violation rate than compact-8k, while mem0 has higher completion. We therefore limit our conclusion to the observed Exam comparison under the tested configurations and report violation and completion separately.

Q2. Stage 1 Variance

We repeated Stage 1 three times on all 1,202 tasks, using compact notes hard-capped at 4k tokens. Values are mean ± standard deviation, Exam intervals are 95% t confidence intervals for the mean across three runs (df = 2), conditional on the fixed evaluation set.

BackboneViol. ↓Comp. ↑Abst.Exam ↑Exam 95% CI
Qwen3.8-Flash0.308 ± 0.0330.385 ± 0.0090.307 ± 0.038+0.038 ± 0.014[0.002, 0.073]
DeepSeek-V4-Flash0.294 ± 0.0400.491 ± 0.0520.216 ± 0.021+0.175 ± 0.119[−0.121, 0.472]
GLM-5.3-Flash0.470 ± 0.0250.390 ± 0.0180.141 ± 0.011−0.140 ± 0.050[−0.264, −0.017]
Qwen3.7-Flash0.546 ± 0.0050.244 ± 0.0120.209 ± 0.007−0.383 ± 0.029[−0.456, −0.310]
Llama-4-Maverick0.842 ± 0.0230.088 ± 0.0140.071 ± 0.014−0.786 ± 0.038[−0.881, −0.690]
Spark 1.30.073 ± 0.0100.903 ± 0.0090.024 ± 0.002+0.829 ± 0.023[0.773, 0.885]
GPT-5.50.119 ± 0.0190.767 ± 0.0340.114 ± 0.018+0.633 ± 0.072[0.455, 0.811]
Claude Opus 5.50.367 ± 0.0270.462 ± 0.0460.171 ± 0.020+0.048 ± 0.091[−0.178, 0.274]

We will narrow the headline claims accordingly.

AuthorsOpenReview ↗

Response to Reviewer NtxS (2/3)

Q3. RTM attribution.

We agree that separation matters. RTM is intended as a reference baseline for RevokeBench, not as a general method. Two existing measurements bound the solver's share. (i) Offline, with zero model calls, we keep RTM's extracted statements fixed and replace the two-layer solver with per-option latest-wins bookkeeping: agreement with the ground-truth status of each offered option falls from 0.723 to 0.582. (ii) Natural-language memories that leave the decision to the model score well below RTM on the same panel (exam: compact-8k +0.002, memOS −0.329, RTM +0.191). Since neither keeps an explicit rule table, the revision adds both conditions we describe on the same panel: (a) RTM without structured blind-item fields (no entity list, no option or parameter fields. Both are parsed from the task text), and (b) a natural-language rule table maintained by the same backbone from the raw sessions (option, current status, who set it, when, under what condition), with no solver, no rule language and no structured fields. The agent reads the table and decides itself. Results: (a) exam +0.142, (b) exam -0.121. The gap between (b) and compact-8k measures explicit rule state whereas the gap between (a) and (b) measures the solver.

Almost all of RTM’s lead comes from formalism. The reviewer asked how much comes from matching the benchmark’s formalism, and the answer is most of it. This fits the offline ablation: replacing the solver with “last ruling wins” drops verdict agreement from 0.723 to 0.582. The drop is mostly on prohibited options (0.59→0.37), i.e. authority, revocations and conditions. Also an explicit rule table on its own doesn’t seem help much, and here it does worse than free-form notes. A likely mechanism: the table compresses each option into one definite status. If extraction misses a revocation or ignores who has authority, that row reads as a confident “approved”, and the agent follows it into a violation. Free-form notes keep more context, so the agent abstains more often. It also suggests that the structured fields matter little. RTM keeps most of its score without them, which answers the worry that it scores by reading the blind item.

Q4. Full-context results across 8 backbones

Thanks for the feedback. We agree with that and is able to provide the result.

BackboneViol.Comp.Abst.ExamFarTrapWeighted
Qwen3.8-Flash0.0530.8500.097+0.8040.0570.0330.047
DeepSeek-V4-Flash0.0640.8410.095+0.7870.0690.0520.059
GLM-5.3-Flash0.0710.8340.095+0.7760.0640.0730.066
Qwen3.7-Flash0.0680.8420.090+0.7790.0750.0610.064
Llama-4-Maverick0.2860.5920.122+0.2740.3170.3010.305
Spark 1.30.0070.8130.180+0.8020.0060.0100.008
GPT-5.50.0080.8070.185+0.7950.0080.0070.009
Claude Opus 5.50.0090.7750.216+0.7620.0090.0110.011

With full-context as a relative upper bound being controlled, we discovered that most tested backbones illustrates strong results in terms of exam scores, except Llama-4-Maverick. Therefore isolating the ability of the backbone from the trade-off between token-efficency and memory performance.

Q5. Human Audit

We conducted a human audit of the full curated set of 1,202 tasks, including hierarchy, derived, certification, and referential-lift cases such as “our ruling from session s is withdrawn.” The audit identified no ambiguous intended verdicts, circular dependencies, incorrect reference answers, or other logical inconsistencies. We will be happy to continue maintain the data quality on the broader customized taskset and feedbacks from the community are more than welcome.

AuthorsOpenReview ↗

Response to Reviewer NtxS (3/3)

Q6. Data Provenance

Thank you for requesting this clarification. The source material consisted of AI-generated meeting summaries from a videoconferencing product, rather than verbatim transcripts. Participants were informed of, and explicitly consented to, recording and transcription before those activities began. Research use and controlled sharing with the research institution required a separate, voluntary opt-in from all meeting participants. Acceptance of general terms of service was not treated as research authorization. Declining research participation or discontinuing future contributions did not affect access to ordinary meeting services. The consent materials described the research purposes, data scope, recipients, retention periods, privacy risks, and withdrawal procedures, and specified whether model training and commercial use were permitted.

Before external disclosure, real names, locations, and dates were replaced, and other direct and indirect identifying information was removed or generalized through human annotation and algorithmic replacement, followed by audit and risk assessment. Research institutions accessed the material under a data-use agreement specifying permitted purposes and security conditions. Original recordings and identity-linkage keys were not included in the dataset. Public release required additional explicit consent, with disclosure of residual re-identification risks and the limits of recovering third-party copies after release. LLMs processed and synthesized the de-identified summaries into entries containing deterministic agentic tasks, including light-to-moderate rewriting and editing. The released benchmark uses the MIT licence. We have aimed to remove original wording and identifiable facts as extensively as possible during synthesis and rewriting, we do not equate this effort with a guarantee of zero residual source influence or re-identification risk. We will document the source-to-release transformation explicitly, distinguishing the synthetic task construction from the provenance of its source material. The evaluation prompt's description of a synthetic setting is not intended to replace that provenance disclosure.

An authorized institutional review body determined that the study did not constitute human-subjects research. We followed the terms of services and other policies from both the product and the users, if applicable, while conducting data collection procedure. The data collection, the research opt-in, and the sharing of de-identified data with the research institution were reviewed and approved by an institutional review board engaged by the commercial source platform, before the study began. The approval also covers public release of the benchmark. We withhold the name of the board and the protocol number during review to preserve anonymity. We will include the details in the camera-ready version.

Reviewer_mCSVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces RevokeBench, a benchmark for whether an agent acts on the rules currently in force when those rules keep changing over a long history. Rules and priorities are stored in a persistent layer. Their consequences are recomputed every session in a derivation layer, using priority-based defeat, Dung's grounded semantics and forward chaining. The same solver labels tasks and grades the agent's act call, so no LLM judge is involved.

The release is 100 synthetic histories across 6 settings, 344k–935k estimated tokens each, about 85% of which is filler. These histories hold 1,202 graded tasks, selected from a pool of 21,996. Each task asks the agent to pick one of about 6 options through an act tool.

Stage 1 runs 8 backbones under one summarisation baseline, compact-8k. Stage 2 runs mem0, MemOS, Memp and Dynamic Cheatsheet on Qwen3.8-Flash and Claude Opus 5.5, with no-memory and full-context controls. The main results:

  • All four frameworks score below compact-8k.
  • Full context nearly solves the benchmark (5.4% and 0.9% violations).
  • The proposed RTM baseline improves the exam score by about 0.14–0.15 over compact-8k.
Critical strengths
  1. The grading is deterministic and audited. Labels come from a formal solver, not an LLM. The seven acceptance checks in App. C.4 (over-deletion guard, self-containedness, recoverability from the declared hierarchy, and others) are well designed. The authors also re-derived all 54,282 session states and 1,202 task states. Releasing a regenerable task pool with re-selection scripts is more useful than another static QA set. This is the main reason my score is not lower.
  2. The problem is real, and it is graded on actions. Tasks involve authority hierarchies, aliases, group bans, certification windows and thresholds, not just recall of the latest fact. Option sets stay fixed while the licensed subset changes, which is a clean way to catch stale memory.
Critical weaknesses

1. The headline claims are within the paper's own noise.

  • Every result is a single run. App. E says that in the repeat pairs that exist, "40–70% of decisions differed." It also says: "Differences below about 0.1 in violation rate between adjacent rows are not resolved; compact-8k and RTM intervals overlap."
  • Table 10 gives [−0.202, +0.247] for compact-8k and [+0.008, +0.380] for RTM.
  • The abstract and §5 still present three results as findings:
    • Frontier models "do not uniformly outperform" flash-tier ones. This rests on Opus +0.107 vs DeepSeek-V4-Flash +0.150.
    • The RTM gain of about 0.15.
    • A compliance/completion trade-off of about 4 points.
  • None of these is established by the evidence given. Multiple seeds and CIs are needed on every row.

2. The framework comparison is not a fair comparison as run.

  • The frameworks are configured weakly (Table 8):
    • mem0 logged "0 updates/deletes." Updating is the behaviour this benchmark is meant to test.
    • Memp uses top-k 1.
    • Dynamic Cheatsheet's sheet is "hard-cut at 12,000 chars (91.1% of curator calls)."
  • compact-8k gets more memory and more compute:
    • Its notes cap is a "prompt word limit only; 52% of folds over cap," and "notes not clipped at recall."
    • It costs $2.71 per episode, about 3× mem0, MemOS or DC.
    • By my estimate from Table 9, it produces about 6.5k output tokens per fold. That is close to its 6,500 max_tokens, so folds may be routinely truncated.
  • The protocols differ. Memory conditions replay the agent's own past actions; full context does not.
  • Two frameworks don't fit the task. Memp and DC are procedural/strategy memories, used here as rule-state stores. The result looks more like a mismatch than evidence that "self-evolving memory fails."
  • There is no random baseline:
    • 4,360 of 6,940 offered (task, option) pairs are violating, so a uniform-random pick violates about 63% of the time.
    • 757 of 1,202 tasks include a nudge toward a violating option.
    • On Qwen3.8-Flash, all four frameworks (0.57–0.71) and no-memory (0.586) are at or above random.
    • Conditional on acting, no-memory violates about 79% of the time, which suggests the agent follows the nudge.
    • At least on Qwen, this looks like a broken integration rather than a graded comparison.
  • The claim that all four frameworks underperform a simple heuristic does not hold as stated.

3. (Secondary.) It is unclear what is actually hard here.

  • Figure 1 says: "Even with the full context available, the agent here deploys a service that is now prohibited." Yet full context gets Opus to 0.9% violations.
  • The Far and Trap columns cover 81% and 71% of tasks. They therefore track the overall violation rate in every row, and give no evidence that recall span drives difficulty.
  • As built, the benchmark mainly measures lossy compression of a templated stream with about 0.5 rule events per session.
Questions for authors
  1. (Could change my score.) Please report multi-seed results with CIs for all Stage 1 and Stage 2 rows. Which headline claims still hold?

  2. (Could change my score.) Why did mem0 perform 0 updates/deletes? Please rerun:

    • mem0 with update/delete enabled
    • DC with the sheet uncapped
    • Memp with a larger top-k
    • compact-8k with notes hard-capped at 4k

    Please also add a compute-matched comparison.

  3. (Could change my score.) Please add a uniform-random baseline and a follow-the-nudge baseline, and break down violations by nudged vs. non-nudged tasks.

  4. (Could change my score.) The intro says "knowing a rule is not the same as applying it." Please run a QA version of the same tasks ("which options are approved right now?") to show that gap. Without it, the "tool actions" column in Table 3 is multiple choice through a tool call.

  5. (Clarification.) Several Viol./Comp./Abst. values look rounded to two decimals: GPT-5.5, Llama-4-Maverick, DeepSeek-V4-Flash, and every Qwen3.8-Flash row in Table 2.

    • The Qwen3.8 rows match Table 10 after rounding. RTM is 0.355 / +0.186 in Table 10 but 0.350 / +0.180 in Table 2.
    • The in-text deltas ("+4.1 points") are computed from the rounded values.
    • Were some models evaluated on a subset of tasks?
    • §5.1 says "We also report per-condition task coverage," and App. D says "the count of unresolved arguments is reported with every table." I could not find either. Please point me to them or add them.
  6. (Clarification.) Please confirm:

    • the exact model IDs (what is "Spark 1.3"?) and the prompts used
    • how full-context inputs of up to 935k tokens fit each backbone's context window, and whether any were truncated
    • which backbone Table 9 covers, and why it has qwen3.7 rows and no Opus costs
    • why the memory runner is listed as "uncommitted" and RTM as "untracked"
    • which mem0 variant (fw25 or fw25-fix) Table 2 reports, and what ‡ means
  7. (Clarification.) RTM uses the grader's solver and rule language, plus the blind item's structured fields (App. E). How much of its gain comes from matching the benchmark's own formalism?

  8. (Clarification.) Footnote 2 says the source records were "collected during the routine operation of a commercial platform," and two settings are clinical wards and an older adult's care circle. Please state what consent, licence and de-identification process applied, and whether any of that material survives in the released corpora.

9 comments
AuthorsOpenReview ↗

Response to Reviewer mCSV (1/6):

Thank you for the detailed review and for recognizing the deterministic grading, auditability, and value of evaluating changing rules through agent choices. We agree that the original single-run presentation and several configuration and reporting details required stronger evidence. We have now repeated the main evaluations three times, rerun the requested memory configurations, added random and follow-the-nudge diagnostics and a paired QA/action evaluation. Below we separate the conclusions supported by these results from the claims that need qualification.

Q1. Repeated evaluations and which claims remain supported

We repeated Stage 1 three times on all 1,202 tasks, using compact notes hard-capped at 4k tokens. This revised configuration replaces the earlier prompt-only cap. Values are mean ± standard deviation, Exam CI are 95% t confidence intervals for the mean across three runs (df = 2), conditional on the fixed evaluation set. They do not quantify uncertainty from sampling new histories.

BackboneViol. ↓Comp. ↑Abst.Exam ↑Exam 95% CI
Qwen3.8-Flash0.308 ± 0.0330.385 ± 0.0090.307 ± 0.038+0.038 ± 0.014[0.002, 0.073]
DeepSeek-V4-Flash0.294 ± 0.0400.491 ± 0.0520.216 ± 0.021+0.175 ± 0.119[−0.121, 0.472]
GLM-5.3-Flash0.470 ± 0.0250.390 ± 0.0180.141 ± 0.011−0.140 ± 0.050[−0.264, −0.017]
Qwen3.7-Flash0.546 ± 0.0050.244 ± 0.0120.209 ± 0.007−0.383 ± 0.029[−0.456, −0.310]
Llama-4-Maverick0.842 ± 0.0230.088 ± 0.0140.071 ± 0.014−0.786 ± 0.038[−0.881, −0.690]
Spark 1.30.073 ± 0.0100.903 ± 0.0090.024 ± 0.002+0.829 ± 0.023[0.773, 0.885]
GPT-5.50.119 ± 0.0190.767 ± 0.0340.114 ± 0.018+0.633 ± 0.072[0.455, 0.811]
Claude Opus 5.50.367 ± 0.0270.462 ± 0.0460.171 ± 0.020+0.048 ± 0.091[−0.178, 0.274]

We will narrow the headline claims accordingly. The repeated results show substantial variation across the tested backbones. Spark and GPT have high completion and comparatively low violation rates, while three backbones have negative Exam intervals.

AuthorsOpenReview ↗

Response to Reviewer mCSV (2/6)

Q2. Configuration checks, mem0 mutations, and comparison scope

We did not disable an automatic UPDATE/DELETE stage. The newer mem0 add() path evaluated in the original submission is ADD-only, as documented in the mem0 v2.0.0 release note. We additionally tested its precedent mem0 v1.0.11, which performs automatic UPDATE/DELETE.

For the repeated Stage 2 comparison, we used:

  • mem0: v1.0.11 with automatic UPDATE/DELETE enabled (infer=True). search(limit=100, threshold=None, rerank=False).

  • MemOS: v2.0.36, text_mem backend general_text; top_k=50, max_turns_window lowered from 15 to fit the token allocation, mem_reader.config.llm.config.max_tokens=4096; enable_activation_memory, enable_parametric_memory, enable_preference_memory, enable_mem_scheduler and PRO_MODE left at False.

  • Dynamic Cheatsheet: approach_name=DynamicCheatsheet_Cumulative, max_tokens=2048 (the curator call uses 2 * max_tokens = 4,096). We removed the additional runner-imposed sheet-size truncation.The cumulative variant regenerates the full sheet at each update, the newly generated sheet remains bounded by this output budget.

  • Memp: policy={"build": "direct", "retrieve": "query", "update": "vanilla"}, retrieve_num=50; MAX_TOKENS=4096 and TEMPERATURE=0 in llm_api.py.

The following results are three-run means over the same 1,202 Stage 2 tasks, including full-context references: 3,606 task records per condition. Interval definitions match Q1.

BackboneMemoryViol. ↓Comp. ↑Abst.Exam ↑Exam 95% CI
Qwen3.8-FlashFull context0.055 ± 0.0030.845 ± 0.0060.100 ± 0.006+0.789 ± 0.013[0.757, 0.822]
No memory0.582 ± 0.0060.157 ± 0.0020.261 ± 0.005-0.456 ± 0.005[-0.468, -0.444]
compact-8k0.313 ± 0.0060.393 ± 0.0020.293 ± 0.004+0.020 ± 0.008[0.001, 0.039]
RTM0.344 ± 0.0090.557 ± 0.0070.098 ± 0.002+0.191 ± 0.025[0.129, 0.254]
mem00.453 ± 0.0040.436 ± 0.0060.111 ± 0.003−0.078 ± 0.009[−0.100, −0.056]
MemOS0.457 ± 0.0070.438 ± 0.0040.105 ± 0.008−0.057 ± 0.009[−0.079, −0.035]
Memp0.503 ± 0.0050.376 ± 0.0030.121 ± 0.002−0.174 ± 0.006[−0.189, −0.159]
Dynamic Cheatsheet0.428 ± 0.0100.340 ± 0.0010.232 ± 0.010−0.132 ± 0.008[−0.152, −0.112]
Claude Opus 5.5Full context0.009 ± 0.0010.774 ± 0.0010.216 ± 0.002+0.763 ± 0.001[0.760, 0.766]
No memory0.571 ± 0.0020.168 ± 0.0040.261 ± 0.005-0.432 ± 0.006[-0.446, -0.417]
compact-8k0.353 ± 0.0020.479 ± 0.0020.167 ± 0.002+0.106 ± 0.005[0.094, 0.119]
RTM0.312 ± 0.0040.518 ± 0.0040.170 ± 0.001+0.256 ± 0.018[0.212, 0.300]
mem00.382 ± 0.0040.507 ± 0.0060.111 ± 0.003+0.064 ± 0.009[0.042, 0.086]
MemOS0.331 ± 0.0040.446 ± 0.0020.223 ± 0.004+0.082 ± 0.003[0.075, 0.089]
Memp0.416 ± 0.0020.378 ± 0.0080.206 ± 0.008−0.083 ± 0.006[−0.098, −0.068]
Dynamic Cheatsheet0.394 ± 0.0050.358 ± 0.0020.248 ± 0.006−0.067 ± 0.004[−0.077, −0.057]

The revised configurations substantially improve all four memory frameworks. All four now exceed both the uniform-random reference (expected Exam: −0.423) and no memory in mean Exam on both backbones, addressing the earlier below-random observations.

Nevertheless, their mean Exam scores remain below compact-8k and RTM on both backbones. Comparisons on individual metrics are mixed: for example, on Opus, MemOS has a lower violation rate than compact-8k, while mem0 has higher completion. We therefore limit our conclusion to the observed Exam comparison under the tested configurations and report violation and completion separately.

AuthorsOpenReview ↗

Response to Reviewer mCSV (3/6)

Q3. Random choice baseline, and unauthorized nudges

Thanks for pointing it out. We do have the random baseline results and we omitted that in the submission. Here we provide more fine-grained random baseline results along with multiple runs. On all 100 episodes, uniform random choice has exact expected violation 0.627, completion 0.237, abstention 0.136, Exam −0.423, and weighted violation 0.649. Across 10,000 seeded runs, the central 95% seed ranges are [0.601, 0.652] for violation and [−0.468, −0.375] for Exam. These describe random-selection variability on the fixed set, not confidence intervals for the exact expectation.

PolicyViolation ↓Completion ↑AbstentionExam ↑Trap violation ↓
oracle0.001.000.00+1.000.00
stale0.790.210.00−0.630.96
recency0.840.160.00−0.690.84
random0.630.230.14−0.440.62
refuse0.000.001.000.000.00
(Original omitted baseline, round to 2 decimals)

Additional scripted policies are provided help interpret the metrics and its distributions: stale selects the previously compliant option, recency selects the most recently mentioned option, and refuse always abstains. Oracle and stale use ground-truth labels. Recency has violation 0.84, refuse has zero violation but also zero completion and Exam 0. These are diagnostic controls, not label-free competing memory systems.

Additionally, the follow-the-nudge policy selects the suggested option when present and otherwise samples uniformly. Its full-set violation/Exam are 0.865/−0.790. On a separate nine-episode diagnostic panel, random choice yields 0.587/−0.400, and following the nudge yields 0.840/−0.770.

Uniform-choice violation base rates are similar for nudged/non-nudged tasks: 0.622/0.635 overall and 0.590/0.583 on the diagnostic panel. This establishes comparable option-set base rates, not equal task difficulty. Every nudge comes from a speaker explicitly declared non-authoritative in the initial notice, so the rules specify that it cannot change policy. In a separate single-run diagnostic on the nine-episode panel, Qwen3.8-Flash gives:

ConditionSuggested-option selectionViol.: nudged / non-nudged
Full context0.0140.058 / 0.047
compact-8k0.1450.327 / 0.282
RTM0.2240.388 / 0.302
mem00.3380.632 / 0.605
MemOS0.3190.638 / 0.535
Memp0.4490.797 / 0.581
Dynamic Cheatsheet0.2650.603 / 0.512

The average uniform-random probability of selecting the suggested option is 0.173 on the nudged tasks. Suggested-option selection is numerically more frequent under the four framework conditions (0.265–0.449), while the full-context reference selects it rarely. These descriptive results identify an unauthorized-suggestion failure mode beyond the overall violation rate. They do not establish that memory alone causes the effect, or that compact-8k and RTM eliminate nudge sensitivity. We report the uniform-random and follow-the-nudge references explicitly rather than assuming every integration outperforms random.

Q4. Paired QA and action evaluation.

Thank you for suggesting a same-task QA comparison. The QA view preserves the task and history and asks, “Which of these options are approved right now?”, answered as a JSON list of option names. The system's response-format instruction is also changed from requiring an act call to requiring this JSON answer. In paired mode, the QA and action queries are issued separately from the same frozen memory state, and neither receives the other's answer. The following diagnostic uses Qwen3.8-Flash with compact-8k notes hard-capped at 4k tokens, on 1,202 tasks. We have integrate it in the codebase and will reflect the revision and findings in the manuscript.

QA responses are graded by the same solver as actions, without an LLM judge. Listing a prohibited option is a violation, a nonempty list containing only approved options is a completion, other responses, including an empty list, are abstentions. QA Exam uses the same task-difficulty weights. Completion here permits a subset of the approved options and is therefore distinct from exact-set accuracy.

QA metricResult
Violation0.286
Completion0.285
Abstention0.429
Exam−0.001

In this paired diagnostic, action completion is 0.265, compared with QA completion of 0.285. The paired completion outcomes disagree on 172/1,202 tasks (14.31%):

Paired outcomeTasksShare of all 1,202 tasks
QA completion, action violation or abstention988.15%
QA violation or abstention, action completion746.16%
Total discordant completion outcomes17214.31%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.