- Rating
- 4
- Confidence
- 1(−2)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This project introduces RevokeBench, a benchmark for evaluating whether long-running LLM agents can correctly track evolving constraints and act according to the currently valid state. The benchmark contains long multi-session histories with rule additions, revocations, conflicts, authority changes, and tool-use tasks. The authors find that several existing memory frameworks underperform a simple compact-summary baseline, and propose Rule-Tracking Memory (RTM) as a structured state-tracking baseline.
I find the problem important and the observation that stale but historically valid memories can lead to invalid actions interesting. However, I currently lean toward Weak Reject because the novelty relative to recent state-updating memory benchmarks is not sufficiently established, the necessity of the proposed formalization is unclear, and some experimental conclusions require stronger evidence.
Critical strengths
-
The distinction between remembering historical information and maintaining what is currently valid for action is important for long-running agents.
-
Evaluating actual tool actions rather than only QA responses is valuable.
-
The full-context experiment is a useful control, suggesting that many failures come from memory construction/retrieval rather than purely from model reasoning.
-
The finding that revoked permissions can remain actionable through memory retrieval is interesting and practically relevant.
Critical weaknesses
-
Insufficient positioning against related state-updating benchmarks. Recent benchmarks such as AMA-Bench already explicitly evaluate state updating over long agent trajectories and also use graph-based memory representations. RevokeBench appears to differ through its emphasis on evolving authorization, authority hierarchy, rule interaction, and action-level compliance, but this distinction should be much more clearly articulated. The current novelty framing around long-horizon evolving-state tracking feels somewhat overstated.
-
The necessity of the two-layer formalism is unclear. The paper argues that defeated rules should remain stored because they may become active again later. I agree with this requirement, but it does not seem to require the proposed grounded-argumentation and fixed-point machinery. A simpler persistent rule/event graph could retain all historical rules and dependencies, maintain active/inactive labels, and expose only the latest materialized state to the agent. When a rule changes, affected nodes could simply be recomputed. Since the benchmark itself is generated from a finite set of predefined transition patterns, the paper should demonstrate which cases genuinely require the proposed formalism rather than a simpler state-transition/dependency graph.
-
Experimental robustness and baseline fairness need improvement. The main experiments are single runs, while the appendix reports substantial variation across the limited repeated runs. In addition, some memory baselines appear constrained by their configurations; for example, Dynamic Cheatsheet frequently reaches its memory cap. RTM is also relatively benchmark-aware, as it uses structured benchmark fields and the same rule solver used by the grader. These factors make the broad conclusion that existing memory frameworks systematically fail less convincing.
-
Presentation. Several figures are visually dense and overly infographic-like, with small text and unnecessary decorative elements. Simpler scientific diagrams would make the benchmark and method much easier to understand.
Questions for authors
-
How does RevokeBench differ fundamentally from AMA-Bench and other recent benchmarks that already study state updating over long agent trajectories?
-
Can the authors provide a concrete RevokeBench example that cannot be correctly represented using a persistent event/rule graph with dependency edges, active/inactive labels, and a materialized current-state view?
-
What fraction of tasks actually require the proposed multi-rule conflict/fixed-point reasoning, rather than simple state transitions?
-
Given that RTM uses the benchmark's structured representation and solver, how should its gains be interpreted as evidence for a general memory method rather than a benchmark-specific diagnostic baseline?
3 comments
Response to Reviewer cFnm (2/3)
Q4. Intepreting RTM
Our intent is that RTM is not being proposed for a general memory method on such class of problems yet, and we will present it as a privileged-structure reference baseline for baselining future works. RTM borrows the grader's rule language and solver, a rank-over-recency priority convention, the entity list as an extraction hint, and each task's options. It never sees events or labels, and the backbone extracts every ruling from the raw transcript. Its role is diagnostic, and three findings carry beyond RTM. (1) Having the grader's logic does not solve the benchmark: RTM scores +0.191 against +0.789 with full context, and offline its verdicts match the ground truth on 0.723 of task–option pairs, against 1.000 by construction on the true rules. The remaining error is extraction from long dialogue, the general memory problem. (2) Update semantics matter independently of RTM: on the same extracted statements, with perfect extraction a delete-on-conflict store(the policy Mem0 applies to contradicted memories) mislabels some option on 24% of tasks. It suggests current methods might have methodological flaws against RevokeBench class problems. (3) Structure is needed: given the true rules, conditions plus one-pass priority are fully right on 98.1% of tasks. We read this, as a hypothesis, that it might suggest that evolving rule state suits LLM extraction plus explicit resolution, as in solver-backed reasoning (PAL, Logic-LM) and in policy engines that resolve conflicts by declared precedence (XACML, Cedar). So RTM might carries some methodological values on evolving state problems in agent memory that RevokeBench introduces if the longer-to-full-context could be well handled to provision the raw information that a reliable structural representation like RTM requires.
On AI-use statement
We do have an AI-use statement which was placed right after references per the guideline. We can move it to be right after conclusion and/or page limits so it reads a bit better.
On presentation
Thanks you for your feedback. We will be sure to refine and polish the figures read more scientifically informative.
On experimental robustness and baseline fairness need improvement.
Thank you for the feedback, we agree with that. Therefore we have run multiple runs on stage1 and updated configurations on stage2 for a more aligned provisioning. The results are the following:
We repeated Stage 1 three times on all 1,202 tasks, using compact notes hard-capped at 4k tokens. This revised configuration replaces the earlier prompt-only cap. Values are mean ± standard deviation, Exam intervals are 95% t confidence intervals for the mean across three runs (df = 2), conditional on the fixed evaluation set. They do not quantify uncertainty from sampling new histories.
| Backbone | Viol. ↓ | Comp. ↑ | Abst. | Exam ↑ | Exam 95% CI |
|---|---|---|---|---|---|
| Qwen3.8-Flash | 0.308 ± 0.033 | 0.385 ± 0.009 | 0.307 ± 0.038 | +0.038 ± 0.014 | [0.002, 0.073] |
| DeepSeek-V4-Flash | 0.294 ± 0.040 | 0.491 ± 0.052 | 0.216 ± 0.021 | +0.175 ± 0.119 | [−0.121, 0.472] |
| GLM-5.3-Flash | 0.470 ± 0.025 | 0.390 ± 0.018 | 0.141 ± 0.011 | −0.140 ± 0.050 | [−0.264, −0.017] |
| Qwen3.7-Flash | 0.546 ± 0.005 | 0.244 ± 0.012 | 0.209 ± 0.007 | −0.383 ± 0.029 | [−0.456, −0.310] |
| Llama-4-Maverick | 0.842 ± 0.023 | 0.088 ± 0.014 | 0.071 ± 0.014 | −0.786 ± 0.038 | [−0.881, −0.690] |
| Spark 1.3 | 0.073 ± 0.010 | 0.903 ± 0.009 | 0.024 ± 0.002 | +0.829 ± 0.023 | [0.773, 0.885] |
| GPT-5.5 | 0.119 ± 0.019 | 0.767 ± 0.034 | 0.114 ± 0.018 | +0.633 ± 0.072 | [0.455, 0.811] |
| Claude Opus 5.5 | 0.367 ± 0.027 | 0.462 ± 0.046 | 0.171 ± 0.020 | +0.048 ± 0.091 | [−0.178, 0.274] |
Response to Reviewer cFnm (3/3)
(continued)
For the repeated Stage 2 comparison, we used:
-
mem0: v1.0.11 with automatic UPDATE/DELETE enabled (
infer=True).search(limit=100, threshold=None, rerank=False). -
MemOS: v2.0.36,
text_membackendgeneral_text;top_k=50,max_turns_windowlowered from 15 to fit the token allocation,mem_reader.config.llm.config.max_tokens=4096;enable_activation_memory,enable_parametric_memory,enable_preference_memory,enable_mem_schedulerandPRO_MODEleft atFalse. -
Dynamic Cheatsheet:
approach_name=DynamicCheatsheet_Cumulative,max_tokens=2048(the curator call uses2 * max_tokens= 4,096). We removed the additional runner-imposed sheet-size truncation.The cumulative variant regenerates the full sheet at each update, the newly generated sheet remains bounded by this output budget. -
Memp:
policy={"build": "direct", "retrieve": "query", "update": "vanilla"},retrieve_num=50;MAX_TOKENS=4096andTEMPERATURE=0inllm_api.py.
The following results are three-run means over the same 1,202 Stage 2 tasks, including full-context references: 3,606 task records per condition. Interval definitions match Q1.
| Backbone | Memory | Viol. ↓ | Comp. ↑ | Abst. | Exam ↑ | Exam 95% CI |
|---|---|---|---|---|---|---|
| Qwen3.8-Flash | Full context | 0.055 ± 0.003 | 0.845 ± 0.006 | 0.100 ± 0.006 | +0.789 ± 0.013 | [0.757, 0.822] |
| No memory | 0.582 ± 0.006 | 0.157 ± 0.002 | 0.261 ± 0.005 | -0.456 ± 0.005 | [-0.468, -0.444] | |
| compact-8k | 0.313 ± 0.006 | 0.393 ± 0.002 | 0.293 ± 0.004 | +0.020 ± 0.008 | [0.001, 0.039] | |
| RTM | 0.344 ± 0.009 | 0.557 ± 0.007 | 0.098 ± 0.002 | +0.191 ± 0.025 | [0.129, 0.254] | |
| mem0 | 0.453 ± 0.004 | 0.436 ± 0.006 | 0.111 ± 0.003 | −0.078 ± 0.009 | [−0.100, −0.056] | |
| MemOS | 0.457 ± 0.007 | 0.438 ± 0.004 | 0.105 ± 0.008 | −0.057 ± 0.009 | [−0.079, −0.035] | |
| Memp | 0.503 ± 0.005 | 0.376 ± 0.003 | 0.121 ± 0.002 | −0.174 ± 0.006 | [−0.189, −0.159] | |
| Dynamic Cheatsheet | 0.428 ± 0.010 | 0.340 ± 0.001 | 0.232 ± 0.010 | −0.132 ± 0.008 | [−0.152, −0.112] | |
| Claude Opus 5.5 | Full context | 0.009 ± 0.001 | 0.774 ± 0.001 | 0.216 ± 0.002 | +0.763 ± 0.001 | [0.760, 0.766] |
| No memory | 0.571 ± 0.002 | 0.168 ± 0.004 | 0.261 ± 0.005 | -0.432 ± 0.006 | [-0.446, -0.417] | |
| compact-8k | 0.353 ± 0.002 | 0.479 ± 0.002 | 0.167 ± 0.002 | +0.106 ± 0.005 | [0.094, 0.119] | |
| RTM | 0.312 ± 0.004 | 0.518 ± 0.004 | 0.170 ± 0.001 | +0.256 ± 0.018 | [0.212, 0.300] | |
| mem0 | 0.382 ± 0.004 | 0.507 ± 0.006 | 0.111 ± 0.003 | +0.064 ± 0.009 | [0.042, 0.086] | |
| MemOS | 0.331 ± 0.004 | 0.446 ± 0.002 | 0.223 ± 0.004 | +0.082 ± 0.003 | [0.075, 0.089] | |
| Memp | 0.416 ± 0.002 | 0.378 ± 0.008 | 0.206 ± 0.008 | −0.083 ± 0.006 | [−0.098, −0.068] | |
| Dynamic Cheatsheet | 0.394 ± 0.005 | 0.358 ± 0.002 | 0.248 ± 0.006 | −0.067 ± 0.004 | [−0.077, −0.057] |
Some revised framework results improve, but the central observed comparison remains.
Response to Reviewer cFnm (1/3)
We thank the reviewer for the insightful review, and appreciate the acknowledgement on our contributions. To your questions, we have the following answers.
Q1. Positioning on AMA-Bench and others
Thank you for pointing this. We now cite AMA-Bench. Briefly the difference lies in what state evolves and what is graded. (1) What evolves. AMA-Bench's State Updating category tracks environment state (inventories, counters, container contents) within a single-task log, and conversational benchmarks (LongMemEval, etc.) track user facts where the newest value wins. RevokeBench's state is normative: which instructions are in force, decided by issuer authority, priority, conditions, group membership, withdrawal and reinstatement, so the newest statement is often not the operative one. In 68% of tasks, at least one option's correct status was set by an event that does not mention that option. (2) What is graded. AMA-Bench asks a memory system questions about a frozen trajectory produced by another agent and scores free-text answers with an LLM judge. RevokeBench grades the agent's own action at the moment it must act, against a solver over the rule history, with no judge, and separates acting on stale state (violation) from not acting (abstention). (3) Closest work. StateMemBench models retractions and scoped exceptions, but in conversations of at most ~15K tokens, graded on answers, without issuer hierarchy or reinstatement, etc. To our best knowledge, none combines normative state that evolves under defeasible semantics over hundreds of sessions with judge-free, action-level grading. AMA-Bench's finding that similarity-based retrieval is lossy on agent logs is complementary to ours.
Q2. Why not one persistent graph with active/inactive labels?
Such a graph can represent RevokeBench, but only if its labels are recomputed from the whole rule history at every step, which is exactly our two-layer construction. The intuitive alternative, deleting or deactivating the defeated rule when a conflict arises, is what the construction was designed to avoid. Considering an extended example of workplace decision-makings: the team lead approves vendor A (s2, π=1); compliance bans A during low-budget periods (s5, π=2), defeating the approval; the budget recovers (s7); it tightens again (s8); an executive endorses A, raising the lead's rule to π=3 (s9). The correct status of A is approved, banned, approved, banned, approved. A store that deletes the defeated approval at s5 answers approved, banned, no rule, banned, banned: when the ban lapses there is nothing left to restore, and the endorsement has no rule to raise. Keeping every rule in a persistent layer and recomputing the defeat relation at each step (the grounded extension, which is unique and independent of deletion order) restores the approval in both cases. Whole-set recomputation also avoids over-deleting conclusions that keep an alternative derivation (the situation Delete-and-Rederive handles). On the benchmark itself, replaying the ground-truth event logs with perfect extraction, a delete-on-conflict store mislabels at least one option in 284 of 1,202 tasks (24%), and in 118 tasks (10%) it no longer approves any option that is in fact approved.
Q3. How many tasks need conflict resolution.
We measured this on the ground-truth logs with a ladder of resolvers that are given the true rule base at each task (no memory loss), recording the simplest one that gets every offered option right:
So 90% of tasks cannot be answered by per-option state transitions, and 78% require resolving conflicting rules by priority rather than recency. Most of these involve a standing list regime (only certified or listed items) that outranks later approvals of individual items; in 5% of tasks, an earlier ruling by a higher-ranked speaker outranks a later one. Iterated fixed-point reasoning, where a rule's condition is itself decided by a conflict, is needed in only 23 tasks (1.9%). With strict priorities, the grounded extension reduces to "the highest applicable priority wins" for a single literal; we use the full construction because it gives every state the generator can produce a unique, order-independent ground truth, not because most tasks exercise deep chains. These resolvers see the true rule base; an agent must first reconstruct it from 344k–935k tokens of history.