- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper draws a distinction that is easy to overlook: in a multi-agent system with shared memory, having correct and consistent records does not mean you have enough evidence to decide whether the agents actually did what the task required. Memory governance keeps state consistent and failure diagnosis inspects it, but neither one by itself establishes task-duty fulfillment. The authors call the missing notion "execution consistency" — judging whether a task duty was fulfilled from the available execution evidence — and instantiate it across three contract families (strict-temporal state use, causal-semantic information handoffs, and eventual-semantic final-state agreement) covering twelve candidate mechanisms, with verdicts of violation, no violation, or undetermined when the evidence is insufficient.
Their central argument is that identical retained records can correspond to both a compliant and a violating execution under the same task rule, so better reasoning over those records cannot recover the distinction — the record has to preserve the execution relationships (receipt, downstream use, lasting effect), not just the final state. They demonstrate this directly with a controlled removal-and-restoration study, then show the defined violations occur in natural logs, and finally build CAVERT, a framework that recovers these relationships from existing logs with a two-stage LLM extraction, applies deterministic verdict rules, and optionally gates uncommitted writes before they commit.
The evaluation is organized around three questions. For necessity, removing selected evidence from 450 controlled pairs makes 82.4% of opposite-labeled (compliant versus violating) pairs indistinguishable, and restoring the relationships re-separates 97.9%; an independent annotation of 360 natural trajectories finds 43 violations, dominated by "unsupported assumption." For diagnosis, on 720 executions across benchmarks and four executor models CAVERT reaches about 87% accuracy and leads the LLM and rule baselines across all twelve settings, including under a matched per-execution spending limit and under recoding and missing-evidence stress tests. For practical value, under a shared pre-commit gate CAVERT improves recovery over a rules-guided gate in all four environments (for example +18.3 points on AppWorld), beats a full memory-system comparator by about 8.75 points, stays within a predeclared non-inferiority margin on clean tasks, and costs roughly 25% more tokens and five seconds per task.
Critical strengths
The removal-and-restoration demonstration of the core claim. This is the cleanest and most convincing part of the paper. By stripping specific execution relationships from controlled pairs and watching 82.4% of opposite-labeled compliant and violating executions collapse into identical inputs — then restoring the relationships and recovering the distinction in 97.9% — the authors turn an abstract point ("correct memory is not sufficient evidence") into a concrete, almost measurement-level result. It holds independently of CAVERT and of any particular judge, which is exactly what makes it credible. This is the contribution I would cite.
Breadth and honesty of the evaluation. The empirical work is unusually disciplined: matched-gate and matched-spending-limit comparisons, an oracle ceiling that exposes the extraction gap rather than hiding it, diagnosis across four executor models and several benchmarks, format and missing-evidence robustness, non-inferiority testing on clean tasks, and explicit cost and latency accounting, all with clustered-bootstrap intervals and null results reported as nulls. Taken together this makes the directional claims — that the recovered relationships help diagnosis and that gating guided by them helps recovery — trustworthy rather than asserted.
Critical weaknesses
The deployable method inherits an extraction bottleneck that the paper's own oracle exposes. CAVERT recovers the needed relationships from existing logs with a two-stage LLM extraction, and the reference-semantic oracle reaches about 98% accuracy where end-to-end CAVERT lands around 83–87%. So a sizable share of the achievable signal is lost in parsing messy logs, and the paper states plainly that diagnostic quality is bounded by extraction quality. That means the strong necessity result is really an argument for instrumenting agent memory to emit these relationships directly; CAVERT as a log-recovery tool is a reasonable stopgap, but its ceiling is set by LLM extraction, which tempers how much of the clean conceptual message transfers to deployments whose logs do not already preserve the relationships.
Much of the practical case rests on self-authored material, and the real-world magnitude is honestly mixed. The contract families, mechanisms, predicates, natural-log labels, and recovery environments are all team-constructed, and the natural-log prevalence is explicitly not meant to represent deployment-wide rates, so we do not learn how often these violations actually arise in the wild. Several practical gains are real but modest or not significant — the EntCollabBench gate-versus-prompt effect is not significant, and the clean-task non-inferiority leans on a fairly loose margin that one benchmark only just meets. The direction is consistent across a lot of settings, which is reassuring, but prevalence and effect size outside the authors' own constructions remain open.
Questions for authors
- On the extraction gap: the oracle reaches roughly 98% while end-to-end CAVERT reaches about 83–87%. Where do the lost cases concentrate — particular mechanisms or evidence factors, such as the within-context "unsupported assumption" cases — and is there a realistic path to closing that gap from logs, or is the honest conclusion that reliable judgment needs memory to emit these relationships natively? This matters for how I weigh the practical contribution against the (strong) conceptual one.
- The natural-log study reports violation rates as descriptive of the sampled sources. Could you say more about how the 360 traces and the 120-per-source quotas were selected, so a reader can gauge how much the 43-of-360 rate and the mechanism mix reflect the sampling choices versus the sources themselves? This is a request to clarify the existing data, not to run more.
- For the gated recovery results: when CAVERT and the rules baseline disagree on a block under the same gate, is CAVERT's advantage coming mainly from catching violations the rules miss, or from avoiding false blocks on normal tasks? A short decomposition from runs you already have would clarify where the gating benefit originates.