acceptodds
Under review as a conference paper at ICLR 2027

Beyond Corrected Memory: Execution Consistency in Multi-Agent Systems

Abstract

Shared memory coordinates agents’ actions, but correct records do not establish that those actions satisfy task requirements. Memory governance and failure diagnosis regulate or inspect recorded information; they do not by themselves establish whether it is sufficient to judge task duties. We define execution consistency through duties governing state use, information handoffs, and final-state agreement, with explicit evidence conditions for judging fulfillment. Our core claim is that identical retained records can correspond to compliant and violating executions under the same task rule. Controlled removal of evidence such as receipt, action dependence, or response validity leaves 82.4% of opposite-label pairs indistinguishable; restoration separates 97.9% of the merged pairs. Natural-log annotations identify the defined violations in actual executions. However, existing logs do not always explicitly represent the execution relationships needed for these judgments. To assess the definition’s practical value, we use CAVERT, a framework for consistency diagnosis and recovery, to extract supported relationships from logs and apply these criteria. It consistently outperforms contract-prompted LLM and rule-based baselines in diagnosis across all 12 benchmark–executor settings. Under the same gate and executor limits, it also outperforms rule-guided recovery in all four evaluated environments. These findings identify execution evidence that agent-memory and execution interfaces should preserve for reliable judgment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-agent systems, agent memory, execution consistency, evidence sufficiency, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FWQuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper draws a distinction that is easy to overlook: in a multi-agent system with shared memory, having correct and consistent records does not mean you have enough evidence to decide whether the agents actually did what the task required. Memory governance keeps state consistent and failure diagnosis inspects it, but neither one by itself establishes task-duty fulfillment. The authors call the missing notion "execution consistency" — judging whether a task duty was fulfilled from the available execution evidence — and instantiate it across three contract families (strict-temporal state use, causal-semantic information handoffs, and eventual-semantic final-state agreement) covering twelve candidate mechanisms, with verdicts of violation, no violation, or undetermined when the evidence is insufficient.

Their central argument is that identical retained records can correspond to both a compliant and a violating execution under the same task rule, so better reasoning over those records cannot recover the distinction — the record has to preserve the execution relationships (receipt, downstream use, lasting effect), not just the final state. They demonstrate this directly with a controlled removal-and-restoration study, then show the defined violations occur in natural logs, and finally build CAVERT, a framework that recovers these relationships from existing logs with a two-stage LLM extraction, applies deterministic verdict rules, and optionally gates uncommitted writes before they commit.

The evaluation is organized around three questions. For necessity, removing selected evidence from 450 controlled pairs makes 82.4% of opposite-labeled (compliant versus violating) pairs indistinguishable, and restoring the relationships re-separates 97.9%; an independent annotation of 360 natural trajectories finds 43 violations, dominated by "unsupported assumption." For diagnosis, on 720 executions across benchmarks and four executor models CAVERT reaches about 87% accuracy and leads the LLM and rule baselines across all twelve settings, including under a matched per-execution spending limit and under recoding and missing-evidence stress tests. For practical value, under a shared pre-commit gate CAVERT improves recovery over a rules-guided gate in all four environments (for example +18.3 points on AppWorld), beats a full memory-system comparator by about 8.75 points, stays within a predeclared non-inferiority margin on clean tasks, and costs roughly 25% more tokens and five seconds per task.

Critical strengths

The removal-and-restoration demonstration of the core claim. This is the cleanest and most convincing part of the paper. By stripping specific execution relationships from controlled pairs and watching 82.4% of opposite-labeled compliant and violating executions collapse into identical inputs — then restoring the relationships and recovering the distinction in 97.9% — the authors turn an abstract point ("correct memory is not sufficient evidence") into a concrete, almost measurement-level result. It holds independently of CAVERT and of any particular judge, which is exactly what makes it credible. This is the contribution I would cite.

Breadth and honesty of the evaluation. The empirical work is unusually disciplined: matched-gate and matched-spending-limit comparisons, an oracle ceiling that exposes the extraction gap rather than hiding it, diagnosis across four executor models and several benchmarks, format and missing-evidence robustness, non-inferiority testing on clean tasks, and explicit cost and latency accounting, all with clustered-bootstrap intervals and null results reported as nulls. Taken together this makes the directional claims — that the recovered relationships help diagnosis and that gating guided by them helps recovery — trustworthy rather than asserted.

Critical weaknesses

The deployable method inherits an extraction bottleneck that the paper's own oracle exposes. CAVERT recovers the needed relationships from existing logs with a two-stage LLM extraction, and the reference-semantic oracle reaches about 98% accuracy where end-to-end CAVERT lands around 83–87%. So a sizable share of the achievable signal is lost in parsing messy logs, and the paper states plainly that diagnostic quality is bounded by extraction quality. That means the strong necessity result is really an argument for instrumenting agent memory to emit these relationships directly; CAVERT as a log-recovery tool is a reasonable stopgap, but its ceiling is set by LLM extraction, which tempers how much of the clean conceptual message transfers to deployments whose logs do not already preserve the relationships.

Much of the practical case rests on self-authored material, and the real-world magnitude is honestly mixed. The contract families, mechanisms, predicates, natural-log labels, and recovery environments are all team-constructed, and the natural-log prevalence is explicitly not meant to represent deployment-wide rates, so we do not learn how often these violations actually arise in the wild. Several practical gains are real but modest or not significant — the EntCollabBench gate-versus-prompt effect is not significant, and the clean-task non-inferiority leans on a fairly loose margin that one benchmark only just meets. The direction is consistent across a lot of settings, which is reassuring, but prevalence and effect size outside the authors' own constructions remain open.

Questions for authors
  1. On the extraction gap: the oracle reaches roughly 98% while end-to-end CAVERT reaches about 83–87%. Where do the lost cases concentrate — particular mechanisms or evidence factors, such as the within-context "unsupported assumption" cases — and is there a realistic path to closing that gap from logs, or is the honest conclusion that reliable judgment needs memory to emit these relationships natively? This matters for how I weigh the practical contribution against the (strong) conceptual one.
  2. The natural-log study reports violation rates as descriptive of the sampled sources. Could you say more about how the 360 traces and the 120-per-source quotas were selected, so a reader can gauge how much the 43-of-360 rate and the mechanism mix reflect the sampling choices versus the sources themselves? This is a request to clarify the existing data, not to run more.
  3. For the gated recovery results: when CAVERT and the rules baseline disagree on a block under the same gate, is CAVERT's advantage coming mainly from catching violations the rules miss, or from avoiding false blocks on normal tasks? A short decomposition from runs you already have would clarify where the gating benefit originates.
Reviewer_gPBmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Cavert discusses execution correctness in the context of shared memory. A key part of the work is concerned with auditing. How can evidence from logs be used to check whether an agent (or multi-agent system) fulfilled its duties? Connected to that is a gating mechanism that can use the findings of the auditor to block potentially problematic actions. Benchmarks show improved diagnosis and error recovery compared to selected baselines.

Critical strengths

The problem is highly relevant. The paper cites some (though not all!) relevant prior work.

Critical weaknesses

The paper appears a bit sprawling. There are multiple contributions and each could profit from more careful grounding in related works and presentation of improvements and novel contributions.

I will first discuss the auditing mechanism. There is highly relevant prior work that investigates how to audit whether agents fulfil their duty. The authors don't cite: Willful Disobedience: Automatically Detecting Failures in Agentic Traces They do cite: Beyond task completion: Revealing corrupt success in LLM agents through procedure-aware evaluation. But the discussion of this mechanism stays very superficial (a single sentence "PAE audits procedural integrity, including agreement between action claims and tool execution") To motivate how the present manuscript is different, the authors claim "We focus on shared-memory duties ..." But I have the impression that the shared memory angle stays largely unexplored. In fact, I found a table in the appendix that supports this impression. Table A9 discusses which violation mechanisms show up in natural logs. And most of the categories defined by the authors have simply 0 occurences. As far as I can see, the only violations found are not related to shared memory.

Next I would like to discuss the safety feature of adding an execution gate. This is a very active field of research. The seminal work from the field SagaLLM (2025) is mentioned in the appendix. If the authors want to discuss recovery mechanisms (and they do this in the main body of the paper, see Figure 7) then they need to add a careful related works section and cite the current state of the field. Next to SagaLLM I think highly related would also be Atomix: Timely, Transactional Tool Use for Reliable Agentic Workflows. This works in a multi-agent setting where execution relationships are used to coordinate gating and approval. The issues raised in this present manuscript, such as missed handoffs, stale dependencies, conflicting writes, ... should be handled by such an agent runtime.

Finally a key weakness is the evaluation. The CAVERT results presented here miss strong baselines. I have already mentioned Figure 7: It discusses recovery but only compares CAVERT against a single relevant baselines, MemTX. There are scenarios such as "Retry", "Direct + G", "Rules + G", but no other competing integrated systems. Similar criticism holds for Table 2, which compares only to prompted LLMs and simple Rules.

Questions for authors

Is there a compelling reason for not adding more baselines (in particular carefully chosen competing systems) to your benchmarks?

How do you justify the taxonomy of violations, given that most mechanisms never seem to occur in the wild (Table A9)

Reviewer_uP3iOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an execution-consistency framework that uses task contracts and COWD evidence states to classify executions as no violation, violation, or undetermined. It introduces CAVERT to extract execution relationships from logs, produce auditable diagnoses, and support pre-commit intervention through a read-before-write gate. Validation includes controlled evidence-removal and restoration studies, natural-log annotations, and benchmark comparisons of diagnosis and recovery performance.

Critical strengths

The distinction between correct shared-memory contents and compliant downstream execution raises a potentially valuable research question: what evidence must agent systems preserve to establish that shared information was appropriately acted upon? A well-supported treatment of this question could inform the design of more reliable and auditable agent systems.

Critical weaknesses
  1. The empirical evidence does not substantiate the paper’s central motivating failure mode. The paper motivates its framework through situations in which updated information is available in shared memory but downstream agents fail to act accordingly. However, its natural-log analysis provides inadequate evidence for this specific phenomenon. Table A8 reports 43 violating traces, of which 38 come from TraceElephant, a sample restricted to task-failure trajectories. AFTraj and MATM contribute only 2/120 and 3/120 confirmed violations, respectively. These figures concern the paper’s broader violation taxonomy; they do not establish the prevalence of the motivating memory-to-action failure. More consequentially, Table A9 reports 0 primary violations for both stale dependency use and missed handoff, while the observed violations predominantly involve unsupported assumptions and evidence-status escalation—problems that can also arise without shared memory or multi-agent coordination. Therefore, the paper has not adequately demonstrated that its extensive formal machinery addresses an empirically established, distinctive problem. This weakness undermines both the motivation and the claimed scope of the contribution.
  2. The connection between the formal judgment framework and the operational recovery method is insufficiently specified and validated. The paper devotes substantial attention to COWD and its mapping to N/V/U, yet does not provide a comparably explicit account of how these outputs determine CAVERT’s interventions. Section 4.3 describes blocking a write that depends on superseded state, refreshing that state, and requesting a revised action. This short description does not supplie a general policy specifying when an agent should retrieve additional information, when an action should be blocked, or what evidence permits execution to resume. In particular, the respective operational consequences of N, V, and U remain unspecified. Thus, the paper leaves a central methodological gap: its elaborate judgment framework has not been shown to be necessary for the comparatively simple control mechanism used to demonstrate practical value. This is a substantive limitation in methodological completeness and contribution attribution.
Questions for authors
  1. Is the motivating failure mode common in real-world agent systems, and what empirical evidence supports its prevalence?
  2. Why does the paper emphasize stale dependency use when its natural-log analysis identifies no primary violations of this type?
  3. How exactly does CAVERT use the computed COWD states and N/V/U verdicts to decide when to block actions, retrieve information, or resume execution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.