- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how errors propagate in LLM-based multi-agent systems and when persistent memory can enable recovery. It models the feedback loop between agent communication and environment interaction, establishes information-theoretic limits on recovery without new evidence, and constructs a task where minimal persistent memory enables exact recovery while memory-free interaction remains at chance. These results motivate RECOVER, an in-task memory mechanism that retains verified evidence for action preconditions and selectively injects it when it resolves an otherwise undetermined check. Experiments across five benchmarks and three multi-agent frameworks report the highest mean performance among the evaluated baselines, while injection-timing comparisons and efficiency analyses support improved token and storage efficiency.
Critical strengths
- Clear theoretical insight. The paper establishes when communication alone cannot recover lost information and how sufficient memory can restore correct decisions, providing a principled basis for memory design.
- Consistent practical benefits. RECOVER achieves the highest mean performance across all 15 benchmark–framework combinations, with timing and efficiency experiments supporting selective evidence injection.
Critical weaknesses
- The connection between the theory and RECOVER remains limited. Theorem 3 establishes separation through a construction that stores the already-computed target before erasing it from all transient inputs (Appendices B.2–B.4). This demonstrates the value of preserving information, but does not establish that RECOVER’s precondition-based representation is sufficient for general task recovery. Proposition 3 guarantees only local verdict correctness under strong evidence-validity and retrieval assumptions. Consequently, the theory provides motivation rather than a substantive guarantee for the proposed system.
- The experiments do not isolate the central mechanism. RECOVER combines structured evidence retention, precondition checking, selective injection, and an additional revision call (Appendix C.1). Although Figure 6 evaluates injection timing, the reported comparisons do not separate the contribution of precondition-based memory from structured state tracking or additional deliberation. This leaves uncertainty about whether the gains specifically validate the proposed memory-design principles.
Questions for authors
- Clarification: Beyond the constructed erasure task, what result or measurement connects the propagation analysis to RECOVER’s recovery benefits? In particular, how does correctness of the declared preconditions translate into recovery of the task-level target?
- Clarification: How are evidence extraction, invalidation, and precondition checking implemented and validated, particularly for semantic judgments in FEVER? What evidence shows that the assumptions underlying Proposition 3 hold sufficiently often in actual runs?
- Targeted validation: Can you provide a controlled comparison against compact factual memory with the same checking and revision budget, changing only the precondition-based retention or injection rule? This would help establish whether the proposed mechanism explains the gains.
6 comments
W1 & Q1. (continued)
2. Precondition correctness translate into task-level target recovery
Controlled experiments demonstrate that RECOVER translates correct precondition judgments into reduced errors during task execution and improved final task success.
- Recovery of task-level target errors. We quantify RECOVER's recovery benefits by measuring the reduction in per-step error rates during subsequent task execution with precondition-based memory insertion compared to fact-based memory insertion.
Method Error Reduction ↓ Task Accuracy ↑ Fact-based Memory 5.12 pp 79.85% RECOVER (precondition-based) 28.36 pp 91.54%
- Precondition-oriented memory and task-level recovery. We further compare different memory designs under identical Check-step insertion settings:
Method Memory Type ACC ↑ Tokens/Task ↓ Revision-based Fact-based 70.90% 57,654 Precondition-based 73.88% 56,197 Retrieval-based Fact-based 75.37% 54,594 Precondition-based 77.61% 52,975 RECOVER Fact-based 79.85% 51,276 RECOVER (Ours) Precondition-based 91.54% 38,406 Precondition-based RECOVER memory achieves the highest task accuracy and lowest token consumption, improving accuracy by up to 20.65 percentage points while reducing tokens per task by 33.39% compared with fact-based revision.
Overall, these results demonstrate that correct precondition recovery reduces errors during task execution and supports task-level target recovery, improving end-to-end task success and token efficiency.
Q2.
Clarification: How are evidence extraction, invalidation, and precondition checking implemented and validated, particularly for semantic judgments in FEVER? What evidence shows that the assumptions underlying Proposition 3 hold sufficiently often in actual runs?
We thank the reviewer for this important question. Independent assessments validate the reliability of RECOVER's evidence extraction and precondition checking. Controlled experiments further support the practical applicability of Proposition 3's assumptions in task-level recovery. The implementation details and experimental evidence are presented below.
1. Implementation of evidence extraction and precondition checking
Extract/Check Implementation. Both components use straightforward implementations without specialized architectures or additional training.
| Component | Implementation | Example |
|---|---|---|
| Extract | Uses fixed regular-expression templates tailored to each environment's action grammar and observation format. Extracts action arguments and observed state values while preserving source spans. | FEVER: Given claim (q): “Global warming threatens food security.” and agent responded action “finish support the claim,” extracts Claim[q], Finish[Support], and Support[Claim]. |
| Check | Evaluates preconditions using fixed action schemas. Reuses benchmark regular-expression templates for PDDL-like environments (e.g., ALFWorld) and direct field lookup/value comparison for structured conditions. For textual conditions (primarily FEVER), uses a frozen DeBERTa-v3-base-mnli entailment model. | FEVER: Map entailment → true, contradiction → false, and neutral → unknown. |
2. Empirical validation of Proposition 3's assumptions
- Independent correctness assessment. We independently evaluated sampled Extract and Check outputs using GPT-6/Astra-based assessments:
Independent Evaluator Extract Accuracy ↑ Check Accuracy ↑ GPT-6/Astra 92.68% 100.00% DeepSeek-V4-Flash 90.91% 96.10% These results provide independent empirical support for the correctness of evidence extraction and precondition checking, two key operations underlying Proposition 3.
- Sensitivity to Extract/Check errors and assumption violations. Precondition-based retention consistently outperforms fact-based retention under imperfect Extract/Check conditions, supporting the robustness of RECOVER's recovery mechanism.
To ensure a fair comparison, both methods use identical Extract implementations and RECOVER's Check-step insertion rule. We introduce controlled errors by randomly dropping Extract outputs or flipping Check verdicts, and compare the resulting accuracy degradation across different memory strategies.
Setting Fact-based memory ACC ↑ RECOVER ACC ↑ Gain (pp) ↑ Original Extract/Check 78.36% 91.54% +13.18 Extract-drop-10% 75.87% 91.04% +15.17 Extract-drop-20% 70.40% 90.05% +19.65 Check-flip-10% 73.38% 85.32% +11.94 Check-flip-20% 69.90% 82.84% +12.94
Overall, the independent correctness assessments and sensitivity analysis provide empirical support for evidence extraction and precondition checking mechanisms, and demonstrate the sufficiency of Proposition 3’s assumptions for effective recovery in actual runs.
W1 & Q1.
We thank the reviewer for this important question.
1. Empirical connection between error propagation and RECOVER's recovery benefits
Beyond the constructed erasure task, empirical analysis validates error propagation in MAS, while complementary experiments demonstrate RECOVER's improvements in task success and token efficiency.
The overall recovery comparison further supports the practical value of the theory-guided design. With Extract-based retention and Check-step insertion, RECOVER improves task accuracy from 77.61% with retrieval-based memory to 91.54% (+13.93 pp) while using 27.50% fewer tokens. It also outperforms revision-based recovery by 17.66 pp with 31.66% fewer tokens.