- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SEIAR-D, a hierarchical model for error propagation in LLM multi-agent systems. It distinguishes six agent-level states, including error exposure, acceptance, amplification, deviation, and recovery. The authors analyze relay propagation, semantic amplification, population-level criticality, and early prediction of error consolidation. Experiments across four LLMs suggest that agents can amplify errors through new supporting reasoning and that intermediate propagation states can help predict final system failures.
Critical strengths
-
Distinguishing error repetition from semantic amplification is meaningful. LLM agents can generate new reasoning that reinforces an incorrect claim, and the proposed state taxonomy captures this behavior more explicitly than binary propagation models.
-
The empirical evaluation is relatively extensive, covering four LLMs and multiple intervention settings. The early-warning results are also promising, suggesting that intermediate agent states contain useful predictive signals.
Critical weaknesses
-
The overall novelty is limited, as the framework mainly integrates existing ideas rather than introducing fundamentally new insights.
-
Most of the theorems follow directly from standard probability or dynamical-systems results. For example, the multi-path result follows from the monotonicity of event unions, the amplification result assumes nondecreasing propagation probabilities, and the criticality condition uses the standard spectral-radius criterion. Their specific contribution to understanding LLM behavior is therefore unclear.
-
The framework relies heavily on automatically distinguishing states such as Infected, Amplifying, and Deviated, but the reliability of these labels is not adequately evaluated. The assumption that aggregate state counts form a sufficient Markov state is also strong.
-
Moreover, the static-state baseline already achieves very high AUROC, making the added benefit of transition modeling relatively modest.
Questions for authors
- Can the authors report human annotation agreement, particularly for Infected, Amplifying, and Deviated?
- What evidence supports the Markov assumption on aggregate state counts? Would including topology or interaction history improve prediction?
- Can the authors isolate the contribution of transition modeling over the strong static-state baseline and report statistical significance?
- Since the amplification theorem assumes nondecreasing acceptance probabilities, can this assumption be validated empirically rather than imposed?
3 comments
Additional Analysis
5. Empirical Validation of the Monotonicity Assumption
We thank the reviewer for this suggestion. To test the monotonicity assumption in Theorem 3, we conduct a length-matched, paired hop-level experiment. We fix the task, receiving agent, interaction history, and experimental settings. We only change whether the message contains new supporting reasoning, keeping the token length difference within ±5%. For each model, we select 150 paired samples and repeat each condition five times.
Experimental results:
| Model | I Acceptance (%) | A Acceptance (%) | Δ (pp) | Negative Cases |
|---|---|---|---|---|
| MiniMax-M3 | 31.1 | 41.4 | +10.3 | 4/150 |
| GPT-5.6-Luna | 35.1 | 45.8 | +10.7 | 2/150 |
| DeepSeek-V4-Pro | 29.4 | 35.3 | +5.9 | 9/150 |
| Gemini-2.5-Flash | 17.4 | 21.3 | +3.9 | 6/150 |
New supporting reasoning increases error acceptance by 3.9–10.7 percentage points across all four models. These results provide direct hop-level empirical support for the monotonicity assumption in Theorem 3.
However, a small number of negative cases remain. In these cases, new arguments may expose factual errors or logical flaws, prompting agents to reject or correct the target error.
Thus, our results support monotonicity at the aggregate level, but do not imply that it holds for every interaction. Theorem 3 guarantees cascading gains under shared effective exposure and the hop-level condition . We will include these results and clarify the theorem's scope in the revised paper.
Thank you for your response. While some of my concerns have been addressed, my primary concern regarding the paper's limited novelty remains unresolved. The additional analysis does not fundamentally change my assessment of the paper's original contribution. I will maintain my current score.
We appreciate the reviewer's insightful comments, which have motivated us to strengthen the theoretical analysis and provide more rigorous empirical validation. We address the key concerns below.
1. LLM-Specific Theoretical Insights
Our theoretical analysis builds on standard mathematical tools. Its main contribution is to characterize how error acceptance (I) and semantic amplification (A) interact in LLM-based multi-agent systems.
New Proposition: Cross-State Semantic Feedback Criticality
Consider the SEIAR-D next-generation matrix:
k_{II} & k_{IA}\\\\ k_{AI} & k_{AA} \end{pmatrix}\geq 0,$$ where $k_{uv}$ is the expected number of new state-$u$ carriers generated by a state-$v$ carrier. When $k_{II}<1$ and $k_{AA}<1$, neither state can sustain propagation alone. However, $\rho(K)>1$ if and only if $$\boxed{k_{IA}k_{AI}>(1-k_{II})(1-k_{AA})}.$$ **Theoretical Insight:** Error acceptance and semantic amplification can reinforce each other. Agents may accept an error, generate new supporting arguments, and induce further error acceptance. This cross-state feedback can sustain propagation even when neither state can do so independently. This proposition provides a theoretical explanation of how newly generated reasoning can contribute to persistent errors in LLM multi-agent systems. We will include its proof in the revised paper. ### 2. Reliability of Human Annotations We use a Rule-first Hybrid Evaluator. Rules identify S, E, D, and R, while an LLM distinguishes I from A. We stratified 360 samples from four LLMs. Two human annotators labeled them independently and resolved disagreements through arbitration. Overall human agreement reached 92.2%, with Cohen's $\kappa=0.906$. Class-level results for I, A, and D are: | State | Human Agreement (%) | Cohen's κ | Evaluator F1 (%) | |---|---:|---:|---:| | I (Infected) | 88.8 | 0.857 | 89.2 | | A (Amplifying) | 88.1 | 0.854 | 90.0 | | D (Deviated) | 95.6 | 0.945 | 88.3 | Human Agreement denotes positive agreement. Class-level $\kappa$ is computed using one-vs-rest labels. Against the arbitrated labels, the evaluator achieved 93.3% accuracy and 93.3% Macro-F1. In addition, 115 of 120 I/A samples received identical labels across three independent evaluations, showing 95.8% stability. These results support the reliability of our annotations. We will add these results and clarify the I/A/D labeling criteria in the revised paper. ### 3. Markov Assumption, History, and Network Topology The first-order Markov assumption is an approximation of the system's macro-state dynamics. We conduct two additional experiments to evaluate its effectiveness. (1) Impact of Historical Information | Method | Average AUROC | |---|---:| | Binary Propagation | 0.9109 | | First-order SEIAR-D | 0.9895 | | Second-order SEIAR-D | 0.9936 | | Full-history | 0.9932 | First-order SEIAR-D outperforms Binary Propagation by 0.0786 AUROC. Second-order and Full-history provide only 0.0041 and 0.0037 additional gains. This suggests that first-order states capture most predictive information. (2) Impact of Network Topology and Interaction Exposure Our experiments use a fixed, fully connected topology, so structural features remain constant. We therefore add interaction exposure features, including the Target Agent's error-message exposure, new supporting arguments received, and cumulative error exposure. | Method | Average AUROC | |---|---:| | First-order SEIAR-D | 0.9895 | | First-order SEIAR-D + Interaction Exposure | 0.9923 | Interaction exposure improves AUROC by only 0.0028. These results support first-order SEIAR-D as an effective and compact approximation. We will include these results and clarify the scope of our assumptions in the revised paper. ### 4. Independent Contribution of State-Transition Modeling We introduce a Matched Static State baseline on MiniMax-M3. It uses the same input features as SEIAR-D but does not explicitly model state transitions. | Method | AUROC ↑ | F1 (%) ↑ | Warning Round ↓ | |---|---:|---:|---:| | Static State | 0.9884 | 90.00 | 2.39 | | Matched Static State | 0.9879 | 89.80 | 2.45 | | SEIAR-D (Ours) | **0.9914** | **93.62** | **1.23** | Compared with Matched Static State, SEIAR-D improves AUROC by 0.0035 and F1 by 3.82 percentage points, while providing warnings 1.22 rounds earlier. Paired bootstrap analysis (2,000 resamples) and paired significance tests confirm statistically significant improvements: | Metric | 95% CI | p-value | |---|---:|---:| | ΔAUROC | [0.0006, 0.0067] | 0.018 | | ΔF1 (pp) | [0.9, 6.4] | 0.012 | These results confirm that explicit state-transition modeling improves prediction performance and early warning, even with matched input features.