- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper represents the execution of large language model multi-agent systems (LLM MASs) as a multistage dynamical system. It introduces Stream of Variation (SoV) into failure attribution for agent trajectories through a state-space model and achieves state-of-the-art performance across multiple datasets and orchestration architectures.
Critical strengths
- Modeling LLM MAS execution as a multistage dynamical system and introducing SoV provides an intuitive formulation with meaningful connections to physical systems and actual agent execution.
- The paper is easy to follow, and the experimental evaluation is comprehensive.
Critical weaknesses
- Agent trajectories contain highly diverse text, resulting in heterogeneous vector representations. Many trajectories encountered at deployment may differ substantially from those observed during training. It remains unclear how well the method generalizes to unseen trajectories and handles edge cases.
- The paper does not evaluate on the widely used Who&When benchmark, and the baseline comparisons are insufficient.
Questions for authors
- Please address the concerns raised in the weaknesses above.
4 comments
Official Rebuttal to Reviewer ZoBt (2/2)
Weakness 2
We thank the reviewer for highlighting these points.
- Regarding the Who&When benchmark, both its algorithm-generated and hand-crafted subsets consist of failed executions and therefore lack the fault-free training trajectories required by AgentSoV to estimate normal dynamics.
To address benchmark coverage, we evaluate AgentSoV on AgentHallu's Magentic-One subset, which contains 40 fault-free trajectories. This subset covers the same Magentic-One architecture used in Who&When's hand-crafted subset, and both collections include GAIA tasks. Thus, our evaluation examines a comparable MAS architecture while providing the fault-free trajectories necessary for learning variation-propagation dynamics.
- Regarding baseline comparisons, we have added baseline experiments in Sections 4.3 and 4.4. Specifically, Section 4.3 now compares AgentSoV with six stage-input influence baselines, including centrality-based methods, SVR and GRU input occlusion, KernelSHAP, TimeSHAP, and CAIR. Section 4.4 additionally evaluates five virtual-sensor placement baselines: Random, FastCan, SSPOC, TPGR, and POD + Pivoted QR. These comparisons complement the failure-attribution baselines in Section 4.2 and provide a more comprehensive evaluation across all three system-level analyses supported by AgentSoV. We have also expanded the baseline descriptions in Appendix E. All revisions are highlighted in blue in the revised manuscript, which are also presented below for the reviewer's convenience.
Stage-input influence.
- Section 4.3, Table 7:
We further evaluate whether predicted stage-input influence agrees with measured intervention effects on BFCL. For each intervention, we remove one stage’s input and regenerate the downstream MAS execution. We measure influence as the Euclidean change in aligned downstream measurements. Table 7 shows the Spearman correlations across source-target pairs within each trajectory, averaging across trajectories. AgentSoV achieves the highest mean correlation (0.28), followed by SVR + event KernelSHAP (0.23) and TimeSHAP (Bento et al., 2021) (0.22), while CAIR (Giloni et al., 2025) obtains 0.05. These results provide evidence that AgentSoV’s stage-input influence estimates can help rank downstream intervention effects.
Table 7: Correlation between predicted and measured downstream influence on BFCL. SVR: Support vector regression; GRU: Gated recurrent unit.
| Method | Correlation |
|---|---|
| Betweenness / Eigenvector | -0.16 ± 0.27 |
| SVR Input Occlusion | 0.18 ± 0.09 |
| GRU Input Occlusion | 0.15 ± 0.04 |
| SVR + Event KernelSHAP | 0.23 ± 0.17 |
| CAIR | 0.05 ± 0.22 |
| TimeSHAP | 0.22 ± 0.25 |
| AgentSoV | 0.28 ± 0.22 |
- Appendix E:
For stage-input influence baselines, betweenness and eigenvector centrality rank source stages on the diagnostic chain. SVR and GRU input occlusion measure changes in predicted downstream measurements after zeroing one source input. SVR + event KernelSHAP and TimeSHAP-style GRU attribution (Bento et al., 2021) score each input using the norm of its event-level Shapley attribution. CAIR-inspired query transfer (Giloni et al., 2025) retrieves the most similar other task and transfers its measured intervention effects at matched relative stage positions, excluding the evaluated task.
Virtual-sensor placement.
- Section 4.4, Figure 3:
On BFCL, AgentSoV consistently outperforms the baselines under limited monitoring budgets, achieving approximately 38% accuracy with only 70% of stages, slightly exceeding full-observation performance. On Magentic-One, monitoring just 10% of stages achieves approximately 59% agent accuracy, comparable to using all stages. Regarding dimensions, BFCL achieves its highest accuracy with all four dimensions, whereas Magentic-One maintains comparable or higher agent accuracy with only one of 16 dimensions.
- Appendix E:
We compare five sensor-placement baselines. Random uniformly samples budget-sized subsets. FastCan (Wang et al., 2026) greedily selects features using canonical correlation with training labels. SSPOC (Brunton et al., 2016) identifies sparse measurements that preserve discriminative information for classification. TPGR (Klishin et al., 2023) uses two-point greedy selection to balance individual sensor informativeness against pairwise redundancy. POD + pivoted QR (Manohar et al., 2018) selects sensors through QR pivoting on a learned low-dimensional reconstruction basis. Selected measurements are evaluated using AgentSoV’s failure attribution pipeline.
I thank the authors for their response. However, my concerns remain. The results suggest limited transferability, and the method requires training on normal data, whereas many existing LLM-based methods do not.
Official Rebuttal to Reviewer ZoBt
We thank the reviewer for the follow-up. AgentSoV's primary contribution lies in providing a unified diagnostic framework across heterogeneous MAS architectures. Our additional experiments demonstrate transferability in several settings. On Magentic-One, training on GPT-4.1 achieves comparable or higher accuracy on other backends for several metrics. Similarly, on BFCL, training on Qwen3-32B achieves comparable or higher accuracy when evaluated on GPT-4.1 and Llama-3.3-70B without retraining.
Regarding the requirement for normal training data, this is a deliberate methodological choice without dependence on labeled failures. AgentSoV learns normal variation-propagation dynamics from fault-free executions without requiring failure annotations. Once trained, it performs diagnosis without additional LLM inference or MAS replay, substantially reducing inference costs. Moreover, unlike LLM-based attribution methods, the same learned dynamics support failure attribution, stage-input influence analysis, and virtual-sensor placement within a unified framework. Thus, AgentSoV offers complementary capabilities and computational advantages.
Official Rebuttal to Reviewer ZoBt (1/2)
We thank the reviewer for the detailed and constructive feedback. All revisions are highlighted in blue in the revised manuscript.
Weakness 1
We thank the reviewer for highlighting the importance of generalization. Our original evaluation pools trajectories from different LLM backends before splitting, so both training and test folds contain multiple backends. Specifically, Magentic-One contains 50 GPT-4.1, 15 GPT-4o, 21 GPT-5, and 8 Claude-3.7-Sonnet trajectories, while BFCL contains 60 GPT-4.1, 60 Qwen3-32B, and 44 Llama-3.3-70B trajectories.
To directly evaluate generalization, we conducted additional experiments in which AgentSoV is trained on trajectories from one LLM backend and evaluated on trajectories from the same or different backends without retraining. We have added these results to Section 4.2 of the revised manuscript and also present them below for the reviewer's convenience:
“Tables 4 and 5 evaluate AgentSoV's transferability. AgentSoV is trained on trajectories from one LLM backend and evaluated on trajectories from the same or different backends without retraining. On Magentic-One, training on trajectories generated by GPT-4.1 achieves the same 50.0% step accuracy when tested on GPT-4.1 and GPT-5, while cross-backend agent accuracy (62.5% on GPT-5 and 72.7% on GPT-4o) exceeds the within-backend accuracy (60.0%). On BFCL, although training on GPT-4.1 yields lower cross-backend accuracy, training on Qwen3-32B achieves comparable or higher accuracy on GPT-4.1 and Llama-3.3-70B than on Qwen3-32B itself. These results demonstrate that AgentSoV can transfer learned diagnostic patterns across different LLM backends.”
Table 4: Within- and cross-backend failure attribution on Magentic-One.
Table 5: Within- and cross-backend failure attribution on BFCL.