acceptodds
Under review as a conference paper at ICLR 2027

AgentSoV: Virtual Sensing for LLM Multi-Agent Systems via Stream of Variation

Abstract

Large language model (LLM) multi-agent systems (MASs) solve complex tasks through interacting agents, yet their reliability is difficult to analyze because failures depend on both individual agents and how information and errors propagate across the system. Existing approaches largely treat this as post-hoc failure attribution, but do not provide a common model across heterogeneous MAS architectures or support broader analysis of information influence and observability. We introduce **AgentSoV**, a virtual-sensing framework that models MAS execution as a stochastic dynamic system through Stream of Variation (SoV). AgentSoV maps heterogeneous trajectories into diagnostic stages, constructs virtual measurements from observable outputs, and learns cross-stage variation propagation with a state-space model. Its representation is architecture-aware in stage dependencies yet architecture-agnostic in analytical language, supporting chain-structured, orchestration-based, and directed acyclic graph (DAG)-structured MASs. Without re-executing the MAS, AgentSoV supports three system-level analyses: failure attribution identifies where abnormal variation enters; stage-input influence quantifies how strongly intermediate inputs affect downstream outcomes; and virtual-sensor placement selects complementary observations under a monitoring budget. Experiments across multiple MAS architectures demonstrate a unified approach to fault localization, information influence, and observability. AgentSoV reframes agentic execution as a dynamic system to model rather than merely text to inspect, providing insights for MAS structure and component design.

Primary area:
interpretability and explainable AI
Keywords:
Large language model, Multi-agent system, Stream of variation, Virtual sensing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZoBtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper represents the execution of large language model multi-agent systems (LLM MASs) as a multistage dynamical system. It introduces Stream of Variation (SoV) into failure attribution for agent trajectories through a state-space model and achieves state-of-the-art performance across multiple datasets and orchestration architectures.

Critical strengths
  1. Modeling LLM MAS execution as a multistage dynamical system and introducing SoV provides an intuitive formulation with meaningful connections to physical systems and actual agent execution.
  2. The paper is easy to follow, and the experimental evaluation is comprehensive.
Critical weaknesses
  1. Agent trajectories contain highly diverse text, resulting in heterogeneous vector representations. Many trajectories encountered at deployment may differ substantially from those observed during training. It remains unclear how well the method generalizes to unseen trajectories and handles edge cases.
  2. The paper does not evaluate on the widely used Who&When benchmark, and the baseline comparisons are insufficient.
Questions for authors
  1. Please address the concerns raised in the weaknesses above.
4 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer ZoBt (1/2)

We thank the reviewer for the detailed and constructive feedback. All revisions are highlighted in blue in the revised manuscript.

Weakness 1

We thank the reviewer for highlighting the importance of generalization. Our original evaluation pools trajectories from different LLM backends before splitting, so both training and test folds contain multiple backends. Specifically, Magentic-One contains 50 GPT-4.1, 15 GPT-4o, 21 GPT-5, and 8 Claude-3.7-Sonnet trajectories, while BFCL contains 60 GPT-4.1, 60 Qwen3-32B, and 44 Llama-3.3-70B trajectories.

To directly evaluate generalization, we conducted additional experiments in which AgentSoV is trained on trajectories from one LLM backend and evaluated on trajectories from the same or different backends without retraining. We have added these results to Section 4.2 of the revised manuscript and also present them below for the reviewer's convenience:

“Tables 4 and 5 evaluate AgentSoV's transferability. AgentSoV is trained on trajectories from one LLM backend and evaluated on trajectories from the same or different backends without retraining. On Magentic-One, training on trajectories generated by GPT-4.1 achieves the same 50.0% step accuracy when tested on GPT-4.1 and GPT-5, while cross-backend agent accuracy (62.5% on GPT-5 and 72.7% on GPT-4o) exceeds the within-backend accuracy (60.0%). On BFCL, although training on GPT-4.1 yields lower cross-backend accuracy, training on Qwen3-32B achieves comparable or higher accuracy on GPT-4.1 and Llama-3.3-70B than on Qwen3-32B itself. These results demonstrate that AgentSoV can transfer learned diagnostic patterns across different LLM backends.”

Table 4: Within- and cross-backend failure attribution on Magentic-One.

TrainingTestStep AccuracyAgent Accuracy
GPT-4.1GPT-4.150.0%60.0%
GPT-4.1GPT-4o27.3%72.7%
GPT-4.1GPT-550.0%62.5%

Table 5: Within- and cross-backend failure attribution on BFCL.

TrainingTestAccuracy
GPT-4.1GPT-4.158.3%
GPT-4.1Qwen3-32B17.9%
GPT-4.1Llama-3.3-70B31.0%
Qwen3-32BQwen3-32B16.7%
Qwen3-32BGPT-4.117.4%
Qwen3-32BLlama-3.3-70B24.1%
Llama-3.3-70BLlama-3.3-70B22.2%
Llama-3.3-70BGPT-4.115.2%
Llama-3.3-70BQwen3-32B10.7%
AuthorsOpenReview ↗

Official Rebuttal to Reviewer ZoBt (2/2)

Weakness 2

We thank the reviewer for highlighting these points.

  1. Regarding the Who&When benchmark, both its algorithm-generated and hand-crafted subsets consist of failed executions and therefore lack the fault-free training trajectories required by AgentSoV to estimate normal dynamics.

To address benchmark coverage, we evaluate AgentSoV on AgentHallu's Magentic-One subset, which contains 40 fault-free trajectories. This subset covers the same Magentic-One architecture used in Who&When's hand-crafted subset, and both collections include GAIA tasks. Thus, our evaluation examines a comparable MAS architecture while providing the fault-free trajectories necessary for learning variation-propagation dynamics.

  1. Regarding baseline comparisons, we have added baseline experiments in Sections 4.3 and 4.4. Specifically, Section 4.3 now compares AgentSoV with six stage-input influence baselines, including centrality-based methods, SVR and GRU input occlusion, KernelSHAP, TimeSHAP, and CAIR. Section 4.4 additionally evaluates five virtual-sensor placement baselines: Random, FastCan, SSPOC, TPGR, and POD + Pivoted QR. These comparisons complement the failure-attribution baselines in Section 4.2 and provide a more comprehensive evaluation across all three system-level analyses supported by AgentSoV. We have also expanded the baseline descriptions in Appendix E. All revisions are highlighted in blue in the revised manuscript, which are also presented below for the reviewer's convenience.

Stage-input influence.

  • Section 4.3, Table 7:

We further evaluate whether predicted stage-input influence agrees with measured intervention effects on BFCL. For each intervention, we remove one stage’s input and regenerate the downstream MAS execution. We measure influence as the Euclidean change in aligned downstream measurements. Table 7 shows the Spearman correlations across source-target pairs within each trajectory, averaging across trajectories. AgentSoV achieves the highest mean correlation (0.28), followed by SVR + event KernelSHAP (0.23) and TimeSHAP (Bento et al., 2021) (0.22), while CAIR (Giloni et al., 2025) obtains 0.05. These results provide evidence that AgentSoV’s stage-input influence estimates can help rank downstream intervention effects.

Table 7: Correlation between predicted and measured downstream influence on BFCL. SVR: Support vector regression; GRU: Gated recurrent unit.

MethodCorrelation
Betweenness / Eigenvector-0.16 ± 0.27
SVR Input Occlusion0.18 ± 0.09
GRU Input Occlusion0.15 ± 0.04
SVR + Event KernelSHAP0.23 ± 0.17
CAIR0.05 ± 0.22
TimeSHAP0.22 ± 0.25
AgentSoV0.28 ± 0.22
  • Appendix E:

For stage-input influence baselines, betweenness and eigenvector centrality rank source stages on the diagnostic chain. SVR and GRU input occlusion measure changes in predicted downstream measurements after zeroing one source input. SVR + event KernelSHAP and TimeSHAP-style GRU attribution (Bento et al., 2021) score each input using the norm of its event-level Shapley attribution. CAIR-inspired query transfer (Giloni et al., 2025) retrieves the most similar other task and transfers its measured intervention effects at matched relative stage positions, excluding the evaluated task.

Virtual-sensor placement.

  • Section 4.4, Figure 3:

On BFCL, AgentSoV consistently outperforms the baselines under limited monitoring budgets, achieving approximately 38% accuracy with only 70% of stages, slightly exceeding full-observation performance. On Magentic-One, monitoring just 10% of stages achieves approximately 59% agent accuracy, comparable to using all stages. Regarding yky_k dimensions, BFCL achieves its highest accuracy with all four dimensions, whereas Magentic-One maintains comparable or higher agent accuracy with only one of 16 dimensions.

  • Appendix E:

We compare five sensor-placement baselines. Random uniformly samples budget-sized subsets. FastCan (Wang et al., 2026) greedily selects features using canonical correlation with training labels. SSPOC (Brunton et al., 2016) identifies sparse measurements that preserve discriminative information for classification. TPGR (Klishin et al., 2023) uses two-point greedy selection to balance individual sensor informativeness against pairwise redundancy. POD + pivoted QR (Manohar et al., 2018) selects sensors through QR pivoting on a learned low-dimensional reconstruction basis. Selected measurements are evaluated using AgentSoV’s failure attribution pipeline.

I thank the authors for their response. However, my concerns remain. The results suggest limited transferability, and the method requires training on normal data, whereas many existing LLM-based methods do not.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer ZoBt

We thank the reviewer for the follow-up. AgentSoV's primary contribution lies in providing a unified diagnostic framework across heterogeneous MAS architectures. Our additional experiments demonstrate transferability in several settings. On Magentic-One, training on GPT-4.1 achieves comparable or higher accuracy on other backends for several metrics. Similarly, on BFCL, training on Qwen3-32B achieves comparable or higher accuracy when evaluated on GPT-4.1 and Llama-3.3-70B without retraining.

Regarding the requirement for normal training data, this is a deliberate methodological choice without dependence on labeled failures. AgentSoV learns normal variation-propagation dynamics from fault-free executions without requiring failure annotations. Once trained, it performs diagnosis without additional LLM inference or MAS replay, substantially reducing inference costs. Moreover, unlike LLM-based attribution methods, the same learned dynamics support failure attribution, stage-input influence analysis, and virtual-sensor placement within a unified framework. Thus, AgentSoV offers complementary capabilities and computational advantages.

Reviewer_2fEBOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentSoV, a virtual-sensing framework for diagnosing failures and analyzing information propagation in LLM-based multi-agent systems. The key idea is to represent an execution trajectory as a sequence or graph of diagnostic stages and model the propagation of task-relevant variation using a linear state-space formulation inspired by the Stream-of-Variation literature in multistage manufacturing. The same fitted dynamics are then used for three system-level analyses: failure attribution, stage-input influence estimation, and virtual-sensor placement. The framework is instantiated for chain-structured tool use, centralized orchestration, and DAG-structured multi-agent systems, and is evaluated on BFCL, Magentic-One/AgentHallu, and AEGIS. The reported results suggest that modeling cross-stage propagation improves failure localization and enables low-cost analysis without replaying the underlying agent system or invoking an LLM at inference time.

Critical strengths
  1. A unified and interpretable system-level perspective. AgentSoV provides a coherent abstraction for reasoning about error propagation across heterogeneous multi-agent architectures. In particular, using the same fitted dynamics for failure attribution, stage-input influence, and virtual-sensor placement is conceptually appealing and offers a more systematic perspective than treating these tasks independently.

  2. Broad architectural coverage with low diagnostic cost. The paper instantiates the framework on chain-structured, orchestration-based, and DAG-structured systems and evaluates it on three corresponding benchmarks. The method avoids per-trajectory LLM inference and system replay, while generally improving failure-localization performance over the considered baselines. This combination of interpretability, architectural flexibility, and computational efficiency is practically meaningful.

Critical weaknesses
  1. Backend-Model Invariance of Low-Dimensional Semantic Features

The proposed representation may not be invariant to the underlying LLM backend. Different models can exhibit substantially different wording, response structure, tool-use patterns, and self-correction behavior. These differences may alter the embeddings produced by MiniLM and, consequently, the PCA-projected features and learned transition dynamics.

It is unclear whether the training and test trajectories in AgentHallu were generated using the same backend model and configuration. If so, the current evaluation is largely in-distribution and cannot determine whether AgentSoV captures general error-propagation patterns or merely model-specific behavioral regularities. The authors should clarify the LLM backends and configurations used to generate each data split and, ideally, evaluate cross-backend transfer—for example, training the diagnostic model on trajectories from one LLM and testing it on another. Experiments across model families, or model versions would be necessary to substantiate claims of backend-model invariance.

  1. Validity of the Linear Assumption

AgentSoV fundamentally relies on the assumption that variation propagation in the projected semantic space can be adequately modeled by linear dynamics. However, the paper does not directly validate this assumption, and downstream attribution performance alone is insufficient to establish the adequacy of the linear approximation. The authors should provide empirical or theoretical justification for this assumption, or more clearly discuss its limitations and the conditions under which it is expected to hold.

  1. Additional Minor Comments
  • In Table 2, the MRR of MASC is reported as 86.53% +/- 1.4%, whereas the other results are generally reported to one decimal place. The numerical precision should be made consistent.
  • The state-space formulation requires an initial state x0x_0, but the paper does not specify whether it is fixed, estimated from data, or initialized separately for each trajectory.
  • The parent-aggregation term in Equation 8 is undefined for a root node with ∣pa(k)∣=0|\mathrm{pa}(k)|=0. The paper should explicitly state how root nodes are handled.
Questions for authors
  1. Were the training and test trajectories in AgentHallu generated using the same LLM backend, model version, prompts, and agent configuration? How does a diagnostic model trained on one backend perform on trajectories generated by a different backend or model version?
  2. How are x0x_0 and DAG root nodes initialized in the implementation? Does this choice affect the learned dynamics or attribution results?
4 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer 2fEB (1/2)

We thank the reviewer for the detailed and constructive feedback. All revisions are highlighted in blue in the revised manuscript.

Weakness 1

We thank the reviewer for highlighting the importance of backend-model generalization. Our original evaluation pools trajectories from different LLM backends before splitting, so both training and test folds contain multiple backends. Specifically, Magentic-One contains 50 GPT-4.1, 15 GPT-4o, 21 GPT-5, and 8 Claude-3.7-Sonnet trajectories, while BFCL contains 60 GPT-4.1, 60 Qwen3-32B, and 44 Llama-3.3-70B trajectories. The trajectories share the same agent framework within each dataset.

To directly evaluate transferability, we conducted additional experiments in which AgentSoV is trained on trajectories from one LLM backend and evaluated on trajectories from the same or different backends without retraining. We have added these results to Section 4.2 of the revised manuscript and also present them below for the reviewer's convenience:

“Tables 4 and 5 evaluate AgentSoV's transferability. AgentSoV is trained on trajectories from one LLM backend and evaluated on trajectories from the same or different backends without retraining. On Magentic-One, training on trajectories generated by GPT-4.1 achieves the same 50.0% step accuracy when tested on GPT-4.1 and GPT-5, while cross-backend agent accuracy (62.5% on GPT-5 and 72.7% on GPT-4o) exceeds the within-backend accuracy (60.0%). On BFCL, although training on GPT-4.1 yields lower cross-backend accuracy, training on Qwen3-32B achieves comparable or higher accuracy on GPT-4.1 and Llama-3.3-70B than on Qwen3-32B itself. These results demonstrate that AgentSoV can transfer learned diagnostic patterns across different LLM backends.”

Table 4: Within- and cross-backend failure attribution on Magentic-One.

TrainingTestStep AccuracyAgent Accuracy
GPT-4.1GPT-4.150.0%60.0%
GPT-4.1GPT-4o27.3%72.7%
GPT-4.1GPT-550.0%62.5%

Table 5: Within- and cross-backend failure attribution on BFCL.

TrainingTestAccuracy
GPT-4.1GPT-4.158.3%
GPT-4.1Qwen3-32B17.9%
GPT-4.1Llama-3.3-70B31.0%
Qwen3-32BQwen3-32B16.7%
Qwen3-32BGPT-4.117.4%
Qwen3-32BLlama-3.3-70B24.1%
Llama-3.3-70BLlama-3.3-70B22.2%
Llama-3.3-70BGPT-4.115.2%
Llama-3.3-70BQwen3-32B10.7%

Weakness 2

We thank the reviewer for raising this important point. We have strengthened the manuscript by adding a detailed justification of the linear variation-propagation assumption in Appendix D, which are also presented below for the reviewer's convenience:

Interpretation of the linear assumption. The linear state-space formulation in AgentSoV does not assume that the internal reasoning or text-generation mechanisms of LLMs are inherently linear. Instead, it approximates how task-relevant variation propagates across diagnostic stages within a lower-dimensional representation. Specifically, AgentSoV models the relationship between the diagnostic state xk−1\mathbf{x} _{k-1}, stage input uk\mathbf{u} _k, and subsequent state xk\mathbf{x} _k, rather than directly modeling the nonlinear transformation from input text to generated text. This distinction is important because the diagnostic representations retain selected information relevant to variation propagation rather than the complete semantic complexity of LLM execution.

Theoretical and empirical justification. Consider a general nonlinear stage-transition model

xk=fk(xk−1,uk)+ϵk\mathbf{x} _k = f _k(\mathbf{x} _{k-1}, \mathbf{u} _k) + \boldsymbol{\epsilon} _k

where fk denotes the conditional mean transition in the diagnostic space and ϵk\boldsymbol{\epsilon} _k captures stochastic variation.

Let (xˉk−1,uˉk)(\bar{\mathbf{x}} _{k-1}, \bar{\mathbf{u}} _k) denote a reference operating point associated with fault-free execution. If fk is twice continuously differentiable in a neighborhood of this point, its first-order Taylor expansion gives

Δxk=FkΔxk−1+BkΔuk+rk+ϵk\Delta \mathbf{x} _k = \mathbf{F} _k \Delta \mathbf{x} _{k-1} + \mathbf{B} _k \Delta \mathbf{u} _k + \mathbf{r} _k + \boldsymbol{\epsilon} _k

where

\left. \frac{\partial f _k}{\partial \mathbf{x} _{k-1}} \right| _{\mathrm{ref}}, \qquad \mathbf{B} _k = \left. \frac{\partial f _k}{\partial \mathbf{u} _k} \right| _{\mathrm{ref}}$$ and $\mathbf{r} _k$ represents the higher-order approximation error. If the Hessian norm of fk is bounded by Lk within this neighborhood, the approximation error satisfies $$\\|\mathbf{r} _k\\| _2 \leq \frac{L _k}{2} \left\\| \begin{bmatrix} \Delta \mathbf{x} _{k-1} \\\\ \Delta \mathbf{u} _k \end{bmatrix} \right\\| _2^2$$
AuthorsOpenReview ↗

Official Rebuttal to Reviewer 2fEB (2/2)

Weakness 2 (Continued)

Therefore, the linear approximation has a second-order remainder and becomes increasingly accurate as deviations from the reference operating point decrease. After centering the diagnostic variables, or equivalently incorporating an affine intercept, this approximation corresponds to the state-transition structure in Eq. 1.

Importantly, AgentSoV does not require explicit computation of these Jacobians. Instead, Fk\mathbf{F} _k and Bk\mathbf{B} _k are estimated from fault-free execution trajectories. For directly observed states, the resulting linear regression can also be interpreted as the best affine approximation under a squared prediction-error criterion over the training distribution. This statistical interpretation remains valid even when the underlying transition is nonlinear or contains discrete execution variables.

The objective of AgentSoV is not to reproduce every detail of an LLM-generated response. Rather, it is to characterize sufficiently informative cross-stage dependencies for system analysis. The estimated process noise and residual covariance account for ordinary stochastic fluctuations and part of the remaining model mismatch. Thus, useful diagnostic information may be obtained without an exact global representation of the underlying nonlinear transition function.

The experimental results in Section 4 further demonstrate the effectiveness of AgentSoV across three system-level analyses: failure attribution, stage-input influence, and virtual-sensor placement. AgentSoV achieves competitive or superior performance compared with the evaluated baselines, suggesting that the linear approximation captures meaningful cross-stage relationships for diagnostic purposes. In particular, the F=0\mathbf{F}=0 ablations in Section 4.2 show that removing the learned transition dynamics generally reduces failure-attribution accuracy, indicating that cross-stage propagation contains useful diagnostic information.

Validity conditions and limitations. The linear approximation is most appropriate when diagnostic states remain within the operating region represented by the fault-free training trajectories, the execution architecture and stage definitions remain consistent, and the conditional relationships between consecutive diagnostic states vary smoothly over the relevant range. Its accuracy may decrease under substantial distribution shifts, abrupt changes in agent behavior, nonlinear interactions between multiple stage inputs, or discrete transitions that alter the execution workflow.

These considerations clarify that linear dynamics serve as a tractable approximation for diagnostic variation propagation rather than a universal assumption about LLM behavior. Extending AgentSoV to nonlinear or locally adaptive state-space models is a promising direction for capturing more complex propagation mechanisms.

Weakness 3

  • We have corrected the MRR of MASC in Table 2 to ensure consistent numerical precision.
  • The BFCL and Magentic-One use a fixed prior x0∼N(0,I)\boldsymbol{x}_0 \sim \mathcal{N}(\boldsymbol{0},\boldsymbol{I}) for every trajectory. Magentic-One also estimates a separate planning state from each trajectory’s fact sheet for plan-fault routing. In AEGIS, root calls are initialized from their inputs as x^k=Buk+b\hat{\boldsymbol{x}}_k=\boldsymbol{B}\boldsymbol{u}_k+\boldsymbol{b}. We specified these initialization rules in the revised Appendix B.
  • For a root node with pa(k)=∅\mathrm{pa}(k)=\varnothing, the implementation sets the parent contribution to zero. The prediction is therefore x^k=Buk+b\hat{\boldsymbol{x}}_k=\boldsymbol{B}\boldsymbol{u}_k+\boldsymbol{b}. We added the explanation in Appendix B.3.

Question 1

Please see our response to Weakness 1.

Question 2

Please see our responses to Points 2 and 3 under Weakness 3. These initialization rules are consistent with the respective MAS architectures and prevent artificial upstream dependencies. The fixed prior in BFCL and Magentic-One ensures consistent initialization across trajectories, while AEGIS root nodes are initialized solely from their inputs. Consequently, the learned dynamics capture cross-stage variation propagation without introducing additional trajectory-specific assumptions.

The author conducted an experiment of "training on one backend and testing on another backend", which is a valuable supplement. However, the cross-model performance on BFCL has obviously decreased, and the score brought by changing the back-end fluctuates greatly. This shows that the diagnostic model has a certain back-end dependence and cannot prove that the low-dimensional semantic features have back-end invariance.

The second mathematical response is correct but not sufficient. Your Taylor expansion is of course correct, but it means that "linear approximation can be made" rather than "linear approximation is reasonable and the error is acceptable"; Although it is difficult to quantify the error in mathematics, the author discusses the limitations of this method and the conditions for the expected establishment more clearly, which is also a valuable supplement.

To sum up, I think the author has made a full supplement in rebuttal, and these analyses have given me a deeper understanding of the problem, but the scope of the conclusions that the paper can support is less than the generality of the methods it implies. So I tend to keep my score.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer 2fEB

We thank the reviewer for the follow-up. Regarding cross-backend transferability, we would like to emphasize that the observed performance fluctuations may partly arise from the limited number of fault-free training trajectories: only 17 for Magentic-One and 14, 24, and 15 for BFCL when training on GPT-4.1, Qwen3-32B, and Llama-3.3-70B, respectively. Such small training sets may contribute to performance variability across backends. Nevertheless, the results demonstrate transferability in several settings without retraining. Importantly, AgentSoV's primary generality lies in its unified diagnostic formulation across heterogeneous MAS architectures.

Regarding the linear assumption, beyond the theoretical justification, our experiments demonstrate its empirical effectiveness across three system-level analyses. In particular, the ablation results show that removing learned cross-stage propagation dynamics (FF=0) generally reduces failure-attribution accuracy, supporting the practical utility of the linear approximation.

Reviewer_NkLROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work organizes multi-agent execution logs into diagnostic stages, fits a linear state-space model of normal behavior, and uses sparse disturbance estimates to locate faults. The same model supports stage-input influence analysis and sensor selection. Experiments cover chain, orchestrated, and DAG systems through BFCL, Magentic-One, and AEGIS, with localization results, propagation ablations, and runtime measurements. The main contribution is adapting process diagnosis to heterogeneous multi-agent traces.

Critical strengths
  1. The approach works on existing logs without rerunning the original system. Sections 2–3 separate system-specific stage construction from the subsequent diagnosis, allowing three execution structures to use the same analysis. This is useful for offline debugging. 2. The localization results and computational cost are promising. On BFCL, removing propagation reduces step accuracy from 37.5% to 28.5%, while Table 4 reports 0.0242 seconds of inference per trace. These results give concrete support for propagation modeling and practical use.
Critical weaknesses
  1. I could not reconcile the metrics in Table 2. If Top-1 and MRR use the same ranking, with reciprocal rank measured at the first relevant fault, then MRR ≤ Top-1 + (1−Top-1)/2. The 59.0% Top-1 for GPT-5.5 All-at-Once implies an upper bound of 79.5%, but the reported MRR is 91.5%. Different aggregation units might explain this, but the definitions do not make that clear. This is my main concern about the localization evaluation.

  2. The influence and sensor-selection applications are less well validated. Equation (7) removes an input within the fitted model while fixing other inputs; changing an upstream message in the actual system may also change downstream inputs. Figure 2 does not establish whether this sensitivity predicts intervention effects. Figure 3 reports localization with partial observations, but lacks random or simple heuristic selection at the same budget, leaving the added benefit of log-det selection unclear.

Questions for authors
  1. How are Top-1 and MRR computed per trace in Table 2? Please explain the discrepancy above and specify whether λ is selected using validation data drawn only from each outer training fold. These details would directly affect my assessment of the main results.

  2. Are comparisons available between Equation (7) and actual message interventions, and between log-det and equal-budget random selection? I currently view these two applications as preliminary.

2 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer NkLR (1/2)

We thank the reviewer for the detailed and constructive feedback. All revisions are highlighted in blue in the revised manuscript.

Weakness 1

We thank the reviewer for carefully examining the evaluation metrics and identifying this inconsistency. AEGIS allows multiple faulty agents within a single trajectory. For each trajectory, we rank the five candidate agents in descending order of their likelihood of being faulty. Top-1 Accuracy measures whether the highest-ranked agent belongs to the ground-truth faulty-agent set, while MRR is calculated as the reciprocal rank of the highest-ranked ground-truth faulty agent.

We indeed identified an inconsistency in our previously reported results. Both metrics should be computed from the same agent ranking for each trajectory and then averaged across test trajectories. However, due to limited computing resources, some experiments were conducted on separate PCs, and results from different experimental settings and agent rankings were inadvertently combined during metric calculation. We apologize for this error and have re-evaluated all experiments under a unified evaluation protocol to ensure consistent and fair comparisons across methods. The updated results resolve the inconsistency identified by the reviewer. Specifically, GPT-5.5 (All-at-Once) now achieves 84.2% Top-1 Accuracy. The updated Top-1 Accuracy implies an MRR upper bound of 92.1%, consistent with the reported 91.5%. We will update all affected experimental results.

Importantly, the corrected results continue to support the effectiveness of AgentSoV. On AEGIS, AgentSoV achieves 46.8% Exact-Set Accuracy and 89.7% Top-1 Accuracy, outperforming the strongest non-AgentSoV baselines by 30.3 percentage points in Exact-Set Accuracy and 2.9 percentage points in Top-1 Accuracy. These results further demonstrate the effectiveness of AgentSoV in identifying multiple faulty agents within complex MAS executions.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer NkLR (2/2)

Weakness 2

We thank the reviewer for the constructive suggestions. To address these concerns, we have added comparisons with relevant baselines for both stage-input influence (new Table 7 in Section 4.3) and virtual-sensor placement (updated Figure 3 in Section 4.4). We have also expanded the baseline descriptions in Appendix E. All revisions are highlighted in blue in the revised manuscript, which are also presented below for the reviewer's convenience.

Stage-input influence: Validation against actual interventions.

  • Section 4.3, Table 7:

We further evaluate whether predicted stage-input influence agrees with measured intervention effects on BFCL. For each intervention, we remove one stage’s input and regenerate the downstream MAS execution. We measure influence as the Euclidean change in aligned downstream measurements. Table 7 shows the Spearman correlations across source-target pairs within each trajectory, averaging across trajectories. AgentSoV achieves the highest mean correlation (0.28), followed by SVR + event KernelSHAP (0.23) and TimeSHAP (Bento et al., 2021) (0.22), while CAIR (Giloni et al., 2025) obtains 0.05. These results provide evidence that AgentSoV’s stage-input influence estimates can help rank downstream intervention effects.

Table 7: Correlation between predicted and measured downstream influence on BFCL. SVR: Support vector regression; GRU: Gated recurrent unit.

MethodCorrelation
Betweenness / Eigenvector-0.16 ± 0.27
SVR Input Occlusion0.18 ± 0.09
GRU Input Occlusion0.15 ± 0.04
SVR + Event KernelSHAP0.23 ± 0.17
CAIR0.05 ± 0.22
TimeSHAP0.22 ± 0.25
AgentSoV0.28 ± 0.22
  • Appendix E:

For stage-input influence baselines, betweenness and eigenvector centrality rank source stages on the diagnostic chain. SVR and GRU input occlusion measure changes in predicted downstream measurements after zeroing one source input. SVR + event KernelSHAP and TimeSHAP-style GRU attribution (Bento et al., 2021) score each input using the norm of its event-level Shapley attribution. CAIR-inspired query transfer (Giloni et al., 2025) retrieves the most similar other task and transfers its measured intervention effects at matched relative stage positions, excluding the evaluated task.

Virtual-sensor placement: Baseline comparisons.

  • Section 4.4, Figure 3:

On BFCL, AgentSoV consistently outperforms the baselines under limited monitoring budgets, achieving approximately 38% accuracy with only 70% of stages, slightly exceeding full-observation performance. On Magentic-One, monitoring just 10% of stages achieves approximately 59% agent accuracy, comparable to using all stages. Regarding yky_k dimensions, BFCL achieves its highest accuracy with all four dimensions, whereas Magentic-One maintains comparable or higher agent accuracy with only one of 16 dimensions.

  • Appendix E:

We compare five sensor-placement baselines. Random uniformly samples budget-sized subsets. FastCan (Wang et al., 2026) greedily selects features using canonical correlation with training labels. SSPOC (Brunton et al., 2016) identifies sparse measurements that preserve discriminative information for classification. TPGR (Klishin et al., 2023) uses two-point greedy selection to balance individual sensor informativeness against pairwise redundancy. POD + pivoted QR (Manohar et al., 2018) selects sensors through QR pivoting on a learned low-dimensional reconstruction basis. Selected measurements are evaluated using AgentSoV’s failure attribution pipeline.

Question 1

Regarding Top-1 Accuracy and MRR, please see our response to Weakness 1.

Regarding the selection of λ\lambda, we have clarified in Appendix E of the revised manuscript: “The sparsity parameter λ\lambda in Eq.14 is selected independently from {0.03,0.1,0.3,1.0,3.0} by maximizing step accuracy on a validation subset drawn exclusively from that fold's training data. Neither the held-out test trajectories nor their labels are used for hyperparameter selection.”

Question 2

Please see our response to Weakness 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates

We sincerely thank the reviewers for their valuable and insightful comments. We have revised the manuscript accordingly, with all revisions highlighted in blue. Below, we summarize our major revisions and clarifications:

  1. Cross-backend generalization. We added experiments (Tables 4-5) evaluating transfer across different LLM backends without retraining. The results demonstrate transferability in several settings, while performance fluctuations may arise from the limited number of fault-free training trajectories. We further clarify that AgentSoV's primary generality lies in its unified diagnostic formulation across heterogeneous MAS architectures.

  2. Justification of the linear assumption. We added Appendix D with a theoretical analysis of linear variation propagation, including a first-order approximation of nonlinear stage transitions, a bound on the approximation error, and conditions under which the approximation is expected to hold. Reviewer 2fEB recognized these additions as “a valuable supplement”, stating that we “made a full supplement in rebuttal” and that the analyses provided “a deeper understanding of the problem.”

  3. Additional baseline comparisons. For stage-input influence, we conducted actual message-intervention experiments and introduced six baselines (Table 7). AgentSoV achieves the highest Spearman correlation (0.28) between predicted influence and measured downstream effects. For virtual-sensor placement, we added five baselines (Figure 3), demonstrating that AgentSoV achieves superior performance under limited monitoring budgets.

We thank all reviewers again for their constructive feedback. We believe these revisions substantially strengthen the paper's theoretical foundation and empirical validation, further demonstrating AgentSoV's value as a unified and computationally efficient diagnostic framework across heterogeneous LLM multi-agent architectures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.