- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes REPA, a framework for modeling error propagation in multi-agent LLM systems. REPA uses LLM judges to quantify semantic differences between reference and perturbed agent trajectories, then learns local linear models to capture how errors propagate through agent interactions. By composing these models along the workflow, REPA predicts system-level error propagation and guides targeted interventions, such as prompt refinement and RAG. Experiments across multiple LLMs, MAS workflows, and datasets show promising results in prediction correlation, fault localization, and system recovery.
Critical strengths
-
Meaningful problem. The paper addresses an important problem of understanding error propagation in multi-agent LLM systems.
-
Interpretable framework. REPA provides a structured approach to modeling local error responses and predicting system-level error propagation.
-
Promising experimental results. The method achieves strong improvements in prediction correlation and demonstrates practical benefits in fault localization and targeted recovery.
Critical weaknesses
-
Validity and cost of error measurement. The framework relies on LLM judges to quantify semantic differences between agent trajectories. It is unclear whether these numerical scores can faithfully capture complex error propagation. Additionally, the cost of LLM-based evaluation could be substantial for larger MAS workflows.
-
Limited MAE improvement. Despite significant improvements in Spearman correlation, the reduction in MAE is relatively modest. This raises concerns about the framework's ability to accurately predict the magnitude of error propagation.
-
Insufficient justification for linear modeling. The framework relies on linear transformations to model error propagation. Given the complexity of MAS interactions, it is unclear whether this assumption is sufficient. More empirical justification or comparisons with nonlinear alternatives would strengthen the paper.
Questions for authors
-
Could the authors provide additional validation of the numerical error measurements and clarify the associated evaluation costs?
-
Could the authors provide instance-level prediction results to better evaluate the accuracy of error magnitude estimation?
-
Have the authors considered nonlinear alternatives to validate the linear modeling assumption?
-
Could the authors clarify their reproducibility plans, including whether the code, datasets, and evaluation pipelines will be released?
3 comments
Thank you for the detailed response and additional validation. However, concerns regarding measurement reliability, limited MAE improvements, and the lack of nonlinear comparisons remain only partially addressed. Therefore, I will maintain my current rating.
Dear reviewer, The authors first thank the reviewer for your careful review. We add a nonlinear comparison on Q9 using polynomial regression with ridge regularization. We use a second-degree polynomial with squared and interaction terms.
| Method | Spearman ↑ | MAE ↓ |
|---|---|---|
| REPA-Polynomial | 0.80 | 0.28 |
| REPA (ours) | 0.85 | 0.23 |
REPA achieves a higher correlation and a lower MAE in this comparison. One possible explanation is overfitting. The additional polynomial terms increase the number of parameters and may fit noise in the available training data, even with ridge regularization. These results support our use of the first-order approximation.
The authors also wish to clarify that relatively few studies directly connect local agent errors to system-level propagation predictions, as discussed in Appendix B. The current results provide empirical support for the proposed error propagation model in multi-agent systems. We will extend this work with additional experiments beyond the scope of the present conference paper.
The authors thank the reviewer for your careful review and helpful suggestions.
1. Validity and cost of error measurement
REPA measures changes in error severity between original and perturbed runs using fixed scoring rules for each channel. In Fig. 2a, we form the joint reference by averaging A/B results within GPT and Claude, then averaging the two models. CCC measures agreement in both the pattern and size of these changes. These experiments support the observer's reliability and cost efficiency in error measurement. We also conduct a human validation study. Our human check covers 50 text pairs. Ten people each rate 10 pairs independently on a 0–4 scale, giving 100 pair-level ratings. We use the mean human change for each pair as the reference.
The first two rows report model agreement; the last two compare the observer measurements with human judgments. Against humans, DS is 0.04 below Claude on Fact and 0.06 below it on Hallu. Fig. 2a estimates observer scoring costs at USD 0.17 per 100 question pairs for DS and USD 0.38 for GLM. Both observers have the same Fact/Hallu CCC in Fig. 2a, so DS provides these agreement levels at a lower cost. For larger workflows, the cost of scoring paired runs grows with the number of nodes, rounds and the length of messages. Once the maps are fitted, forward prediction uses source measurements and numerical computation without LLM judging at each downstream node. Together, these results support our use of DS as the observer in REPA. DS combines agreement with human ratings on Fact and Hallu with a lower scoring cost.
2. MAE and instance-level predictions
As described in the paper, REPA estimates statistical error-propagation patterns across samples. It does not identify a separate propagation rule for each individual sample. MAE evaluates the predictions of this shared model on individual test records. For each question and test condition, we compute the absolute error in each of the four endpoint channels, then average across channels and records. Spearman measures the ranking of condition-level gains. A large ranking improvement therefore need not imply an equally large MAE reduction. REPA has the lowest MAE among the four methods in Table 1. Both Fact and Hallu also have lower MAE than other methods.
3. Why use local linear maps?
Agent behavior can be nonlinear. REPA models average error responses across paired samples. For a smooth local mean response, a first-order Taylor expansion gives a linear term plus higher-order effects. This motivates the local linear maps in Section 3.2. We fit these maps with nonnegative ridge regression. Their residuals include higher-order effects, measurement error and fitting error. The first-order form uses few parameters and lets us combine local maps along the workflow. The prediction results in Table 1 and fault-localization results in Fig. 4 support its practical use in the tested settings. Therefore, we use a first-order approximation to study error propagation in this work.
4. Code, data and evaluation pipeline
We will release the code, datasets and evaluation pipeline. It also provides clear instructions and keep expanding the datasets to cover more tasks and agent settings.