- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the distinction between failure attribution and effective repair in LLM-based multi-agent systems, introducing the concept of source–repair mismatch (SRM) to characterize cases where the error source differs from the location of an available correction. The authors formalize this phenomenon and construct a benchmark with event-level responsibility annotations and checker-based repair evaluation. They further propose RAISER, a responsibility-aware repair selection framework that combines trace evidence graphs, learned role estimation, candidate ranking, and LLM-based verification to select corrective answers from failed execution traces. Experiments across multiple datasets evaluate the prevalence of SRM and the effectiveness of responsibility-guided repair selection.
Critical strengths
S1. Important Distinction Between Failure Attribution and Effective Repair
The paper identifies an important limitation of attribution-first agent debugging: accurately locating where an error originated does not necessarily identify where an effective repair should be applied. This distinction is particularly relevant to multi-agent systems, where downstream agents may correct or compensate for upstream errors, making the optimal repair location different from the original error source. Also, such a source-repair mismatch may lead to shallow intervention.
S2. Formalization and Empirical Investigation of Source–Repair Mismatch
I appreciate that the paper goes beyond qualitative observations by formally defining source–repair mismatch and systematically investigating its prevalence and implications (although the current formulation seems to have flaws). The authors construct a benchmark with responsibility annotations and checker-based repair evaluation, and conduct controlled experiments and sensitivity analyses to support their findings.
Critical weaknesses
W1. Limited Generalizability Beyond Final-Answer Replacement
The paper primarily evaluates intervention-based repair by selecting an already-logged correct answer to replace the failed final decision. However, this is a highly constrained setting that differs substantially from realistic agent debugging, where interventions typically modify earlier actions and require re-executing subsequent steps.
W2. Limited Reliability and Cross-Dataset Generalizability of Responsibility Estimation
RAISER relies heavily on learned responsibility scores to guide repair selection, yet the Role Model achieves relatively modest classification performance, and the authors acknowledge weak cross-dataset transfer. Although the paper demonstrates improvements over several baselines, it remains unclear whether the logistic regression-based responsibility estimation captures generalizable failure patterns or dataset-specific characteristics. The lack of comprehensive cross-dataset evaluation, particularly regarding downstream repair performance, raises concerns about the robustness and broader applicability of the proposed framework.
Questions for authors
Q1. Generalizability Beyond Final-Answer Replacement
The paper primarily evaluates intervention-based repair by selecting an already-logged answer to replace the failed final decision. However, this setting is substantially different from realistic agent debugging, where interventions may modify earlier actions and require re-executing subsequent steps. To what extent do the source–repair mismatch findings and RAISER's improvements generalize to such settings?
Q2. Reliability and Cross-Dataset Generalization of Responsibility Estimation
RAISER relies on learned responsibility scores from logistic regression models, but their absolute classification performance is relatively modest, and Appendix C acknowledges weak cross-dataset transfer. How reliably can these models identify responsibility roles in unseen agent frameworks or benchmark distributions?
Q3. Availability and Reliability of Contracts and Obligations
The framework assumes access to agent contracts and cross-agent obligations to construct responsibility evidence, particularly for identifying missed verification duties. How are these obligations obtained in practice, and how reliably can they be extracted from ordinary multi-agent execution traces?