- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ARIA, a framework for detecting and repairing failures during the execution of LLM-based multi-agent systems. The authors construct a dataset of approximately 4,000 execution-verified repair pairs and train a repair model through SFT, DPO, and GRPO to identify failures and generate corrective interventions. Experiments show promising improvements in repair success rates across multiple agent frameworks and benchmarks.
Critical strengths
-
Improving the reliability of LLM-based agent systems is an important and practically relevant problem.
-
The trained repair model demonstrates encouraging improvements over existing approaches.
Critical weaknesses
- The methodological novelty appears limited. The training pipeline primarily combines established techniques, including SFT, DPO, and GRPO, in a relatively standard manner.
- The representativeness of the collected failure trajectories is unclear, particularly given that failure patterns can vary substantially across tasks, models, and agent frameworks.
- The evaluation does not adequately characterize training costs or runtime overhead, including latency and token consumption.
Questions for authors
Thanks for submitting your work. I enjoyed reading the paper and agree that improving the reliability of multi-agent systems is an important research direction. However, I have concerns about the novelty of the proposed approach and the representativeness of the evaluation workloads. Below are my specific comments:
-
Why not leverage real-world software engineering data, such as GitHub issues and their corresponding pull requests, to construct additional failure-repair examples? Such data could provide more realistic repair scenarios, especially for software development agents. It would be interesting to compare the proposed dataset with repairs derived from real-world development workflows.
-
It is unclear how representative the collected failure trajectories are of practical agent workloads. Failure patterns can vary significantly across tasks, agent frameworks, and underlying model capabilities. Although the paper considers multiple host models, the dataset construction process may favor certain types of failures that are relatively easy to repair. How does this selection affect the diversity and difficulty of the resulting dataset?
-
The proposed training pipeline follows a fairly standard SFT–DPO–GRPO recipe. While the execution-based reward design is useful, the overall approach appears incremental.
-
The repair utility score in Proposition 1 follows directly from the definitions of failure probability and conditional repair success probability. The result seems straightforward and may not need to be presented as a formal proposition.
-
The repair models are primarily based on Qwen2.5, which is relatively dated. Evaluating more recent backbone models would help establish whether the proposed training approach remains beneficial as base-model capabilities improve.
-
Constructing the dataset requires generating and validating candidate repairs through execution, followed by multiple post-training stages. What is the total computational cost of data construction and training? At runtime, how much additional latency and token consumption does ARIA introduce?
3 comments
Thanks for the response. However, I feel that my concerns regarding novelty (W1/Q3) have not been adequately addressed, while W2/Q2 remain acknowledged limitations.
Regarding Q5, ARIA does not demonstrate a clear advantage over Spark to Fire, particularly on HIRs. It also remains unclear whether existing methods could achieve better performance with additional training.
Overall, I appreciate the efforts in addressing the feedback, though my concerns remain. My current rating is 2.5. That said, I would not oppose acceptance if other reviewers are supportive.
We thank the reviewer for the follow-up, which gives us the opportunity to clarify.
To W1/Q3: ARIA is novel at two levels. On the data side, ARIA provides the first large-scale dataset that pairs failed multi-agent trajectories with execution-verified repairs (3,953 pairs). Existing corpora such as MAST-Data and Who&When annotate failures but contain no verified repairs. On the method side, each stage is anchored to execution-verified repairs. Stage 2 contrasts verified patches with observed failed continuations, and selects diagnoses by how much they raise the likelihood of verified repairs on related incidents (Eq. 4). Stage 3 rewards proposed repairs by re-executing the system, credits diagnoses by cross-cell transfer, and trains the gate with a matched-seed counterfactual reward that penalizes harmful interventions (Eqs. 5–7). Table 4 shows that each stage is necessary. Stage 2 cuts HIR from 3.13% to 1.49%, and Stage 3 raises SR from 16.6% to 19.2%. We appreciate this concern, as it gave us a valuable opportunity to highlight ARIA’s novelty.
To W2/Q2: A repair-oriented corpus is the motivation of this work, because existing datasets do not connect understanding a failure with recovering from it. This selection does not bias evaluation toward easy cases. Repair-optimality selects source cells, not individual instances. The host models include frontier systems such as GPT-5.5 and Claude-Opus-4.8. Most importantly, repairability filtering applies only to training data. The test set keeps every failure, including those no method can repair, and ARIA still outperforms all baselines on it.
To Q5: ARIA outperforms Spark to Fire by 9.4 SR points (a 21% relative gain) with equal HIR, and halves the HIR of SafeAgent. It also adds much less latency (38.5 s, compared with 90.6 s and 171.5 s for the two baselines) and 30% fewer tokens than Spark to Fire. ARIA therefore matches or beats Spark to Fire on every metric.
We sincerely thank the reviewer for indicating an updated score of 2.5. We hope our responses above have further addressed the reviewer’s concerns and support a more positive assessment, and we would be grateful if the reviewer could reflect this in the official score. If any concerns remain, we are happy to address them promptly.
To W1/Q3: We thank the reviewer for this question. We clarify that our post-training design is specifically developed for ARIA. SFT, DPO, and GRPO are established techniques; Stage 2 DPO corresponds to diagnosis transfer, while Stage 3 GRPO corresponds to cross-cell execution feedback and gate optimization, as shown in Table 4. Thus, the contribution is the execution-grounded ARIA design rather than a new optimizer.
To W2/Q2: We thank the reviewer for raising this important concern. Section 2.1 discusses this issue. We define a repair-optimality principle and use it to collect a large number of samples from four multi-agent frameworks, six benchmarks, and eight host models, which we organize into our datasets. Because this principle targets verified failure-repair supervision, the resulting corpus is designed to support repair rather than to estimate the natural prevalence of all failures.
To W3/Q6: We thank the reviewer for this question. Data construction and the training/inference pipeline are separate stages. Collecting the trajectories and constructing the datasets took several months of data collection and validation. We conducted two cost measurements: full post-training for two backbone models, and matched per-trajectory runtime measurements for Vanilla, Spark to Fire, SafeAgent, and ARIA.
Table A. Training cost (GPU hours).
These values report the GPU time required to complete each post-training stage for one backbone model.
Table B. Runtime cost.
Vanilla denotes uninterrupted execution. Latency is the wall-clock time from the start of execution until the benchmark evaluator returns a result, while tokens sum the
prompt_tokensandcompletion_tokensfrom all model API requests. ARIA has the smallest latency overhead among the repair methods (+38.52 s); its token overhead (+11,536) is below Spark to Fire (+16,590) and above SafeAgent (+6,261), showing a moderate runtime cost. We have added these cost measurements to the Appendix of the revised PDF. We thank the reviewer for this suggestion, which makes the paper more comprehensive.To Q1: We thank the reviewer for this meaningful suggestion. Our current datasets already cover diverse task types, including mathematical problems, question answering, agent benchmarks, and software benchmarks. Since our focus is collecting failure-repair trajectories, we mainly gather data from these established datasets under their existing evaluation protocols. Extending the corpus with real-world issue and pull-request histories would be meaningful, and we will pursue this direction as future work.
To Q4: We thank the reviewer for the suggestion. A definition introduces and fixes the meaning of a quantity or concept, whereas a proposition states a derived claim that follows from the definitions and can be proved under stated assumptions. Proposition 1 records such a relationship for repair utility, it is not intended to claim a new standalone theory.
To Q5: We thank the reviewer for this question. We chose the Qwen2.5 and Llama3.1 families because they provide comprehensive size variants for scaling experiments and are widely used by prior work, enabling comparability. To further test ARIA under the runtime setting, we conducted a matched experiment with the latest Qwen3.5-9B backbone model:
Because of the time limit, we used a subset of the datasets for training and testing. ARIA still achieves the best repair success rate, and the newer Qwen3.5-9B capabilities keep harmful interventions well controlled. Together, these results support ARIA’s improvements under a newer backbone. We have added this new experiment to the Appendix of the revised PDF. We sincerely thank the reviewer for this suggestion, these complementary experiments substantially improve the quality of the paper.
We sincerely thank you for your thoughtful review and are confident that we have addressed all of your concerns, so please feel free to reach out with any further questions, which we will promptly address, and we would be grateful if you would consider raising your score.