acceptodds
Under review as a conference paper at ICLR 2027

ARIA: Runtime Inspection and Amendment for Multi-Agent Systems

Abstract

In LLM-based multi-agent systems (MAS), a local mistake can propagate as other agents build on it, turning a correctable error into a failed task. Diagnosing completed runs or repeatedly re-executing them to search for corrections does not provide a learned policy for intervening while execution is still in progress. We present Agent Runtime Inspection and Amendment (ARIA), a framework for learning when and how to repair ongoing multi-agent executions. Our key intuition is to turn execution-verified repairs into reusable supervision, moving repair search into data construction and training so that a learned model can intervene at runtime without repeatedly testing alternatives. First, we collect over 24,000 successful and failed trajectories across four multi-agent frameworks, six benchmarks, and eight host models. From these trajectories, we construct the ARIA dataset, to our knowledge the first large-scale dataset pairing failed multi-agent trajectories with execution-verified repairs. It contains 3,653 verified repair pairs, each specifying what went wrong, where to intervene, and how execution should continue. Then, we develop the ARIA framework for MAS-agnostic runtime inspection and amendment, designed to integrate with any MAS that exposes execution trajectories and supports continuation replacement. Using verified repairs as training examples, we teach a compact model to recognize when intervention is needed and how to correct an ongoing run. The aim is to learn repair strategies that apply to related failures in new configurations, rather than merely reproduce individual corrections. Finally, the trained ARIA model demonstrates strong repair performance in both post-hoc and runtime settings, with generalization to unseen frameworks and host models. Our code is available at https://anonymous.4open.science/r/ARIA-code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Multi-Agent System, Runtime Inspection and Amendment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_imF8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ARIA, a framework for detecting and repairing failures during the execution of LLM-based multi-agent systems. The authors construct a dataset of approximately 4,000 execution-verified repair pairs and train a repair model through SFT, DPO, and GRPO to identify failures and generate corrective interventions. Experiments show promising improvements in repair success rates across multiple agent frameworks and benchmarks.

Critical strengths
  • Improving the reliability of LLM-based agent systems is an important and practically relevant problem.

  • The trained repair model demonstrates encouraging improvements over existing approaches.

Critical weaknesses
  • The methodological novelty appears limited. The training pipeline primarily combines established techniques, including SFT, DPO, and GRPO, in a relatively standard manner.
  • The representativeness of the collected failure trajectories is unclear, particularly given that failure patterns can vary substantially across tasks, models, and agent frameworks.
  • The evaluation does not adequately characterize training costs or runtime overhead, including latency and token consumption.
Questions for authors

Thanks for submitting your work. I enjoyed reading the paper and agree that improving the reliability of multi-agent systems is an important research direction. However, I have concerns about the novelty of the proposed approach and the representativeness of the evaluation workloads. Below are my specific comments:

  • Why not leverage real-world software engineering data, such as GitHub issues and their corresponding pull requests, to construct additional failure-repair examples? Such data could provide more realistic repair scenarios, especially for software development agents. It would be interesting to compare the proposed dataset with repairs derived from real-world development workflows.

  • It is unclear how representative the collected failure trajectories are of practical agent workloads. Failure patterns can vary significantly across tasks, agent frameworks, and underlying model capabilities. Although the paper considers multiple host models, the dataset construction process may favor certain types of failures that are relatively easy to repair. How does this selection affect the diversity and difficulty of the resulting dataset?

  • The proposed training pipeline follows a fairly standard SFT–DPO–GRPO recipe. While the execution-based reward design is useful, the overall approach appears incremental.

  • The repair utility score in Proposition 1 follows directly from the definitions of failure probability and conditional repair success probability. The result seems straightforward and may not need to be presented as a formal proposition.

  • The repair models are primarily based on Qwen2.5, which is relatively dated. Evaluating more recent backbone models would help establish whether the proposed training approach remains beneficial as base-model capabilities improve.

  • Constructing the dataset requires generating and validating candidate repairs through execution, followed by multiple post-training stages. What is the total computational cost of data construction and training? At runtime, how much additional latency and token consumption does ARIA introduce?

3 comments

W1/Q3: What is novel about the method beyond standard SFT, DPO, and GRPO?

To W1/Q3: We thank the reviewer for this question. We clarify that our post-training design is specifically developed for ARIA. SFT, DPO, and GRPO are established techniques; Stage 2 DPO corresponds to diagnosis transfer, while Stage 3 GRPO corresponds to cross-cell execution feedback and gate optimization, as shown in Table 4. Thus, the contribution is the execution-grounded ARIA design rather than a new optimizer.

W2/Q2: Does selecting validated repairs bias the data toward easier failures?

To W2/Q2: We thank the reviewer for raising this important concern. Section 2.1 discusses this issue. We define a repair-optimality principle and use it to collect a large number of samples from four multi-agent frameworks, six benchmarks, and eight host models, which we organize into our datasets. Because this principle targets verified failure-repair supervision, the resulting corpus is designed to support repair rather than to estimate the natural prevalence of all failures.

W3/Q6: What are the data-construction, training, and runtime computational costs?

To W3/Q6: We thank the reviewer for this question. Data construction and the training/inference pipeline are separate stages. Collecting the trajectories and constructing the datasets took several months of data collection and validation. We conducted two cost measurements: full post-training for two backbone models, and matched per-trajectory runtime measurements for Vanilla, Spark to Fire, SafeAgent, and ARIA.

Table A. Training cost (GPU hours).

Backbone / StageSFTDPOGRPO
Qwen2.5-7B6.8 h1.3 h29.6 h
Llama3.1-8B6.4 h1.2 h27.5 h

These values report the GPU time required to complete each post-training stage for one backbone model.

Table B. Runtime cost.

MetricVanilla (w/o ARIA)Spark to FireSafeAgentARIA
Latency (s)296.01386.59 (+90.58)467.49 (+171.48)334.53 (+38.52)
Tokens26,44943,039 (+16,590)32,710 (+6,261)37,985 (+11,536)

Vanilla denotes uninterrupted execution. Latency is the wall-clock time from the start of execution until the benchmark evaluator returns a result, while tokens sum the prompt_tokens and completion_tokens from all model API requests. ARIA has the smallest latency overhead among the repair methods (+38.52 s); its token overhead (+11,536) is below Spark to Fire (+16,590) and above SafeAgent (+6,261), showing a moderate runtime cost. We have added these cost measurements to the Appendix of the revised PDF. We thank the reviewer for this suggestion, which makes the paper more comprehensive.

Q1: Can issue and pull-request histories provide realistic repair examples?

To Q1: We thank the reviewer for this meaningful suggestion. Our current datasets already cover diverse task types, including mathematical problems, question answering, agent benchmarks, and software benchmarks. Since our focus is collecting failure-repair trajectories, we mainly gather data from these established datasets under their existing evaluation protocols. Extending the corpus with real-world issue and pull-request histories would be meaningful, and we will pursue this direction as future work.

Q4: Is the repair utility expression theoretically novel or just definitional?

To Q4: We thank the reviewer for the suggestion. A definition introduces and fixes the meaning of a quantity or concept, whereas a proposition states a derived claim that follows from the definitions and can be proved under stated assumptions. Proposition 1 records such a relationship for repair utility, it is not intended to claim a new standalone theory.

Q5: Do the gains persist on newer backbone models?

To Q5: We thank the reviewer for this question. We chose the Qwen2.5 and Llama3.1 families because they provide comprehensive size variants for scaling experiments and are widely used by prior work, enabling comparability. To further test ARIA under the runtime setting, we conducted a matched experiment with the latest Qwen3.5-9B backbone model:

MethodSR ↑HIR ↓
Spark to Fire43.75%3.13%
SafeAgent46.88%6.25%
ARIA53.13%3.13%

Because of the time limit, we used a subset of the datasets for training and testing. ARIA still achieves the best repair success rate, and the newer Qwen3.5-9B capabilities keep harmful interventions well controlled. Together, these results support ARIA’s improvements under a newer backbone. We have added this new experiment to the Appendix of the revised PDF. We sincerely thank the reviewer for this suggestion, these complementary experiments substantially improve the quality of the paper.

We sincerely thank you for your thoughtful review and are confident that we have addressed all of your concerns, so please feel free to reach out with any further questions, which we will promptly address, and we would be grateful if you would consider raising your score.

Thanks for the response. However, I feel that my concerns regarding novelty (W1/Q3) have not been adequately addressed, while W2/Q2 remain acknowledged limitations.

Regarding Q5, ARIA does not demonstrate a clear advantage over Spark to Fire, particularly on HIRs. It also remains unclear whether existing methods could achieve better performance with additional training.

Overall, I appreciate the efforts in addressing the feedback, though my concerns remain. My current rating is 2.5. That said, I would not oppose acceptance if other reviewers are supportive.

We thank the reviewer for the follow-up, which gives us the opportunity to clarify.

To W1/Q3: ARIA is novel at two levels. On the data side, ARIA provides the first large-scale dataset that pairs failed multi-agent trajectories with execution-verified repairs (3,953 pairs). Existing corpora such as MAST-Data and Who&When annotate failures but contain no verified repairs. On the method side, each stage is anchored to execution-verified repairs. Stage 2 contrasts verified patches with observed failed continuations, and selects diagnoses by how much they raise the likelihood of verified repairs on related incidents (Eq. 4). Stage 3 rewards proposed repairs by re-executing the system, credits diagnoses by cross-cell transfer, and trains the gate with a matched-seed counterfactual reward that penalizes harmful interventions (Eqs. 5–7). Table 4 shows that each stage is necessary. Stage 2 cuts HIR from 3.13% to 1.49%, and Stage 3 raises SR from 16.6% to 19.2%. We appreciate this concern, as it gave us a valuable opportunity to highlight ARIA’s novelty.

To W2/Q2: A repair-oriented corpus is the motivation of this work, because existing datasets do not connect understanding a failure with recovering from it. This selection does not bias evaluation toward easy cases. Repair-optimality selects source cells, not individual instances. The host models include frontier systems such as GPT-5.5 and Claude-Opus-4.8. Most importantly, repairability filtering applies only to training data. The test set keeps every failure, including those no method can repair, and ARIA still outperforms all baselines on it.

To Q5: ARIA outperforms Spark to Fire by 9.4 SR points (a 21% relative gain) with equal HIR, and halves the HIR of SafeAgent. It also adds much less latency (38.5 s, compared with 90.6 s and 171.5 s for the two baselines) and 30% fewer tokens than Spark to Fire. ARIA therefore matches or beats Spark to Fire on every metric.

We sincerely thank the reviewer for indicating an updated score of 2.5. We hope our responses above have further addressed the reviewer’s concerns and support a more positive assessment, and we would be grateful if the reviewer could reflect this in the official score. If any concerns remain, we are happy to address them promptly.

Reviewer_Cr7MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ARIA is a framework for learning when and how to repair failures in LLM-based multi-agent systems. The paper constructs 3,953 execution-verified repair pairs from over 24,000 trajectories across four MAS frameworks, six benchmarks, and eight host models, linking failure diagnoses to intervention points and corrected continuations. It develops a method, composed of SFT, DPO, and execution-grounded policy optimization, for error diagnosis and repair. Validation covers post-hoc repair and runtime intervention, measuring task success, harmful interventions, and execution costs, with additional held-out-framework evaluations, backbone scaling comparisons, and training-stage ablations.

Critical strengths

The paper addresses an important problem, runtime inspection and amendment of LLM-based multi-agent systems. A notable methodological strength is the use of cross-task repair utility to guide diagnosis learning. ARIA groups related failure incidents using similarities in feedback and repair edits, then evaluates diagnoses by how effectively they support repairs on related incidents. This provides a concrete training mechanism for encouraging transferable diagnoses and effective corrective continuations beyond the original failure.

Critical weaknesses
  1. The results in Figure 4a appear inconsistent with Table 3, although both report runtime generalization across backbone scales under RQ2. For example, Qwen2.5-7B achieves in-domain/unseen-Magentic-One success rates of 20.9%/21.0% in Table 3, versus 18.8%/15.3% in Figure 4a. This discrepancy also affects interpretation. Table 3 suggests essentially no generalization gap at 7B and 14B, whereas Figure 4a shows a consistent decline on held-out Magentic-One. Appendix B.4 mentions separate subsets across research questions but does not clarify whether these two RQ2 results use different evaluation sets or protocols. Please specify the task subsets, sample sizes, checkpoints, seeds, and aggregation procedures underlying each result, and reconcile the reported values.
  2. The paper does not clearly report the number of agents or the role configuration used in each MAS framework, nor whether these vary across tasks. Please provide these details to improve reproducibility. While an additional system-size experiment is not essential, I would be interested in the authors’ perspective on how ARIA’s performance and inspection overhead might change as the number of agents increases, particularly given the fixed observation budget and limited intervention scope.
  3. The experiments demonstrate runtime repair performance, but do not clearly establish its distinct value over post-hoc repair. The two settings use different evaluation populations, making their success rates difficult to compare directly. It would strengthen the paper to evaluate both approaches on matched tasks and quantify benefits specific to early intervention, such as preventing downstream error propagation or avoiding wasted execution, while accounting for inspection overhead and harmful interventions.
  4. At the beginning of Section 2.2 (around lines 207–208), the notation yi=(di,ki,ei)y_i=(d_i,k_i,e_i) and D={(xi,yi)}i=1N\mathcal{D}=\{(x_i,y_i)\}_{i=1}^{N} is introduced without explicitly defining xix_i, did_i, kik_i, and eie_i. Please define these variables when they first appear
  5. There appear to be typographical errors in the indicator-function notation around lines 218 and 246–247.
Questions for authors

See Weaknesses.

2 comments

W1. Table 3 and Figure 4a give inconsistent RQ2 numbers, and the settings behind each are unclear.

To W1: We thank the reviewer for this very careful observation. Table 3 and Figure 4a use different pools: Table 3 evaluates on unseen Magentic-One, whereas Figure 4a holds out both Magentic-One and AG2. They therefore use the same checkpoint, which also explains why the in-domain results in Figure 4a are identical. We have updated the PDF, and the revised appendix now states these two pool definitions explicitly.

W2. The number of agents and the role configuration in each MAS framework are not reported, and how ARIA’s performance and inspection overhead change with system size.

To W2: Thank you for this insightful question. OpenManus uses 3 role labels, MetaGPT 4 roles, Magentic-One 4 roles, and AG2 3 roles. The full specifications are in the revised appendix.

ARIA’s overhead scales with the number of messages, not the number of agents. Each gate check runs once on a prefix truncated to a fixed token budget, so total cost grows linearly with the number of messages, and localization, attribution, and repair run at most once per run without checkpointing or replay. As the system grows, the fixed budget covers less of the history, so far-upstream root causes are more likely to be missed. A missed failure simply leaves the run unchanged, so limited observation can shrink ARIA’s gain but never push performance below the unassisted baseline.

W3. Runtime and post-hoc repair are evaluated on different populations, so the distinct value of runtime repair is not established.

To W3: Thank you for this question. We agree that the two success rates are not directly comparable, and the reason they differ is the distinct value of runtime amendment. Post-hoc repair works with more information. It receives evaluator-confirmed failures together with evaluator feedback, which assumes an oracle at deployment that identifies which runs failed and why. Runtime amendment has no such verdict. It must decide on a partial trajectory whether to intervene, and it risks damaging runs that would otherwise succeed. Post-hoc repair is therefore best viewed as a verdict-assisted reference rather than a competing deployment mode. Runtime amendment’s contribution is recovering failures without a verdict and within a single execution.

W4/W5. Some variables in Section 2.2 are undefined, and the indicator-function notation has typos.

To W4/W5: We thank the reviewer for the careful reading. We have fixed both issues and uploaded a revised PDF.

We sincerely thank you for your thoughtful review and are confident that we have addressed all of your concerns, and we would be happy to promptly address any further questions you may have.

Thanks for the response. I will maintain my score.

Reviewer_AxEtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ARIA, a framework for detecting and repairing failures in LLM-based multi-agent systems. It moves the search for effective repairs into offline data construction and training, so a learned model can intervene during execution without repeatedly testing candidate repairs.

The authors construct 3,953 execution-verified repair pairs from over 24,000 trajectories. Each record specifies the failure, where to intervene, and how execution should continue. A three-stage training procedure teaches a compact model to generate repairs, while an inspection gate decides when to intervene.

Experiments evaluate post-hoc repair, runtime intervention, and transfer to held-out frameworks. The authors report higher task success and fewer harmful interventions than the evaluated baselines.

Critical strengths
  • Repair data validated through execution. The 3,953 repair pairs link failure diagnoses and intervention points to corrections verified through execution. A repair is accepted only when the patched run succeeds and an unpatched control fails (Appendix B.3). Tasks are separated across data splits to limit leakage (Appendix B.4). These checks make the dataset useful for training failure-recovery models.

  • The experiments test both successful recovery and the risk of harmful intervention. ARIA reports higher success in both post-hoc and runtime repair. The runtime evaluation also reports fewer harmful interventions, addressing the risk that a repair disrupts an otherwise successful run. Held-out tests on Magentic-One and AG2 provide preliminary evidence of transfer (Table 3 and Figure 4a). Table 4 also shows gains from each training stage.

Critical weaknesses
  • Limited native runtime repair capability and insufficient justification for the single-intervention constraint. ARIA constructs repair supervision by selecting historical cut points and generating replacement continuations, potentially involving multiple events (Appendix B.3). However, its native runtime implementation only supports a single pending plain-text message substitution per execution (Appendix B.6), without historical rollback, tool-call replacement, or multi-event continuation. The paper does not sufficiently analyze how this discrepancy affects the applicability of learned repairs in real runtime settings. Moreover, the motivation for restricting each execution to a single intervention is unclear. Since errors may arise at multiple stages of a long-running multi-agent task, limiting ARIA to one intervention could restrict its recovery capability. An ablation varying the maximum number of interventions (e.g., 0, 1, 2, and more) would help determine whether this restriction is justified and how additional interventions affect task success, harmful intervention rates, and execution overhead.
  • Insufficient experimental controls to establish the contributions of ARIA-specific mechanisms and the fairness of baseline comparisons. Although Table 4 shows improvements from combining SFT, DPO, and GRPO, the experiments do not isolate the contributions of ARIA-specific mechanisms, such as diagnosis transfer, cross-cell execution feedback, and gate optimization. It remains unclear whether the reported gains primarily arise from these proposed mechanisms or from applying standard post-training techniques to execution-verified repair data. Furthermore, Appendix B.6 indicates that the baselines are evaluated using repository-local adaptations rather than their original implementations, potentially restricting their capabilities. The runtime evaluation also lacks a direct comparison with uninterrupted multi-agent executions, making it difficult to assess the net improvement attributable to ARIA. These limitations weaken the evidence supporting the paper's methodological contributions and comparative performance claims.
Questions for authors
  • Why is ARIA restricted to a single intervention per execution, and could the authors provide an intervention-budget ablation and report the proportion of repair proposals actually enacted?

  • Could the authors provide ablations isolating the contributions of diagnosis transfer, cross-cell execution feedback, and gate optimization?

  • How do the adapted baselines preserve the original methods' capabilities, and what is ARIA's net improvement over uninterrupted host executions?

  • Why does Table 3 report 21.0% success for Qwen2.5-7B on the unseen system, while Figure 4a reports 15.3% for held-out Magentic-One?

  • Do ARIA's improvements persist when using newer LLMs, under matched training and inference budgets?

4 comments

W1/Q1: How does the single-intervention limit affect recovery, and what changes with larger budgets?

To W1/Q1: We thank the reviewer for raising this important question. Table 2 fixes EB=1.00EB=1.00 for every method to ensure a fair matched comparison; EB counts complete multi-agent executions per task. Figure 3b gives the detailed EB sweep, where ARIA leads at every tested budget, showing that its gains do not come from extra replay. We also appreciate the discussion of “single message substitution.” This is central to ARIA’s runtime contribution: it inspects an incomplete execution prefix and directly replaces the pending response before an error propagates, unlike historical rollback or post-hoc repair. We are glad that the reviewer raised these questions, giving us a valuable opportunity to further reinforce ARIA’s contribution.

W2/Q2: Could the authors provide ablations isolating the contributions of diagnosis transfer, cross-cell execution feedback, and gate optimization?

To W2/Q2: We thank the reviewer for this careful observation. We clarify that our entire post-training design is specifically developed for ARIA. Stage 2 DPO corresponds to diagnosis transfer, while Stage 3 GRPO corresponds to cross-cell execution feedback and gate optimization, as shown in Table 4.

W2/Q3: How do the adapted baselines preserve the original methods' capabilities, and what is ARIA's net improvement over uninterrupted host executions?

To W2/Q3: Thank you for raising this question. Our baselines use their original implementations, and our code repository documents the corresponding repository and commit version for each method. To control comparability across methods, we constrain shared conditions such as execution budgets. Tables 1 and 4 already show ARIA’s improvement over uninterrupted host execution, demonstrating its strong performance.

Q4: Why do Table 3 and Figure 4a report different success rates?

To Q4: We thank the reviewer for this very careful observation. Table 3 and Figure 4a use different pools: Table 3 evaluates on unseen Magentic-One, whereas Figure 4a holds out both Magentic-One and AG2. They therefore use the same checkpoint, which also explains why the in-domain results in Figure 4a are identical. The revised appendix now states these two pool definitions explicitly.

Q5: Do the gains persist on newer backbone models?

To Q5: We thank the reviewer for this question. We chose the Qwen2.5 and Llama3.1 families because they provide comprehensive size variants for scaling experiments and are widely used by prior work, enabling comparability. To further test ARIA under the runtime setting, we conducted an experiment with the latest Qwen3.5-9B backbone model:

MethodSR ↑HIR ↓
Spark to Fire43.75%3.13%
SafeAgent46.88%6.25%
ARIA53.13%3.13%

Because of the time limit, we used a subset of the datasets for training and testing. ARIA still achieves the best repair success rate, and the newer Qwen3.5-9B capabilities keep harmful interventions well controlled. Together, these results support ARIA’s improvements under a newer backbone. We have added this new experiment to the Appendix of the revised PDF. We sincerely thank the reviewer for this suggestion, these complementary experiments substantially improve the quality of the paper.

We sincerely thank you for your thoughtful review and are confident that we have addressed all of your concerns, so please feel free to reach out with any further questions, which we will promptly address, and we would be grateful if you would consider raising your score.

As the discussion period is approaching its end, we would very much appreciate hearing your thoughts on our rebuttal, so please let us know if you have any remaining questions, and we would be happy to address them promptly.

Thank you for the authors' responses and the additional Qwen3.5-9B experiment. However, my main concerns remain unresolved.

1. W1+Q1:

The authors refer to Figure 3b and explain ARIA's single-message substitution mechanism. However, this does not address the original concern. Execution Budget measures the number of complete multi-agent executions per task, whereas Intervention Budget refers to the number of interventions within a single execution. Thus, Figure 3b does not show whether allowing multiple interventions would improve recovery performance.

The rationale for the single-intervention constraint remains unclear. It would also be useful to report the proportion of proposed repairs actually enacted at runtime.

2. W2+Q2:

Table 4 provides ablations across different training stages, but mainly shows their combined effects. It does not isolate the individual contributions of ARIA's specific mechanisms. More targeted ablations or further clarification are needed to establish the contributions of these mechanisms.

3. W2+Q3:

The authors state that the baselines use their original implementations, whereas Appendix B.7 describes them as repository-local adaptations rather than official implementations. It remains unclear whether these adaptations preserve the core capabilities of the original methods and whether this affects the comparison.

Furthermore, Table 9 reports runtime and token overhead but lacks a direct comparison of task success rates with uninterrupted host executions. Such a comparison under matched conditions is needed to clarify ARIA's actual improvement over the original execution process.

We thank the reviewer for the follow-up, which gives us the opportunity to clarify.

To W1/Q1: We had misunderstood the intervention budget (IB) in our previous response. To support multiple interventions, we extended the native adapters. We have now varied the maximum number of interventions enacted per execution over IB ∈\in {0, 1, 2, 3}, under the runtime setting of Table 2 with the Qwen2.5-7B backbone and EB = 1.00. IB = 0 denotes uninterrupted host execution, and Enacted / Proposed denotes the fraction of repair proposals that are actually enacted at runtime.

IBSR ↑HIR ↓Enacted / Proposed
012.9%0.00%-
121.4%2.00%68.4%
221.9%2.71%66.1%
321.1%3.58%63.7%

A single intervention recovers most of the gain over uninterrupted execution, while a second intervention adds only a marginal improvement at the cost of a higher HIR. Beyond this point SR fluctuates while HIR keeps rising. Later interventions act on executions that earlier substitutions have already redirected, so they are more likely to overwrite a continuation that was already correct. One or two interventions therefore suffice, and IB = 1 gives the best balance between recovery and harm at the lowest overhead.

To W2/Q2: We further decompose post-training into the phases described in Appendix B.6, and remove one phase at a time on a smaller subset of the Table 4 evaluation set with Qwen2.5-7B. Stage 2a is repair DPO, Stage 2b is diagnosis transfer, Stage 3a is cross-cell execution feedback (w/o set α=0\alpha=0), and Stage 3b is gate optimization.

VariantIn-domain SR ↑In-domain HIR ↓Unseen SR ↑Unseen HIR ↓
Stage 1 only12.8%9.10%11.2%10.4%
w/o Stage 2a18.1%2.36%17.3%2.94%
w/o Stage 2b18.5%1.92%16.9%2.87%
w/o Stage 3a17.9%1.88%15.6%2.81%
w/o Stage 3b18.6%3.42%17.5%4.06%
ARIA (full)20.2%1.21%18.7%2.55%

Removing any single phase degrades both SR and HIR, so every mechanism contributes. Removing diagnosis transfer or cross-cell feedback causes the largest SR drops on the unseen system, and since the latter variant still retains standard repair GRPO, its drop is attributable to the cross-cell term itself. Removing repair DPO or gate optimization causes the largest HIR increases. The gains therefore come from ARIA-specific mechanisms rather than from standard post-training alone.

To W2/Q3: To clarify, for every baseline with publicly released code, we take the source code directly from its official repository and record the exact commit. We retain all of the core code of each such method and only add repository-local adapters that connect it to the shared event and execution interfaces, so that all methods can be evaluated under the same protocol. This is what “repository-local adaptations” refers to in Appendix B.7. For baselines without publicly released code, we implement the method faithfully following the descriptions in its original paper. All code and adaptation details are available in our anonymous repository. We have also revised the baseline description in Appendix B.7 to state this more accurately.

Table 9: We have added the SR of each method, including uninterrupted execution, to Table 9. These values are measured on the same evaluation subset as the cost measurement.

MetricVanilla (w/o ARIA)Spark to FireSafeAgentARIA
SR ↑13.3%14.2%17.5%21.7%
Latency (s)296.01386.59 (+90.58)467.49 (+171.48)334.53 (+38.52)
Tokens26,44943,039 (+16,590)32,710 (+6,261)37,985 (+11,536)

ARIA achieves the highest SR and the smallest latency overhead among the repair methods. All of the above changes have been incorporated into the revised PDF.

We sincerely thank the reviewer for the valuable suggestions, which have helped us further strengthen the paper. If the reviewer has any further questions, please feel free to let us know, and we will address them promptly.

Comments on the paper

2 comments
Area_Chair_vTC5OpenReview ↗

Please engage in the discussion

Reviewers: if you have not responded to the authors' reply, please read it and engage in the discussion. Thanks!

AuthorsOpenReview ↗

Overall Response and Summary of Manuscript Revisions

Dear Area Chair and Reviewers,

We sincerely thank Reviewers AxEt, Cr7M, and imF8 for their thoughtful feedback. We have responded to each reviewer and incorporated the revisions below into the updated manuscript. All manuscript revisions are highlighted in red.

1. Recognized Strengths

  • Important problem and clear presentation: The paper addresses runtime inspection and amendment for improving the reliability of LLM-based multi-agent systems, with a clear and easy-to-follow presentation.
  • Execution-verified repair data: Each repair is accepted only when the patched run succeeds and an unpatched control fails, with task-level splits that limit leakage, making the dataset useful for training failure-recovery models.
  • Transfer-oriented repair learning: Cross-task repair utility evaluates diagnoses by how well they support repairs on related incidents, encouraging transferable diagnoses and effective corrective continuations.
  • Effective and safe recovery: ARIA improves task success over existing methods while reducing harmful interventions, transfers to held-out frameworks, and benefits from each training stage.

2. Summary of Manuscript Revisions

  • Section 2.2 (Cr7M-W4, W5): Defined x_i, d_i, k_i, and e_i at their first appearance and corrected the indicator-function notation.
  • Appendices B.4 and B.5; Table 6 (AxEt-Q4; Cr7M-W1, W2): Stated that Table 3 evaluates on unseen Magentic-One whereas Figure 4a holds out both Magentic-One and AG2, and reported the agent roles of each framework (3, 4, 4, and 3 role labels for OpenManus, MetaGPT, Magentic-One, and AG2).
  • Appendix B.10; Table 7 (AxEt-Q5; imF8-Q5): On the newer Qwen3.5-9B backbone, ARIA achieves the highest SR of 53.13%, a 21% relative gain over Spark to Fire with equal HIR, and halves the HIR of SafeAgent.
  • Appendix B.11; Tables 8–9 (imF8-W3, Q6): Reported training and runtime costs. Complete post-training takes 37.7 GPU hours for Qwen2.5-7B, and ARIA adds the smallest latency overhead among repair methods (+38.52 s) with 30% lower token overhead than Spark to Fire.
  • Sections 2.2 and 3.4; Table 4 (AxEt-W2, Q2; imF8-W1, Q3): Clarified that every post-training stage is anchored to execution-verified repairs, with Stage 2 reducing HIR from 3.13% to 1.49% and Stage 3 raising SR from 16.6% to 19.2%.

We welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.