acceptodds
Under review as a conference paper at ICLR 2027

MAS-Gate: Unlocking Multi-Agent System Lifecycle via Node-Centric Quantification

Abstract

Large language model (LLM)-based multi-agent systems (MAS) can solve complex problems beyond the reach of a single agent by decomposing tasks and coordinating complementary capabilities. Despite extensive research on MAS, existing work largely focuses on specific aspects or individual phases of their development and operation, resulting in fragmented perspectives with objectives and signals that are difficult to reconcile across these different settings. To bridge these disconnected perspectives, we conceptualize MAS development and operation as a three-phase lifecycle spanning design-time orchestration before execution, runtime intervention during execution, and post-hoc attribution after execution, and introduce MAS-Gate, a unified node-centric quantification applicable across these phases. Modeling each execution as a directed acyclic graph, MAS-Gate combines within-node utility, which estimates a node's contribution through changes in reference-answer likelihood, with inter-node strength, which approximates semantic uptake and potential downstream reach. We adapt this shared quantification to three objectives: providing a structure-aware dense reward for Orchestrator training, training an Editor to act on execution prefixes using paired rewards from completed edited and unedited trajectories, and supplying risk-ranked nodes as auxiliary evidence for LLM-based post-hoc attribution. Experiments spanning interaction environments, knowledge-intensive multi-hop question answering, general reasoning, and post-hoc attribution demonstrate that MAS-Gate improves workflow construction, targeted runtime editing, and LLM-assisted error localization over corresponding phase-specific baselines. Together, these results establish MAS-Gate as a unified structural interface for improving MAS generation, dynamic editing, and error attribution throughout the lifecycle.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent System

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dZ97OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce MAS-Gate, a unified node-centric quantification that provides a common formulation for the distinct objectives of the MAS lifecycle. MAS-Gate evaluates the role of an agent from two dimensions: within-node utility and inter-node strength. Based on this, MAS-Gate constructs node influence scores and applies them to three scenarios. Experiments demonstrate improved orchestration and generalization, effective trained runtime intervention, and gains in assisted attribution across most evaluated settings, supporting the use of a shared quantification mechanism for these distinct objectives.

Critical strengths

1.While existing studies often focus on multi-agent workflow design, runtime intervention, or post-hoc error attribution separately, MAS-Gate attempts to connect these three stages through a unified node quantification framework. 2.MAS-Gate evaluates the role of agents from two dimensions: Within-node Utility and Inter-node Strength, which provides a unified quantitative index for analyzing the positive contributions and negative influences in multi-agent cooperation. 3.Experimental results demonstrate the improvement of MAS-Gate over existing baselines in orchestration and failure localization across interactive environments, multi-hop question answering, general reasoning, and failure attribution.

Critical weaknesses

1.MAS-Gate multiplies the internal utility of nodes by the influence strength between nodes to construct the node influence score. This design is intuitive, but it still relies on a heuristic combination. 2.Although MAS-Gate brings workflow design, runtime intervention, and post-hoc error attribution into a unified framework, the three phases are currently still optimized independently. The paper also lists cross-stage feedback and joint optimization as future research directions in the conclusion. 3.184 execution traces of the Who&When dataset and 137 single target test samples selected from the AEGIS dataset are used in the experiment. This provides preliminary evidence to validate the method, but experiments on joint failures caused by multiple nodes and on the interactions of multiple error sources remain insufficient.

Questions for authors

1.Please clarify why the node impact is defined as Ii=Ui(1+Ki)I_i=U_i(1+K_i), and why multiplication better reflects the actual contribution of the agent. 2.The paper uses an NLI model to estimate the semantic entailment probability RijR_{ij} between the outputs of an upstream agent and a downstream agent, using it to measure how much upstream information is absorbed by downstream agents. How can it prove that the downstream agent actually uses the information of the upstream agent? 3.The three modules of the method are trained independently, so what additional advantages does the MAS lifecycle framework bring besides sharing the same scoring mechanism?

8 comments
AuthorsOpenReview ↗

Response to Reviewer dZ97

[W1, Q1] Rationale for multiplicative impact

This multiplicative composition follows the distinct roles of the two quantities: utility determines direction, whereas strength scales magnitude. Accordingly, I=U(1+K)I=U(1+K) preserves the sign of UU and retains local utility when K=0K=0. We therefore adopt this formulation to combine them. The controlled composition ablation (Multiplicative vs Additive) in Appendix D.1.1 supports the effectivess of multiplication design.

[W2, Q3] Benefits of joint lifecycle optimization

Following your suggestion, we add joint design/runtime training, alternating Orchestrator updates from edited execution outcomes with Editor updates on the current workflows. Each policy is evaluated under its original experimental setting for direct comparison of each model: the Orchestrator runs without an Editor, and the Editor uses the original frozen Qwen3-8B Orchestrator for both separate and joint training comparisons. Both policies improve, demonstrating benefits from coordinated training beyond sharing a metric. We have incorporated this experiment into Section 4.2.3 and Appendix D.2.2 of the revised paper.

Evaluated policyTrainingHotpotQA (%)BCP (out-of-task, %)
OrchestratorSeparate69.5013.69
OrchestratorJoint72.0016.67
EditorSeparate75.5015.48
EditorJoint78.0019.64

[W3] Localization of multiple error sources

We extend evaluation to 463 AEGIS trajectories containing multiple faulty agents. Direct and Guided use Qwen3-30B-A3B-Instruct-2507 and gold references, with a guidance budget of k=4k=4. Accuracy requires an exact match of the predicted and annotated agent sets. Guided improves overall accuracy and every target-count group, extending the effectiveness of MAS-Gate to multi-source localization. We have incorporated these results into Appendix D.3.3 of the revised paper.

TargetsExamplesDirect (%)Guided (%)
All multi-target46318.1419.87
222127.6028.05
31997.049.05
44320.9327.91

[Q2] Human validation of semantic support

As for the validation of RR, we complement Figure 1(b)'s qualitative analysis with human validation on 50 HotpotQA trajectories. Three blinded annotators (postgraduates in AI, don't know our paper before) achieve Krippendorff's α>0.8\alpha>0.8 for both quantities, indicating high agreement and reliable annotations. The correlations with mean human ratings confirm alignment with the intended semantic roles of RR. Together, the case analysis and human evaluation yield complementary qualitative and quantitative evidence for the shared quantification used across the three lifecycle applications. We have incorporated these results into Section 3.4 and Appendices D.1.3 and D.1.5 of the revised paper.

QuantityInter-annotator α\alphaMetric–human Spearman's ρ\rho
RR0.80030.7122

Dear Reviewer dZ97,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. Your insights have been instrumental in refining our work.

To address your concerns, we have provided additional clarifications and experiments to enhance the clarity of our contributions. We hope these responses effectively resolve any ambiguities, provide a more comprehensive understanding of our work, and encourage you to reconsider your rating.

If you have any further questions or require additional clarification, we would be happy to discuss them. We truly appreciate your time and consideration and look forward to your feedback.

Best regards,

The Authors

Dear Reviewer dZ97,

Thank you again for your time and constructive feedback.

With about one day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.

We would be happy to clarify any remaining questions and welcome your further feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_xJxpOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a single score as the measure of performance for three different stages in multi-agent systems, i.e., system design, on the fly corrections, and spotting the failures after they happen. Authors claim that the combination of usefulness of an agent's output for getting a more accurate final result and also how widely the other agents use the agent's output can define a unified score to study and improve all those three stages. They use the combination of these two metrics and use it as the overall impact measure for each agent. The reported results show that when they train an agent to design a multi-agent workflow using this parameter as part of the reward, it shows a higher performance. Also better performance is reported for the execution time correction phase and also afterwards failure spotting, in which hints from this score about the ranking of potentially faulty steps is used.

Critical strengths

In my opinion, the most important strength point for this paper is using a cheap score that can contribute effectively to improve all three stages. Also, authors have shown that this score clearly improves the performance for the workflow designing stage compared to the case that only final output correctness is used as the reward. This could be an interesting and promising finding for a RL setting.

Critical weaknesses

I think there is an important issue in the reported benchmarking results for the workflow design stage performance. Seems like some of the methods that the authors' proposed method has been compared against them, are showing notably bad performance in the reported results. I couldn't find much details about how those methods have been utilized in the benchmarking, so there might be a chance that their poor performance comes from using some weaker models, or with less compute budget or less adaptation for these tasks. Also, the small size of some of the test sets (I also couldn't find repeated run results to be reported) makes it a bit difficult to judge about a meaningful performance gain over the baselines.

Questions for authors
  • Following up on what I mentioned in the weakness section, I would like to see more details on how the baselines in the workflow design stage have been set up. It would help if authors can provide more details on model type, compute details, like number of LLM calls, etc., hyperparameter settings, etc. The current results that show much lower performance for some baselines (even sometime worse than a plain chain of thought) makes it a bit unclear that what is the root cause for these outcomes.

  • Due to small size of some of the test sets, I think it is necessary that authors perform multiple runs and report more detailed statistical results in terms of confidence or standard deviation for tables 1, 2 and 3.

  • As I understand the proposed method for the failure spotting part, the within-node utility defined by the authors need the correct final answer. I think this is ok for the training stage, but could authors elaborate more on necessity of knowing this final correct answer for the proposed method to be useful in real settings where this answer is not known beforehand? If this is the case, I would suggest authors clearly mention and explain this point. Can authors come up with an ablation experiment that clarifies this point (like for example what happens if that final correct ground truth answer is replaced by system's own output).

5 comments
AuthorsOpenReview ↗

Response to Reviewer xJxp

[W1, Q1] Baseline settings, resources, and performance below CoT

Except for MAS-GPT, the design-time MAS methods use Qwen3-8B as the Orchestrator and frozen Qwen3-30B-A3B-Instruct-2507 models as sub-agents. MAS-GPT releases trained MAS-GPT-32B weights without training code, so we use the released checkpoint as its Orchestrator while retaining the same sub-agent backbone. ToolOrchestra and MAS-Orchestra share MAS-Gate's training data and RL hyperparameters. Inference-time baselines follow their original method-specific settings under shared decoding settings. The configurations for all methods are listed below. We have incorporated the details into Appendix C.2.1 of the revised paper.

MethodConfiguration
CoT-SC5 samples, majority voting
AFlow20 search rounds, 5 candidate evaluations per round
AOrchestra10 orchestrator attempts, 50 sub-agent steps
Skill-MAS10 evolution rounds, 5 rollouts per task
ToolOrchestra / MAS-Orchestra / MAS-GateLearning rate 10−610^{-6}, batch size 256, rollout group 16, 1 training epoch

Methods differ in their upfront optimization, from validation-set workflow search to GPU-based policy training. In practical deployment, these upfront costs can be amortized over repeated deployments, whereas inference costs recur with each use. We therefore focus on inference as a common basis for comparing deployment costs. The results show that these gains are achieved without a larger inference token budget. We have incorporated this table into Appendix C.4 of the revised paper.

MethodInput (M)Output (M)Total (M)Calls (K)
CoT0.840.060.900.80
CoT-SC4.200.304.504.00
AFlow1.950.192.141.00
AOrchestra1.310.131.440.87
Skill-MAS4.390.915.301.75
ToolOrchestra17.280.5117.791.46
MAS-Orchestra3.670.203.881.14
MAS-Gate1.970.232.212.21

We also ensemble two independent AOrchestra runs per query to obtain comparable inference token budgets. MAS-Gate remains more accurate on both benchmarks, suggesting that additional inference sampling alone does not close the gap.

MethodHotpotQA (%)BCP (out-of-task, %)
AOrchestra self-ensemble (k=2k=2)57.0012.50
MAS-Gate69.5013.69

As prior research has shown [1, 2], using multiple agents does not guarantee an advantage over CoT. [1] finds that automatically generated workflows can add architectural complexity and computational cost without realizing effective specialization or coordination. [2] identifies concrete failures such as repetitive steps, ignored agent inputs, task derailment, and incomplete or incorrect verification. Such variation aligns with these observations that architectural inefficiencies can prevent automatically constructed MAS from realizing the benefits of multi-agent coordination. We have incorporated the baseline settings, resource comparison, and ensemble evaluation into Appendices C.2–C.4 and D.1.4 of the revised paper.

[1] The Illusion of Multi-Agent Advantage. NeurIPS 2026. [2] Why Do Multi-Agent LLM Systems Fail? NeurIPS 2026.

[W1, Q2] Reliability across repeated evaluations

We report mean ± sample standard deviation over three evaluations for all three main experiments. For reference, MAS-Gate retains its overall advantage comparing to the strongest baseline. We have incorporated the full repeated-run results into Sections 4.1–4.3.

EvaluationMAS-GateStrongest baseline
Design, seven-task Avg42.80 ± 1.68AOrchestra: 33.37 ± 0.32
Runtime, six-task Avg47.90 ± 1.70DyLAN: 43.78 ± 2.21

[Q3] Attribution without gold answers

That's a good question. Gold answers are not always avaliable in pratice. We therefore evaluate SC-5 as an alternative: Qwen3-30B-A3B-Instruct-2507 (Our sub-agent solver) generates five independent solutions from the task inputs, and the most frequent answer replaces gold answer (Pseudo-Guided). Pseudo-Guided falls below Direct in eight of ten settings and improves in two, with smaller gains than gold guidance. Thus, pseudo-references can provide an alternative when gold is unavailable, although approximation errors often degrade attribution. We have incorporated this experiment and analysis into Appendix D.3.4 of the revised paper.

LocalizerWW: Direct (%)WW: Pseudo-Guided (%)WW: Gate-Guided (%)AEGIS: Direct (%)AEGIS: Pseudo-Guided (%)AEGIS: Gate-Guided (%)
Qwen3-30B-A3B-Instruct-250733.7031.5237.5043.0741.6143.80
Qwen3.5-4B34.2431.5235.8734.3138.6940.15
Qwen3.6-27B40.7635.3339.6752.5551.8254.01
Mistral-Small-3.2-24B30.9824.4632.0750.3645.9951.82
Llama-3.1-8B21.2020.1126.6338.6939.4240.15

Dear Reviewer xJxp,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. Your insights have been instrumental in refining our work.

To address your concerns, we have provided additional clarifications and experiments to enhance the clarity of our contributions. We hope these responses effectively resolve any ambiguities, provide a more comprehensive understanding of our work, and encourage you to reconsider your rating.

If you have any further questions or require additional clarification, we would be happy to discuss them. We truly appreciate your time and consideration and look forward to your feedback.

Best regards,

The Authors

Dear Reviewer xJxp,

Thank you again for your time and constructive feedback.

With about one day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.

We would be happy to clarify any remaining questions and welcome your further feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_spgHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces MAS-Gate, which is a shared node-level scoring formulation for design-time orchestration, runtime intervention, and post-hoc failure attribution in LLM-based multi-agent systems. Within-node utility measures the change in reference-answer log-likelihood after adding a node’s output. Inter-node strength combines NLI-based entailment scores with graph-based downstream reach. Their product, with a baseline shift, provides training rewards for an Orchestrator and an Editor, and candidate rankings for an LLM failure localizer. The authors evaluates orchestration and intervention across interactive environments, multi-hop QA, and reasoning benchmarks, and attribution on Who&When and a single-target subset of AEGIS. It reports improvements over the selected baselines, with additional ablations on orchestration rewards, impact composition, and attribution guidance.

Critical strengths
  1. A concrete shared formulation with several useful applications. The authors implement the same node-level score as an orchestration reward, a component of paired intervention rewards, and evidence for failure attribution. The runtime setting seems useful , as the Editor operates on observed execution prefixes without requiring reference answers at inference time.

  2. Broad empirical evaluation with a meaningful controlled reward ablation. The evaluation spans several task families and includes held-out transfer benchmarks. In Table 6, replacing accuracy-only training with the full reward improves HotpotQA from 64.50 to 69.50 and BrowseComp-Plus from 9.52 to 13.69 under otherwise matched settings, which proves the additional dense signal can improve orchestration, although it does not isolate the contribution of its structural component.

Critical weaknesses
  1. The experiments do not seem to establish what the distinctive methodological contribution adds. The central claim is a unified, structure-aware node score, but the ablations do not separate the proposed structural contribution from reference-answer likelihood shaping. If there are any omissions in my understanding here, please let me know. This may affect my evaluation. While your Table 6 compares the full method with accuracy-only training and an additive composition, neither comparison establishes whether �0��0� is useful beyond �1��1�. This distinction matters: on HotpotQA, accuracy-only training already achieves 64.50, compared with 54.00 for the strongest baseline in Table 1 and 69.50 for the full method. A substantial part of the headline improvement therefore remains without MAS-Gate’s dense reward. The additional improvement is encouraging, but its source is unresolved. Moreover, �2��2� measures entailment between outputs, which does not establish that the downstream agent used the upstream information, while �3��3� measures potential reach rather than realized influence. Without a utility-only control or direct validation of these proxies, the paper has not demonstrated that its distinctive structural formulation is necessary or informative. Reusing this unvalidated formulation across three applications broadens the evaluation, but does not resolve the contribution question.

  2. The runtime experiments do not support the paper’s attribution of gains to MAS-Gate or its comparison with design-time orchestration. Your Section 4.2.2 uses Tables 1 and 2 to argue runtime intervention improves over design-time orchestration. However, Appendix C.3.2 explicitly states that the runtime experiments use the original frozen Qwen3-8B Orchestrator, independently of the trained Orchestrator in Table 1. I'm confused here. This comparison changes the underlying orchestration policy and therefore cannot isolate the effect of adding an Editor. Table 2 lacks the essential no-intervention control using the exact same Orchestrator. It also lacks an Editor trained with $ β=0\beta=0$, which would establish whether the proposed quantification contributes beyond outcome-based reinforcement learning and intervention costs. Comparing a trained Editor against frozen prompted models does not answer that question. These are missing controls for the central claims, not merely opportunities for additional evaluation. In addition, I cannot find sufficient matched training and inference budgets across the main baselines in your paper, making the reported performance gaps difficult to interpret as methodological advantages. If there is any detailed information mentioned in the article somewhere I haven't noticed, please let me know.

Questions for authors
  1. What does the structural component contribute? Please provide the matched �0��0� control and separately remove entailment weighting and downstream reach. Does the full formulation consistently outperform these simpler alternatives? And what does MAS-Gate contribute to Editor training? With initialization, training budget, action space, and costs fixed, how does the Editor perform when �1��1� in Eq. 11? This comparison is necessary to distinguish the proposed reward from ordinary outcome-based intervention training.

  2. What is the actual incremental effect of runtime intervention? Please report the exact runtime Orchestrator with and without the Editor under matched conditions. How should the comparison between Tables 1 and 2 be interpreted given the different Orchestrator checkpoints specified in Appendix C.3.2?

  3. Does the attribution score carry useful information independently of the prompted localizer? Can you plz report ranking-only localization performance and comparisons with utility-only and position-based rankings. The latter is particularly relevant because the chronological-chain approximation makes downstream reach depend on step position.

  4. Are the main improvements robust under comparable resources? Can you specify baseline-specific training, workflow-search, and inference budgets, and provide repeated-run or paired uncertainty estimates? Several AEGIS gains correspond to only one or two additional correct predictions out of 137, so I think aggregate percentages alone provide weak evidence of a reliable improvement.

9 comments
AuthorsOpenReview ↗

Response to Reviewer spgH (Part 1/2)

[W1, Q1] Structural contribution and validation of the quantification

MAS-Gate provides a shared quantification across the lifecycle by combining local task utility UU with inter-node strength KK. Following your suggestion, we add U-only, no-RR, and no-GG ablations. These results indicate that the retained node-level signals provide useful supervision beyond terminal correctness and the U-only component is a more helpful part.

Training variantHotpotQA (%)BCP (out-of-task, %)
Accuracy-only64.509.52
U-only67.5011.90
No-RR65.009.52
No-GG66.5010.71
Full MAS-Gate69.5013.69

As for the validation of these signal, we complement Figure 1(b)'s qualitative analysis with human validation on 50 HotpotQA trajectories. Three blinded annotators (postgraduates in AI, don't know our paper before) achieve Krippendorff's α>0.8\alpha>0.8 for both quantities, indicating high agreement and reliable annotations. The correlations with mean human ratings confirm alignment with the intended semantic roles of UU and RR. Together, the case analysis and human evaluation yield complementary qualitative and quantitative evidence for the shared quantification used across the three lifecycle applications. We have incorporated these results into Section 3.4 and Appendices D.1.3 and D.1.5 of the revised paper.

QuantityInter-annotator α\alphaMetric–human Spearman's ρ\rho
UU0.83100.7245
RR0.80030.7122

[W2, Q1, Q2] The design/runtime comparison and Editor contribution

The cross-table comparison evaluates two ways of improving a MAS from the same original Qwen3-8B backbone. In the design-time setting, we train the Orchestrator to construct better workflows, which then execute without an Editor. In the runtime setting, we keep the original Orchestrator frozen and train a separate Qwen3-8B Editor to adjust its workflows using evidence observed during execution. Thus, the comparison examines two learning policy, i.e., learning better initial designs versus learning to improve ongoing execution.

Following your suggestion, we add no-intervention and β=0\beta=0 controls using the same original frozen Qwen3-8B Orchestrator. Full Editor improves over no-intervention across all six tasks and also exceeds β=0\beta=0 by 6.33 and 3.17 points on HotpotQA and BCP, demonstrating the quantification reward's added value. We have incorporated these controls and full results into Section 4.2.2 and Appendix D.2.1 of the revised paper.

ConfigurationHotpotQA (%)BCP (out-of-task, %)
No-intervention56.006.55
Editor, β=0\beta=069.0012.90
Full Editor75.3316.07

[Q3] Localization from rankings alone

We add ranking-only evaluation without an LLM localizer. Full I, U-only, and Position-first prioritizes earlier steps. Position-first performs worst on both datasets, showing that temporal position alone is insufficient for localization. U-only captures failure-relevant information beyond position, while Full's further gain demonstrates the complementary value of propagation strength. We have incorporated this evaluation into Appendix D.3.2 of the revised paper.

RankingWho&When Acc (%)AEGIS Acc (%)
Full I21.7440.88
U-only20.6539.42
Position-first10.8732.12
AuthorsOpenReview ↗

Response to Reviewer spgH (Part 2/2)

[W2, Q4] Comparable resources and repeated-run uncertainty

Except for MAS-GPT, the design-time MAS methods use Qwen3-8B as the Orchestrator and frozen Qwen3-30B-A3B-Instruct-2507 models as sub-agents. MAS-GPT releases trained MAS-GPT-32B weights without training code, so we use the released checkpoint as its Orchestrator while retaining the same sub-agent backbone. ToolOrchestra and MAS-Orchestra share MAS-Gate's training data and RL hyperparameters. Inference-time baselines follow their original method-specific settings under shared decoding settings. The configurations for all methods are listed below. We have incorporated the details into Appendix C.2.1 of the revised paper.

MethodConfiguration
CoT-SC5 samples, majority voting
AFlow20 search rounds, 5 candidate evaluations per round
AOrchestra10 orchestrator attempts, 50 sub-agent steps
Skill-MAS10 evolution rounds, 5 rollouts per task
ToolOrchestra / MAS-Orchestra / MAS-GateLearning rate 10−610^{-6}, batch size 256, rollout group 16, 1 training epoch

Methods differ in their upfront optimization, from validation-set workflow search to GPU-based policy training. In practical deployment, these upfront costs can be amortized over repeated deployments, whereas inference costs recur with each use. We therefore focus on inference as a common basis for comparing deployment costs. We report HotpotQA inference tokens and LLM calls as a common basis for deployment costs. The results show that these gains are achieved without a larger inference token budget. We have incorporated this table into Appendix C.4 of the revised paper.

Design: MethodInput (M)Output (M)Total (M)Calls (K)Runtime: MethodInput (M)Output (M)Total (M)Calls (K)
CoT0.840.060.900.80No-Intervention5.000.765.761.40
CoT-SC4.200.304.504.00DeepSeek-V4-Flash Editor8.171.049.212.20
AFlow1.950.192.141.00GPT-5.4-Mini Editor8.321.129.442.33
AOrchestra1.310.131.440.87Self-Refine7.400.998.392.00
Skill-MAS4.390.915.301.75AgentDropoutV28.801.019.812.36
ToolOrchestra17.280.5117.791.46DyLAN9.801.3611.162.60
MAS-Orchestra3.670.203.881.14AgentPrune4.010.764.771.40
MAS-Gate1.970.232.212.21MAS-Gate8.040.918.952.16

Then We additionally compare with a two-run AOrchestra ensemble under comparable inference token budgets. MAS-Gate remains better on both benchmarks, suggesting that its advantage is not explained solely by token usage.

MethodHotpotQA (%)BCP (out-of-task, %)
AOrchestra self-ensemble (k=2k=2)57.0012.50
MAS-Gate69.5013.69

To address uncertainty concern, we add two repeated-run for all the main experiments and report means and standard deviations over three evaluations. We report the comparison between MAS-Gate and Strongest baseline here for Exp1&2. As for Exp3, for example, AEGIS gains range from 0.49 ± 1.12 to 6.33 ± 1.52 points. We have incorporated these full results and settings into Sections 4.1–4.3 of the revised paper.

EvaluationMAS-GateStrongest baseline
Design, seven-task Avg42.80 ± 1.68AOrchestra: 33.37 ± 0.32
Runtime, six-task Avg47.90 ± 1.70DyLAN: 43.78 ± 2.21

Dear Reviewer spgH,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. Your insights have been instrumental in refining our work.

To address your concerns, we have provided additional clarifications and experiments to enhance the clarity of our contributions. We hope these responses effectively resolve any ambiguities, provide a more comprehensive understanding of our work, and encourage you to reconsider your rating.

If you have any further questions or require additional clarification, we would be happy to discuss them. We truly appreciate your time and consideration and look forward to your feedback.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for recognizing the shared quantification across the MAS lifecycle (spgH, xJxp, dZ97), broad empirical evaluation (spgH, dZ97), and the value of dense node-level rewards for orchestration (spgH, xJxp). We address their concerns through four main revisions:

  • Quantification: add component ablations and human validation of utility and semantic support.
  • Lifecycle applications: add no-intervention and beta-zero Editor controls, and evaluate joint Orchestrator/Editor training.
  • Reliability and efficiency: report three-run main results, baseline configurations, inference tokens and LLM calls, and an AOrchestra ensemble comparison under comparable token budgets.
  • Failure attribution: add ranking-only, multi-source, and no-gold evaluations.

The responses below address each reviewer's specific concerns and provide key results, with full details in the revised paper (changes are highlighted in blue).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.