acceptodds
Under review as a conference paper at ICLR 2027

MAS-Loop: Beyond One-Shot Orchestration towards Self-Evolving Multi-Agent Systems

Abstract

Training-time multi-agent system (MAS) design seeks to learn reusable orchestration policies, yet prevailing approaches remain largely one-shot: an Orchestrator generates a system but only observes and learns from its terminal outcome. This paradigm discards execution trajectories that expose system-level failures and contain fine-grained information beyond the terminal outcome, leaving no direct basis for revising the source MAS, evaluating the updated MAS, or learning from the resulting contrast. To this end, we introduce MAS-Loop, an execution-grounded framework that closes this feedback loop by coupling an Orchestrator with a trajectory-conditioned Refiner. For each task, the Orchestrator constructs and executes a source MAS to obtain an outcome score and an intermediate trajectory. Conditioned on the trajectory, the Refiner localizes an actionable failure, formulates a trajectory-grounded diagnosis, and proposes a bounded patch to the agents, information flow, or execution configuration. The executable updated MAS is re-executed on the same task to measure its realized outcome under stochastic execution. This pair-MAS comparison provides complementary learning signals: the Orchestrator learns to generate strong systems with fewer recoverable failures, whereas the Refiner learns to expose and repair observed weaknesses, thereby driving policy-level co-evolution across training iterations. We evaluate MAS-Loop across interaction environments, knowledge-intensive multi-hop reasoning, and general reasoning, examining both the complete framework and the independent utility of these two policies. Experiments show that MAS-Loop achieves the strongest overall performance among the evaluated baselines, with consistent gains from refinement across all evaluated benchmarks. The co-evolution process also yields MAS-JudgeCorpus, a dataset of execution-grounded source-updated pairs with orchestration- and trajectory-level supervision, which can provide the community with support for MAS reward modeling and trajectory-aware investigation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent System

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_DdLiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MAS-Loop jointly trains an Orchestrator that constructs a multi-agent system and a Refiner that diagnoses its execution and proposes a bounded graph or configuration edit. The revised system is executed again, and source/updated outcomes supply separate rewards for the two policies. The Refiner receives execution evidence but no gold answer. Experiments cover interactive tasks, multi-hop QA, and reasoning, including transfer to held-out task families and other MAS generators. The paired rollouts also form MAS-JudgeCorpus for training a system-selection reward model.

Critical strengths
  1. The paper connects execution diagnosis to an explicit, executable edit interface and learns both system construction and refinement. The results are useful beyond running two policies together: the standalone Orchestrator is strong, and the Refiner improves systems from several other generators. This makes the contribution more convincing than an improvement tied to one bespoke workflow.

  2. The evaluation includes several relevant controls. Appendix D doubles MAS-Orchestra’s rollout group to match the number of system executions and includes a prompted reflection pass with the same gold-free diagnostic input and bounded edit interface. Both remain below MAS-Loop on HotpotQA and BrowseComp-Plus. Together with the reward-coefficient ablations and held-out transfer results, these comparisons provide meaningful support for learning execution-grounded refinement. I lean toward acceptance on this evidence.

Critical weaknesses
  1. The reward signal does not separate a patch effect from execution randomness. Each source graph has one source execution and one independent updated execution; a training group samples different systems rather than repeating the same intervention. Consequently, max(Δs,0) in Eq. (11) can penalize apparent recoverability caused by noise, even when a valid edit has no effect on expected performance. The unchanged-graph branch avoids this penalty, but does not establish how much of the valid-patch signal comes from actual repairs. The paper correctly calls Δs a realized score change; its interpretation as evidence of a recoverable architectural weakness still needs support.

  2. Several incremental gains from adding the Refiner are small relative to the evaluation sets: AIME has 57 questions and GPQA has 198, and the overall average gain over the standalone Orchestrator is 1.96 points. Repeated training/evaluation results and complete inference costs are not reported. These limitations weaken the strength of the co-evolution and deployment claims, although the larger standalone-policy gains and matched-execution controls keep my recommendation positive.

Questions for authors
  1. For fixed source graphs, how does the distribution of improvements after a valid patch compare with independent unchanged-graph reruns? A repeated-continuation control would clarify how much the rewards measure actual repair rather than execution noise.

  2. How stable are the standalone and Refiner gains across independent training/evaluation seeds? Please report Accept, invalid-edit, improvement, and regression rates alongside the principal task scores.

  3. What is the total inference token/latency cost of source execution, diagnosis, and re-execution relative to the strongest baselines? For the best-of-16 selection experiment, please distinguish verification-only tokens from candidate generation and execution costs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DdLi

We thank the reviewer for recognizing policy transfer and the existing execution-matched controls.

[W1 + Q1] Fixed-patch effects versus unchanged-graph reruns

We agree that single-sample feedback can be affected by execution noise. To test whether initial gains persist, we randomly select 200 initially improving pairs from HotpotQA training log. Both graphs and the original patch remain fixed, and each source MAS and updated MAS are independently executed eight times. Comparing each pair's eight-execution means, 90.50% retain the initial positive direction. These results provide empirical support for using a single Refiner sample per source MAS in our RL training. We have added the sampling trade-off to Section 3.3 and the experiment to Appendix D.3.1 in the revised paper.

Source meanUpdated meanImproved (%)Tied (%)Degraded (%)
28.937566.125090.508.001.50

[W2: stability + Q2] Repeated evaluation and Refiner decisions

Following your suggestion, we report mean ± sample standard deviation over three evaluations. MAS-Loop achieves the highest overall average under both O-only and O+R deployment. Refinement further improves MAS-Loop and all three external orchestrators. These results align with our original findings. We have incorporated the complete repeated-evaluation results into Section 4 in the revised paper.

MethodSeven-metric Avg.
CoT27.95 ± 1.03
CoT-SC29.91 ± 0.77
AFlow24.92 ± 0.85
AOrchestra33.59 ± 0.44
Skill-MAS25.02 ± 0.24
Skill-MAS + Refiner30.73 ± 0.19
MAS-GPT29.65 ± 0.88
MAS-GPT + Refiner33.18 ± 0.89
ToolOrchestra21.62 ± 0.75
MAS-Orchestra23.43 ± 0.66
MAS-Orchestra + Refiner28.92 ± 0.55
MAS-Loop (O-only)41.74 ± 0.41
MAS-Loop (O+R)44.31 ± 0.75

We also report the requested HotpotQA behavior rates. All rates use the complete HotpotQA test set, and Edit denotes valid non-Accept edits. Larger improvement proportions (Imporve - Regress) on external orchestrators agree with their larger average refinement gains. This supports R's independent utility and is consistent with a stronger co-trained O leaving less room for refinement. We have incorporated the repeated evaluations into Section 4 and the uncertainty and behavior analyses into Appendices D.3.2–D.3.4 in the revised paper.

Source orchestratorAccept (%)Edit (%)Invalid (%)Improve (%)Tie (%)Regress (%)
MAS-Loop9.5089.001.5011.5078.5010.00
Skill-MAS11.0088.500.5014.5077.508.00
MAS-GPT9.0090.001.0014.0080.006.00
MAS-Orchestra10.5087.502.0015.5079.005.50

[W2: costs + Q3] Deployment costs and verification-only accounting

Methods differ in their upfront optimization, from validation-set workflow search to GPU-based policy training. In practical deployment, these upfront costs can be amortized over repeated deployments, whereas inference costs recur with each use. We therefore focus on inference as a common basis for comparing deployment costs. Following your suggestion, we add HotpotQA inference costs analysis for the strongest overall baseline AOrchestra and each MAS-Loop stage. Source execution includes O construction and execution, diagnosis includes R diagnosis/editing, and re-execution runs the updated system. We have added this breakdown to Appendix D.4.1 in the revised paper.

Method / stageInput (M)Output (M)Total (M)LLM calls (K)
AOrchestra1.310.131.440.87
MAS-Loop: source execution1.190.151.341.35
MAS-Loop: diagnosis1.070.101.170.24
MAS-Loop: re-execution0.990.101.091.09
MAS-Loop: full3.250.353.602.67

We also compare against AOrchestra under comparable inference token budgets, using three-sample majority voting to increase its token budget. MAS-Loop exceeds the ensemble by 8.00 percentage points while using fewer tokens, indicating that its advantage beyond increased token budget. We have added this result to Appendix D.4.2 in the revised paper.

MethodHotpotQA accuracy (%)Inference tokens (M)
AOrchestra49.001.44
AOrchestra, 3-sample majority vote56.504.31
MAS-Loop (full)64.503.60

The best-of-16 experiment already reports verification-only tokens. All judges use the same 16 MAS candidates, so shared generation and execution costs are excluded and the Token (M) in Table 2 is actually the verification-only tokens, as specified in Section 5.2.

Dear Reviewer DdLi,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. Your insights have been instrumental in refining our work.

To address your concerns, we have provided additional clarifications and experiments to enhance the clarity of our contributions. We hope these responses effectively resolve any ambiguities, provide a more comprehensive understanding of our work, and encourage you to reconsider your rating.

If you have any further questions or require additional clarification, we would be happy to discuss them. We truly appreciate your time and consideration and look forward to your feedback.

Best regards,

The Authors

Dear Reviewer DdLi,

Thank you again for your time and constructive feedback.

With about one day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.

We would be happy to clarify any remaining questions and welcome your further feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_BQrDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MAS-Loop trains an Orchestrator that constructs multi-agent execution graphs and a Refiner that uses execution traces to make one bounded graph revision. The Refiner does not observe gold answers or correctness rewards. Both policies are trained with GRPO while the executing agents remain frozen. The paper also constructs a preference corpus from original and revised systems and trains MAS-JudgeRM for candidate selection. The empirical study evaluates both jointly trained policies, transfer of the Refiner to other orchestrators, and graph-level versus trajectory-level verification.

Critical strengths
  1. An executable, reusable revision interface. The Orchestrator constructs graphs and the gold-free Refiner makes bounded changes to roles, agents, dependencies, reasoning budgets, or tools. Separate policies support component-level testing, and Refiner gains on other orchestrators show usefulness beyond the jointly trained pair.

  2. The controls test more than another reflection prompt. Appendix D matches 32 baseline graph executions against 16 original plus 16 revised executions, and includes one-step prompted reflection using the same information and editing interface. On HotpotQA/BCP, MAS-Loop obtains 64.50/15.48, versus 54.00/12.50 for the execution-matched baseline and 59.50/11.90 for prompted reflection (Table 5). These comparisons support the empirical framework.

Critical weaknesses
  1. Execution-grounded rewards and preferences are not necessarily reproducible edit effects. Original and revised graphs are executed independently, and Eq. 11 penalizes the positive part of observed improvement. For a valid edit with unchanged success probability p and independent binary outcomes, E[max(s_updated − s_source, 0)] = p(1 − p) > 0 for 0 < p < 1. This does not apply to the explicitly unchanged Accept branch or prove failure, but shows that noise need not cancel. Similarly, filtering local preference pairs to agree with an aggregate task direction (§5.1) does not independently validate those pairs. Aggregate score units and the zero-score exclusion rule also need clarification.

  2. Incremental gains need to be distinguished from overall training benefits and their cost. The seven-metric mean rises from 41.47 for the Orchestrator to 43.43 with the Refiner. The HotpotQA and AIME increments correspond to approximately three of 200 and one of 57 questions. Paired uncertainty matters for these increments. Matching execution counts is valuable, but graph size and reasoning budgets can change, and the Refiner adds training and inference work. Best-of-16 verification token counts do not measure this full cost.

Questions for authors
  1. On a small fixed sample of original graphs and edits, how repeatable are improvement signs under independent re-execution? A legal behavior-preserving edit control would help distinguish revision benefit from resampling benefit.

  2. How are corpus training, validation, and testing separated at the task level? Please define aggregate score units, explain the zero-denominator branch, and report any independent validation of retained preference labels.

  3. What are the paired task-level uncertainty and full token/tool costs for Orchestrator-only versus Orchestrator-plus-Refiner? Please separate the incremental revision gains from the overall training gains.

4 comments
AuthorsOpenReview ↗

Response to Reviewer BQrD

We thank the reviewer for the questions about reproducible edit gains, preference reliability, and deployment costs.

[W1: edit effects + Q1] Reward noise and fixed-edit repeatability

We agree that single-sample feedback can be affected by execution noise. To test whether initial gains persist, we randomly select 200 initially improving pairs from HotpotQA training log. Both graphs and the original patch remain fixed, and each source MAS and updated MAS are independently executed eight times. Comparing each pair's eight-execution means, 90.50% retain the initial positive direction. These results provide empirical support for using a single Refiner sample per source MAS in our RL training. We have added the sampling trade-off to Section 3.3 and the experiment to Appendix D.3.1 in the revised paper.

Source meanUpdated meanImproved (%)Tied (%)Degraded (%)
28.937566.125090.508.001.50

[W1: corpus concerns + Q2] Preference labels and data separation

All 8,223 training pairs come from HotpotQA training tasks. Evaluation uses disjoint held-out HotpotQA tasks and out-of-task BCP. We use no validation split, train MAS-JudgeRM for one epoch, and evaluate the final checkpoint. The aggregate scores S_src and S_upd each sum 16 rollout scores for the same task and lie in [0,16]. When both are positive, retention requires an absolute gap ≥2.0 and a relative gap ≥25%. If the smaller aggregate is zero, we skip the relative condition and use only the absolute-gap threshold. We have clarified the score units, zero-denominator rule, and training protocol in Sections 5.1–5.2 in the revised paper.

[W2 + Q3] Refinement gains, uncertainty, and costs

Following your suggestion, we report paired task-level refinement gains from the original single evaluation. Representative results for all four source orchestrators on ALFW, SciW, VITA, and HotpotQA are shown below as mean gain ± 95% t-interval half-width. Mean refinement gains are positive across these settings and are generally larger on external orchestrators. This descriptive pattern agrees with the repeated evaluations in Table 1 and is consistent with co-evolution producing a stronger Orchestrator with less scope for subsequent refinement, while the Refiner remains more useful when deployed independently. We have included the complete paired results and uncertainty analysis in Appendix D.3.2 in the revised paper.

Source orchestratorALFWSciWVITAHotpotQA
MAS-Loop0.32 ± 3.631.67 ± 2.544.98 ± 1.381.50 ± 6.48
Skill-MAS14.10 ± 5.053.33 ± 2.432.84 ± 0.806.50 ± 6.57
MAS-GPT1.28 ± 4.193.33 ± 2.762.89 ± 1.008.00 ± 6.15
MAS-Orchestra4.17 ± 3.375.00 ± 2.654.16 ± 1.0810.00 ± 6.25

HotpotQA outcome rates further illustrate the Refiner's behavior. The larger improvement proportions (Improve - Regress) on external orchestrators agree with their larger average gains, supporting the Refiner's independent utility. Full decision and outcome statistics are included in Appendix D.3.3 of the revised paper.

Source orchestratorImprove (%)Tie (%)Regress (%)
MAS-Loop11.5078.5010.00
Skill-MAS14.5077.508.00
MAS-GPT14.0080.006.00
MAS-Orchestra15.5079.005.50

For deployment costs, we directly compare the token cost of construction/execution, diagnosis/editing, and re-execution. Source execution corresponds to O-only deployment, while diagnosis and re-execution account for the additional cost of refinement. These results support flexible deployment: O-only provides a lower-cost option, while O+R enables further performance gains through execution-grounded refinement. We have added the full stage breakdown to Appendix D.4.1 in the revised paper.

Method / stageInput (M)Output (M)Total (M)LLM calls (K)
MAS-Loop (source execution)1.190.151.341.35
MAS-Loop (diagnosis)1.070.101.170.24
MAS-Loop (re-execution)0.990.101.091.09
MAS-Loop (full)3.250.353.602.67

Dear Reviewer BQrD,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. Your insights have been instrumental in refining our work.

To address your concerns, we have provided additional clarifications and experiments to enhance the clarity of our contributions. We hope these responses effectively resolve any ambiguities, provide a more comprehensive understanding of our work, and encourage you to reconsider your rating.

If you have any further questions or require additional clarification, we would be happy to discuss them. We truly appreciate your time and consideration and look forward to your feedback.

Best regards,

The Authors

Thank you for the additional experiments and clarifications. The reruns strengthen the evidence for initially positive edits, and the corpus and cost details are helpful. I will maintain my recommendation.

Thank you for your feedback. We greatly appreciate the recognition that the additional experiments and analyses strengthen the evidence for MAS-Loop and address the main concerns.

Reviewer_twAROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper introduces a multi-agent system that is execution graded and improves through a orchestrator and a refiner that is trajectory-grounded. The orchestrator and refiner co-evolve using rewards from the source and system executions that are update. The paper demonstrates improvements with this method across reasoning and interactive benchmarks.

Critical strengths

The paper designs a practical framework for multi-agent system evolution through trajectory based diagnosis and refinment and includes a broad evaluation with transfer to other multi-agent system generators.

Critical weaknesses

The novelty of the proposed method is limited with respect to prior work in single and multi agent self-improvement. Furthermore, the claimed benefits of co-evolving the orchestrator and refiner are not fully isolated experimentally.

Questions for authors
  1. What happens if only 1 component is trained while the other is frozen? How can be the improvements be attributed specifically to co-evolution?
  2. Are the gains statistically significant across independent training runs? How does MAS-Loop compare with the strongest baselines under matched total training and inference compute, including all agent executions?
5 comments
AuthorsOpenReview ↗

Response to Reviewer twAR

We thank the reviewer for the questions about novelty, co-evolution, and computational fairness.

[W: novelty] Distinction from prior self-improvement

Prior training-time MAS methods learn orchestration from successful designs, task rewards, structured credit assignment, or graph optimization. They improve future constructions but lack a direct connection between trajectory-guided revision of the current system and learning both construction and revision. The Orchestrator must therefore discover better designs indirectly through repeated rollouts and outcome rewards.

MAS-Loop closes this loop: a trajectory-conditioned Refiner revises the current MAS, and paired re-execution supplies distinct learning signals to both policies. By revealing coordination and information-flow failures that emerge during execution and are difficult to infer from system design alone, this process provides richer, more targeted training signals for both policies. MAS construction and executable revision are jointly learned, and both policies remain independently deployable. We have clarified this contribution in Sections 1 and 2.1 in the revised paper.

[W: attribution + Q1] Isolating co-evolution

Following your suggestion, we add the frozen-policy controls, initializing both policies from Qwen3-8B in all settings. Each control trains one policy while keeping the other at its initial weights. O-only evaluates the source MAS, and O+R evaluates the updated MAS. Joint training outperforms both controls in every column. Freezing O yields the weakest source systems, while freezing R yields the smallest refinement gains, supporting the value of co-evolution. We have added these results and analysis to Appendix D.2.2 in the revised paper.

Training settingHotpotQA O-onlyHotpotQA O+RBCP O-onlyBCP O+R
MAS-Loop (full)63.0064.5013.1015.48
Train O, freeze R60.5060.009.5210.71
Train R, freeze O59.0061.508.3313.10

[Q2] Repeated evaluation and computational fairness

Following your suggestion, we report mean ± sample standard deviation over three evaluations. MAS-Loop achieves the highest overall average under both O-only and O+R deployment. Refinement further improves MAS-Loop and all three external orchestrators. These results align with our original findings. We have incorporated the complete repeated-evaluation results into Section 4 in the revised paper.

MethodSeven-metric Avg.
CoT27.95 ± 1.03
CoT-SC29.91 ± 0.77
AFlow24.92 ± 0.85
AOrchestra33.59 ± 0.44
Skill-MAS25.02 ± 0.24
Skill-MAS + Refiner30.73 ± 0.19
MAS-GPT29.65 ± 0.88
MAS-GPT + Refiner33.18 ± 0.89
ToolOrchestra21.62 ± 0.75
MAS-Orchestra23.43 ± 0.66
MAS-Orchestra + Refiner28.92 ± 0.55
MAS-Loop (O-only)41.74 ± 0.41
MAS-Loop (O+R)44.31 ± 0.75

We also report paired task-level refinement gains from the original single evaluation. Representative results for all four source orchestrators on ALFW, SciW, VITA, and HotpotQA are shown below as mean gain ± 95% t-interval half-width. Mean refinement gains are positive across these settings and are generally larger on external orchestrators. This descriptive pattern agrees with the repeated evaluations in Table 1 and is consistent with co-evolution producing a stronger Orchestrator with less scope for subsequent refinement, while the Refiner remains more useful when deployed independently. We have included the complete paired results and uncertainty analysis in Appendix D.3.2 in the revised paper.

Source orchestratorALFWSciWVITAHotpotQA
MAS-Loop0.32 ± 3.631.67 ± 2.544.98 ± 1.381.50 ± 6.48
Skill-MAS14.10 ± 5.053.33 ± 2.432.84 ± 0.806.50 ± 6.57
MAS-GPT1.28 ± 4.193.33 ± 2.762.89 ± 1.008.00 ± 6.15
MAS-Orchestra4.17 ± 3.375.00 ± 2.654.16 ± 1.0810.00 ± 6.25

For computational fairness, we increase the inference budget of AOrchestra, the strongest overall baseline, using three-sample majority voting ensemble. The result shows that MAS-Loop exceeds the ensemble by 8.00% on HotpotQA, 2.38% on BCP with fewer tokens, supporting the effectiveness of MAS-Loop beyond additional token budget. We have added the cost breakdown and comparison to Appendices D.4.1–D.4.2 in the revised paper.

MethodHotpotQABCP (out-of-task)HotpotQA tokens (M)
AOrchestra49.0010.711.44
AOrchestra, 3-sample majority vote56.5013.104.31
MAS-Loop (full)64.5015.483.60

Dear Reviewer twAR,

We sincerely appreciate your time and effort in reviewing our paper and providing valuable comments. Your insights have been instrumental in refining our work.

To address your concerns, we have provided additional clarifications and experiments to enhance the clarity of our contributions. We hope these responses effectively resolve any ambiguities, provide a more comprehensive understanding of our work, and encourage you to reconsider your rating.

If you have any further questions or require additional clarification, we would be happy to discuss them. We truly appreciate your time and consideration and look forward to your feedback.

Best regards,

The Authors

Dear Reviewer twAR,

Thank you again for your time and constructive feedback.

With about one day remaining in the author–reviewer discussion period, we would greatly appreciate it if you could take a moment to review our rebuttal and revised manuscript. We have addressed your concerns through additional analyses, experiments, and clarifications.

We would be happy to clarify any remaining questions and welcome your further feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_N7E1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies learning to generate and refine multi-agent systems from execution feedback. An Orchestrator constructs a task-specific agent graph, and a Refiner examines its execution record to propose a bounded modification or preserve the original design. Re-execution on the same task provides learning signals for both policies: successful repairs reward the Refiner and reduce the Orchestrator’s reward, encouraging stronger initial designs. Both policies are trained through reinforcement learning. Experiments cover interactive environments, multi-hop question answering, general reasoning, and out-of-task transfer, with stronger overall performance than the evaluated baselines. The Orchestrator remains effective when deployed alone, while the Refiner also improves systems produced by other generators. The collected execution pairs additionally form MAS-JudgeCorpus, which is used to train reward models for selecting candidate MAS designs.

Critical strengths
  1. The method jointly trains MAS construction and refinement, using execution feedback to revise the current system and the effects of those revisions to improve future orchestration. This provides a coherent way to move beyond one-shot system generation.

  2. The experiments cover multiple task families and out-of-task transfer, with the strongest overall performance among the evaluated baselines. Ablations and controls for execution count and prompted reflection also support the benefits of learning and coupled training.

  3. The co-trained Orchestrator remains effective when deployed alone, while the Refiner also improves systems produced by other methods. These results show that the two policies have independent value, particularly that the refinement policy can be reused across different MAS generators.

Critical weaknesses
  1. The experiments do not directly isolate the contribution of execution trajectories to refinement. The existing ablations mainly assess the trained Refiner and the adversarial reward, but do not compare refinement with and without execution trajectories under the same training conditions. Beyond a single case study, the paper also provides limited analysis of common failure types and refinement outcomes. The results show that the Refiner improves task performance, but it remains unclear how much of this improvement comes from using execution evidence to identify and repair specific system failures.

  2. The paper lacks sufficient discussion and comparison with closely related work, particularly LEMON [1], which it already cites. LEMON likewise trains an orchestration policy through local modifications to MAS designs, re-execution, and reward comparisons. MAS-Loop introduces a trajectory-conditioned Refiner and coupled training of the two policies, but the paper does not sufficiently discuss these differences or provide a direct experimental comparison. It is therefore difficult to assess the additional benefit of these components over existing training methods based on local interventions.

  3. Could the authors report the total training cost and the average inference cost of deploying the Orchestrator alone versus the complete MAS-Loop, in terms of token usage or runtime? Refinement introduces additional model calls and system executions, while the control in the appendix matches the number of MAS executions. These measurements would help readers understand the performance gains relative to the computational cost of the additional refinement step.

[1] LEMON: Learning Executable Multi-Agent Orchestration via Counterfactual Reinforcement Learning

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer N7E1

We thank the reviewer for recognizing the learning loop, evaluation breadth, and independent value of both policies.

[W1] Execution trajectories and refinement outcomes

We add a controlled comparison in which R-answer excludes trajectories and derived summaries, while R-full includes them. R-full performs better on both datasets, supporting the value of execution evidence beyond system structure and final answers. We have incorporated this result into Appendix D.2.1 in the revised version.

MethodHotpotQABCP (out-of-task)
R-answer61.5014.29
MAS-Loop (R-full)64.5015.48

We also analyze refinement outcomes by the seven supported operations. RewriteRole is the most frequent operation (33.00%) and has the highest improvement rate among edits (15.15%). Operations with more improvements than regressions account for 65.00% of decisions, consistent with the training objective encouraging the Refiner to favor score-improving revisions. We have added this analysis to Appendix D.3.4 in the revised paper.

OperationFreq. (%)Improve (%)Tie (%)Regress (%)
Accept9.505.2689.475.26
AddAgent14.5010.3475.8613.79
RewriteRole33.0015.1572.7312.12
AddEdge7.000.0092.867.14
RemoveEdge4.000.0087.5012.50
AdjustReasoningBudget21.0014.2976.199.52
ChangeTool11.0013.6481.824.55

[W2] Relationship to LEMON

The key distinction is how execution feedback guides system improvement. LEMON uses predefined local interventions, re-execution, and reward comparisons for localized credit assignment when learning an orchestration policy. Its edit selection is not conditioned on the current execution trajectory. The Orchestrator therefore still needs to discover better designs indirectly through vanilla repeated rollouts.

Our Refiner reads the current trajectory to select the edit target, operation, and content. Re-execution trains both policies: R learns targeted revision, while O learns stronger future constructions. This connects current-system improvement and future construction through execution-grounded co-evolution.

The deployment scope also differs. LEMON retains the learned Orchestrator, whereas MAS-Loop retains both policies for O-only, O+R, and independent refinement of other orchestrators. Our transfer results demonstrate this reusable refinement capability. We have added the comparison of learning mechanisms to Section 2.1 in the revised paper.

[W3] Token costs

Methods differ in their upfront optimization, from validation-set workflow search to GPU-based policy training. In practical deployment, these upfront costs can be amortized over repeated deployments, whereas inference costs recur with each use. We therefore focus on inference as a common basis for comparing deployment costs. The results show that these gains are achieved without a larger inference token budget. We have added the baseline comparison and full stage breakdown to Appendix D.4.1 in the revised paper.

Method / stageInput (M)Output (M)Total (M)LLM calls (K)
CoT0.840.060.900.80
CoT-SC4.200.304.504.00
AFlow1.832.424.250.79
AOrchestra1.310.131.440.87
Skill-MAS6.131.147.272.26
MAS-GPT2.710.653.361.40
MAS-Orchestra0.980.181.161.20
ToolOrchestra0.590.060.650.60
MAS-Loop (source execution)1.190.151.341.35
MAS-Loop (diagnosis)1.070.101.170.24
MAS-Loop (re-execution)0.990.101.091.09
MAS-Loop (full)3.250.353.602.67

We further compare against AOrchestra, the strongest overall baseline, under comparable inference token budgets by increasing its sampling budget through three-sample majority voting ensemble. MAS-Loop exceeds the ensemble by 8.00 percentage points on HotpotQA while using fewer tokens (3.60M versus 4.31M), supporting the effectiveness of MAS-Loop beyond additional inference sampling. We have added this experiment to Appendix D.4.2 in the revised paper.

MethodHotpotQABCP (out-of-task)HotpotQA tokens (M)
AOrchestra49.0010.711.44
AOrchestra, 3-sample majority vote56.5013.104.31
MAS-Loop (full)64.5015.483.60

Thank you for the response. The clarifications have largely addressed my concerns. I will maintain my positive rating.

Thank you for your feedback. We greatly appreciate the recognition that the additional experiments and analyses strengthen the evidence for MAS-Loop and address the main concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for recognizing the execution-grounded learning framework (N7E1, twAR, BQrD, DdLi), broad evaluation (N7E1, twAR), and independent Orchestrator and transferable Refiner (N7E1, BQrD, DdLi). We address their concerns through three main revisions:

  • Contribution and attribution: clarify the distinction from prior self-improvement and LEMON, and add trajectory-input and frozen-policy ablations.
  • Reliability: report three evaluations of all Table 1 methods, paired task-level uncertainty, fixed-patch re-execution, and Refiner decision and outcome rates.
  • Efficiency: report inference tokens and LLM calls by stage, and compare with the strongest overall baseline under comparable HotpotQA inference budgets.

The responses below address each reviewer's specific concerns and provide key results, with full details in the revised paper (changes are highlighted in blue).

Area_Chair_R5t8OpenReview ↗

Reviewer Feedback on Rebuttal Before Discussion Closes

Hi, all. I noticed that some reviewers have not yet provided follow-up feedback. As the discussion period is approaching its end, I would appreciate it if twAR and DdLi could take a moment to review the authors' responses and revised manuscript, and share whether their concerns have been addressed. Please also update your ratings if warranted. It would be helpful to have your feedback before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.