acceptodds
Under review as a conference paper at ICLR 2027

MetaAgent-X : Breaking the Ceiling of Automatic Multi-Agent Systems via End-to-End Reinforcement Learning

Abstract

Automatic multi-agent systems aim to instantiate agent workflows without relying on manually designed or fixed orchestration. However, existing automatic MAS approaches remain only partially adaptive: they either perform training-free test-time search or optimize the meta-level designer while keeping downstream execution agents frozen, which creating a frozen-executor ceiling and leaving the end-to-end training of self-designing and self-executing agentic models unexplored. To address this, we introduce \name, an end-to-end reinforcement learning framework that jointly optimizes automatic MAS design and execution. \name enables script-based MAS generation, execution rollout collection, and credit assignment for both designer and executor trajectories. To support stable and scalable optimization, we propose Executor Designer Hierarchical Rollout and Stagewise Co-evolution to improve training stability and expose the dynamics of designer-executor co-evolution. \name consistently outperforms existing automatic MAS baselines, improving the six-benchmark average by 6.2 and 7.5 points over the strongest automatic MAS baseline on Qwen3-8B and Qwen3-4B respectively, and by 11.2 and 12.8 points over a single-agent baseline; it also outperforms single-agent best-of- and fixed multi-agent workflows evaluated under a comparable test-time token budget. Comprehensive ablations show that both designer and executor improve throughout training, and that effective automatic MAS learning follows a stagewise co-evolution process. These results establish end-to-end trainable automatic MAS as a practical paradigm for building self-designing and self-executing agentic models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic RL, Multi-Agent System, Dynamic Workflow

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iVxoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces MetaAgent-X, an end-to-end reinforcement learning framework that jointly optimizes automatic MAS design and execution, since it claims that recent works either perform training-free test-time search or optimize the meta-level designer with frozen executors. The method contains two components: Executor Designer Hierarchical Rollout and Stagewise Co-evolution. The hierarchical rollout procedure samples M designs for every query. Each sampled design is executed N times, with decomposed GRPO advantages computed for each run. Stagewise co-evolution alternates every K steps. One role’s trajectories update the policy at each interval, so the two roles do not send gradients simultaneously. The main setup uses a single shared backbone, and prompts distinguish the roles. Training begins with a supervised cold start using DeepSeek-V3.2 trajectories. For Qwen3-8B and Qwen3-4B, the RL model achieves a higher six-benchmark average than both the single-agent baseline and automatic-MAS baselines. The ablation studies compare hierarchical rollout against flatter allocation, stagewise updates against coupled updates, etc.

Critical strengths

The paper notes the co-evolution of designers and executors in multi-agent systems, which may have a wide range of applications in the construction of MAS. It proposes a method that has the same backbone for both writing and executing workflows. For better credit assignment and stable training, it separates the advantages into two sets: "whether the design is good" and "whether this execution is good", and uses stagewise alternating updates since simultaneous updates will collapse into repeated tokens. The experimental ablation is relatively comprehensive, which basically demonstrates the function of all designed components.

Critical weaknesses
  1. The writing is somewhat hard to follow, with some grammar errors. e.g., in line 16 "while keeping downstream execution agents frozen, which creating a frozen-executor ceiling and leaving the end-to-end training of self-designing and self-executing agentic models unexplored"; in line 48 "recent automatic MAS begin to embrace these paradigms, their transition remains incomplete."; in line 75 "The dynamics by which designer and executor could co-evolve under joint training, and where each role’s improvement remains unclear in practice and in understanding the mechanism." is not a complete sentence; in line 194 "we build a training framework contains predefined coordination structures"; The objective is notationally inconsistent, so its dependencies are unclear. Equation (1) defines the return as R(q,d,e)R(q, d, e), with d∼πϑDD(⋅∣q)d \sim \pi^{D}_{\vartheta_D}(\cdot \mid q) and

e∼πϑEE(⋅∣q,d)e \sim \pi^{E}_{\vartheta_E}(\cdot \mid q, d).

Equations (4)–(7) and Appendix B instead write R(ei,j,di)R(e_{i,j}, d_i), dropping qq and reversing the argument order. Equation (8) writes R(e,d)R(e, d) and conditions the executor only on dd, as in πϑEE(⋅∣d)\pi^{E}_{\vartheta_E}(\cdot \mid d), so the query disappears from both the policy and the return. The objective is also denoted J(θ)J(\theta) while the surrounding text uses ϑ\vartheta, ϑD\vartheta_D, and ϑE\vartheta_E. The paper does not state explicitly how J(θ)J(\theta) relates to the parameters in the shared- and split-policy settings.

  1. Inconsistent experimental data analysis: "Search-based Auto MAS baselines degrade when instantiated with these backbones: ADAS falls −6.81 points to 20.35% on 8B, and AFlow reaches only 22.52% on 4B." The fact that AFlow outperforms the single-agent baseline (21.38) on the 4B model cannot be regarded as degradation.

  2. Logical contradiction: The last paragraph of Section 3.3 reverses its own causal claim. The stagewise schedule is introduced to isolate each phase to one reward distribution and to reduce gradient interference between the two roles. The next sentence then describes the co-evolutionary loop and concludes, “As a result, the effective reward distribution becomes non-stationary, and the two role-specific objectives can produce noisy or conflicting updates.” As written, “as a result” attributes non-stationarity and conflicting updates to the proposed loop, which contradicts the stated motivation. If the claim is that the environment remains non-stationary even under alternation, that contrast should be explicit. If it instead describes simultaneous updates, it should not follow the positive description of the stagewise loop.

  3. The hierarchical-rollout ablation does not match the budget it claims to control. Table 3 and Section 4.3 state that rollout variants are matched at M × N = 16 trajectories per query, but the flat variant is M=8, N=1, i.e., 8 trajectories, against M=4, N=4.

  4. The paper does not clarify how the Separate Policy is updated. If strictly following the ablation setup, the Separate Policy is also updated in a stagewise manner. However, the Separate Policy has no issue of mutual gradient interference at all. Stagewise updates are likely a side effect, and the results of normal updates for each Separate Policy should be compared.

  5. Regarding the fairness of the experiments, the method proposed in the paper adopts SFT, while other RL-based methods do not. The paper also fails to compare the performance of its own method when only RL is applied, or the performance of other RL-based methods using SFT-preprocessed models. This introduces a certain unfairness. Since SFT clearly brings performance gains, and as it learns from the outputs of a stronger model, it most likely amplifies the performance improvement brought by the subsequent RL stage. These aspects are not analyzed in the paper.

  6. The paper only identifies three types of workflow topologies. For "automatically designing multi-agent systems", this action space is too small and lacks sufficient persuasiveness. Tables 6 and 7 categorize all workflows into single, reflection, and ensemble. The labels in the case studies also belong to these three categories: math_single_agent, ensemble_judge, and reflection; code_solver_unittest in the code examples is still marked as reflection. The template library in Appendix A also mentions solver-tester and "and etc.", yet the subsequent text provides no distributions for agent counts, edges, or control flow graphs. The authors refer to Table 7 as the coarse skeleton and state that 54 role names appeared during evaluation, 77.5% of prompts were unique per problem, and scripts were also generated per problem. These points illustrate that role names and prompts within the same skeleton vary, but they do not prove that the topology changes. All 54 names may be attached to solver–critic or ensemble–judge. On APPS, single accounts for 55.2%, meaning that more than half of these outputs are not even multi-agent. The routing modified by reinforcement learning is also limited: ensemble drops by roughly 30 percentage points, mostly shifting to reflection. This is more similar to selecting among three predefined templates and using an executor to run the chosen template more effectively, rather than openly designing multi-agent organizations.

Questions for authors
  1. For the hierarchical-rollout ablation, which configuration was actually run? Table 3 lists M=8, N=1 against M=4, N=4, while the caption says both use M × N = 16 trajectories per query. If the flat run used 8 trajectories, the reported gain cannot be attributed to how the budget is allocated.

  2. Was the separate-policy run also updated stagewise? Separate parameters do not write both roles' gradients into one backbone, so stagewise masking can itself discard half of each role's updates.

  3. Did any training-based baseline receive the same DeepSeek-V3.2 trajectories? If not, how much of the gain over those baselines remains after giving them the same cold start?

Reviewer_1mqNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MetaAgent-X, an RL framework that trains two roles of an automatic multi-agent system with one shared Qwen3 backbone:

  • a designer, which writes a Python workflow script per query (roles, prompts, interaction structure);
  • the executor agents, which run that workflow.

The goal is to provide a robust multi-agent post-training framework.

Training starts from an SFT cold start distilled from DeepSeek-V3.2. It then uses GRPO with two mechanisms:

  1. A hierarchical rollout of M=4 designs × N=4 executions per query. The designer’s advantage comes from each design’s mean reward; executor advantages are normalized per question.
  2. A stagewise schedule that alternates, every K=30 steps, which role’s trajectories receive gradients.

Experiments use Qwen3-4B and 8B in non-thinking mode on three math and three code benchmarks. The paper reports average gains of +6.2/+7.5 points over the strongest automatic-MAS baseline and +11.2/+12.8 over a single agent.

It also includes:

  • ablations on rollout allocation, coupled vs stagewise training, and shared vs separate parameters;
  • a token-matched test-time comparison;
  • a continued-SFT control;
  • routing statistics and case studies.
Critical strengths
  1. A relevant, well-posed question. Most existing automatic-MAS methods either search at test time or train only the designer over frozen executors. Training both roles end-to-end, with rollouts separable by role and hierarchy, is a natural next step that requires deep investigation.

  2. Several of the right controls are present.

  • A token-matched test-time comparison (App. E).
  • A continued-SFT control that sees the same data without the RL objective (App. F). Gains are significantly lower than RL (+1.6/+1.5 vs +7.3/+5.0).
  • An explicit acknowledgment that the cold start is necessary.
  • A caveat about AIME granularity.
  • Honest reporting of cells where baselines win.

I think these results suggest the RL stage contributes beyond distillation, but they don't settle the paper’s main claims.

Critical weaknesses
  1. The experimental scale is too small to support the mechanistic conclusions. Several explanations are not tested, just asserted.

On the scale:

  • Training runs on a single 8×H200 node with 4B/8B models.
  • Each step uses 8 queries, for about 180–210 steps. That is fewer than 1,700 training queries in total, half of them math.
  • Every result is a single seed, with no variance or confidence intervals. With small models, one would assume that more seeds can be collected.
  • The recipe has many knobs (M, N, K, learning rate, batch size, SFT data, stage order), and almost all are fixed at one value.
  • K is ablated over three values with one run each. In Fig. 4 the 10-step schedule matches or beats the 30-step one for most of training, so the choice of K=30 rests on the last few steps of one run. Training collapsing in one seed is hard to pin to a single factor, and one may question if another factor is not at play in such cases (see below).

Stagewise vs coupled training: The main empirical case for stagewise training is that the Coupled run collapses (Fig. 3). That is one seed. Collapse into repeated tokens up to the length limit is a common GRPO instability with many possible causes, such as the learning rate or entropy collapse. One may observe no collapse in some seeds for the same setting as the one that did collapse, or collapse for those who did not. Seeds should be reported and error bars displayed.

I am also not entirely convinced by the conceptual motivation:

  • With shared parameters, Eq. 8 is a single objective. Its policy gradient is simply the sum of the designer-token and executor-token terms. Coupled training is the plain gradient estimator; stagewise training drops one of the two terms for 30 steps at a time but doesn't meaningfully change the objective.
  • Because the parameters are shared, an executor-stage update also changes the designer, so nothing is actually isolated.
  • Non-stationarity is inherent to any on-policy RL problem. The cited MARL work is wrongfully interpreted as a legitimate justification for the two stage training. It concerns independent learners that treat each other as part of the environment, not one policy optimizing one objective. It is common for objectives to be the sum of multiple sub-objectives and I fail to understand how things differ in the current context.
  • "Gradient interference" is never measured, for example through per-role gradient norms or the cosine similarity between role gradients. That could help to show that there is interference, although it should be compared with other multi-objective training recipes, even bare regularization terms in the loss as they are usually used.

There is a confound specific to coupling is not discussed: per query, the designer contributes 4 short trajectories while the executor contributes 16 rollouts × several agents, so how the two role losses are weighted matters a lot. This factor may explain why there is some apparent benefit in multi-stage training. The Coupled scores 36.7/25.2 in Table 3 are close to the SFT starting point (37.4/27.0). That is hard to square with a run whose training reward fell to roughly zero, it is therefore unclear on which checkpoint it was evaluated.

Hierarchical rollout budget: Something doesn't really make sense in the budget: the flat variant is described as budget-matched at 16 trajectories per query, but M=8, N=1 gives 8, not 16.

Post hoc explanations:

  • The shared-vs-separate gap (+5.8/+2.6) is attributed to each role's signal acting as an inductive bias for the other, but this is never tested. There could be a simpler explanation: the shared model receives a gradient update at every step, while each separate model is updated only during its own stage.
  • The "frozen-executor ceiling" is also asserted. Its only within-framework support is a single designer-only run.
  1. The baseline comparison is not reliable enough to support the headline gains.

Choice of setting: All experiments use non-thinking mode. For reference, the Qwen3 technical report gives the following AIME'24/'25 scores:

Qwen3-8B Thinking: 76.0/67.3 vs Non-thinking: 29.1/20.9 Qwen3-4B Thinking: 73.8/65.6 vs Non-thinking: 25.0/19.1

The proposed MetaAgent-X's best score is 40.0/33.3 at about 24K tokens per math problem, which is well within thinking-mode budgets. Without a thinking-mode single-agent baseline at a matched token budget, it is unclear whether the learned workflow adds anything beyond simply enabling the backbone’s native reasoning.

Evaluation provenance: Results from different protocols and test sets seem to be mixed in one table. More precisely, with 30 problems averaged over three runs, every valid AIME score is a multiple of 1/90, i.e. a step of about 1.11 points. Several entries are not:

  • The single-agent AIME25 scores of 20.90 (8B) and 19.10 (4B) exactly equal the Qwen3 report's non-thinking figures. Meanwhile the single-agent AIME24 scores (18.3/20.0) sit well below the report's (29.1/25.0).
  • MaAS's 45.8, 29.2 and 25.0 equal 11/24, 7/24 and 6/24. That matches Appendix D's 1:4 train/test split: a 24-problem test set, with the controller trained on the other 6.
  • MaAS's LiveCodeBench score is identical (24.29) for 4B and 8B.

A rigorous setup should be put in place and used for comparison.

Questions for authors

Clarifications

  1. For each cell in Tables 1–2: was it measured by you or taken from prior work? How many problems, how many samples, what temperature? Infra details are very important.
  2. In Table 3, M=8, N=1 gives 8 executions per query, versus 16 for M=4, N=4. What exactly is matched?
  3. How were checkpoints selected for Tables 1–3, and was the test set involved? Why do Fig. 3's stagewise curve and Fig. 4's 30-step curve differ at step 180 (roughly 0.4–0.5 vs 0.34)?
  4. In the Coupled run, how are the designer and executor losses combined and (crucially) weighted? How many seeds were run? Does the collapse persist with loss reweighting, a lower learning rate, or KL/entropy regularization? Did the authors exclude other factors that may cause a collapse?
  5. Eq. 9 starts with an executor stage, while Figs. 2–3 start with a designer stage. Which order was used? Does it matter?
  6. Eq. 2 needs three clarifications:
    • Is the ratio computed at the sequence or token level?
    • Are tool-observation tokens masked?
    • Since G^E_q pools all agent trajectories, designs with more agents dominate μ^E_q. Is that intended?

Missing experiments:

  1. At least three seeds (matched across settings) for Tables 1–3, with confidence intervals or paired per-problem tests.
  2. Two single-agent baselines:
    • thinking mode at a matched token budget;
    • a single agent distilled from the same DeepSeek-V3.2 teacher, then trained with GRPO on the same data for the same number of steps.
  3. The RL-trained executor run inside a fixed reflection workflow, and a fixed single-agent one.
  4. [Optionally] A 2×2 cross-evaluation of {SFT, RL} designer × {SFT, RL} executor. This directly tests the co-adaptation and "ceiling" claims.

Minor:

  • "MAS" is used undefined in the abstract.
  • Notation: N means both stage length (Fig. 2) and executions per design; θ and ϑ are mixed in §3.3; the α terms in Eq. 9 are never used.
  • The last sentence of §3.3 reads as an outcome of stagewise training rather than the problem it addresses.
  • Table 4's motivation rests on a single AIME problem (19/30 → 20/30).
  • "77.5% byte-unique prompts" is automatic, since the designer pastes the problem statement into system prompts.
  • In G.2, the critic's feedback is generic and partly wrong, so "the critic localizes the error" is overstated.
  • Typos: "which creating", "scode"
Reviewer_76FwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a single policy to act both as a Designer, which writes an executable Python multi-agent workflow for a query, and as the Executor agents that run inside that workflow, using GRPO with outcome rewards. The paper proposes: (i) a two-level rollout (M=4 designs × N=4 executions per question) where the designer's advantage is computed from the mean execution reward of each design and the executor's advantage is normalized over all executions of the question; (ii) stagewise co-evolution that alternates executor-only and designer-only updates every K=30 steps. The policy is cold-started by SFT on DeepSeek-V3.2-generated designer/executor trajectories (3K + 8K). On three code and three math benchmarks, MetaAgent-X RL reaches 38.39 / 34.18 average on 8B / 4B, +6.2 / +7.5 over the best auto-MAS baseline (MaAS, ScoreFlow) and +11.2 / +12.8 over direct prompting. Ablations cover hierarchical vs flat rollout, stagewise vs coupled / single-role training, shared vs separate policies, stage length, a token-matched comparison and a continued-SFT control.

Critical strengths
  1. The credit assignment design is a clean way to get role-specific signals without a critic, which targets the limitation of freezing the executor in previous works (MAS-GPT / FlowReasoner / MAS²-style methods).

  2. Ablation studies and experiment results show the gain is not just extra test-time tokens or extra supervised data within the MetaAgent-X pipeline. This is better practice than some auto-MAS papers.

Critical weaknesses
  1. Single-agent baselines are not given the same data and recipe. Qwen3-8B SFT'd on DeepSeek-V3.2 single-agent solutions for the same questions, then GRPO on the same mixture is absent, which would be a fair comparison. In addition, all runs use Qwen3 in non-thinking mode; Qwen3-8B in thinking mode is far above 40% on AIME24, so a single thinking-mode agent would likely beat the entire trained MAS at a comparable token budget.

  2. The search-based baselines are not meaningful. AFlow and ADAS are evaluated with workflows searched under other models and then executed with Qwen3-8B, so they fall far below the single agent (OlympiadBench 35.3 / 32.9 vs 55.0); this does not tell us anything about search-based auto-MAS and should be re-run with the Qwen executor in the loop.

Questions for authors
  1. Could you add a single-agent baseline distilled from DeepSeek-V3.2 single-agent solutions on the same questions, followed by the same GRPO, and report Qwen3-8B thinking-mode single-agent accuracy on the same benchmarks with its token usage?

  2. In Table 8, is "self-consistency / best-of-8" majority voting or oracle pass@8, and which checkpoint (base, MetaAgent-X SFT, or SA+GRPO) is used for those rows?

  3. Could you re-run AFlow search with the Qwen3-8B executor and the same rollout budget, so the search-based rows are a fair comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.