acceptodds
Under review as a conference paper at ICLR 2027

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs

Abstract

Outcome-reward reinforcement learning (RL) can train every role of a multi-agent LLM workflow end to end. Its payoff is hard to predict, because the reward does not credit individual roles and a shared policy trades pooled data against specialization. We ask when this training improves a workflow over its base model and single-agent RL, and what each role learns. We train three workflows (Eval-Opt, Voting, Orch-Workers) with GRPO on math and code at three Qwen3 scales, with one policy shared by all roles (Shared-Policy, SP) or one per role type (Isolated-Policy, IP). RL lifts every workflow above its untrained version. The solving roles, which write candidate solutions, end no better than single-agent RL, so a workflow exceeds single-agent RL only through the structure above them, whose gain is set by the simple rule its decision role learns. Eval-Opt gains most because its revision loop gives truncated first attempts another turn, extending the token budget. IP learns faster and peaks higher than SP. Under SP, averaging each output's loss over its tokens lets a decision role with much shorter outputs take well over its uniform share of the shared gradient, and the solving roles learn slowly. Under IP, the adapter of a role run three times per episode can become unstable after the peak, and only that role's policy-ratio and perplexity rise. A stopping rule on training success fires only after the peak but can return a checkpoint near it, and SP runs decline little.

Primary area:
reinforcement learning
Keywords:
Agentic RL, LLM Agent, Multi-agent System, Multi-agent Reinforcement Learning, LLM Workflows, Credit Assignment, Parameter Sharing, GRPO, Training Stability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CuwhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies outcome-reward GRPO across 36 configurations of three workflows, three Qwen3 sizes, math/code tasks, and shared or role-specific LoRA policies. Comparisons with single-agent RL identify gains from extra generation or aggregation. Role-level analyses examine gradient imbalance under shared policies and post-peak decline under isolated policies.

Critical strengths
  1. The systematic comparisons and role-level measurements clarify where workflow gains come from. In Table 4, three-sample SA-RL voting reaches 57.8% at 1.7B, matching Eval-Opt-IP’s 57.6%.

  2. Gradient diagnostics and control experiments help separate broadcast-reward issues from training instability. The monitoring analysis also shows that training-success alarms can recover a near-peak checkpoint, although they fire after the peak.

Critical weaknesses

My main concern is that the narrow experimental scope limits the paper’s general insights into when multi-agent RL is useful.

  1. Only three fixed workflows are evaluated on non-interactive math and code tasks. It is unclear whether the findings extend to more general systems involving dynamic delegation, tool use, or longer interactions.

  2. SP/IP comparisons are restricted to Qwen3 models at 0.6B, 1.7B, and 4B. The conclusions lack validation on stronger models and other model families, and may depend on the specific training configuration.

  3. The truncation-based explanation is tested only on these small models under fixed response caps. Its relevance with longer generation budgets remains unclear. Although Table 4 includes a three-sample baseline, comparisons against longer single-agent responses or continuation without an evaluator are missing.

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response to Reviewer Cuwh (1/3)

Response to Reviewer Cuwh

Thank you for the review. W3 is answered by two new experiments (Tables C and D of the general comment). W1 and W2 concern scope. Within that scope, the controlled grid separates the gain of a workflow from effects of the training recipe and of the generation budget, and Tables A to D test these effects by intervention.

W3: longer single-agent responses, and continuation without an evaluator

"Although Table 4 includes a three-sample baseline, comparisons against longer single-agent responses or continuation without an evaluator are missing."

We ran both comparisons (Tables C and D). Table C evaluates the SA-RL checkpoint with a longer response cap. Table D replaces the trained evaluator by a fixed rule that starts a new attempt only after a cut-off attempt, with no evaluator call.

  • At 1.7B and 4B, a longer single response closes the lead of Eval-Opt-IP to within about one point at about half the generated tokens, and a fixed implementation of the retry rule performs about as the trained evaluator does.
  • At 0.6B, the lead of Eval-Opt-IP remains at both caps, at 14.1k against 3.9k generated tokens per episode. The fixed rule is about five points below the trained evaluator, whose feedback often states the answer.

These results confirm the truncation explanation. If Eval-Opt's lead comes from giving truncated attempts another turn, the lead should shrink where the single model can use a longer response and stay where it cannot. That is what Table C shows at 1.7B and 4B against 0.6B.

The general point is that multi-agent RL learned the workflow's strategy. In this setting a new attempt after a cut-off attempt is an effective strategy: it lifts the single-agent checkpoint by two to seven points (Table D). Nobody specified it, and two kinds of evidence show that the evaluator learned it from the outcome reward. Table 2 gives the evaluator's decisions on wrong first attempts: under isolated policies it rejects at least 90% of the cut-off ones and accepts most finished ones. Table D tests the replacement of the evaluator by this rule (first item above). At 1.7B the trained generator is also 4.5 points above the single-agent generator under the same rule, with 6.7k against 4.1k generated tokens per episode (Table D, last row). We will add Tables C and D to Section 4.2.

AuthorsOpenReview ↗

Response to Reviewer Cuwh (2/3)

W1: three fixed workflows on non-interactive tasks

"Only three fixed workflows are evaluated on non-interactive math and code tasks. It is unclear whether the findings extend to more general systems involving dynamic delegation, tool use, or longer interactions."

What carries over to tool use and longer interactions is the diagnostic. Each training effect depends on one quantity of the training recipe, not on a topology or a task: a role's share of a shared gradient under the per-sequence mean, and the optimizer steps an adapter takes per training step. Both quantities can be measured in any system that trains several roles, and tool use and longer interactions change their values.

  • Tool use. A role that issues tool calls adds short sequences to a shared policy's batch, which is the condition of Section 4.3.
  • Longer interactions. A longer interaction gives an adapter more sequences per training step and therefore more optimizer steps, which is the condition of Section 4.4. Eval-Opt is the instance in the grid, where the trained evaluator sets the number of turns.

A system that shares one policy across roles with unequal output lengths, or runs several calls of a role on one adapter, meets the same conditions. Tables A and B show that changing the condition changes the behavior.

We fixed the set of roles so that the comparison is controlled and each result can be attributed to a role. With the same roles and the same number of instances in every episode, we can grade each role on its own answer, measure per-role gradient shares, and compare the same workflow under a shared and an isolated policy. Dynamic delegation, where the policy chooses the sub-agents and their number during training, is the next step. We will name it in the Limitations section and state the fixed-role reason in the setup.

The three workflows are representative patterns of the taxonomy we cite (Anthropic's "Building effective agents"): one iterative pattern (evaluator-optimizer), one parallel pattern (voting) and one hierarchical pattern (orchestrator-workers; the orchestrator writes input-dependent instructions for a fixed set of three workers). They differ in the two structural properties the findings depend on: whether a short-output decision role shares a policy with long-output solving roles, and whether a role runs several instances on one adapter.

The controlled experiments run in one cell, and the grid shows that their effects are not specific to it. We ran them in the cell where both training effects are strongest (Voting at 1.7B on Math). The shared policy's slower learning appears in 17 of 18 matched pairs on both tasks, and the post-peak decline under isolated policies appears in four grid cells and three separately trained variants (Table 19). The control cell is one instance of a pattern the grid shows, which is what makes it representative.

AuthorsOpenReview ↗

Response to Reviewer Cuwh (3/3)

W2: Qwen3 at 0.6B, 1.7B and 4B, and the training configuration

"SP/IP comparisons are restricted to Qwen3 models at 0.6B, 1.7B, and 4B. The conclusions lack validation on stronger models and other model families, and may depend on the specific training configuration."

Scale is an axis of the study, and we report where conclusions change with it. Within the grid, three conclusions change with scale:

  • a longer single response closes most of the lead of Eval-Opt-IP at 1.7B and 4B but not at 0.6B (W3 above);
  • under isolated policies, Orch-Workers exceeds one SA-RL response on Math by about five points at 0.6B and 1.7B and by about one point at 4B (Table 1);
  • isolated policies peak above shared policies in every Math pair except Orch-Workers at 4B (Table 1).

Training configuration. All 36 cells and the single-agent baselines use the training framework's default GRPO and LoRA configuration for each task, with no per-cell tuning. The only hyperparameter difference is a short warmup on nine Code cells and the 0.6B single-agent Code baseline (App. B.1). The shared configuration is what makes the comparison controlled, since differences between cells, and between a cell and its single-agent baseline, are then not tuning differences. AT-GRPO and Dr. MAS likewise train all the methods they compare with one shared configuration. Whether a finding depends on a component of the recipe is a separate question, and we test it by varying that component directly: the reward (per-voter reward, Table 19), the loss weighting across roles (Table A), the repeated role's optimizer steps (Table B) and the response cap (Table C).

Model family and range. Controlled studies of end-to-end RL for multi-agent LLM systems use the same model family at overlapping sizes. AT-GRPO compares shared and role-specific policies on Qwen3 at 1.7B and 8B. Dr. MAS compares them on Qwen3 at 4B and 8B and Qwen2.5 at 3B and 7B. MAPoRL trains models of 3B to 8B. Each uses one to four fixed workflows. Our contribution within this range is the controlled cross of workflow, scale, task and policy sharing.

Cost of the study

Training every role end to end in 36 cells and six single-agent baselines took about 12,400 GPU-hours, and each controlled experiment is a further run of several days. This cost is the reason the grid has three workflows and stops at 4B, and the reason the controlled experiments run in one cell.

Reviewer_ouK3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates when end-to-end outcome-reward RL benefits multi-agent LLM workflows and what individual roles learn. It trains 36 configurations spanning three workflows (Eval-Opt, Voting, and Orch-Workers), math and code tasks, three Qwen3 model sizes, and shared versus role-specific LoRA policies using GRPO with a final-answer reward broadcast to all roles.

All workflows improve over their untrained versions. However, when candidate solutions are graded individually before revision or aggregation, the solving roles at the latest sampled checkpoints perform no better than a model trained with single-agent RL. Eval-Opt’s gains primarily come from additional generation turns for truncated attempts. On math, independently sampling the single-agent RL model three times and taking a plurality vote achieves accuracy close to Eval-Opt with role-specific policies and matches or exceeds all trained Voting and Orch-Workers configurations. Role-specific policies generally learn faster and reach higher peaks, while shared-policy gradient measurements reveal disproportionate contributions from short-output decision roles under sequence-mean loss aggregation. Several role-specific math runs subsequently exhibit performance declines associated with instability in repeated solving roles.

The analysis combines baseline comparisons, role-level output grading, token and truncation statistics, gradient diagnostics, and control experiments varying rewards, prompts, and adapter assignment or removing the workflow. It also evaluates a training-success-based rule for selecting near-peak checkpoints.

Critical strengths
  1. The study distinguishes improvements in individual solving ability from gains provided by workflow structure and additional sampling. Comparing untrained workflows, single-agent RL, and three-sample single-agent voting on math, together with role-level output grading, shows that higher workflow accuracy does not necessarily reflect stronger individual solvers or effective learned coordination. In particular, the analysis traces Eval-Opt’s gains primarily to recovering truncated attempts. This provides useful evidence for deciding whether joint workflow training offers benefits beyond simpler uses of a single-agent policy.

  2. The paper goes beyond final accuracy to characterize how policy sharing affects learning. Per-role gradient measurements and a length-based model support the observation that short-output decision roles receive disproportionate gradient shares under shared policies. Separately, controls involving per-voter rewards, different prompts, and removal of the workflow show that post-peak decline can persist without either the workflow or the broadcast reward. These analyses narrow plausible explanations and provide actionable guidance on role-level monitoring and checkpoint selection, even though the causes are not fully established.

Critical weaknesses
  1. While the role-level analyses provide valuable evidence of the policy-sharing tradeoffs, the causal explanations would benefit from more targeted controls. In particular, changing the loss aggregation would help establish whether the measured gradient concentration contributes to slower solving-role learning under shared policies. For the post-peak decline under role-specific policies, the current controls do not fully separate optimizer-step count from sample count. The single-role control retains both the increased sample count and the additional optimizer steps, while the one-adapter-per-voter experiment changes the reward as well as the adapter assignment. These remaining ambiguities limit the extent to which the findings support specific recommendations for improving training.

  2. The truncation analysis provides a convincing account of Eval-Opt’s gains under the studied response limit, but its applicability to larger token budgets remains uncertain. The three-sample single-agent voting baseline is a useful comparison; however, each sample remains subject to the same 5,120-token math cap. It therefore does not establish whether similar gains could be obtained by allowing a single reasoning trajectory to continue beyond that limit. A comparison with longer single-agent responses would help clarify how much of the observed advantage depends on the per-call token limit and the conditions under which the revision workflow remains beneficial.

Questions for authors
  1. The gradient analysis suggests that short-output decision roles receive a disproportionate share of the shared-policy gradient, potentially slowing learning in the solving roles. Have you tested whether switching from sequence-mean to token-level loss aggregation changes the gradient distribution and improves solving-role learning in a representative setting? This would help establish whether the measured imbalance contributes to the learning-speed gap between shared and role-specific policies.

  2. The controls show that post-peak decline can occur without the workflow or the broadcast reward, but the repeated role still receives more samples and takes more optimizer steps per training step than the single-agent baseline. Have you tested keeping the rollout samples and reward unchanged while increasing the repeated role’s mini-batch size so that it takes one optimizer step per training step, matching the single-agent baseline? Whether the decline persists would help clarify the role of the update schedule in the observed instability.

  3. The truncation analysis shows that Eval-Opt often benefits from giving incomplete attempts additional generation turns. However, the three-sample voting baseline does not test whether allowing one uninterrupted reasoning trajectory to run longer would provide similar benefits. Could you compare against single-agent RL with a larger response cap and report both accuracy and actual generated tokens? Clarifying whether the larger cap is applied only during evaluation or also during training would help interpret how much the observed advantage depends on the original token limit.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ouK3 (1/2)

Response to Reviewer ouK3

Thank you for the careful reading. We ran the three experiments you ask for; the tables are in the general comment.

  • Q1. Reweighting the roles' loss speeds up solving-role learning under a shared policy (Table A).
  • Q2. With one optimizer step per training step, the decline of the reference does not appear and the learning speed is kept (Table B).
  • Q3. One longer single-agent response reaches about the accuracy of Eval-Opt-IP at 1.7B and 4B, and not at 0.6B (Table C).

Q1 and W1: loss aggregation under a shared policy

"Have you tested whether switching from sequence-mean to token-level loss aggregation changes the gradient distribution and improves solving-role learning in a representative setting?"

(a) Yes, the aggregation changes the gradient distribution. We computed the decision role's share of the gradient norm in two ways on the same sampled outputs of shared-policy checkpoints. One is the training aggregation (mean over each sequence's tokens, then over sequences); the other is a mean over all tokens of the batch.

Shared-policy run (probed steps)Decision role's share as trainedShare under an all-token meanIts share of the sequences
Voting-SP-1.7B-Math (230, 300), aggregator0.80–0.910.25–0.310.25
Eval-Opt-SP-1.7B-Math (150), evaluator0.930.260.50

As trained, the decision role holds most of the gradient. Under an all-token mean it holds about its share of the sequences or less. The 4B Voting and Orch-Workers shared runs show the same direction: the decision role holds well over its share of the sequences as trained and about that share under an all-token mean.

(b) Yes, giving each role its all-token-mean weight speeds up solving-role learning. We retrained Voting-SP-1.7B-Math with the per-role weights of Table A, under which each role's total weight equals its total under an all-token mean. On the outputs above, the voters then hold 0.71 to 0.78 of the gradient (0.09 to 0.20 as trained). Table A shows three things.

  1. The gradient share contributes to the slow early learning of the solving roles. The reweighted shared policy reaches 90% of the common gain in about half the steps of the shared reference, and its voters close a little more than half of their early gap to the isolated voters.
  2. With the imbalance removed, the shared policy learns at about the isolated run's speed. It reaches its peak at step 190, the step at which the isolated run peaks, while the shared reference is still rising at step 300.
  3. Speed and stability are separate outcomes. Table A tests the speed claim of Section 4.3 and confirms it. The reweighted shared policy then declines as the isolated run does. This is the post-peak decline that Section 4.4 studies, so the run's accuracy at step 300 does not measure the learning-speed effect.

Three token-level aggregations appear in this answer, and each tests a different thing.

  • The probe of (a) takes one mean over all tokens of the batch, offline, on fixed outputs. It measures how the aggregation changes the roles' gradient shares.
  • The role-length weights of Table A give each role the total weight it has under that all-token mean and keep the sequences within a role equal. They test the role shares alone. Within a role, a token mean also up-weights the longest outputs, including the truncated ones, and this would mix a second change into the role-share question.
  • The training framework's token-level aggregation, which we also ran, takes the token mean within each micro-batch of a mini-batch. A token is then weighted by the inverse of the mean response length of its micro-batch. This run tests the switch you name, in which the role shares and the weights within a role change together.

With only the loss aggregation changed, the token-mean run reaches 90% of the common gain of Table A at step 160, against 281 for the shared reference and 136 for the reweighted shared policy. The run is validated to step 200 so far, and we will add the remaining steps to 300 when it finishes.

Voting-SP-1.7B-MathSteps to 90% of the common gainAccuracy at step 100Step 150Step 200
Shared reference28134.939.242.4
Role-length weights (Table A)13639.944.548.5
Framework token mean (new)16039.944.245.5
AuthorsOpenReview ↗

Response to Reviewer ouK3 (2/2)

Q2 and W1: optimizer steps against sample count

"Have you tested keeping the rollout samples and reward unchanged while increasing the repeated role’s mini-batch size so that it takes one optimizer step per training step ...?"

Yes, we ran this experiment (Table B). The decline of the reference does not appear, and the learning speed is kept. The samples and the reward are unchanged. Only the update schedule changes: the repeated role's mini-batch is enlarged so that its adapter takes one optimizer step per training step.

  • From step 190 to 300 the reference loses 10.2 points and the one-step run gains 3 points.
  • Both runs reach 90% of the common gain at about the same step (143 against 146) and are level at step 190.

The update schedule of the repeated role is therefore the component behind the decline. We will state this control in Section 4.4 in place of the one-adapter-per-voter run.

Q3 and W2: one longer single-agent response

"Could you compare against single-agent RL with a larger response cap and report both accuracy and actual generated tokens? Clarifying whether the larger cap is applied only during evaluation or also during training would help ..."

Yes (Table C). At 1.7B and 4B one longer response reaches about the accuracy of Eval-Opt-IP, and at 0.6B Eval-Opt-IP keeps its lead. Table C reports accuracy and generated tokens. The larger cap is applied only at evaluation; we have not trained SA-RL with a longer cap.

  • At 1.7B and 4B, one longer response needs about half the generated tokens of Eval-Opt-IP and one call instead of about four. The differences in accuracy are within about one point at both caps (Table C, last two rows). Under the same retry rule (Table D, last row), the Eval-Opt-IP generator is 4.5 points above the SA-RL generator at 1.7B and generates 6.7k against 4.1k tokens per episode. At that scale the two generators use the retry structure differently.
  • At 0.6B, Eval-Opt-IP keeps a lead of about five points at both caps, at 14.1k against 3.9k generated tokens per episode.

So your reading is correct. At 1.7B and 4B the advantage of Eval-Opt-IP over single-agent RL holds at the same per-call cap and closes against a single response with a larger budget. This agrees with the explanation in Section 4.2. The revision workflow remains ahead where a longer single response does not help the single agent enough, which in our grid is 0.6B. We will state this condition next to the Eval-Opt result.

W1: recommendations for training

"These remaining ambiguities limit the extent to which the findings support specific recommendations for improving training."

With Tables A and B the paper identifies two settings of multi-agent training that single-agent RL does not have: a role's share of a shared gradient under the per-sequence mean, and the optimizer steps a repeated role's adapter takes per training step. It measures an intervention on each. The recommendation we draw is one check per strategy.

  • Under a shared policy, measure each role's share of the gradient. If the solving roles hold only a small share, reweight the loss by role (Table A). This speeds up the solving roles. The reweighted run then shows the post-peak decline, which Table B addresses under isolated policies.
  • Under isolated policies, set the mini-batch so that a repeated role's adapter takes one optimizer step per training step (Table B).

Thank you for the additional experiments and for incorporating them into the revision. Most of my questions have been resolved, and I will maintain my current score.

Thank you for reading our response and for your thoughtful feedback.

Reviewer_sftVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares different multi-agent systems in different workflows, model sizes and policy-sharing strategies and analyzes their gain compared to a single agent. Phenomena such as role-level gradient imbalance and late-stage training degradation are also studied.

Critical strengths
  1. This work not only reports a final score, but also analyzes the role performance, gradient and training strategy, providing valuable diagnoses.
  2. This work provides comparators including multiple samples from a single agent and individual rewards, which are helpful to distinguish collaboration gains, additional generation opportunities and reward misalignment.
Critical weaknesses
  1. The core mechanism lacks direct validation. For example, it is still unclear whether .... This harms the convincingness that ...
  2. There is only one training seed for each setting, and the best checkpoint is selected using the validation set. The computational budget is also not matched. So robustness across settings is still unclear.
Questions for authors
  1. Could the authors provide controlled experiments that only change the loss weight of each role or ...?
  2. Can the conclusion be supported using multiple training seeds, an independent test, and computational budget matching?
5 comments
AuthorsOpenReview ↗

Response to Reviewer sftV (1/3)

Response to Reviewer sftV

Thank you for the review. Q1 is answered by Table A of the general comment, with a second control of the same kind in Table B. Q2 is answered by two additional training seeds, replication counts across the grid, held-out checkpoint selection, independent test sets and an accounting of the budget.

Q1 and W1: controlled experiments on the mechanisms

"The core mechanism lacks direct validation."

"Could the authors provide controlled experiments that only change the loss weight of each role or ...?"

Yes. Table A changes only the roles' loss weights, and Table B changes only the repeated role's optimizer steps. Both experiments run in one cell (Voting at 1.7B on Math) and keep everything else fixed. The table below lists the test behind each explanation in Section 4.

Explanation in the submissionTestResult
§4.3: under a shared policy the decision role takes most of the gradient and the solving roles learn slowlyreweight the roles' loss so that each role has the total it would have under an all-token mean (Table A, new)reaches 90% of the common gain in about half the steps (136 against 281); the voters close about half their early gap to the isolated voters (+4.1 of +7.3); peaks at step 190 and then declines, both as the isolated run does
§4.4: under isolated policies the repeated role's extra optimizer steps are tied to the declineone optimizer step per training step, samples and reward unchanged (Table B, new); single-role and per-voter-reward controls (Table 19)Table B: the decline does not appear (from step 190 to 300 the run gains 3 points and the reference loses 10.2). Table 19: the decline persists without the workflow or the broadcast reward
§4.2: Eval-Opt gains by giving truncated attempts another turnthe evaluator's decisions on wrong first attempts (Table 2); a fixed rule "new attempt only after a cut-off attempt", no evaluator (Table D, new); SA-RL with one longer response (Table C, new)Table 2: under isolated policies the evaluator rejects at least 90% of the cut-off attempts and accepts most finished ones. Table D: a fixed implementation of this rule performs about as the trained evaluator does at 1.7B and 4B. Table C: a longer single response closes the lead to within about one point at 1.7B and 4B, not at 0.6B
§4.1: decision roles learn simple rulesaggregator against a plurality vote over its inputs (Table 2, offline on the same outputs)the trained aggregator is within one point of plurality at 0.6B and 1.7B, and about three points above it at 4B under isolated policies

Q2 and W2: seeds, checkpoint selection, independent tests, budget

"Can the conclusion be supported using multiple training seeds, an independent test, and computational budget matching?"

We address each of the three requests, and the checkpoint selection raised in W2.

  • Multiple training seeds. We trained two additional seeds of Voting-IP-1.7B-Math, the cell the decline and both new controls rest on. Both peak at about the same step and accuracy as the submitted run and then decline.
  • Consistency across the grid. Each headline finding is a pattern across independently trained runs.
  • Checkpoint selection and independent tests. Held-out selection changes the selected peaks by a median of 0.8 points, and the main claims hold on four independent test sets.
  • Budget. IP and SP receive the same training inputs. Their training compute and generated tokens are not matched, and we report the token ratio. Against three single-agent comparators, Eval-Opt-IP is within about one point at 1.7B and 4B and keeps about three points at 0.6B.

Training seeds. We trained two additional seeds of Voting-IP-1.7B-Math, the cell on which the post-peak decline and both new controls rest.

RunPeak accuracy (step)Accuracy at step 300Change from peak to step 300
Submitted run50.9 (190)40.7−10.2
Seed 1 (new)51.2 (180)37.1−14.1
Seed 2 (new)50.9 (190)33.6−17.3

The post-peak decline appears in both new seeds. Each peaks within ten steps of the submitted run, at about the same accuracy, and is 14 to 17 points below its peak at step 300.

AuthorsOpenReview ↗

Response to Reviewer sftV (2/3)

Consistency across the grid. This is a second kind of support, separate from the seeds. Each headline finding is a pattern across independently trained runs:

FindingIndependently trained runs showing it
RL lifts every workflow above its untrained version36 of 36 cells
Solving roles end below SA-RL when graded on their own answers18 of 18 Math cells
Eval-Opt-IP exceeds one SA-RL response3 of 3 Math scales, by 7.6 to 8.6 points
IP reaches 90% of the common gain sooner than SP17 of 18 pairs
Post-peak decline under IP with a repeated role at 1.7B and 4B4 grid runs (Voting and Orch-Workers at both scales), plus three separately trained variants of Voting-IP-1.7B-Math that change the reward, the prompts or the workflow (Table 19, peaks at steps 150 to 240, drops of six to seven points) and the reweighted shared policy (Table A)

The grid itself is trained once per cell because of cost. The 36 cells and six single-agent baselines took about 12,400 GPU-hours of training plus about 1,200 GPU-hours of Math checkpoint evaluation. Two more seeds of the grid would cost about 20,000 to 25,000 GPU-hours. We spent the budget on the grid because the paper's question is where multi-agent RL helps, which is a question about the axes.

Checkpoint selection. We re-selected every Math peak on held-out problems: the peak is chosen on a random half of the 1412 problems and scored on the other half, over 200 splits. The optimism of in-sample selection is small (median 0.8 points, at most 2.7). Under held-out selection:

  • 17 of the 18 Math cells keep the sign of their margin over one SA-RL response;
  • Eval-Opt-IP keeps +8.6 to +8.8 over SA-RL at every scale;
  • IP is above SP in 7 of 9 Math pairs;
  • the declines of Voting-IP and Orch-Workers-IP at 1.7B and 4B appear on the held-out half in every split.

Independent test sets. We evaluated the validation-selected checkpoints, without re-selecting them, on four test sets: MATH-500 (474 problems after removing training duplicates, 4 samples each), AIME 2024 and AIME 2025 (30 problems, 32 samples each), and the DeepCoder test split (687 problems, no overlap with training).

ClaimValidationTest sets
Eval-Opt-IP minus SA-RL on Math, at 0.6B / 1.7B / 4Babove at 3 of 3 scales (+7.6 to +8.6)MATH-500: +3.2 / +1.6 / +2.5; AIME 2024: +6.6 / +10.4 / +5.2; AIME 2025: +6.4 / +4.2 / +9.0
Sign of IP − SP on Math9 pairsmatches validation in every pair-by-set comparison but one (Voting at 0.6B on AIME 2025); for Eval-Opt, IP − SP is +3.6 to +12.7 over the nine scale-by-set comparisons
Post-peak decline of Voting-IP-4Byesthe last checkpoint is below the peak on MATH-500, AIME 2024 and AIME 2025 (−4.1, −13.9, −7.0)
Eval-Opt-IP minus SA-RL on Code, at 0.6B / 1.7B / 4Babove at 0.6B and 1.7BDeepCoder test: +3.3 / +3.2 / +0.3
AuthorsOpenReview ↗

Response to Reviewer sftV (3/3)

Computational budget. The comparisons fix what each system is given; the tokens a trained system generates are a result of training. We therefore separate three quantities.

  • Training inputs are matched. An IP run and its SP counterpart use the same workflow, the same calls per episode (four in Voting, five in Orch-Workers, the same maximum number of turns in Eval-Opt) and the same response cap per call. They train on the same problems, with the same episodes per problem and training steps. Before training they are the same system and generate the same tokens.
  • Training compute is not matched. It grows with the tokens the roles generate, and these differ once the roles are trained (table below).
  • Inference tokens are reported next to accuracy. Tables C and D give the generated tokens per episode of each system.

IP against SP. The table gives the ratio of IP to SP in generated tokens per episode after training. It uses the full validation set with each run at its selected peak (Eval-Opt at 4B on Math: the evaluated checkpoint within 10 steps of it).

IP / SP, generated tokens per episode0.6B1.7B4B
Voting, Math1.121.301.25
Eval-Opt, Math1.231.071.71
Orch-Workers, Math1.011.120.87
Voting, Code1.050.971.50
Eval-Opt, Code0.581.770.69
Orch-Workers, Code0.920.821.15

IP generates more tokens than SP in most Math pairs and in about half of the Code pairs. Training produces this difference. In Voting at 1.7B the untrained workflow generates 5.1k tokens per episode. The number rises in every run as the voters learn to write longer solutions: to 7.7k by step 300 under SP and to 10.0k by step 190 under IP. The control of Table A changes only the loss weights of the SP run, and its token use then rises as under IP (9.6k at step 190). The token difference therefore follows from how fast the solving roles learn, which is the difference Section 4.3 studies. In Eval-Opt the trained evaluator also decides the number of turns (Section 4.2), and the ratios vary most there.

Workflows against single-agent RL. Here the additional calls are the workflow itself, so we compare against SA-RL in three forms: three samples with a plurality vote (Table 4), one longer response (Table C) and the same retry rule (Table D). Against the strongest of these, Eval-Opt-IP keeps a lead of about three points at 0.6B and is within about one point at 1.7B and 4B. SA-RL's three-sample plurality matches or exceeds every trained Voting and Orch-Workers cell on Math (Table 4).

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to All Reviewers (1/2)

General comment to all reviewers

We thank the reviewers for three careful reviews. The new experiments sharpen three conclusions of the paper.

  1. Why a shared policy learns slowly (Section 4.3, Table A). Reweighting the roles' loss, with nothing else changed, halves the training steps the shared policy needs to reach 90% of the common gain (136 against 281).
  2. The post-peak decline follows the update schedule (Section 4.4, Table B). With samples and reward unchanged, one optimizer step per training step for the repeated role keeps the learning speed, and the decline of the reference does not appear (54.2 against 40.7 at step 300).
  3. Where the accuracy gain of Eval-Opt needs the workflow (Section 4.2, Tables C and D). At 1.7B and 4B, one longer single-agent response comes within about one point of Eval-Opt-IP at about half the generated tokens. At 0.6B Eval-Opt-IP keeps a lead of about five points, at 14.1k against 3.9k generated tokens.

Learning speed and stability are separate outcomes. The reweighted shared policy of Table A learns faster and then shows the post-peak decline that Section 4.4 studies. Table B addresses that decline under isolated policies.

Robustness. Held-out checkpoint selection, four independent test sets and two additional training seeds of one cell test the paper's main claims (in the response to Reviewer sftV). Both new seeds show the post-peak decline.

Tables. Several questions ask for the same experiments. We report them once here, and each response refers to the tables by letter.

  • Table A: the controlled experiment on the roles' loss weights that Reviewers sftV and ouK3 ask for.
  • Table B: the experiment Reviewer ouK3 describes in Q2.
  • Tables C and D: the single-agent comparators that Reviewers ouK3 and Cuwh ask for.

Table A. Shared policy, loss reweighted by role length (Voting at 1.7B on Math)

The reference is Voting-SP-1.7B-Math. The new run multiplies each sequence's loss by its role's mean output length divided by the overall mean, computed per training step. Each role then has the total weight it would have under a mean over all tokens of the batch, while sequences within a role keep equal weight. Data, batch size, reward, learning rate and optimizer steps are unchanged. The common gain of two runs is the smaller of their two gains over the untrained workflow.

Shared referenceShared, role-length weights (new)Isolated reference
Training steps to 90% of the common gain281136109
Voters' accuracy on their own answers at step 120, minus the shared referencen/a+4.1+7.3 (step 130)
Peak accuracy (step)46.5 (300)49.2 (190)50.9 (190)
Change from step 190 to step 300+5.4−14.5−10.2

Table B. Isolated policies, one optimizer step per training step for the repeated role (Voting at 1.7B on Math)

The reference is Voting-IP-1.7B-Math, in which the voters' adapter takes three optimizer steps per training step. The new run enlarges the mini-batch threefold so that this adapter takes one step per training step, as the single-agent baseline does. The rollouts (64 problems, 8 episodes each, 3 voters), the broadcast reward, the learning rate and the aggregator's single step are unchanged.

Training step100190250300
Reference: three steps per training step42.950.9 (peak)47.040.7
One step per training step (new)41.951.152.054.2

From step 190 to 300 the reference loses 10.2 points. The new run gains 3 points over the same steps and is 13.5 points above the reference at step 300. The learning speed is the same. Both runs reach 90% of their common gain at about the same step (143 against 146) and are level at step 190 (50.9 against 51.1).

AuthorsOpenReview ↗

Response to All Reviewers (2/2)

Table C. Single-agent RL with one longer response

The SA-RL checkpoint at its selected peak, trained with the 5,120 cap, is evaluated on Math with a larger response cap. Each cell gives accuracy / truncated / tokens. Rows average three samples per problem. "Truncated" is the share of responses that reach the cap (for Eval-Opt-IP, of first attempts). Tokens are generated tokens per episode, summed over all calls. Eval-Opt-IP keeps its 5,120 cap per turn.

System0.6B1.7B4B
SA-RL, one response, cap 5,120 (as trained)31.4 / 24% / 3.5k49.9 / 29% / 3.3k66.6 / 23% / 2.9k
SA-RL, one response, cap 10,240 (new)33.8 / 1% / 3.9k57.4 / 4% / 4.0k74.6 / 4% / 3.5k
SA-RL, one response, cap 15,360 (new)34.6 / 0% / 3.9k56.4 / 1% / 4.1k76.3 / 1% / 3.6k
Eval-Opt-IP, trained, cap 5,120 per turn39.5 / 90% / 14.1k57.6 / 58% / 7.5k75.2 / 60% / 7.6k
Accuracy: Eval-Opt-IP minus SA-RL at cap 10,240+5.7+0.2+0.6
Accuracy: Eval-Opt-IP minus SA-RL at cap 15,360+4.9+1.2−1.1

Table D. A fixed retry rule in place of the evaluator

"New attempt after a cut-off attempt" gives the generator a new response of up to 5,120 tokens whenever its attempt reached the cap, up to three attempts. The cut-off attempt stays in context, and a fixed message replaces the evaluator's feedback. No evaluator is called. This is the retry structure of Eval-Opt without the trained evaluator. The rule is applied at evaluation, on Math, to the trained checkpoints. Each cell gives accuracy / tokens.

System0.6B1.7B4B
SA-RL, one response, cap 5,12031.4 / 3.5k49.9 / 3.3k66.6 / 2.9k
SA-RL, new attempt after a cut-off attempt (new)33.3 / 4.0k55.9 / 4.1k73.3 / 3.4k
Eval-Opt-IP generator, new attempt after a cut-off attempt (new)34.3 / 11.2k60.4 / 6.7k74.9 / 7.1k
Eval-Opt-IP, trained evaluator39.5 / 14.1k57.6 / 7.5k75.2 / 7.6k
Accuracy: fixed rule minus trained evaluator, Eval-Opt-IP generator−5.2+2.8−0.3
Accuracy: Eval-Opt-IP generator minus SA-RL generator, both under the fixed rule+1.0+4.5+1.6
AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded a revised PDF. Text added or changed in response to the reviews is shown in blue. The section, table and figure numbers of the submission are unchanged.

New experiments (Appendix F)

  • Tables C and D (§4.2; method in F.1): single-agent RL with one longer response, and a fixed retry rule in place of the trained evaluator.
  • Figure 10 (a) and Table E (F.2): loss weighting under a shared policy (Table A of our responses), and the decision role's share of the shared gradient.
  • Figure 10 (b) (F.3): one optimizer step per training step for the repeated role (Table B of our responses).
  • Figure 10 (c) and Table F (F.4): two additional training seeds of Voting-IP-1.7B-Math, and the main claims under held-out checkpoint selection and on independent test sets.
  • Table G (F.5): generated tokens per episode under isolated and shared policies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.