- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies outcome-reward GRPO across 36 configurations of three workflows, three Qwen3 sizes, math/code tasks, and shared or role-specific LoRA policies. Comparisons with single-agent RL identify gains from extra generation or aggregation. Role-level analyses examine gradient imbalance under shared policies and post-peak decline under isolated policies.
Critical strengths
-
The systematic comparisons and role-level measurements clarify where workflow gains come from. In Table 4, three-sample SA-RL voting reaches 57.8% at 1.7B, matching Eval-Opt-IP’s 57.6%.
-
Gradient diagnostics and control experiments help separate broadcast-reward issues from training instability. The monitoring analysis also shows that training-success alarms can recover a near-peak checkpoint, although they fire after the peak.
Critical weaknesses
My main concern is that the narrow experimental scope limits the paper’s general insights into when multi-agent RL is useful.
-
Only three fixed workflows are evaluated on non-interactive math and code tasks. It is unclear whether the findings extend to more general systems involving dynamic delegation, tool use, or longer interactions.
-
SP/IP comparisons are restricted to Qwen3 models at 0.6B, 1.7B, and 4B. The conclusions lack validation on stronger models and other model families, and may depend on the specific training configuration.
-
The truncation-based explanation is tested only on these small models under fixed response caps. Its relevance with longer generation budgets remains unclear. Although Table 4 includes a three-sample baseline, comparisons against longer single-agent responses or continuation without an evaluator are missing.
Questions for authors
N/A
5 comments
Response to Reviewer Cuwh (2/3)
W1: three fixed workflows on non-interactive tasks
"Only three fixed workflows are evaluated on non-interactive math and code tasks. It is unclear whether the findings extend to more general systems involving dynamic delegation, tool use, or longer interactions."
What carries over to tool use and longer interactions is the diagnostic. Each training effect depends on one quantity of the training recipe, not on a topology or a task: a role's share of a shared gradient under the per-sequence mean, and the optimizer steps an adapter takes per training step. Both quantities can be measured in any system that trains several roles, and tool use and longer interactions change their values.
- Tool use. A role that issues tool calls adds short sequences to a shared policy's batch, which is the condition of Section 4.3.
- Longer interactions. A longer interaction gives an adapter more sequences per training step and therefore more optimizer steps, which is the condition of Section 4.4. Eval-Opt is the instance in the grid, where the trained evaluator sets the number of turns.
A system that shares one policy across roles with unequal output lengths, or runs several calls of a role on one adapter, meets the same conditions. Tables A and B show that changing the condition changes the behavior.
We fixed the set of roles so that the comparison is controlled and each result can be attributed to a role. With the same roles and the same number of instances in every episode, we can grade each role on its own answer, measure per-role gradient shares, and compare the same workflow under a shared and an isolated policy. Dynamic delegation, where the policy chooses the sub-agents and their number during training, is the next step. We will name it in the Limitations section and state the fixed-role reason in the setup.
The three workflows are representative patterns of the taxonomy we cite (Anthropic's "Building effective agents"): one iterative pattern (evaluator-optimizer), one parallel pattern (voting) and one hierarchical pattern (orchestrator-workers; the orchestrator writes input-dependent instructions for a fixed set of three workers). They differ in the two structural properties the findings depend on: whether a short-output decision role shares a policy with long-output solving roles, and whether a role runs several instances on one adapter.
The controlled experiments run in one cell, and the grid shows that their effects are not specific to it. We ran them in the cell where both training effects are strongest (Voting at 1.7B on Math). The shared policy's slower learning appears in 17 of 18 matched pairs on both tasks, and the post-peak decline under isolated policies appears in four grid cells and three separately trained variants (Table 19). The control cell is one instance of a pattern the grid shows, which is what makes it representative.
Response to Reviewer Cuwh (3/3)
W2: Qwen3 at 0.6B, 1.7B and 4B, and the training configuration
"SP/IP comparisons are restricted to Qwen3 models at 0.6B, 1.7B, and 4B. The conclusions lack validation on stronger models and other model families, and may depend on the specific training configuration."
Scale is an axis of the study, and we report where conclusions change with it. Within the grid, three conclusions change with scale:
- a longer single response closes most of the lead of Eval-Opt-IP at 1.7B and 4B but not at 0.6B (W3 above);
- under isolated policies, Orch-Workers exceeds one SA-RL response on Math by about five points at 0.6B and 1.7B and by about one point at 4B (Table 1);
- isolated policies peak above shared policies in every Math pair except Orch-Workers at 4B (Table 1).
Training configuration. All 36 cells and the single-agent baselines use the training framework's default GRPO and LoRA configuration for each task, with no per-cell tuning. The only hyperparameter difference is a short warmup on nine Code cells and the 0.6B single-agent Code baseline (App. B.1). The shared configuration is what makes the comparison controlled, since differences between cells, and between a cell and its single-agent baseline, are then not tuning differences. AT-GRPO and Dr. MAS likewise train all the methods they compare with one shared configuration. Whether a finding depends on a component of the recipe is a separate question, and we test it by varying that component directly: the reward (per-voter reward, Table 19), the loss weighting across roles (Table A), the repeated role's optimizer steps (Table B) and the response cap (Table C).
Model family and range. Controlled studies of end-to-end RL for multi-agent LLM systems use the same model family at overlapping sizes. AT-GRPO compares shared and role-specific policies on Qwen3 at 1.7B and 8B. Dr. MAS compares them on Qwen3 at 4B and 8B and Qwen2.5 at 3B and 7B. MAPoRL trains models of 3B to 8B. Each uses one to four fixed workflows. Our contribution within this range is the controlled cross of workflow, scale, task and policy sharing.
Cost of the study
Training every role end to end in 36 cells and six single-agent baselines took about 12,400 GPU-hours, and each controlled experiment is a further run of several days. This cost is the reason the grid has three workflows and stops at 4B, and the reason the controlled experiments run in one cell.
Response to Reviewer Cuwh (1/3)
Response to Reviewer Cuwh
Thank you for the review. W3 is answered by two new experiments (Tables C and D of the general comment). W1 and W2 concern scope. Within that scope, the controlled grid separates the gain of a workflow from effects of the training recipe and of the generation budget, and Tables A to D test these effects by intervention.
W3: longer single-agent responses, and continuation without an evaluator
We ran both comparisons (Tables C and D). Table C evaluates the SA-RL checkpoint with a longer response cap. Table D replaces the trained evaluator by a fixed rule that starts a new attempt only after a cut-off attempt, with no evaluator call.
These results confirm the truncation explanation. If Eval-Opt's lead comes from giving truncated attempts another turn, the lead should shrink where the single model can use a longer response and stay where it cannot. That is what Table C shows at 1.7B and 4B against 0.6B.
The general point is that multi-agent RL learned the workflow's strategy. In this setting a new attempt after a cut-off attempt is an effective strategy: it lifts the single-agent checkpoint by two to seven points (Table D). Nobody specified it, and two kinds of evidence show that the evaluator learned it from the outcome reward. Table 2 gives the evaluator's decisions on wrong first attempts: under isolated policies it rejects at least 90% of the cut-off ones and accepts most finished ones. Table D tests the replacement of the evaluator by this rule (first item above). At 1.7B the trained generator is also 4.5 points above the single-agent generator under the same rule, with 6.7k against 4.1k generated tokens per episode (Table D, last row). We will add Tables C and D to Section 4.2.