acceptodds
Under review as a conference paper at ICLR 2027

Learning Across Roles: Role-Balanced Shared-Policy Optimization for Multi-Agent Code Generation

Abstract

Shared-policy multi-agent learning enables collaborative code generation without maintaining a separate policy for each role. However, common actor-loss reductions implicitly assign greater weight to roles that generate longer responses or are invoked more frequently, coupling optimization to workflow exposure. We propose Role-Balanced Shared-Policy Optimization (RBPO), which jointly trains a Planner, a Coder, and a Reflector through a single role-conditioned policy in an execution-guided refinement loop. RBPO averages token losses within each response and responses within each role, then combines the resulting role objectives with explicit weights. This hierarchical objective makes each represented role's nominal weight independent of its response lengths and invocation count, without adding role-specific policy parameters or rollout calls. On LiveCodeBench (LCB-v6), RBPO with Qwen3-8B achieves 60.7% accuracy, improving over Qwen3-8B GRPO by 9.8 percentage points and the same workflow without reinforcement learning by 10.4 points. It also exceeds a strong learning-based multi-agent method using Qwen3-14B by 1.3 points under the evaluated inference configurations. Controlled ablations support the contribution of both sequence and role balancing, while role-removal analyses demonstrate the importance of reflection despite its lower invocation frequency. These results highlight explicit role weighting as an effective design choice for shared-policy multi-agent code generation.

Primary area:
reinforcement learning
Keywords:
Multi-agent reinforcement learning, Large language models, Shared-policy optimization, Role balancing, Code generation, Execution feedback

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_s6nJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes RBPO method to train one language model acting as a Planner, Coder, and Reflector. It computes the average PPO token loss for each generated response, then averages these response losses separately for each role and gives each role equal weight. This prevents longer responses or more frequently called roles from automatically contributing more to the training objective.

Critical strengths
  1. Clear practical results do definitely exist. RBPO improves DeepCoder accuracy from 26.3% to 30.1%. Role balancing contributes the larger gain. LCB-v6 accuracy reaches 60.7%, versus 50.3% for the same workflow without RL. Reflection removal also substantially reduces revision success.
  2. The paper shows how workflow choices, such as stopping after successful execution, implicitly reduce reflection’s share of the training loss.
Critical weaknesses
  1. The new contribution is unclear. The main change is a simple way of averaging training losses across responses and roles. This appears related to similar loss-weighting approaches with small engineering change.
  2. Results are averaged over only two runs. Smaller gains (like 0.7 from balancing response legnths) are difficult to assess.

I would change my rating if suggested ablations will be added to the paper. Right now the gain evidence is unclear.

Questions for authors

Can you provide std numbers in tables 2-4? Can you repeat Table 2’s four-way comparison on LCB-v6? Full RBPO vs removing response-length balancing, role balancing, or both, keeping the model, workflow, and training budget fixed?

3 comments

Thank you for recognizing the practical results and specifying the additional evidence needed to assess the gains. Following your suggestions, we have expanded the LCB-v6 evaluations to three seeds and completed the requested four-way aggregation ablation across three independent training seeds. We also clarify the contribution in relation to existing loss-weighting ideas below.

1. Updated LCB-v6 results with variability reporting

The updated system-comparison results are reported as mean ± sample standard deviation across three seeds.

MethodLCB-v6 accuracy (%)
Qwen3-8B Direct48.19 ± 2.31
Single-agent PPO50.30 ± 3.00
Single-agent GRPO51.05 ± 1.75
DyLAN57.71 ± 2.62
AgentVerse48.76 ± 1.75
MetaGPT49.52 ± 1.84
MapCoder37.71 ± 8.57
AFlow50.48 ± 0.80
MARS² (MARTI, Qwen3-8B)56.96 ± 2.37
RBPO60.95 ± 1.19

RBPO achieves the highest mean accuracy among the evaluated methods. These standard deviations make the variability across seeds explicit, addressing the missing variability information in the original Table 4.

For configurations retained from the original evaluation, changes in the reported means reflect the expanded seed set. The updated MARS² result additionally uses Qwen3-8B to match our backbone.

2. Requested four-way aggregation ablation on LCB-v6

To isolate the aggregation contribution, all four configurations use the same Qwen3-8B model, workflow, reward, advantage estimator, PPO settings, training data, and training budget. Only the actor-loss aggregation rule changes.

AggregationSequence balancingRole balancingLCB-v6 accuracy (%)
Full RBPOYesYes60.95 ± 1.19
w/o Sequence BalancingNoYes56.86 ± 2.02
w/o Role BalancingYesNo57.71 ± 1.62
Token PoolingNoNo54.00 ± 2.02

Results are mean ± sample standard deviation across three independent training seeds.

Full RBPO improves over token pooling by 6.95 percentage points. Both components improve mean accuracy under both settings of the other component:

  • Sequence balancing: +4.09 pp with role balancing and +3.71 pp without it.
  • Role balancing: +3.24 pp with sequence balancing and +2.86 pp without it.

These results directly address the concern that the evidence for sequence balancing rests on the small +0.7 pp DeepCoder margin. On LCB-v6, its gains are larger in both controlled comparisons. The experiment also separates the aggregation benefit from improvements due to introducing RL or changing the workflow.

3. Contribution and relevance to shared-policy multi-agent training

RBPO builds on established group-wise loss weighting and investigates a specific optimization problem in shared-policy multi-agent training: the role weights implicitly induced by the execution workflow.

Conditional invocation, early stopping, and response lengths determine each role’s contribution to a pooled actor loss. Under token pooling, a role’s total objective coefficient follows its token share. Response pooling removes the length effect, but still assigns total role coefficients according to response frequency. Thus, even after choosing the reward and RL algorithm, workflow decisions continue to determine how the training objective is distributed across roles.

Our contribution is to make this coupling explicit, distinguish its length and invocation-frequency components, and establish their performance consequences through controlled experiments. The role-level effect remains after response-length normalization: adding role balancing to response pooling raises LCB-v6 accuracy from 57.71% to 60.95%, a gain of 3.24 pp. Together with the full +6.95 pp aggregation gain, this demonstrates that the choice materially affects learning under otherwise matched conditions.

The resulting lesson for the field is concrete: equal weighting of responses can still leave unequal, workflow-dependent weighting of roles. Shared-policy training should therefore make the intended role coefficients explicit and examine how they relate to observed invocation frequencies. RBPO provides a direct implementation through hierarchical averaging, allowing role weights to be specified while retaining the existing workflow and PPO components.

This combination of an explicit analysis of workflow-induced weighting, a method for controlling it, and controlled evidence of substantial gains establishes the method’s practical research value. The improvement requires no additional model parameters or inference calls. We will foreground these findings and their connection to existing weighting ideas in the revised manuscript.

Thank you for the constructive suggestions. The requested LCB-v6 ablation and independent-multi-seed results provide more direct evidence for the gains. We hope the clarified contribution addresses the concerns underlying your assessment.

Thank you for your reply. The new experiments indeed provide stronger evidence that the method improves performance. While the novelty remains limited (the method is a simple technique for weighting losses across roles), this work might still be beneficial to the community. I will raise my score from 2 to 3.

Thank you for reconsidering your assessment and recognizing the potential value of our work to the community. We appreciate your constructive feedback, which helped us strengthen the experimental evidence. We will incorporate the new results into the revised manuscript and clarify the contribution.

Reviewer_n3iPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-agent code generation in which a single LLM plays the Planner, Coder, and Reflector roles and is trained with reinforcement learning. It argues that how the actor loss is aggregated implicitly sets how much each role is trained, so roles that are called more often or produce longer responses dominate training regardless of their functional value. To address this, the authors propose Role-Balanced Shared-Policy Optimization (RBPO), which averages the loss hierarchically: first over tokens within each response, then over responses within each role, and finally across roles with designer-specified weights. It adds no parameters, calls, or search budget. Experiments with Qwen3-8B show that RBPO outperforms single-agent RL, training-free multi-agent systems, and a learned multi-agent baseline, and ablations suggest that role balancing contributes more than sequence balancing.

Critical strengths
  1. The paper clearly identifies a training imbalance problem that can arise when training models in a shared-policy multi-agent setup. The motivation is well-reasoned, and I agree with it.

  2. The proposed aggregation method for addressing this problem is natural and intuitive, and it is well supported by the formulation.

  3. The paper compares against a diverse set of baselines, both training-based and training-free, and shows reasonably well that it resolves issues that other methods do not adequately address.

Critical weaknesses
  1. The paper assumes a single fixed workflow (Planner → Coder → Reflector). Since role imbalance depends on the workflow structure, RBPO should also be evaluated on one or two other workflows or module sequences to show that it generalizes.

  2. All results are reported only as averages over two seeds. Showing the results of each individual run would help assess how robust the findings are.

  3. The only evidence for sequence balancing comes from two contrasts in Table 2: +0.7pp with role balancing on (RBPO 30.1 vs. w/o Seq. Bal. 29.4) and +1.1pp with it off (w/o Role Bal. 27.4 vs. token pooling 26.3). Both are small, so the claim that both components contribute seems weakly supported.

  4. To support the premise that the Reflector is under-trained due to imbalance, the role-removal ablation should also show a smaller Reflector contribution under token/response pooling than under RBPO. However, it is only run on the RBPO model.

Questions for authors
  1. The +10.4pp gain on LCB-v6 in Table 4 mixes the effect of RL training itself with that of the RBPO aggregation method. Could you also report the other aggregation variants in Table 2 (e.g., token pooling) on LCB-v6?

  2. Does RBPO remain effective under workflows or module sequences other than Planner → Coder → Reflector, especially ones with different role-imbalance patterns?

4 comments

Thank you for recognizing the motivation and formulation of RBPO. Following your suggestions, we have added the complete aggregation ablation on LCB-v6, a new two-agent workflow comparison, and Reflector-removal experiments across all four aggregation configurations. These results provide more direct evidence for the aggregation gains and their applicability beyond the original workflow.

1. Controlled LCB-v6 ablation and training-seed variability

We agree that the original +10.4 pp comparison combines the effects of RL training and aggregation. To isolate the aggregation contribution, we evaluated all four variants on LCB-v6 with the same model, workflow, reward, advantage estimator, PPO settings, and training budget. Only the actor-loss aggregation rule changes.

Actor-loss aggregationSequence balancingRole balancingLCB-v6 accuracy (%)
Full RBPOYesYes60.95 ± 1.19
w/o Sequence BalancingNoYes56.86 ± 2.02
w/o Role BalancingYesNo57.71 ± 1.62
Token PoolingNoNo54.00 ± 2.02

The results in this table are mean ± sample standard deviation across three independent training seeds.

Full RBPO improves over token pooling by 6.95 percentage points. Both balancing levels improve mean accuracy in both settings:

  • Sequence balancing contributes +4.09 pp with role balancing enabled and +3.71 pp with it disabled.
  • Role balancing contributes +3.24 pp with sequence balancing enabled and +2.86 pp with it disabled.

These results address the concern that the evidence for sequence balancing relies on the small +0.7/+1.1 pp gains on DeepCoder. They also isolate the aggregation benefit from introducing RL or changing the workflow. The relative contributions of the two balancing levels differ across benchmarks, and we will clarify this in the manuscript.

2. New workflow comparison with Coder and Reflector agents

To address your question about alternative workflows, we evaluated a two-agent workflow containing a Coder agent and a Reflector agent, removing the Planner from the original three-role design. We compared RBPO and token pooling within this new workflow on LCB-v6.

Two-agent workflowActor-loss aggregationLCB-v6 accuracy (%)
Coder + ReflectorToken Pooling55.30
Coder + ReflectorRBPO58.86

RBPO improves accuracy by 3.56 percentage points in the two-agent workflow. This provides additional empirical evidence that its benefit extends beyond the original Planner–Coder–Reflector configuration.

The comparison is performed within the two-agent workflow, so the improvement is measured against that workflow’s own token-pooling baseline. Together with the +6.95 pp gain in the original three-role workflow, this result supports the effectiveness of explicit role-level aggregation across two different role compositions.

3. Reflector removal across all four aggregation configurations

Following your suggestion, we evaluated inference-time Reflector removal for all four training configurations on DeepCoder.

Training configurationFull workflow (%)No Reflector (%)Removal drop (pp)
Full RBPO32.0227.664.36
w/o Sequence Balancing31.0024.756.25
w/o Role Balancing28.9724.314.66
Token Pooling26.3025.141.16

Removing the Reflector causes a larger accuracy drop under Full RBPO than under token pooling: 4.36 versus 1.16 pp. This is consistent with a greater contribution from reflection to the RBPO-trained system.

The comparison with response pooling is more nuanced. Its removal drop is 4.66 pp, slightly larger than Full RBPO’s 4.36 pp. Removal drop depends on performance both with and without reflection, and aggregation changes the shared policy used by all roles. Full RBPO achieves the highest accuracy in both conditions, while the largest removal drop occurs for the configuration without sequence balancing.

The direct full-workflow comparisons also show that role balancing improves accuracy by 3.05 pp when sequence balancing is enabled and by 4.70 pp when it is disabled. These results support the benefit of role balancing for overall shared-policy training. We will clarify that the removal experiment provides supporting evidence about the system’s use of reflection, while leaving the precise attribution of the gains to individual roles open.

4. Reflector benefit under approximately matched token usage

We further examined whether the Reflector’s benefit could be explained by additional generation tokens. On DeepCoder, we adjusted the No-Reflector inference budget to approximately match the full workflow’s average token usage.

ConfigurationGenerated tokens/problemAccuracy (%)
Full RBPO workflow3,490.832.02
No Reflector, approximately token-matched3,577.827.95

The full workflow retains a 4.07 pp advantage even though the No-Reflector configuration generates 2.5% more tokens. Thus, the observed benefit of reflection persists after compensating for its generation budget.

Thank you again for the constructive suggestions. We will incorporate these experiments and the clarified interpretation into the revision, with particular emphasis on the controlled LCB-v6 aggregation gains and the additional evidence from the two-agent workflow.

Thank you again for your helpful suggestions. Our updated response includes the complete LCB-v6 aggregation ablation, Reflector-removal comparisons, and a new two-agent workflow experiment, where RBPO improves over token pooling by 3.56 percentage points. We would appreciate your thoughts on these additions and would be happy to address any remaining concerns.

Thank you for the thorough response and the additional experiments. They address my concerns, and I will raise my score accordingly.

Reviewer_BRjSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper authors propose an optimization method that leverages learning across distinct roles and the paper is squarely targeting coding scenarios. The roles being planning, coding and reflecting. The paper proposes building a shared policy as opposed to building role specific policies and using this in a multi agent code generation harness setup. Authors correctly observe that the standard PPO loss gives way more importance to roles that produce more tokens or are called more often as a result of which reflector role, which is usually called only after a failure, gets mere ~20% weight. The motivation behind the paper is very intuitive and I am in agreement with the authors that treating the optimization at every role level is the right approach to solving this skewed credit assignment problem.

Critical strengths

The motivation is intuitive and I agree with authors that it is important to distinguish between how often a role gets called from just how useful the role is to the overall coding execution.

I also found the 2x2 ablations clean. The proposed change is a drop in replacement largely and so implementation of this method is straightforward.

Critical weaknesses

I found the paper extremely difficult to follow. The idea is intuitive and you could have simplified the exposition considerably. Terms like role marginal, nominal mass and represented roles are never explained in plain english. I would have really liked a simple intuition paragraph and a worked example showing the weights for one trajectory.

You seem to be mixing several changes. The ~10 pt gain over GRPO you mention however combines the proposed workflow with PPO instead of GRPO. There are far too many changes here and comparing to GRPO just feels odd unless you ablate it against PPO.

The biggest weakness, however, is that you have never trained separate policies. You do list it as future work, but I dont think punting it to future completes this paper given this is THE point you are after. Similarly, in your setup, currently, the planner and the coder get the same reward and the reflector shares it reward with the next round's plan and code. So if you wanted to train each role separately, not clear to me how you would do it.

Questions for authors

How would you train separate role policies, and what reward would each role get? Can you report a matched separate-policy baseline? I feel this is a very important baseline to establish here.

You have mentioned GRPO although you never built your methods on top of this optimization algorithm. Does the averaging change still help when applied on top of GRPO ?

4 comments

Thank you for recognizing the motivation and the controlled ablation design. Your comments helped us clarify the presentation and the central research question. We have added a worked example and a complete LCB-v6 aggregation ablation across three independent training seeds. The new controlled experiment shows a 6.95-percentage-point improvement from aggregation alone under the same model, workflow, and training budget.

1. Intuition and a worked example

A role should not automatically receive more training weight merely because it is invoked more often or generates longer responses.

Consider a two-round trajectory in which the Planner and Coder each respond twice, while the Reflector responds once between rounds. Suppose each Planner response contains 100 tokens, each Coder response contains 400 tokens, and the Reflector response contains 100 tokens.

AggregationPlanner’s objective weightCoder’s objective weightReflector’s objective weight
Token pooling18.2%72.7%9.1%
Response pooling40.0%40.0%20.0%
RBPO with uniform role weights1/31/31/3

Token pooling gives the Coder most of the weight because it generates 800 of the 1,100 tokens. Response pooling removes this length effect, but the Reflector still receives less weight because it contributes only one of five responses. RBPO averages token losses within each response, responses within each role, and then the role averages using explicit weights.

“Represented roles” means roles present in the training batch. “Role marginal” describes how the total objective weight is distributed across roles, and “nominal mass” is a role’s assigned share of that weight. These coefficients describe objective weighting, rather than gradient magnitude or functional importance.

We will incorporate this example and these plain-language definitions into the revised manuscript.

2. Separate policies and the practical motivation for shared-policy optimization

How separate policies receive rewards. Separate policies can use the same execution-aligned rewards defined in Section 3.2. Let qkq_k denote the execution pass rate at round kk. The Planner and Coder receive rkP=rkC=qk−qk−1r_k^P = r_k^C = q_k - q_{k-1}, while the Reflector receives the improvement from the subsequent revision, rkR=qk+1−qkr_k^R = q_{k+1} - q_k.

Each policy is updated using its own role’s responses and the corresponding rewards and advantages. Sharing an execution outcome as a reward is therefore compatible with maintaining separate policy parameters. This reward assignment associates responses with execution transitions without assuming that it identifies each role’s individual causal contribution.

Why shared-policy training is a valuable setting. Maintaining a separate full-size policy for every role requires additional policy parameters, optimizer states, and checkpoints as the number of roles grows. A shared policy supports multiple roles through one set of model weights, providing a practical motivation for improving its training.

The central claim of our paper is that explicit role-level loss aggregation improves shared-policy multi-agent training. Conventional pooling lets response length and invocation frequency implicitly determine each role’s objective weight. RBPO controls this allocation while retaining the shared architecture. Our controlled experiment directly tests this claim and improves LCB-v6 accuracy from 54.00% to 60.95% under the same training budget.

Comparison with an existing separate-policy system. Our evaluation also includes a MARS² configuration with two separately optimized agent policies.

MetricRBPOMARS²
LCB-v6 accuracy (%)60.95 ± 1.1956.96 ± 2.37
LLM calls/problem2.6616.00
Generated tokens/problem40,600201,388
Execution batches/problem1.5532.00

Under these evaluated configurations, RBPO achieves 3.99 percentage points higher mean accuracy, with 79.8% fewer generated tokens, 83.4% fewer LLM calls, and 95.2% fewer execution batches. These results support the practical value of our single-policy system for multi-agent collaboration.

This system-level comparison involves different workflows and search procedures. A matched separate-policy version of our own Planner–Coder–Reflector workflow will further clarify the effect of parameter sharing. We agree that this comparison would strengthen the evaluation and are currently training the baseline. We will report its accuracy, parameter count, and training-resource usage when training and evaluation are complete.

3. Isolating the aggregation benefit and applicability to GRPO

We agree that the original comparison with single-agent GRPO combines changes in the workflow, RL algorithm, and aggregation. We have therefore added the following controlled experiment to identify the aggregation contribution directly.

Four-way aggregation ablation under the same PPO workflow. All configurations use the same model, Planner–Coder–Reflector workflow, reward, advantage estimator, PPO settings, and training budget. Only actor-loss aggregation changes.

Actor-loss aggregationSequence balancingRole balancingLCB-v6 accuracy (%)
Full RBPOYesYes60.95 ± 1.19
w/o Sequence BalancingNoYes56.86 ± 2.02
w/o Role BalancingYesNo57.71 ± 1.62
Token PoolingNoNo54.00 ± 2.02

Results are mean ± sample standard deviation across three independent training seeds.

Full RBPO improves over same-workflow token-pooling PPO by 6.95 percentage points. Removing sequence balancing reduces mean accuracy by 4.09 points, and removing role balancing reduces it by 3.24 points. Both components also improve mean accuracy when the other is disabled. This directly addresses the attribution concern while holding the workflow and RL algorithm fixed.

Why PPO and how the method applies to GRPO. We selected PPO because its learned value baseline supports advantage estimation without requiring a group of sampled responses for every input. This is useful in our multi-round workflow, where additional rollouts involve both model generation and code execution.

RBPO operates at the step where per-token actor losses are aggregated into the training objective. The same role-weighting issue can arise in shared-policy GRPO when this reduction depends on response lengths or invocation frequencies. RBPO can be applied at that step while retaining GRPO’s group sampling, rewards, group-relative advantages, and per-token loss definition.

The completed PPO ablation establishes the aggregation benefit in our evaluated setting. To test whether this benefit extends to GRPO, we will add the requested comparison of GRPO with token pooling versus GRPO with RBPO aggregation under the same multi-agent workflow, and report the results when available.

We will foreground the controlled aggregation results in the revision and make the shared-policy research question explicit. Thank you again for helping us improve both the explanation and the experimental evidence.

Thank you again for your thoughtful feedback. We would appreciate any further thoughts on the worked example, the controlled LCB-v6 aggregation results, and our clarification of the shared-policy setting in the updated response. Please let us know if there are any aspects of the current results or explanations that you would like us to clarify. We would be happy to discuss them further.

Thank you again for your time and thoughtful feedback. Before the discussion period closes, we would appreciate your perspective on whether the updated evidence addresses your concerns about the contribution and experimental comparisons.

We would particularly like to highlight two findings:

  • Controlled evidence for the central contribution. With the model, workflow, reward, PPO settings, and training budget fixed, RBPO improves LCB-v6 accuracy from 54.00 ± 2.02% to 60.95 ± 1.19% across three independent training seeds. This directly supports the value of explicit role-level aggregation within shared-policy training.

  • Comparison against independently trained policies. Our evaluated Heter-MARS² baseline uses separately optimized policies. RBPO achieves 3.99 percentage points higher mean accuracy while using 79.8% fewer generated tokens under the evaluated configurations, providing complementary evidence of its practical value.

We would appreciate your thoughts on whether these results and our clarification of the shared-policy research question address your main concerns. If any questions remain, we would be happy to provide further clarification during the remaining discussion period.

Reviewer_N4VuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a reweighting of the different components of a coding agent (Planner, Coder, and Reflector) so that bias stemming from rollout exposure (how often or how long each role is invoked) does not distort how each component is trained. All three roles share a single policy, and RBPO averages token losses within each response, then responses within each role, then combines roles with explicit weights.

Critical strengths

The paper is well written and easy to digest, and the problem and approach are well motivated: coupling the training objective to workflow exposure is a real and clearly identified issue. The main ablation is clean. It holds the policy, workflow, reward, advantage estimator, and training budget fixed and varies only the two averaging levels, so the reported gains can be attributed to the aggregation rule itself. The role-removal result is a nice complement, showing the least-invoked role (Reflector) matters most.

Critical weaknesses

The contribution does not seem substantial enough on its own. With only 2 seeds and no variance reported, it is hard to judge how reliable the margins are; several of the key gaps are small enough to plausibly fall within seed noise (e.g. the 0.7 pp sequence-balancing effect and the 1.3 pp margin over MARS2-14B). The win over MARS2-14B is also not a controlled comparison: RBPO uses ≤3 refinement rounds while MARS2 uses a 16-node MCTS search at a larger model size, so it compares two different systems at two different compute budgets.

Questions for authors
  • Can you report variance / error bars across seeds, or run more than 2 seeds, for Table 2 and Table 4? Several of the reported margins are small relative to likely seed variance.
  • Can the aggregation ablation (Table 2) be run on LCB-v6, so the "why it works" and "how much it helps" results share a benchmark?
  • In the role-removal study, can the comparison be matched on generated tokens (not just rounds), given that removing a role changes the total token budget?
8 comments

Thank you for recognizing the motivation, clarity, and controlled ablation design. Your comments helped us strengthen the evidence for both the reliability and the substance of the contribution. We have added three-seed variability reporting for the LCB-v6 system comparison, a complete controlled aggregation ablation, approximately token-matched role removal, and a same-backbone comparison with MARS².

1. Three-seed results and the complete aggregation ablation on LCB-v6

1.1 Updated system comparison with variability reporting

Following your suggestion, we expanded the LCB-v6 system evaluations to three seeds and report mean ± sample standard deviation for all methods in the updated Table 4.

MethodLCB-v6 accuracy (%)
Qwen3-8B Direct48.19 ± 2.31
Single-agent PPO50.30 ± 3.00
Single-agent GRPO51.05 ± 1.75
DyLAN57.71 ± 2.62
AgentVerse48.76 ± 1.75
MetaGPT49.52 ± 1.84
MapCoder37.71 ± 8.57
AFlow50.48 ± 0.80
MARS² (MARTI, Qwen3-8B)56.96 ± 2.37
RBPO60.95 ± 1.19

RBPO achieves the highest mean accuracy among the evaluated methods. Reporting the standard deviations makes the variability across seeds explicit and supplies the variability information missing from the original Table 4.

For configurations retained from the original evaluation, changes in the reported means reflect the expanded seed set. The MARS² result additionally uses Qwen3-8B for the same-backbone comparison described in Section 4.

1.2 Controlled aggregation ablation across three independent training seeds

To isolate the contribution of aggregation, we evaluated all four aggregation configurations on LCB-v6. They use the same Qwen3-8B model, workflow, reward, advantage estimator, PPO settings, training data, and training budget. Only the actor-loss aggregation rule changes.

AggregationSequence balancingRole balancingLCB-v6 accuracy (%)
Full RBPOYesYes60.95 ± 1.19
w/o Sequence BalancingNoYes56.86 ± 2.02
w/o Role BalancingYesNo57.71 ± 1.62
Token PoolingNoNo54.00 ± 2.02

Results are mean ± sample standard deviation across three independent training seeds.

Full RBPO improves over token pooling by 6.95 percentage points. Both components improve mean accuracy under both settings of the other component:

  • Sequence balancing: +4.09 pp with role balancing and +3.71 pp without it.
  • Role balancing: +3.24 pp with sequence balancing and +2.86 pp without it.

These results extend the evidence beyond the original +0.7 pp sequence-balancing effect on DeepCoder. The larger gains on LCB-v6, together with explicit variability reporting, strengthen the evidence for both averaging levels. They also place the controlled explanation of the method’s benefit on the same benchmark as the main system evaluation.

2. Why the contribution is substantive despite its simple implementation

We appreciate your concern about the contribution’s scope. RBPO builds on established group-wise loss weighting. The contribution is to identify and control how a multi-agent workflow implicitly determines role weights in shared-policy training, and to establish the performance consequences through controlled experiments.

Workflow decisions also become optimization decisions. Conditional invocation and early stopping determine how often each role appears in the training data. Under token pooling, a role’s total objective coefficient follows its share of generated tokens. Response pooling removes the response-length effect, but the role’s coefficient still follows its share of responses. Thus, even with the reward and underlying RL algorithm fixed, the execution structure determines how the training objective is distributed across roles.

For example, a failure-triggered Reflector receives fewer opportunities to contribute when execution succeeds early. Its resulting objective coefficient reflects invocation frequency, although that frequency does not directly measure its functional value. Our formulation makes this coupling explicit and separates its two sources—response length and role invocation frequency.

The new experiments show that the role-level effect remains after length normalization. Adding role balancing to response pooling improves LCB-v6 accuracy from 57.71% to 60.95%, a gain of 3.24 pp. Role balancing also provides +2.86 pp without sequence balancing. Together, the two levels yield the controlled +6.95 pp improvement above.

This gives the field a concrete training-design principle: specify the intended role weights explicitly and examine how they relate to the role exposure produced by the workflow. RBPO makes these coefficients controllable through hierarchical averaging. Researchers can therefore choose invocation rules for execution and role weights for training, while retaining the existing rollout procedure and PPO components.

The practical value is that this simple change produces a substantial controlled gain without adding model parameters or inference calls. We will foreground the connection between workflow structure, implicit role weighting, and learning performance in the revision.

3. Role removal with approximately matched generated tokens

Following your suggestion, we adjusted the inference budgets of the role-removed configurations on DeepCoder to approximately match the full workflow’s average generated tokens.

ConfigurationGenerated tokens/problemAverage roundsAccuracy (%)
Full workflow3,490.82.43832.02
No Planner, approximately token-matched3,355.03.48031.00
No Reflector, approximately token-matched3,577.82.88227.95

Removing the Reflector still reduces accuracy by 4.07 pp, despite generating 2.5% more tokens. Removing the Planner reduces accuracy by 1.02 pp, with 3.9% fewer tokens.

The Reflector’s benefit therefore persists after compensating for its generation budget, supporting its functional importance beyond the additional tokens it supplies. The match is approximate in average token usage across problems, with the adjusted average rounds reported explicitly.

4. Same-backbone comparison with MARS² and explicit inference-resource usage

We agree that the original comparison against MARS²-14B combined differences in model size and search configuration. We now compare both systems using Qwen3-8B and report their measured inference-resource usage.

MetricRBPOMARS²
BackboneQwen3-8BQwen3-8B
Search/refinement configuration≤3 rounds16-node MCTS
LLM calls/problem2.6616.00
Generated tokens/problem40,600201,388
Execution batches/problem1.5532.00
LCB-v6 accuracy (%)60.95 ± 1.1956.96 ± 2.37

With the backbone matched, RBPO achieves 3.99 pp higher mean accuracy, while using 79.8% fewer generated tokens, 83.4% fewer LLM calls, and 95.2% fewer execution batches.

The search procedures and budgets still differ. Nevertheless, at these evaluated configurations, RBPO achieves higher accuracy with lower usage of all three reported resources. This supports the practical value of the trained multi-agent system under a smaller generation and execution budget. We express the resource advantage specifically through these measured quantities.

The same-workflow aggregation ablation independently establishes the training-method contribution: changing only aggregation improves LCB-v6 accuracy by 6.95 pp. Together, these experiments provide evidence for both the aggregation benefit under controlled training conditions and the system’s favorable accuracy–resource tradeoff in the evaluated comparison.

Thank you again for the constructive suggestions. We will incorporate these results and the clearer contribution statement into the revised manuscript. We hope they address the concerns about the reliability of the gains, the interpretation of the system comparisons, and the significance of the aggregation choice.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for overseeing the review of our paper. We would like to summarize how the rebuttal has strengthened the evidence for RBPO and clarified its contribution to shared-policy multi-agent training.

Core contribution. Workflow execution implicitly shapes the training objective: roles that generate longer responses or are invoked more frequently receive greater aggregate weight under conventional loss pooling. Response-length normalization addresses the first effect but leaves the second unresolved. Our contribution is to make this workflow–objective coupling explicit, provide a simple mechanism for controlling it, and demonstrate its consequences through controlled experiments. RBPO requires no additional model parameters or inference calls.

The rebuttal adds four complementary pieces of evidence:

1. Controlled, multi-seed evidence for the aggregation benefit.

We completed a four-way LCB-v6 ablation across three independent training seeds, keeping the model, workflow, reward, advantage estimator, PPO settings, training data, and training budget fixed.

AggregationAccuracy, mean ± SD (%)
Token pooling54.00 ± 2.02
Sequence balancing only57.71 ± 1.62
Role balancing only56.86 ± 2.02
Full RBPO60.95 ± 1.19

Changing only aggregation yields a 6.95 percentage-point improvement. Both balancing levels improve mean accuracy with the other enabled or disabled. Adding role balancing after response-length normalization provides a further 3.24 pp gain. This directly tests the central research question and separates the aggregation benefit from gains due to introducing RL, changing workflows, or switching optimization algorithms.

2. Positive results in a second workflow.

In an additional Coder–Reflector workflow, RBPO improves accuracy from 55.30% to 58.86% against that workflow’s own token-pooling baseline. The observed benefit therefore extends beyond the original Planner–Coder–Reflector configuration.

3. Reflection benefits beyond additional generation tokens.

We expanded Reflector-removal experiments to all four aggregation configurations and added an approximately token-matched comparison. The full workflow retains a 4.07 pp advantage even when the No-Reflector configuration generates 2.5% more tokens. This provides supporting evidence that a conditionally invoked role can contribute substantial functional value, reinforcing the motivation for distinguishing invocation frequency from training priority.

4. Practical accuracy–resource advantages.

Against MARS² with the same Qwen3-8B backbone, RBPO achieves 3.99 pp higher mean accuracy, with 79.8% fewer generated tokens and 83.4% fewer LLM calls under the evaluated configurations. While this comparison involves different search procedures, it complements the controlled aggregation study by demonstrating a favorable practical accuracy–resource tradeoff.

Following the additional experiments, Reviewers n3iP and s6nJ raised their scores from 2 to 3, recognizing the strengthened empirical evidence and the work’s potential value to the community.

Together, these results show that loss aggregation is a consequential training-design choice in shared-policy multi-agent systems. The paper offers an explicit account of the underlying issue, a readily adoptable remedy, and controlled evidence of substantial gains. We believe this combination of methodological insight and practical utility makes the work a valuable contribution worthy of acceptance.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.