- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes RBPO method to train one language model acting as a Planner, Coder, and Reflector. It computes the average PPO token loss for each generated response, then averages these response losses separately for each role and gives each role equal weight. This prevents longer responses or more frequently called roles from automatically contributing more to the training objective.
Critical strengths
- Clear practical results do definitely exist. RBPO improves DeepCoder accuracy from 26.3% to 30.1%. Role balancing contributes the larger gain. LCB-v6 accuracy reaches 60.7%, versus 50.3% for the same workflow without RL. Reflection removal also substantially reduces revision success.
- The paper shows how workflow choices, such as stopping after successful execution, implicitly reduce reflection’s share of the training loss.
Critical weaknesses
- The new contribution is unclear. The main change is a simple way of averaging training losses across responses and roles. This appears related to similar loss-weighting approaches with small engineering change.
- Results are averaged over only two runs. Smaller gains (like 0.7 from balancing response legnths) are difficult to assess.
I would change my rating if suggested ablations will be added to the paper. Right now the gain evidence is unclear.
Questions for authors
Can you provide std numbers in tables 2-4? Can you repeat Table 2’s four-way comparison on LCB-v6? Full RBPO vs removing response-length balancing, role balancing, or both, keeping the model, workflow, and training budget fixed?
3 comments
Thank you for your reply. The new experiments indeed provide stronger evidence that the method improves performance. While the novelty remains limited (the method is a simple technique for weighting losses across roles), this work might still be beneficial to the community. I will raise my score from 2 to 3.
Thank you for reconsidering your assessment and recognizing the potential value of our work to the community. We appreciate your constructive feedback, which helped us strengthen the experimental evidence. We will incorporate the new results into the revised manuscript and clarify the contribution.
Thank you for recognizing the practical results and specifying the additional evidence needed to assess the gains. Following your suggestions, we have expanded the LCB-v6 evaluations to three seeds and completed the requested four-way aggregation ablation across three independent training seeds. We also clarify the contribution in relation to existing loss-weighting ideas below.
1. Updated LCB-v6 results with variability reporting
The updated system-comparison results are reported as mean ± sample standard deviation across three seeds.
RBPO achieves the highest mean accuracy among the evaluated methods. These standard deviations make the variability across seeds explicit, addressing the missing variability information in the original Table 4.
For configurations retained from the original evaluation, changes in the reported means reflect the expanded seed set. The updated MARS² result additionally uses Qwen3-8B to match our backbone.
2. Requested four-way aggregation ablation on LCB-v6
To isolate the aggregation contribution, all four configurations use the same Qwen3-8B model, workflow, reward, advantage estimator, PPO settings, training data, and training budget. Only the actor-loss aggregation rule changes.
Results are mean ± sample standard deviation across three independent training seeds.
Full RBPO improves over token pooling by 6.95 percentage points. Both components improve mean accuracy under both settings of the other component:
These results directly address the concern that the evidence for sequence balancing rests on the small +0.7 pp DeepCoder margin. On LCB-v6, its gains are larger in both controlled comparisons. The experiment also separates the aggregation benefit from improvements due to introducing RL or changing the workflow.
3. Contribution and relevance to shared-policy multi-agent training
RBPO builds on established group-wise loss weighting and investigates a specific optimization problem in shared-policy multi-agent training: the role weights implicitly induced by the execution workflow.
Conditional invocation, early stopping, and response lengths determine each role’s contribution to a pooled actor loss. Under token pooling, a role’s total objective coefficient follows its token share. Response pooling removes the length effect, but still assigns total role coefficients according to response frequency. Thus, even after choosing the reward and RL algorithm, workflow decisions continue to determine how the training objective is distributed across roles.
Our contribution is to make this coupling explicit, distinguish its length and invocation-frequency components, and establish their performance consequences through controlled experiments. The role-level effect remains after response-length normalization: adding role balancing to response pooling raises LCB-v6 accuracy from 57.71% to 60.95%, a gain of 3.24 pp. Together with the full +6.95 pp aggregation gain, this demonstrates that the choice materially affects learning under otherwise matched conditions.
The resulting lesson for the field is concrete: equal weighting of responses can still leave unequal, workflow-dependent weighting of roles. Shared-policy training should therefore make the intended role coefficients explicit and examine how they relate to observed invocation frequencies. RBPO provides a direct implementation through hierarchical averaging, allowing role weights to be specified while retaining the existing workflow and PPO components.
This combination of an explicit analysis of workflow-induced weighting, a method for controlling it, and controlled evidence of substantial gains establishes the method’s practical research value. The improvement requires no additional model parameters or inference calls. We will foreground these findings and their connection to existing weighting ideas in the revised manuscript.
Thank you for the constructive suggestions. The requested LCB-v6 ablation and independent-multi-seed results provide more direct evidence for the gains. We hope the clarified contribution addresses the concerns underlying your assessment.