- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this submission, the authors propose CoBA-RL, a reinforcement learning algorithm that dynamically allocates the rollout budget in accordance with the model’s evolving capability. They introduce a capability-oriented value function based on instance-level pass rates and global model capability, and employs a heap-based greedy algorithm to optimize budget allocation. The main contributions include the reinforcement learning algorithm, the Capability-Oriented Value function, and related experimental results.
Critical strengths
-
Clear motivation. The paper addresses a practical and important problem of improving rollout budget allocation in RLVR. The paper is well structured and easy to follow.
-
Practical design and impressive results. The proposed framework is computationally efficient and can be integrated into existing RL pipelines. Experiments across different model scales and benchmarks demonstrate consistent performance improvements.
Critical weaknesses
- Insufficient justification of the proposed training-value function.
The Capability-Oriented Value function (Eqs. 5–9) is largely heuristic, modeling training value using a capability-dependent Beta density and an exponential budget saturation factor. However, the submission does not directly validate whether the estimated gains reflect actual learning improvements. Moreover, the theoretical optimality of the greedy algorithm only applies to this predefined surrogate objective, rather than actual training gains.
- Unclear computational fairness.
In table 9, the submission shows that CoBA-RL generates more tokens and requires longer generation time than GRPO. Without token- or wall-clock-matched comparisons, it remains unclear whether the performance gains persist under equivalent computational costs.
Questions for authors
-
Could the authors provide evidence that the proposed training-value function, particularly its estimated gains, correlates with actual learning improvements?
-
Could the authors provide token- or wall-clock-matched comparisons with GRPO to demonstrate the computational efficiency of CoBA-RL?
7 comments
Author Response to Reviewer UExw (Part 2/2)
W2 & Q2. Token- and wall-clock-matched computational efficiency comparisons
In Table 9, CoBA-RL generates more tokens and takes longer per step than GRPO under the same trajectory count because GRPO's uniform allocation expends a large share of rollouts on already-mastered prompts, whereas CoBA-RL adaptively shifts the rollout budget toward more informative, higher-value prompts based on global capability feedback and prompt pass rates. Generating more effective output tokens and requiring longer generation time per step are therefore expected consequences of concentrating compute on valuable training samples.
Token- and GPU-hour-matched comparisons. To directly verify whether the performance gains persist under equivalent computational cost, we compare Qwen2.5-7B-Instruct (Avg@16, %) at matched cumulative output-token and training GPU-hour budgets under both 2,048 and 4,096 rollouts per step.
| Budget type | Budget cap | GRPO | CoBA-RL |
|---|---|---|---|
| 2,048 rollouts per step | |||
| Output tokens (B) | 0.75 | 39.22 | 40.79 |
| Output tokens (B) | 1.25 | 41.17 | 42.75 |
| Output tokens (B) | 1.75 | 40.67 | 43.37 |
| Output tokens (B) | 2.25 | 42.07 | 43.59 |
| Training GPU-hours | 100 | 39.53 | 42.06 |
| Training GPU-hours | 140 | 40.67 | 42.91 |
| Training GPU-hours | 160 | 40.67 | 43.37 |
| Training GPU-hours | 180 | 41.05 | 43.61 |
| 4,096 rollouts per step | |||
| Output tokens (B) | 1.00 | 39.48 | 41.42 |
| Output tokens (B) | 2.00 | 42.05 | 42.91 |
| Output tokens (B) | 3.00 | 41.10 | 43.37 |
| Output tokens (B) | 3.50 | 41.76 | 44.53 |
| Training GPU-hours | 100 | 39.48 | 41.42 |
| Training GPU-hours | 140 | 40.88 | 42.90 |
| Training GPU-hours | 180 | 41.19 | 42.91 |
| Training GPU-hours | 220 | 42.30 | 43.41 |
At every matched token and GPU-hour budget across both rollout settings, CoBA-RL consistently surpasses GRPO and continues to improve as compute scales, confirming that capability-adaptive rollout allocation translates into clear performance gains under equal computational budgets.
Cross-budget comparison (CoBA-RL-2048 vs. GRPO-4096). We also compare the total training compute and final accuracy of CoBA-RL with 2,048 rollouts per step against GRPO with 4,096 rollouts per step:
| Method | Rollouts/step | Output tokens (B) | Training GPU-hours | Avg@16 (%) |
|---|---|---|---|---|
| GRPO | 4,096 | 4.41 | 424.64 | 42.24 |
| CoBA-RL | 2,048 | 3.62 | 318.23 | 45.52 |
Even with half the per-step rollout budget, CoBA-RL-2048 consumes 18.02% fewer output tokens and 25.06% fewer training GPU-hours than GRPO-4096, while achieving a 3.28-point higher Avg@16. Combined with the matched-budget results above, this confirms the superior computational efficiency of CoBA-RL in both generated tokens and wall-clock training time.
Once again, we appreciate your thoughtful feedback, which helped us improve our experimental evaluation and strengthen the manuscript. We hope our responses resolve your concerns and welcome any additional questions during the discussion period.
Dear Reviewer UExw,
We sincerely thank you again for your thoughtful comments and for engaging with our submission.
Since the discussion phase will conclude soon, we wanted to briefly check in on whether our responses have sufficiently addressed the concerns you raised. Please let us know if anything requires further explanation.
If you find that your primary concerns have been resolved, we kindly hope you will factor our responses into your final recommendation.
Author Response to Reviewer UExw (Part 1/2)
We thank the reviewer for the thoughtful review and helpful questions. We have highlighted all changes in red in the revised PDF and address each of your points below:
Motivation from training dynamics. Our design is motivated by a fundamental RL training dynamic: as the policy masters certain prompts, allocating additional rollouts to them yields diminishing learning signal. In particular, when all responses to a prompt receive identical rewards, their group-relative advantages vanish. Because rollout generation dominates training cost, budget allocation should dynamically track the shifting set of learnable prompts that remain informative for the current policy.
Capability-adaptive value function. Our goal is to achieve dynamic budget allocation without introducing extra computational budget, while capturing the RL training dynamic described above. To this end, CoBA-RL models sample value using a capability-conditioned Beta preference density and an exponential budget saturation factor. Updated via global capability feedback, the Beta density dynamically shifts the preferred pass-rate region as the model improves, while the saturation factor captures diminishing marginal returns from repeatedly allocating rollouts to the same prompt.
Figure 5 tracks the evolution of αt, which directly reflects how the Beta preference distribution changes throughout training. The allocator initially focuses on easier, learnable prompts to consolidate foundational capability, and progressively shifts exploration toward harder prompts as global capability improves, adapting continuously to real-time training feedback rather than following a rigid schedule. Moreover, Table 14 shows that this capability-oriented value function increases the learnable ratio, i.e., the fraction of rollouts allocated to prompts with intermediate pass rates (0<pi<1), from 44.26% under GRPO to 72.87% under CoBA-RL, directly demonstrating the value of dynamic allocation during training.
Comparison with different methods. In Tables 3 and 4, we compare CoBA-RL on Qwen2.5-7B-Instruct against fixed Beta preferences, Linear Step Decay, curriculum sampling via advantage feedback (SEC), and rollout allocation methods based on gradient or reward variance (GVM-RAFT, DynaMO, and VIP):
CoBA-RL achieves the highest Avg@16 across all comparisons. Together with the higher learnable ratio in Table 14, outperforming fixed preferences, predefined decay schedules, and variance-based allocation methods under identical training budgets confirms that reshaping the value function with global capability feedback effectively captures actual downstream learning improvements.