acceptodds
Under review as a conference paper at ICLR 2027

CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key approach for enhancing LLM reasoning. However, standard frameworks like Group Relative Policy Optimization (GRPO) typically employ a uniform rollout budget, leading to resource inefficiency. Moreover, existing adaptive methods often rely on instance-level metrics, such as task pass rates, failing to capture the model's dynamic learning state. To address these limitations, we propose CoBA-RL, a reinforcement learning algorithm designed to adaptively allocate rollout budgets based on the model's evolving capability. Specifically, CoBA-RL utilizes a Capability-Oriented Value function to map tasks to their potential training gains and employs a heap-based greedy strategy to efficiently self-calibrate the distribution of computational resources to samples with high training value. Extensive experiments demonstrate that our approach effectively orchestrates the trade-off between exploration and exploitation, delivering consistent generalization improvements across multiple challenging benchmarks. These findings underscore that quantifying sample training value and optimizing budget allocation are pivotal for advancing LLM post-training efficiency.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, LLM, Budget Allocation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_UExwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this submission, the authors propose CoBA-RL, a reinforcement learning algorithm that dynamically allocates the rollout budget in accordance with the model’s evolving capability. They introduce a capability-oriented value function based on instance-level pass rates and global model capability, and employs a heap-based greedy algorithm to optimize budget allocation. The main contributions include the reinforcement learning algorithm, the Capability-Oriented Value function, and related experimental results.

Critical strengths
  1. Clear motivation. The paper addresses a practical and important problem of improving rollout budget allocation in RLVR. The paper is well structured and easy to follow.

  2. Practical design and impressive results. The proposed framework is computationally efficient and can be integrated into existing RL pipelines. Experiments across different model scales and benchmarks demonstrate consistent performance improvements.

Critical weaknesses
  1. Insufficient justification of the proposed training-value function.

The Capability-Oriented Value function (Eqs. 5–9) is largely heuristic, modeling training value using a capability-dependent Beta density and an exponential budget saturation factor. However, the submission does not directly validate whether the estimated gains reflect actual learning improvements. Moreover, the theoretical optimality of the greedy algorithm only applies to this predefined surrogate objective, rather than actual training gains.

  1. Unclear computational fairness.

In table 9, the submission shows that CoBA-RL generates more tokens and requires longer generation time than GRPO. Without token- or wall-clock-matched comparisons, it remains unclear whether the performance gains persist under equivalent computational costs.

Questions for authors
  1. Could the authors provide evidence that the proposed training-value function, particularly its estimated gains, correlates with actual learning improvements?

  2. Could the authors provide token- or wall-clock-matched comparisons with GRPO to demonstrate the computational efficiency of CoBA-RL?

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer UExw (Part 1/2)

We thank the reviewer for the thoughtful review and helpful questions. We have highlighted all changes in red in the revised PDF and address each of your points below:

W1 & Q1. Motivation and empirical support for the training-value function

Motivation from training dynamics. Our design is motivated by a fundamental RL training dynamic: as the policy masters certain prompts, allocating additional rollouts to them yields diminishing learning signal. In particular, when all responses to a prompt receive identical rewards, their group-relative advantages vanish. Because rollout generation dominates training cost, budget allocation should dynamically track the shifting set of learnable prompts that remain informative for the current policy.

Capability-adaptive value function. Our goal is to achieve dynamic budget allocation without introducing extra computational budget, while capturing the RL training dynamic described above. To this end, CoBA-RL models sample value using a capability-conditioned Beta preference density and an exponential budget saturation factor. Updated via global capability feedback, the Beta density dynamically shifts the preferred pass-rate region as the model improves, while the saturation factor captures diminishing marginal returns from repeatedly allocating rollouts to the same prompt.

Figure 5 tracks the evolution of αt\alpha_t, which directly reflects how the Beta preference distribution changes throughout training. The allocator initially focuses on easier, learnable prompts to consolidate foundational capability, and progressively shifts exploration toward harder prompts as global capability improves, adapting continuously to real-time training feedback rather than following a rigid schedule. Moreover, Table 14 shows that this capability-oriented value function increases the learnable ratio, i.e., the fraction of rollouts allocated to prompts with intermediate pass rates (0<pi<10 < p_i < 1), from 44.26% under GRPO to 72.87% under CoBA-RL, directly demonstrating the value of dynamic allocation during training.

Comparison with different methods. In Tables 3 and 4, we compare CoBA-RL on Qwen2.5-7B-Instruct against fixed Beta preferences, Linear Step Decay, curriculum sampling via advantage feedback (SEC), and rollout allocation methods based on gradient or reward variance (GVM-RAFT, DynaMO, and VIP):

MethodAvg@16 (%)
Fixed α=10.5, β=1.5\alpha=10.5,\ \beta=1.544.22
Fixed α=1.5, β=10.5\alpha=1.5,\ \beta=10.545.21
Linear Step Decay45.39
SEC45.43
GVM-RAFT45.68
DynaMO45.95
VIP46.67
CoBA-RL47.29

CoBA-RL achieves the highest Avg@16 across all comparisons. Together with the higher learnable ratio in Table 14, outperforming fixed preferences, predefined decay schedules, and variance-based allocation methods under identical training budgets confirms that reshaping the value function with global capability feedback effectively captures actual downstream learning improvements.

AuthorsOpenReview ↗

Author Response to Reviewer UExw (Part 2/2)

W2 & Q2. Token- and wall-clock-matched computational efficiency comparisons

In Table 9, CoBA-RL generates more tokens and takes longer per step than GRPO under the same trajectory count because GRPO's uniform allocation expends a large share of rollouts on already-mastered prompts, whereas CoBA-RL adaptively shifts the rollout budget toward more informative, higher-value prompts based on global capability feedback and prompt pass rates. Generating more effective output tokens and requiring longer generation time per step are therefore expected consequences of concentrating compute on valuable training samples.

Token- and GPU-hour-matched comparisons. To directly verify whether the performance gains persist under equivalent computational cost, we compare Qwen2.5-7B-Instruct (Avg@16, %) at matched cumulative output-token and training GPU-hour budgets under both 2,048 and 4,096 rollouts per step.

Budget typeBudget capGRPOCoBA-RL
2,048 rollouts per step
Output tokens (B)0.7539.2240.79
Output tokens (B)1.2541.1742.75
Output tokens (B)1.7540.6743.37
Output tokens (B)2.2542.0743.59
Training GPU-hours10039.5342.06
Training GPU-hours14040.6742.91
Training GPU-hours16040.6743.37
Training GPU-hours18041.0543.61
4,096 rollouts per step
Output tokens (B)1.0039.4841.42
Output tokens (B)2.0042.0542.91
Output tokens (B)3.0041.1043.37
Output tokens (B)3.5041.7644.53
Training GPU-hours10039.4841.42
Training GPU-hours14040.8842.90
Training GPU-hours18041.1942.91
Training GPU-hours22042.3043.41

At every matched token and GPU-hour budget across both rollout settings, CoBA-RL consistently surpasses GRPO and continues to improve as compute scales, confirming that capability-adaptive rollout allocation translates into clear performance gains under equal computational budgets.

Cross-budget comparison (CoBA-RL-2048 vs. GRPO-4096). We also compare the total training compute and final accuracy of CoBA-RL with 2,048 rollouts per step against GRPO with 4,096 rollouts per step:

MethodRollouts/stepOutput tokens (B)Training GPU-hoursAvg@16 (%)
GRPO4,0964.41424.6442.24
CoBA-RL2,0483.62318.2345.52

Even with half the per-step rollout budget, CoBA-RL-2048 consumes 18.02% fewer output tokens and 25.06% fewer training GPU-hours than GRPO-4096, while achieving a 3.28-point higher Avg@16. Combined with the matched-budget results above, this confirms the superior computational efficiency of CoBA-RL in both generated tokens and wall-clock training time.

Once again, we appreciate your thoughtful feedback, which helped us improve our experimental evaluation and strengthen the manuscript. We hope our responses resolve your concerns and welcome any additional questions during the discussion period.

Dear Reviewer UExw,

We sincerely thank you again for your thoughtful comments and for engaging with our submission.

Since the discussion phase will conclude soon, we wanted to briefly check in on whether our responses have sufficiently addressed the concerns you raised. Please let us know if anything requires further explanation.

If you find that your primary concerns have been resolved, we kindly hope you will factor our responses into your final recommendation.

Reviewer_LKfwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoBA-RL, a method for dynamically allocating a fixed rollout budget across training tasks in reinforcement learning for verifiable rewards. The method introduces a capability-oriented value function that combines task-level pass rates with the model’s global failure rate to adaptively shift the allocation from easier to harder tasks as the model’s capabilities improve. The authors then implement a heap-based greedy algorithm to efficiently distribute the rollout budget toward samples with high estimated training value. Experiments on Qwen2.5 and Qwen3 models of different sizes demonstrate improvements over GRPO and other adaptive allocation methods on mathematical reasoning benchmarks.

Critical strengths

The method is clearly explained, and adapting rollout budgets based on the model’s current capabilities is well motivated. The approach could also be useful in practice, since it does not require major changes to the RL training process.

The paper introduces a budget allocation criterion that adjusts the preference curve using a batch-level capability indicator, rather than relying solely on individual task pass rates. Experiments across multiple models, together with comparisons against fixed preference curves and predefined decay schedules, demonstrate that the approach can improve learning effectiveness under a limited rollout budget.

The method exploits the diminishing marginal gains of its value function to solve the constrained budget allocation problem using a heap-based greedy strategy. This enables efficient computation of an optimal allocation for the defined objective, with low allocation overhead that makes the method practical for RL training.

Critical weaknesses

Exploration and Exploitation Trade-off

  • The authors frame the proposed approach in terms of the exploration and exploitation trade-off, particularly in Section 3.3. This framing is reasonable, but its meaning differs from classical RL. CoBA-RL prioritizes tasks based on their estimated training value, rather than explicitly maximizing rewards or gathering information.
  • There is insufficient evidence to support the interpretation of adaptive budget allocation as balancing exploration and exploitation. Pass rates and rollout counts alone do not clearly distinguish the utilization of existing capabilities, the discovery of new solution paths, and trajectory diversity. Thus, while the reported results show that difficulty-based rollout allocation improves performance, they do not sufficiently establish that these gains arise from a better exploration-exploitation balance. This also leaves the method’s distinction from existing adaptive curriculum and resource allocation approaches unclear.
  • In line 357, the authors state that the “overall Exploit-to-Explore trend emerges from capability feedback rather than a predefined training schedule”. However, while the transition depends on the model’s capabilities, the preference for easier tasks at high global failure rates and harder tasks at lower failure rates is already defined in the capability-oriented value function.
  • In Section 3.3 and Table 2, the implementation of the Explore-to-Exploit strategy is not clearly explained, making it difficult to interpret the comparison.

Contribution of the Capability-Oriented Value Function

  • The budget saturation function already seems to favor tasks with intermediate pass rates through the (pi(1−pi)p_i(1-p_i)) term. Therefore, it is unclear how much of the improvement comes from this component rather than from shifting the preference from easier to harder tasks as training progresses.
  • In Section 3.4, the authors compare different Beta distributions and scheduling strategies. However, including a uniform Beta distribution ((α=β=1\alpha=\beta=1)) would help show whether adapting the task preference to the model’s capabilities provides additional benefits over simply favoring tasks with intermediate pass rates.

Comparison with Recent Adaptive Methods

  • Section 3.5 compares CoBA-RL with several recent adaptive methods, but the improvement over VIP is relatively small (0.62 pp). It is unclear whether the results in Table 4 are based on multiple training runs or how much they vary across runs. Without this information, it is difficult to tell how reliable the improvement is. The paper also does not clearly explain what CoBA-RL adds compared to VIP or the other methods.

Minor comments

  • Section 3.5 would fit more naturally together with the main results section.
  • In related works, the discussion on general budget allocation could be shortened to focus more on recent methods, particularly VIP and DynaMO, and what CoBA-RL adds compared to them.
Questions for authors
  • How many independent training runs underlie the results in Table 2 and Figure 5, respectively? Please specify the training seeds, how the results were aggregated, and the variability across runs. If different budget allocation trends were observed under identical experimental settings, please report how frequently each trend occurred.
  • Are the results in Table 4 from multiple training runs? If so, how much do the results vary across runs?
7 comments
AuthorsOpenReview ↗

Author Response to Reviewer LKfw (Part 1/2)

We sincerely thank the reviewer for the positive assessment and constructive suggestions. We have marked all revisions in red in the updated PDF and respond to your concerns as follows:

W1. Exploration–exploitation trade-off

1. Meaning of exploration and exploitation. In Section 3.3, we use exploration and exploitation at the task-allocation level rather than the classical action level. Specifically, exploitation allocates rollouts to solvable prompts that still offer learning room, obtaining reliable gradient signals to consolidate the model's current capability. Exploration assigns more rollouts to challenging, low-pass-rate prompts to increase the chance of sampling correct solutions and expanding the model's capability boundary.

2. Empirical evidence for exploration–exploitation balance and distinction from adaptive baselines. Figure 6 tracks how training prompts transition across pass-rate categories. Under CoBA-RL, the proportion of prompts reaching full mastery pi=1p_i = 1 after training reaches 36.7% on initially hard prompts, compared with 17.3% for GRPO and 20.4% for Knapsack-RL, and reaches 71.2% on medium prompts, compared with 46.8% for GRPO and 50.0% for Knapsack-RL, while retaining 95.2% on initially mastered prompts. In addition, Table 14 shows that CoBA-RL increases the learnable ratio to 72.87%, compared with 44.26% under GRPO. These gains in solving harder prompts while consolidating existing capability provide outcome-level evidence for our interpretation of exploration and exploitation.

Regarding the distinction from existing adaptive approaches, curriculum methods such as SEC decide which prompts are sampled for training, whereas CoBA-RL determines how many rollouts each prompt receives. The allocation mechanisms in VIP, GVM-RAFT, and DynaMO are motivated by gradient variance minimization, using variance estimates, bounds, or proxies to guide allocation. CoBA-RL explicitly incorporates global capability feedback to dynamically reshape the mapping from pass rates to sample value. Consequently, even if a prompt's pass rate pip_i remains unchanged, its estimated value adapts as the model's global capability evolves, as illustrated by Figure 5 and detailed in Point 3 below.

3. Dynamic preference adjustment driven by model-capability feedback. We agree with the reviewer that the mapping direction—favoring easier prompts when the global failure rate FtF_t is high and harder prompts when FtF_t is low—is specified by our value function. However, the allocation preference is not a predefined function of the training step. As shown by the trajectory in Figure 5, αt\alpha_t changes continuously throughout training and fluctuates even across adjacent steps in response to the model's real-time global capability FtF_t, dynamically shifting the preferred pass-rate region as capability varies.

4. Implementation of Explore-to-Exploit. This baseline reverses the capability-to-preference mapping by driving αt\alpha_t with the global success rate St=1−FtS_t = 1 - F_t and the corresponding mirrored nonlinear transformation, while keeping βt=κ−αt\beta_t = \kappa - \alpha_t, the budget saturation factor, and the greedy allocator unchanged. Thus, it starts with a small αt\alpha_t that favors harder prompts, and as StS_t increases during training, αt\alpha_t increases as shown on the right of Figure 5, shifting the preference toward easier prompts. In Table 2, this reverse mapping achieves 44.50% Avg@16 compared with 47.29% for our Exploit-to-Explore strategy.

AuthorsOpenReview ↗

Author Response to Reviewer LKfw (Part 2/2)

W2. Contribution of the capability-oriented value function

1. Joint effect of the budget saturation factor and the Beta preference. Our value function V(Bi,πθ,pi)V(B_i, \pi_\theta, p_i) in Eq. (9) is determined by the joint effect of two terms: the budget saturation factor models the diminishing marginal returns as more rollouts BiB_i are allocated to a prompt, while the capability-conditioned Beta density dynamically shapes the preference across pass rates pip_i based on global capability feedback.

To illustrate how the two terms work together, the table below lists the values of V(Bi,πθ,pi)V(B_i,\pi_\theta,p_i) across different pass rates and Beta parameters at Bi=16B_i=16:

Beta preferencepi=0.1p_i=0.1pi=0.2p_i=0.2pi=0.3p_i=0.3pi=0.4p_i=0.4pi=0.5p_i=0.5pi=0.6p_i=0.6pi=0.7p_i=0.7pi=0.8p_i=0.8pi=0.9p_i=0.9
α=8, β=3\alpha=8,\ \beta=30.000010.001390.021920.131030.444460.995041.516361.427530.52056
α=6, β=5\alpha=6,\ \beta=50.002500.078070.417771.031901.555611.547840.974800.312270.02249
α=2, β=9\alpha=2,\ \beta=91.171261.427530.884540.373140.111110.021840.002350.000092.45e-7

As (α,β)(\alpha, \beta) shifts from (8,3)(8, 3) to (6,5)(6, 5) and (2,9)(2, 9), the highest value among the listed pass rates moves from pi=0.7p_i=0.7 to 0.50.5 and then to 0.20.2. Combined with Figure 5, which shows how global capability feedback drives (αt,βt)(\alpha_t, \beta_t) during training, this illustrates how the two terms jointly guide budget allocation as model capability evolves.

2. Uniform-Beta ablation. To directly verify the additional benefit of the capability-adaptive Beta preference, we re-ran CoBA-RL and a Uniform Beta baseline (α=β=1\alpha=\beta=1) on Qwen3-4B-Base under identical training and allocation settings, where the Beta density in Uniform Beta reduces to a constant and the value function relies solely on the budget saturation factor:

Method (Qwen3-4B-Base)Avg@16 (%)
Uniform Beta (α=1, β=1\alpha=1,\ \beta=1)48.25
CoBA-RL50.17

CoBA-RL outperforms the Uniform Beta baseline, confirming that dynamically reshaping task preferences via global capability feedback provides clear gains beyond the budget saturation factor alone.

W3 & Q2. Comparison with recent adaptive methods and training variability

Training stability across independent runs. To assess run-to-run variability, we conducted multiple independent training runs for our main experimental configuration and observed a relative standard deviation of approximately 0.9% in Avg@16. This low variance confirms that our experimental findings are reliable and that CoBA-RL's improvement over competing baselines in Table 4, including VIP, is stable.

Key distinction from VIP and other allocation methods. As discussed in W1.2 above, CoBA-RL's key distinction from VIP and other adaptive allocators is that, beyond instance-level statistics, it explicitly incorporates global capability feedback to dynamically reshape the mapping from pass rates to sample values. As the global failure rate changes, the Beta preference distribution shifts accordingly so that the allocator emphasizes different pass-rate intervals at different capability stages, aligning rollout allocation with the model's current learning state.

W4 (Minor comments). Section organization and related work

We thank the reviewer for these helpful suggestions. In the revised manuscript, we have moved Section 3.5 into the main results section (Section 3.2) for a more natural flow, and streamlined the general budget-allocation discussion in Section 4.2 to expand the discussion of recent adaptive methods—particularly VIP and DynaMO—and CoBA-RL's contributions relative to them as outlined above.

Q1. Independent training runs and consistency of budget allocation trends

For our main experiments, we conducted three independent training runs using seeds 1, 42, and 123, and evaluated checkpoints at the same converged late-stage training step across runs, where the relative standard deviation in Avg@16 is within 0.9%. Across all runs, the budget allocation trend remains consistent with the Exploit-to-Explore trajectory shown in Figure 5: allocation initially focuses on exploiting easier prompts and progressively shifts toward exploring harder prompts, with local step-to-step fluctuations driven by real-time capability feedback.

We thank the reviewer once again for the constructive suggestions, which have helped us improve our experiments and the clarity of the manuscript. We hope these responses address your concerns and would be happy to discuss any remaining questions.

Dear Reviewer LKfw,

Thank you again for your comments and for taking the time to engage with our work.

As the discussion period is coming to an end, we wanted to check whether our responses have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.

If you feel that your concerns have been adequately addressed, we would also appreciate it if you could take our responses into account when finalizing your evaluation.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Author Response to Reviewer phce (Part 1/2)

We appreciate the reviewer's insightful comments and valuable feedback. All corresponding revisions are highlighted in red in the updated manuscript, and our detailed responses are provided below:

W1 & Q1. Computational efficiency under cumulative token and GPU-hour budgets

Uniform allocation in GRPO wastes a substantial portion of the rollout budget on already-mastered prompts. In contrast, CoBA-RL adaptively calibrates budget allocation based on global capability feedback and prompt pass rates, directing rollouts toward samples with higher learning value. Consequently, generating more effective output tokens and taking longer per step are expected results of focusing compute on more informative samples.

Performance under cumulative compute budgets. To evaluate efficiency under matched actual compute, we compare Qwen2.5-7B-Instruct (Avg@16, %) across cumulative output-token and training GPU-hour budgets under both 2,048 and 4,096 rollouts per step.

Budget typeBudget capGRPOCoBA-RL
2,048 rollouts per step
Output tokens (B)0.7539.2240.79
Output tokens (B)1.2541.1742.75
Output tokens (B)1.7540.6743.37
Output tokens (B)2.2542.0743.59
Training GPU-hours10039.5342.06
Training GPU-hours14040.6742.91
Training GPU-hours16040.6743.37
Training GPU-hours18041.0543.61
4,096 rollouts per step
Output tokens (B)1.0039.4841.42
Output tokens (B)2.0042.0542.91
Output tokens (B)3.0041.1043.37
Output tokens (B)3.5041.7644.53
Training GPU-hours10039.4841.42
Training GPU-hours14040.8842.90
Training GPU-hours18041.1942.91
Training GPU-hours22042.3043.41

Across all evaluated output-token and GPU-hour caps in both rollout settings, CoBA-RL consistently outperforms GRPO and improves steadily as compute increases, indicating that adaptively allocating the rollout budget based on model capability effectively translates into downstream performance gains.

Efficiency comparison across rollout budgets (CoBA-RL-2048 vs. GRPO-4096). We further compare the total training cost and final accuracy of CoBA-RL at 2,048 rollouts per step against GRPO at 4,096 rollouts per step:

MethodRollouts/stepOutput tokens (B)Training GPU-hoursAvg@16 (%)
GRPO4,0964.41424.6442.24
CoBA-RL2,0483.62318.2345.52

With half the rollout budget per step, CoBA-RL-2048 reduces total output tokens by 18.02% and training GPU-hours by 25.06% relative to GRPO-4096, while improving Avg@16 by 3.28 percentage points. Together with the above comparisons, these results demonstrate that CoBA-RL is more compute-efficient than GRPO in both token usage and training GPU time.

AuthorsOpenReview ↗

Author Response to Reviewer phce (Part 2/2)

W2 & Q2. Motivation, implementation, and reliability of capability-oriented budget allocation

Motivation and training dynamics. Our design follows a common RL training dynamic: as prompts become mastered, additional rollouts are less likely to produce useful group-relative gradients. In particular, a group with identical rewards has zero relative advantages. Since rollout generation is costly, its allocation should adapt as the set of learnable prompts changes.

CoBA-RL captures this dynamic through a capability-conditioned Beta preference. Figure 5 shows an overall shift from consolidating capability on easier prompts toward allocating more exploration to harder prompts as training progresses. Rather than following a fixed monotonic schedule, the preference adapts dynamically to observed capability feedback throughout training. Moreover, Table 14 shows that this capability-oriented value function increases the learnable ratio—the rollout share assigned to prompts with 0<pi<10 < p_i < 1—from 44.26% under GRPO to 72.87% under CoBA-RL, directly demonstrating the value of dynamic allocation during training. The comparison with fixed preferences and a predefined decay schedule in Table 3 of our paper further demonstrates the effectiveness of adapting budget allocation to the model's evolving capability. As reproduced below, CoBA-RL achieves the highest average performance.

MethodAvg@16 (%)
Fixed α=10.5, β=1.5\alpha=10.5,\ \beta=1.544.22
Fixed α=1.5, β=10.5\alpha=1.5,\ \beta=10.545.21
Linear step decay45.39
CoBA-RL47.29

Normalization with variable BiB_i. For each prompt, advantages are normalized over its own BiB_i responses:

Aij=rij−μisi+ϵ,A_{ij}=\frac{r_{ij}-\mu_i}{s_i+\epsilon},

where μi\mu_i and sis_i are the mean and standard deviation of rewards within that prompt's group. As in the GRPO baseline, policy losses are summed over valid response tokens and divided by their count within each microbatch, followed by gradient accumulation.

Nonlinear mapping Ψ\Psi. As training progresses, changes in the global failure rate become smaller. The nonlinear mapping is designed to make the allocation preference more responsive to these changes, so that budget allocation can continue to adapt as model capability improves. Table 12 shows that both sigmoid and log-like mappings outperform the linear baseline, indicating that our method is not restricted to a particular form of nonlinear mapping.

For example, when the failure rate FF is at or below 0.5, we use

Ψ(F)=11+exp⁡[−γ(F−0.5)],\Psi(F)=\frac{1}{1+\exp[-\gamma(F-0.5)]},

where γ\gamma controls the slope and is set to 10 in our experiments.

Dynamic τ\tau. We adjust τ\tau using the pass-rate variance vv of the current batch's candidate prompts:

τ=τmin⁡+(τmax⁡−τmin⁡)clip⁡(vvmax⁡,0,1).\tau=\tau_{\min}+(\tau_{\max}-\tau_{\min})\operatorname{clip}\left(\frac{v}{v_{\max}},0,1\right).

Here, τmin⁡\tau_{\min} and τmax⁡\tau_{\max} bound the coefficient and are set to 3 and 12, respectively, in our experiments; vmax⁡v_{\max} normalizes the variance. Greater difficulty diversity increases τ\tau, allowing larger budget gaps between high- and low-value prompts. Lower variance reduces τ\tau, making gains saturate sooner and limiting further budget concentration.

Batch failure rate computation. The batch failure rate FtF_t is computed from fresh rollouts. Specifically, after the newly generated rollouts of the current batch are scored, we compute each prompt's failure rate directly from these fresh outcomes and average them across the batch to update the Beta parameters for the next step (while also refreshing each prompt's cached pass rate pip_i for future encounters).

Training variability. We conducted multiple independent training runs for our main experiments. The results show a relative standard deviation of approximately 0.9%, supporting the reliability of our experimental findings and confirming that CoBA-RL's improvement over competing baselines such as VIP is stable.

Thank you again for the precise questions, which helped us sharpen both the analysis and its presentation. We hope these additions address your concerns and would be glad to discuss any further questions.

Dear reviewers,

The authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.