acceptodds
Under review as a conference paper at ICLR 2027

Event-Driven Compute Allocation in Fully Asynchronous LLM Training

Abstract

Reinforcement learning (RL) for large language models is computationally intensive, with rollout generation consuming substantial computational resources. Recent fully asynchronous RL systems improve hardware utilization by decoupling rollout generation from model updates, but they leave open a central algorithmic question: once a rollout job finishes and a generation slot becomes available, how should the system allocate the next unit of compute to maximize learning efficiency? This compute-allocation question is particularly important in large-scale RL runs, where the goal is to achieve the best possible performance under a fixed compute budget. We propose EDCA (Event-Driven Compute Allocation), a unified framework for prompt sampling and rollout budget allocation in fully asynchronous RL. Unlike existing methods, which typically study either prompt selection or adaptive rollout allocation in synchronous settings, our framework treats job completion as the fundamental control point of the learning system. At each completion event, the scheduler chooses between two actions: continue, which launches more rollouts for the current prompt, and replace, which retires the current prompt and fills the freed slot with a newly sampled one. Across three model backbones and six mathematical reasoning benchmarks, EDCA improves average accuracy over fixed-rollout GRPO, including a 2.81% point gain on DeepSeek-R1-Distill-Qwen-1.5B. EDCA effective gradient ratio reaches 92.58%, compared with 56.92% for GRPO. When combined with dynamic sampling, EDCA reduces rejected samples by 76.1% while completing more policy updates within the same wall-clock budget.

Primary area:
reinforcement learning
Keywords:
Large language model, Reinforcement Learning, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZyL5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Event-Driven Compute Allocation (EDCA) for fully asynchronous LLM reinforcement learning. EDCA combines value-guided prompt sampling, adaptive rollout continuation/replacement, and a training trigger based on accumulated trajectories. Its central goal is to avoid zero group-relative advantages under GRPO.

Critical strengths
  1. The motivation is reasonable. Treating rollout completions as opportunities to reconsider allocation is a straightforward way to exploit streaming feedback.

  2. Improved average benchmark accuracy. The effective-gradient ratios, rejection counts, and trainer-idleness measurements provide complementary evidence consistent with the proposed mechanism.

Critical weaknesses
  1. The core allocation algorithm is not specified sufficiently for reproducibility. If the continuation rule uses the empirical success rate of a currently all-incorrect group, the estimated probability of obtaining a correct response is zero, preventing the intended compensation behavior. The initial budget is also unclear. Section 5.2 defines (K_i) by maximizing (4p_i(N)(1-p_i(N))) over budgets from 4 to 14, but does not adequately define (p_i(N)) or explain how it is available before selecting the budget.

  2. The connection between the proposed proxies and compute-efficient learning is insufficiently established. The prompt-selection proxy (p(1-p)^2) peaks at (p=1/3), whereas binary reward variance and the probability of obtaining a mixed-reward group at a fixed rollout count peak at (p=1/2). These optimize different quantities and are not inherently contradictory, but the paper does not establish why its asymmetric proxy is preferable for this setting.

Questions for authors
  1. How exactly is the success probability estimated during training? Please specify its initialization, update equation. For an all-incorrect active group, what estimate is used to decide whether additional sampling is worthwhile?

  2. How is the initial rollout budget (K_i) actually computed? What precisely does (p_i(N)) denote, and how is it obtained?

4 comments

Weaknesses

Q1:The core allocation algorithm is not specified sufficiently for reproducibility. If the continuation rule uses the empirical success rate of a currently all-incorrect group, the estimated probability of obtaining a correct response is zero, preventing the intended compensation behavior. The initial budget is also unclear. Section 5.2 defines (K_i) by maximizing (4p_i(N)(1-p_i(N))) over budgets from 4 to 14, but does not adequately define (p_i(N)) or explain how it is available before selecting the budget.

A1: Thank you for pointing out these ambiguities. We have made targeted revisions to clarify the implementation and the distinct motivations behind prompt selection and rollout allocation.

1. Clarification of the allocation algorithm.

Thank you for pointing out this ambiguity. The compensation decision uses a historical per-prompt success-rate estimate, rather than the empirical pass rate of the current initial group. This estimate remains fixed throughout the group’s initial and compensation stages and is updated only after the entire group finishes. Therefore, observing an all-incorrect initial group does not immediately set the estimate to zero or automatically prevent compensation. We will clarify this distinction and the order of these operations in Section 5.2 and Algorithm 1.

2. How compensation is performed.

The scheduler first completes all KiK_i initial rollouts. If the group already contains both correct and incorrect responses, it replaces the prompt. For an all-incorrect initial group, it uses the fixed historical estimate p^i\hat p_i to calculate the probability that cc additional rollouts will produce at least one correct response:

Pi(c)=1−(1−p^i)c,P_i(c)=1-(1-\hat p_i)^c,

under an independent-sampling approximation. It then chooses the smallest feasible positive integer cc satisfying

Pi(c)≥0.85,P_i(c)\ge0.85,

subject to the total group-size cap and sliding-window budget. If such a count exists, the scheduler performs a single compensation stage; otherwise, it replaces the prompt without additional sampling.

The historical estimate is initialized from the initial model’s pass rate over eight responses per prompt. Only after the complete group finishes, including any compensation responses, do we update it:

\qquad \alpha=0.1,$$ where $\bar r_i$ is the complete group’s observed pass rate and $\alpha$ weights the previous estimate. This separates the estimate used to make the compensation decision from the observations subsequently used to update it. **Q2:The connection between the proposed proxies and compute-efficient learning is insufficiently established** **A2:**The two components address different questions: **which prompts offer greater learning potential**, and **how to obtain a usable group-relative training signal from a selected prompt within the available budget**. For rollout allocation, both reward outcomes are needed for nonzero group-relative advantages. Bernoulli variance $p(1-p)$, as well as the mixed-group probability at a fixed rollout count, peaks at $p=0.5$. This motivates the variance-based initial allocation and the subsequent compensation stage. Prompt selection instead targets improvement in success probability. Following the local-update motivation of Knapsack RL, we added an explicit derivation in Appendix B. Under the stated softmax positive-update assumptions, $$p^+-p=Cp(1-p)^2+O(\eta^2),$$ where $C>0$ is independent of $p$ when the advantage and conditional failure distribution are held fixed. Since $$\frac{d}{dp}\bigl[p(1-p)^2\bigr]=(1-p)(1-3p),$$ the leading-order gain peaks at $p=1/3$. This provides a theoretical motivation for the asymmetric proxy under these assumptions, rather than a universal optimality guarantee. We also compared our sampler against a PCL-based selection strategy favoring prompts with success probabilities near $0.5$. Only the prompt-selection component was replaced; the rollout scheduler, training trigger, and other settings were unchanged. On Qwen2.5-Math-1.5B after 500 updates: | **Prompt-selection configuration** | **AIME24** | **AIME25** | **AMC** | **MATH** | **Minerva** | **Olympiad** | **Avg.** | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | EDCA with PCL-style sampler [1] | 15.41 | 9.79 | 52.78 | 71.81 | 24.81 | 36.12 | 35.12 | | EDCA | 16.25 | 11.67 | 50.75 | 72.44 | 26 | 36.31 | **35.57** | This controlled comparison provides empirical support for our selection strategy, complementing the local theoretical motivation. [1]. Prompt Curriculum Learning for Efficient LLM Post-Training.

Questions For Authors:

Q1:How exactly is the success probability estimated during training?

A: We maintain a historical success-rate estimate p^i\hat p_i for each prompt.

Initialization and update. We sample eight responses per prompt using the initial model and initialize p^i\hat p_i from their observed pass rate. The initial estimates are clipped to [0.01,0.99][0.01,0.99].

After a group finishes, we update the estimate using its complete observed pass rate, including compensation responses:

\qquad \alpha=0.1.$$ Here, $\bar r_i$ is the complete group’s observed pass rate, and $\alpha$ weights the **previous estimate**. **Initial allocation and compensation for all-incorrect groups.** Before generating the current group, we compute its initial budget from the historical estimate: $$K_i= \left\lfloor N_{\min} +4\hat p_i(1-\hat p_i) \left(N_{\mathrm{peak}}-N_{\min}\right) \right\rfloor, \qquad N_{\min}=4.$$ Thus, prompts with low estimated success rates receive an initial budget closer to the minimum of four rollouts. The scheduler completes all $K_i$ initial responses before deciding whether to compensate. If these responses are all incorrect, the compensation decision uses the same historical estimate, rather than the current group’s observed pass rate of zero. Under an independent-sampling approximation, the estimated probability of obtaining at least one correct response in $c$ additional rollouts is $$P_i(c)=1-(1-\hat p_i)^c.$$ We select the smallest feasible $c\ge1$ satisfying $P_i(c)\ge0.85$, subject to the total group-size cap and sliding-window budget. If no feasible count meets this threshold, we replace the prompt without compensation. Otherwise, we complete a single compensation stage. Consequently, an all-incorrect initial group can still receive compensation. Prompts with very low historical success estimates receive a small initial allocation and no additional rollouts when the required probability cannot be reached within the available budget. **Q2: How is the initial rollout budget (K_i) actually computed?** **A2:** The original maximization expression did not accurately describe the implementation. The initial budget is computed directly from the historical success-rate estimate: $$K_i= \left\lfloor N_{\min} +4\hat p_i(1-\hat p_i) \left(N_{\mathrm{peak}}-N_{\min}\right) \right\rfloor.$$ Here, $N_{\min}$ is the minimum initial budget, $N_{\mathrm{peak}}$ is the maximum initial budget and $N_{\mathrm{max}}$ caps the total group size. The notation $p_i(N)$ was intended to refer to the prompt’s estimated success probability, but its apparent dependence on the candidate budget $N$ was misleading. We will replace it with $\hat p_i$ and correct the budget formula.

Dear Reviewer ZyL5,

Thank you again for your thoughtful feedback. The ICLR rebuttal deadline is approaching. When you have a chance, could you let us know whether these additional results and clarifications address your main concerns? We would be grateful to hear about any remaining questions.

Best regards, The Authors

Dear Reviewer ZyL5,

We sincerely appreciate your careful review and detailed suggestions. Your feedback helped us clarify the historical success-rate estimation, initial rollout-budget calculation, and compensation procedure. We also provided a derivation motivating the prompt-selection proxy and a controlled comparison with a PCL-style sampler to strengthen its theoretical and empirical support. Reviewer rUer has also indicated that the additional experiments largely addressed their concerns about initialization cost and the stopping rule.

Please let us know if any part of the implementation or motivation remains unclear. We would be glad to discuss it before the discussion closes. If these clarifications and additional results address your main concerns, we sincerely hope you will consider a higher overall score.

Best regards, The Authors

Reviewer_rTbnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduced EDCA, an efficient async RL orchestration system. The system combines a value guided prompt selection system for choose the most effective sample for training, an adaptive rollout scheduler for deciding whether to generate more responses for a prompt or replace it, and a training-trigger for managing the group of samples for each step update.

Critical strengths

The proposed EDCA method is effective compare to vanilla GRPO in improving the training effectiveness. The author performed ablation study showing that each of the component are contributing to the performance.

Critical weaknesses

The baselines used by the author is very weak: Only GRPO and Reinforce-Ada, both at least one year old and is no longer sota methods. The improvement over GRPO and Reinforce-Ada is mild, mostly by 1-3%.

Questions for authors

Are all training performed convergent? If not, does the efficiency and performance of EDCA still holds at the plateau? What are the reason for the chosen step number? It seems like there are existing works for all of the value guided prompt selection, Adaptive rollout scheduling, and the training trigger. What are the novelty compared to the existing methods. Are there experiments to establish each of the component is better performing than existing work on itself?

5 comments

Weakness

Q1-1:Only GRPO and Reinforce-Ada, both at least one year old and is no longer sota methods.

A1-1: We thank the reviewer for suggesting a broader baseline comparison. We have added Knapsack RL [1] and VIP [2], two directly relevant methods for adaptive rollout allocation, and evaluated them on both Qwen2.5-Math-1.5B and Qwen3-4B-Base. All results below are reported at 500 training steps.

For Qwen2.5-Math-1.5B, the results are:

MethodAIME24AIME25AMCMATHMinervaOlympiadAvg.
GRPO13.5410.6349.7771.3122.534.8133.76
Reinforce-Ada15.636.8847.147021.8834.9432.74
Knapsack RL [1]14.168.5450.6771.3124.3137.7534.46
VIP [2]15.8310.4149.2471.8723.3736.7534.58
EDCA16.2511.6750.7572.442636.3135.57

For Qwen3-4B-Base, the results are:

MethodAIME24AIME25AMCMATHMinervaOlympiadAvg.
GRPO26.4621.0458.7382.7535.3345.7545.01
Reinforce-Ada20.6324.1756.5580.6933.1345.6943.47
Knapsack RL [1]23.5424.5859.1982.0633.8847.8145.18
VIP [2]22.2923.3360.3281.2531.9446.3144.24
EDCA27.521.4660.6984.8737.0647.2546.47

EDCA achieves the highest average accuracy in both model settings. On Qwen2.5-Math-1.5B, it exceeds Knapsack RL and VIP by 1.11 and 0.99 percentage points, respectively. On Qwen3-4B-Base, these improvements are 1.80 and 2.23 percentage points, respectively. These additional comparisons extend our evaluation beyond GRPO and Reinforce-Ada and show that EDCA’s advantages also hold against the newly included baselines across both model settings.

Q1-2:The improvement over GRPO and Reinforce-Ada is mild, mostly by 1-3%.

A1-2: Thank you for this comment. We think the accuracy gains should be considered together with the training resources required to obtain them. This perspective is consistent with prior asynchronous RL research [3, 4]. EDCA reduces the average training-step time from 517.1 to 366.7 seconds. Summing the logged step durations gives 50.56 hours for GRPO versus 35.86 hours for EDCA, saving 14.70 hours (29.1%) during training.

And we find that the additional 250 steps yield less than one percentage point of improvement. Moreover, EDCA’s 500-step accuracy remains higher than the baselines’ 750-step accuracy, indicating that its advantage is not recovered simply by extending baseline training within this range.

For Qwen3-4B-Base, the results are:

Method500 steps (%)750 steps (%)Change (percentage points)
GRPO45.0145.210.2
Reinforce-Ada (Pos)43.4743.780.31
VIP44.2445.190.95
EDCA46.4747.120.65

GRPO, Reinforce-Ada, and VIP improve by only 0.20, 0.31, and 0.95 percentage points, respectively. EDCA at 500 steps already outperforms all three baselines at 750 steps.

We observe a similar pattern on *Qwen2.5-Math-1.5B:

Method500 steps (%)750 steps (%)Change (percentage points)
GRPO33.7634.560.8
EDCA35.5736.180.61

[1] Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation.

[2] Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards.

[3] AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning.

[4] Laminar: A Scalable Asynchronous RL Post-Training Framework.

Questions For Authors

Q1-1:Are all training performed convergent? If not, does the efficiency and performance of EDCA still holds at the plateau? What are the reason for the chosen step number?

A1-1: We train all methods for 500 steps and fix the number of trajectories used per update, ensuring a controlled comparison under the same update and training-sample budgets. To examine whether EDCA’s advantage persists with longer training, we extended both EDCA and GRPO to 750 training steps, adding 250 steps to the original budget on Qwen2.5-Math-1.5B. EDCA’s average accuracy across the six mathematical benchmarks increased from 35.57% to 36.18%, a gain of 0.61 percentage points, while GRPO improved from 33.76% to 34.56%, a gain of 0.80 percentage points. EDCA therefore retained an advantage of 1.62 percentage points. They show that additional training yields limited average performance gains for both methods and that EDCA’s performance advantage persists beyond the original 500-step training budget.

Furthermore, we also extended training from 500 to 750 steps and added two baselines, Knapsack RL and VIP. The table below reports the results on Qwen3-4B-Base at step 750. We observe only modest changes in performance compared with step 500, suggesting limited additional gains from longer training.

MethodStepsAIME24AIME25AMCMATHMinervaOlympiadAvg.
GRPO50026.4621.0458.7382.7535.3345.7545.01
GRPO75024.7918.7559.6484.5637.1946.3145.21
Reinforce-Ada50020.6324.1756.5580.6933.1345.6943.47
Reinforce-Ada75021.0421.2560.4779.8136.134443.78
VIP50022.2923.3360.3281.2531.9446.3144.24
VIP75026.0417.9260.3984.1335.6347.0645.19
Knapsack RL50023.5424.5859.1982.0633.8847.8145.18
Knapsack RL75024.7917.2958.0682.6335.6945.2543.95
EDCA w/o VS50027.9222.7158.1384.5635.4448.7546.25
EDCA w/o VS75026.4621.6760.9984.8835.7547.6346.23
EDCA50027.521.4660.6984.8737.0647.2546.47
EDCA75027.2925.6361.684.5634.5649.0647.12

Second, this training horizon is comparable to those used in related mathematical reasoning studies: the GRPO stage of CurioSFT [1] uses 500 steps, while the initial Reinforce-Ada [2] reports 600 steps and its official training script [3] specifies 400 steps.

[1] CurioSFT: Learning While Staying Curious

[2] Reinforce-Ada: An Adaptive Sampling Framework for Reinforce-Style LLM Training, v1

[3] Reinforce-Ada official training script

Q1-2:It seems like there are existing works for all of the value guided prompt selection, Adaptive rollout scheduling, and the training trigger. What are the novelty compared to the existing methods. Are there experiments to establish each of the component is better performing than existing work on itself?

A1-2: We do not attribute EDCA’s novelty simply to the introduction of three modules. We formulate and address the system-level compute-allocation problem in fully asynchronous RL. Our contribution lies in studying a decision that arises naturally in fully asynchronous RL: for a given prompt, when should the system continue generating rollouts, and when should it switch to another prompt? This decision fits naturally into fully asynchronous execution, where individual rollout completions provide new outcome information and opportunities to reassign available generation capacity.

Within this formulation, the native GRPO baseline follows a count-based rule: it samples a fixed number of nn responses for each prompt, then replaces it with the next prompt in the data sequence. We ask whether this allocation can instead use observed outcomes to reduce uninformative sampling and increase the proportion of groups that provide useful training signals under a fixed rollout budget. The three components are designed around this shared objective: value-guided prompt selection determines which prompt to select upon replacement; adaptive rollout allocation determines whether to extend a prompt’s initial sampling budget or switch to another prompt; and the batch-based training trigger fixes the number of trajectories used per update, enabling a fair comparison across allocation strategies with different group sizes.

To directly evaluate our design choices against existing methods, we conduct controlled component-wise comparisons on Qwen2.5-Math-1.5B.

For prompt selection, we replace only EDCA’s value-guided prompt sampler with the PCL-style intermediate-difficulty selection strategy [1], retaining the same adaptive rollout scheduler and training trigger. EDCA achieves 35.57% average accuracy, compared with 35.12% for the PCL-style sampler.

Prompt-selection configurationAIME24AIME25AMCMATHMinervaOlympiadAvg.
EDCA with PCL-style sampler [1]15.419.7952.7871.8124.8136.1235.12
EDCA16.2511.6750.7572.442636.3135.57

For rollout allocation, all configurations disable value-guided prompt selection and use the same sequential prompt-selection rule. We replace only the rollout allocation mechanism, comparing EDCA w/o value sampler against existing allocation strategies. EDCA’s scheduler achieves SOTA 35.41% average accuracy, compared with 34.46% for Knapsack RL allocation [2]. This comparison isolates the scheduling contribution from that of the value-guided prompt sampler.

Rollout allocationAIME24AIME25AMCMATHMinervaOlympiadAvg.
Knapsack RL14.168.5450.6771.3124.3137.7534.46
Reinforce-Ada (Pos)15.636.8847.147021.8834.9432.74
1/p1/p-based allocation14.378.9646.2369.6222.4337.3133.15
EDCA w/o value sampler16.8810.4252.4171.3125.3136.1335.41

The training trigger primarily serves to ensure a fair comparison between allocation strategies which is a replaceable component of EDCA. We also evaluate an alternative training trigger based on DAPO’s dynamic sampling, which filters out zero-variance groups and continues sampling until sufficient valid training data have been collected. EDCA and GRPO use the same trigger in this comparison. As reported in Section 6.3 and Figure 4, EDCA reduces the average training-step time from 517.1 s to 366.7 s, a 29.1% reduction, while maintaining competitive accuracy. The trainer idle ratio also decreases from 43.0% to 19.0%. These results demonstrate that EDCA’s allocation policy remains effective under an alternative training trigger.

[1]. Prompt Curriculum Learning for Efficient LLM Post-Training

Reviewer_rUerOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EDCA, a framework for fully asynchronous RL that decides whether to continue sampling the current prompt or switch to a new one whenever a rollout completes. It combines this adaptive allocation rule with value-guided prompt selection and a training trigger that initiates policy updates once enough data are ready for training.

Critical strengths
  1. The paper addresses the practical problem of reducing uninformative rollout groups in fully asynchronous RL.
  2. EDCA combines value-guided prompt sampling, adaptive rollout allocation, and training triggers in a unified framework.
  3. Experiments across three model backbones show improved average mathematical reasoning accuracy and higher effective-gradient ratios.
Critical weaknesses
  1. The cost of the initial sampling stage is not clearly accounted for. The method first generates eight responses per prompt to estimate its success rate. With approximately 17,000 prompts, this amounts to roughly 136,000 responses. However, the paper does not report how long this step takes, whether its cost is included in the efficiency comparison, or whether these responses are reused for training. The preparation and training stages should be evaluated together to establish the overall cost savings.

  2. It is unclear why sampling should stop once a prompt has both correct and incorrect responses. For example, if the first four responses contain one correct and three incorrect answers, the method stops sampling that prompt and moves to another. This shows that the group can provide a training signal, but it does not establish that additional responses would offer no further benefit or that a new prompt would be a better use of compute. The authors should compare different stopping rules under the same budget, such as generating a few more responses versus immediately switching prompts, to demonstrate why their chosen stopping point is more effective.

Questions for authors

See weakness

3 comments

Weakness

**Q1:The cost of the initial sampling stage is not clearly accounted for. **

A1: We agree that the initialization cost should be explicitly accounted for when evaluating end-to-end efficiency.

For each model, offline initialization generates eight responses for each of the 17,291 prompt instances in the filtered training set which are not reused for gradient updates. Using 16 NVIDIA H20 GPUs (96 GB per GPU), we estimate the following initialization costs:

ModelInitialization timeGPU-hours
Qwen2.5-Math-1.5B0.25 hours4
DeepSeek-R1-Distill-Qwen-1.5B4 hours64
Qwen3-4B-Base1.5 hours24

To quantify the training-phase savings, we examined our dynamic-sampling experiment with DeepSeek-R1-Distill-Qwen-1.5B. EDCA reduces the average training-step time from 517.1 to 366.7 seconds. Summing the logged step durations gives 50.56 hours for GRPO versus 35.86 hours for EDCA, saving 14.70 hours (29.1%) during training. Including the estimated initialization overhead of 4 hours, EDCA’s combined initialization and logged training time is 39.86 hours, which still represents a saving of 10.70 hours (21.2%) compared with GRPO. Thus, the time savings remain substantial after accounting for initialization.

We also evaluated an online initialization variant that eliminates the separate offline sampling pass. During the first epoch, we train with GRPO and reuse its training rollouts to estimate per-prompt success rates, after which training switches to EDCA. This avoids generating additional rollouts solely for initialization. The results on Qwen2.5-Math-1.5B are shown below:

MethodAIME24AIME25AMCMATHMinervaOlympiadAvg.
GRPO13.5410.6349.7771.3122.5034.8133.76
Reinforce-Ada15.636.8847.1470.0021.8834.9432.74
EDCA16.2511.6750.7572.4426.0036.3135.57
EDCA with online initialization17.919.3852.2572.2523.3136.3735.24

EDCA with online initialization achieves higher average accuracy than GRPO and Reinforce-Ada. These results show that EDCA can retain its accuracy advantage over these baselines without a separate offline initialization pass.

Q2: It is unclear why sampling should stop once a prompt has both correct and incorrect responses.

A2: Thank you for your question. Our stopping rule is a practical allocation heuristic intended to obtain useful group-relative training signals across more prompts under a fixed trajectory budget. A similar motivation appears in Knapsack RL [1]. To examine this tradeoff empirically, we evaluated two increased-sampling variants on Qwen2.5-Math-1.5B :

  • 2× compensation budget: doubling the additional sampling budget in the compensation stage.
  • 2× initial-sampling and compensation caps: doubling the upper bounds for both stages. Larger initial groups are completed even when mixed rewards appear earlier, allowing additional sampling beyond the first occurrence of mixed rewards.

All accuracies below are percentages. Mean nn denotes the logged average rollout count per group at step 500.

ConfigurationAIME24AIME25AMCMATHMinervaOlympiadAvg.Mean nn
EDCA16.2511.6750.7572.4426.0036.3135.577.03
2× compensation budget16.879.7951.3570.6824.4334.8734.6711.04
2× initial-sampling and compensation caps16.667.29151.6571.6824.0035.7534.5121.06

Doubling the compensation budget improves some benchmarks but reduces average accuracy from 35.57% to 34.67%. Doubling both the initial-sampling and compensation caps increases the logged mean rollout count per group from 7.03 to 21.06, while yielding an average accuracy of 34.51%. Thus, both variants yield lower average accuracy under the same trajectory budget.

Under a fixed trajectory budget, larger groups reduce prompt coverage and may limit training diversity. Excessively large nn may also delay group completion, increasing policy staleness. These tradeoffs offer a plausible explanation for the observed performance degradation and support our stopping rule as a practical balance between sampling depth and prompt coverage.

[1] Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation.

Thank you for the detailed rebuttal and additional experiments. The new results have largely addressed my concerns regarding the initialization cost and stopping rule. I appreciate the authors' efforts and have raised my rating to 3.

Thank you for your thoughtful reassessment and constructive feedback. We are pleased that the additional experiments and clarifications have largely addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.