- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Event-Driven Compute Allocation (EDCA) for fully asynchronous LLM reinforcement learning. EDCA combines value-guided prompt sampling, adaptive rollout continuation/replacement, and a training trigger based on accumulated trajectories. Its central goal is to avoid zero group-relative advantages under GRPO.
Critical strengths
-
The motivation is reasonable. Treating rollout completions as opportunities to reconsider allocation is a straightforward way to exploit streaming feedback.
-
Improved average benchmark accuracy. The effective-gradient ratios, rejection counts, and trainer-idleness measurements provide complementary evidence consistent with the proposed mechanism.
Critical weaknesses
-
The core allocation algorithm is not specified sufficiently for reproducibility. If the continuation rule uses the empirical success rate of a currently all-incorrect group, the estimated probability of obtaining a correct response is zero, preventing the intended compensation behavior. The initial budget is also unclear. Section 5.2 defines (K_i) by maximizing (4p_i(N)(1-p_i(N))) over budgets from 4 to 14, but does not adequately define (p_i(N)) or explain how it is available before selecting the budget.
-
The connection between the proposed proxies and compute-efficient learning is insufficiently established. The prompt-selection proxy (p(1-p)^2) peaks at (p=1/3), whereas binary reward variance and the probability of obtaining a mixed-reward group at a fixed rollout count peak at (p=1/2). These optimize different quantities and are not inherently contradictory, but the paper does not establish why its asymmetric proxy is preferable for this setting.
Questions for authors
-
How exactly is the success probability estimated during training? Please specify its initialization, update equation. For an all-incorrect active group, what estimate is used to decide whether additional sampling is worthwhile?
-
How is the initial rollout budget (K_i) actually computed? What precisely does (p_i(N)) denote, and how is it obtained?
4 comments
Questions For Authors:
Q1:How exactly is the success probability estimated during training?
A: We maintain a historical success-rate estimate for each prompt.
Initialization and update. We sample eight responses per prompt using the initial model and initialize from their observed pass rate. The initial estimates are clipped to .
After a group finishes, we update the estimate using its complete observed pass rate, including compensation responses:
\qquad \alpha=0.1.$$ Here, $\bar r_i$ is the complete group’s observed pass rate, and $\alpha$ weights the **previous estimate**. **Initial allocation and compensation for all-incorrect groups.** Before generating the current group, we compute its initial budget from the historical estimate: $$K_i= \left\lfloor N_{\min} +4\hat p_i(1-\hat p_i) \left(N_{\mathrm{peak}}-N_{\min}\right) \right\rfloor, \qquad N_{\min}=4.$$ Thus, prompts with low estimated success rates receive an initial budget closer to the minimum of four rollouts. The scheduler completes all $K_i$ initial responses before deciding whether to compensate. If these responses are all incorrect, the compensation decision uses the same historical estimate, rather than the current group’s observed pass rate of zero. Under an independent-sampling approximation, the estimated probability of obtaining at least one correct response in $c$ additional rollouts is $$P_i(c)=1-(1-\hat p_i)^c.$$ We select the smallest feasible $c\ge1$ satisfying $P_i(c)\ge0.85$, subject to the total group-size cap and sliding-window budget. If no feasible count meets this threshold, we replace the prompt without compensation. Otherwise, we complete a single compensation stage. Consequently, an all-incorrect initial group can still receive compensation. Prompts with very low historical success estimates receive a small initial allocation and no additional rollouts when the required probability cannot be reached within the available budget. **Q2: How is the initial rollout budget (K_i) actually computed?** **A2:** The original maximization expression did not accurately describe the implementation. The initial budget is computed directly from the historical success-rate estimate: $$K_i= \left\lfloor N_{\min} +4\hat p_i(1-\hat p_i) \left(N_{\mathrm{peak}}-N_{\min}\right) \right\rfloor.$$ Here, $N_{\min}$ is the minimum initial budget, $N_{\mathrm{peak}}$ is the maximum initial budget and $N_{\mathrm{max}}$ caps the total group size. The notation $p_i(N)$ was intended to refer to the prompt’s estimated success probability, but its apparent dependence on the candidate budget $N$ was misleading. We will replace it with $\hat p_i$ and correct the budget formula.Dear Reviewer ZyL5,
Thank you again for your thoughtful feedback. The ICLR rebuttal deadline is approaching. When you have a chance, could you let us know whether these additional results and clarifications address your main concerns? We would be grateful to hear about any remaining questions.
Best regards, The Authors
Dear Reviewer ZyL5,
We sincerely appreciate your careful review and detailed suggestions. Your feedback helped us clarify the historical success-rate estimation, initial rollout-budget calculation, and compensation procedure. We also provided a derivation motivating the prompt-selection proxy and a controlled comparison with a PCL-style sampler to strengthen its theoretical and empirical support. Reviewer rUer has also indicated that the additional experiments largely addressed their concerns about initialization cost and the stopping rule.
Please let us know if any part of the implementation or motivation remains unclear. We would be glad to discuss it before the discussion closes. If these clarifications and additional results address your main concerns, we sincerely hope you will consider a higher overall score.
Best regards, The Authors
Weaknesses
Q1:The core allocation algorithm is not specified sufficiently for reproducibility. If the continuation rule uses the empirical success rate of a currently all-incorrect group, the estimated probability of obtaining a correct response is zero, preventing the intended compensation behavior. The initial budget is also unclear. Section 5.2 defines (K_i) by maximizing (4p_i(N)(1-p_i(N))) over budgets from 4 to 14, but does not adequately define (p_i(N)) or explain how it is available before selecting the budget.
A1: Thank you for pointing out these ambiguities. We have made targeted revisions to clarify the implementation and the distinct motivations behind prompt selection and rollout allocation.
1. Clarification of the allocation algorithm.
Thank you for pointing out this ambiguity. The compensation decision uses a historical per-prompt success-rate estimate, rather than the empirical pass rate of the current initial group. This estimate remains fixed throughout the group’s initial and compensation stages and is updated only after the entire group finishes. Therefore, observing an all-incorrect initial group does not immediately set the estimate to zero or automatically prevent compensation. We will clarify this distinction and the order of these operations in Section 5.2 and Algorithm 1.
2. How compensation is performed.
The scheduler first completes all Ki initial rollouts. If the group already contains both correct and incorrect responses, it replaces the prompt. For an all-incorrect initial group, it uses the fixed historical estimate p^i to calculate the probability that c additional rollouts will produce at least one correct response:
Pi(c)=1−(1−p^i)c,
under an independent-sampling approximation. It then chooses the smallest feasible positive integer c satisfying
Pi(c)≥0.85,
subject to the total group-size cap and sliding-window budget. If such a count exists, the scheduler performs a single compensation stage; otherwise, it replaces the prompt without additional sampling.
The historical estimate is initialized from the initial model’s pass rate over eight responses per prompt. Only after the complete group finishes, including any compensation responses, do we update it:
\qquad \alpha=0.1,$$ where $\bar r_i$ is the complete group’s observed pass rate and $\alpha$ weights the previous estimate. This separates the estimate used to make the compensation decision from the observations subsequently used to update it. **Q2:The connection between the proposed proxies and compute-efficient learning is insufficiently established** **A2:**The two components address different questions: **which prompts offer greater learning potential**, and **how to obtain a usable group-relative training signal from a selected prompt within the available budget**. For rollout allocation, both reward outcomes are needed for nonzero group-relative advantages. Bernoulli variance $p(1-p)$, as well as the mixed-group probability at a fixed rollout count, peaks at $p=0.5$. This motivates the variance-based initial allocation and the subsequent compensation stage. Prompt selection instead targets improvement in success probability. Following the local-update motivation of Knapsack RL, we added an explicit derivation in Appendix B. Under the stated softmax positive-update assumptions, $$p^+-p=Cp(1-p)^2+O(\eta^2),$$ where $C>0$ is independent of $p$ when the advantage and conditional failure distribution are held fixed. Since $$\frac{d}{dp}\bigl[p(1-p)^2\bigr]=(1-p)(1-3p),$$ the leading-order gain peaks at $p=1/3$. This provides a theoretical motivation for the asymmetric proxy under these assumptions, rather than a universal optimality guarantee. We also compared our sampler against a PCL-based selection strategy favoring prompts with success probabilities near $0.5$. Only the prompt-selection component was replaced; the rollout scheduler, training trigger, and other settings were unchanged. On Qwen2.5-Math-1.5B after 500 updates: | **Prompt-selection configuration** | **AIME24** | **AIME25** | **AMC** | **MATH** | **Minerva** | **Olympiad** | **Avg.** | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | EDCA with PCL-style sampler [1] | 15.41 | 9.79 | 52.78 | 71.81 | 24.81 | 36.12 | 35.12 | | EDCA | 16.25 | 11.67 | 50.75 | 72.44 | 26 | 36.31 | **35.57** | This controlled comparison provides empirical support for our selection strategy, complementing the local theoretical motivation. [1]. Prompt Curriculum Learning for Efficient LLM Post-Training.