- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a training budget reallocation framework ReAlloc for LLM post-training. It revisits objective aggregation as a hierarchical budget allocation problem based on three identified signals, including prompts, rollouts, and tokens. The proposed ReAlloc show superior results compare to baselines with less computational overhead. Ablation study further demonstrates the effectiveness of the three signals.
Critical strengths
- This paper revisit the objective aggregation as a hierarchical budget allocation for LLM post-training and identifies three key signals.
- ReAlloc achieves better results with minimal computational overhead.
- This paper is easy to understand and ReAlloc can be implemented simply.
Critical weaknesses
- ReAlloc appears to be limited to group-based training paradigms, so the phrase “for LLM post-training” in the title is somewhat overclaim.
- The theoretical motivation does not fully justify the specific reallocation signals used by ReAlloc. The first-order analysis only shows that shifting weight toward units with higher true gradient alignment can help, but it does not establish that outcome variance, rollout-centroid distance, or local entropy prominence reliably approximate that quantity. This paper itself notes that outcome variance alone determines neither gradient magnitude nor direction.
- "Optimization budget reallocation" is somewhat misleading, since the method mainly reallocates objective weights rather than computation. At the prompt level, all rollouts must already be generated before outcome variance can be computed, after which low-variance prompts are simply assigned zero optimization weight. Thus, ReAlloc discards optimization contribution after paying the rollout cost, rather than reallocating the actual sampling budget.
- The numerical improvements reported in Sec. 4.2 appear to be computed relative to the vanilla GRPO/OPD baselines, although the preceding sentences jointly discuss improvements over stronger baselines such as FORK, TIP, and BRTS. This makes the reported gains potentially misleading. For example, on Qwen3-4B AMC23, the reported +8.44 Avg@8 gain is relative to vanilla GRPO (57.50→65.94), whereas the gain over the strongest competing baseline FORK is only +4.69. Similarly, under OPD on AIME25, the reported +5.42 gain is relative to vanilla OPD, while the gain over BRTS is +3.75. This paper should explicitly state the reference baseline for each reported improvement and preferably also report gains over the strongest baseline.
Questions for authors
Please refer to the Weaknesses.
4 comments
Response to Reviewer TzPn (2/2)
W2: Theoretical Motivation of Reallocation Signals The theoretical motivation does not fully justify the specific reallocation signals used by ReAlloc...The first-order analysis only shows that shifting weight toward units with higher true gradient alignment can help, but it does not establish that outcome variance, rollout-centroid distance, or local entropy prominence reliably approximate that quantity.
Thank you for your comment! Average aggregation is a standard practice in LLM post-training. Our analysis of how aggregation weights shape the parameter update shows that average allocation is locally suboptimal, and motivates the search for informative reallocation signals.
- For a fixed batch and local gradients , let , , where is a differentiable target objective. Starting from the baseline’s effective weights , transferring mass from a unit to a unit with produces weights satisfying $ F\bigl(\theta+\eta G(q')\bigr)-F\bigl(\theta+\eta G(q^0)\bigr) =\eta\delta(a_u-a_v)+o(\eta)>0 $, for sufficiently small positive step sizes .
Motivated by this opportunity to improve aggregation, we reallocates weight at three levels. The selection of three signals is supported by empirical investigations in Sec. 2.3–2.5, and their effectiveness is further validated through empirical experiments under both GRPO and OPD. Table R3 below reports average scores over seven benchmarks on Qwen3-4B-Base. Compared with average aggregation, ReAlloc improves Avg@8/Pass@8 by 4.92/8.89 points under GRPO and 2.85/7.23 points under OPD. Removing any level lowers both metrics under both algorithms, showing that all three levels contribute to the best observed performance.
Table R3: Comparison of standard average aggregation and hierarchical reallocation under GRPO and OPD on Qwen3-4B-Base. Results report average scores across seven benchmarks.
| Method | GRPO Avg@8 | GRPO Pass@8 | OPD Avg@8 | OPD Pass@8 |
|---|---|---|---|---|
| average aggregation | 29.11 | 43.26 | 31.25 | 46.62 |
| Full ReAlloc | 34.03 | 52.15 | 34.10 | 53.85 |
| ↳ w/ L-P only | 30.75 | 47.07 | 32.31 | 47.51 |
| ↳ w/ L-R only | 30.80 | 46.71 | 32.68 | 47.57 |
| ↳ w/ L-T only | 31.60 | 47.50 | 32.35 | 47.99 |
| ↳ w/o L-P | 32.95 | 47.87 | 33.00 | 50.60 |
| ↳ w/o L-R | 32.73 | 48.11 | 32.82 | 49.69 |
| ↳ w/o L-T | 32.86 | 49.90 | 32.98 | 51.11 |
W3: Terminological Precision "Optimization budget reallocation" is somewhat misleading, since the method mainly reallocates objective weights rather than computation...ReAlloc discards optimization contribution after paying the rollout cost, rather than reallocating the actual sampling budget.
Thank you for your constructive comment. Following your suggestion, we have revised the writing throughout the paper, using "objective weight reallocation" instead of "optimization budget reallocation" to more accurately characterize our method. We emphasize that the core contribution of ReAlloc lies in hierarchical signal-guided objective aggregation rather than computational budget savings. All related descriptions have been updated accordingly in the revised manuscript.
W4: Writing of Performance Improvements The numerical improvements reported in Sec. 4.2 appear to be computed relative to the vanilla GRPO/OPD baselines...This makes the reported gains potentially misleading...This paper should explicitly state the reference baseline for each reported improvement and preferably also report gains over the strongest baseline.
We sincerely appreciate your thoughtful review. Following your suggestion, we have clarified that the reported performance gains are measured relative to vanilla GRPO and vanilla OPD. We have also added the corresponding improvements over the strongest baselines. Such changes have been incorporated into Sec. 4.2 of the revised manuscript:
- Under GRPO, ReAlloc outperforms vanilla GRPO as well as DAPO and FORK. On Qwen3-4B-Base, ReAlloc improves Avg@8 by 8.44 points on AMC23 and 5.00 points on AIME26 over vanilla GRPO, corresponding to gains of 4.69 and 4.17 points over FORK, respectively. It also increases Pass@8 on AIME26 by 13.33 points over vanilla GRPO and 6.67 points over FORK. The average Avg@8 improves by 4.92 points over vanilla GRPO and 2.54 points over FORK.
- Under OPD, ReAlloc similarly improves over vanilla OPD, TIP and BRTS. On Qwen3-4B-Base, it achieves gains of 5.42 points over vanilla OPD and 3.75 points over BRTS in AIME25 Avg@8. It also increases Pass@8 from 23.33 to 46.67, corresponding to gains of 23.34 points over vanilla OPD and 10.00 points over BRTS. The average Avg@8 improves by 2.85 points over vanilla OPD and 1.82 points over BRTS.
We hope these revisions and our detailed clarifications help to address your concerns. We appreciate your valuable suggestions, which have enhanced the clarity of our manuscript. Thanks again for your constructive comments.
Dear Reviewer TzPn,
Thank you for your time and consideration. As the discussion period is drawing to a close, we would sincerely appreciate it if you could let us know whether our rebuttal and clarifications have addressed your concerns.
Many thanks,
Authors of #30028
Dear Reviewer TzPn,
Thank you for acknowledging that most of your concerns have been addressed. We apologize for the late response, as we conducted extensive experiments. We sincerely appreciate your time, understanding and consideration.
Best regards,
Authors of #30028
Response to Reviewer TzPn (1/2)
Dear Reviewer TzPn,
Thank you for your valuable comments and review! We sincerely appreciate your positive assessment of ReAlloc's effectiveness, minimal computational overhead and ease of implementation. We will respond point to point.
Thank you for your valuable comment. We respectfully clarify that ReAlloc is not limited to group-based training paradigms. We conducted experiments with single-rollout PPO and OPD, where only one response is sampled per prompt (K=1). L-P is disabled because outcome variance cannot be estimated from a single response. Accordingly, we evaluate four configurations: vanilla, L-T, L-R and L-R + L-T. Notably, without L-P, L-R is adapted to perform batch-global rollout weighting. We report Avg@8 and Pass@8 on Qwen3-4B-Base.
As shown in Table R1 and Table R2 below, the results support the applicability of ReAlloc beyond group-based optimization, demonstrating that the rollout and token level reallocation mechanisms remain effective even without group-based sampling. Specifically, L-R + L-T improves the average Avg@8 by 3.02 points under PPO and 3.69 points under OPD, with corresponding gains of 5.72 and 7.40 points in Pass@8, respectively. We have incorporated the experiments and discussion into Sec. 4.6 (Line 482) of the revised manuscript.
Table R1: Results of ReAlloc under single-rollout PPO and OPD on Qwen3-4B-Base, evaluated by Avg@8.
Table R2: Results of ReAlloc under single-rollout PPO and OPD on Qwen3-4B-Base, evaluated by Pass@8.