acceptodds
Under review as a conference paper at ICLR 2027

ReAlloc: Hierarchical Optimization Budget Reallocation for LLM post-training

Abstract

Post-training is central to improving the reasoning capabilities of large language models (LLMs). While existing post-training methods differ in optimization objectives, they all require aggregating these objectives across sampled training data to form the batch objective. Standard practices typically aggregate objectives through averaging, with recent research generally prioritizing sampled data at a single granularity level of training units and tailoring to specific algorithms. In this work, we revisit objective aggregation as a hierarchical budget allocation problem over three granularity levels, i.e., prompts, rollouts and tokens, and identify signals that can guide the allocation of optimization weight at each level. At the prompt level, we connect outcome variance to the gradient of expected accuracy and find that prompts with mixed outcomes exhibit greater learning potential. At the rollout level, we show that incorrect trajectories closer to the centroid of correct rollouts preserve more coherent reasoning and exhibit less redundant gradient structure. At the token level, we find that local entropy prominence identifies reasoning transitions more precisely than entropy magnitude alone. Based on these findings, we propose **ReAlloc**, a hierarchical optimization budget reallocation framework that leverages these level-specific signals into aggregation weights over three levels while leaving the underlying objective unchanged. This design allows ReAlloc to be applied to different post-training algorithms. Experiments with GRPO and OPD across three models and seven benchmarks demonstrate consistent improvements, with only minor extra computational overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Post-Training, Reinforcement Learning, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TzPnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training budget reallocation framework ReAlloc for LLM post-training. It revisits objective aggregation as a hierarchical budget allocation problem based on three identified signals, including prompts, rollouts, and tokens. The proposed ReAlloc show superior results compare to baselines with less computational overhead. Ablation study further demonstrates the effectiveness of the three signals.

Critical strengths
  1. This paper revisit the objective aggregation as a hierarchical budget allocation for LLM post-training and identifies three key signals.
  2. ReAlloc achieves better results with minimal computational overhead.
  3. This paper is easy to understand and ReAlloc can be implemented simply.
Critical weaknesses
  1. ReAlloc appears to be limited to group-based training paradigms, so the phrase “for LLM post-training” in the title is somewhat overclaim.
  2. The theoretical motivation does not fully justify the specific reallocation signals used by ReAlloc. The first-order analysis only shows that shifting weight toward units with higher true gradient alignment can help, but it does not establish that outcome variance, rollout-centroid distance, or local entropy prominence reliably approximate that quantity. This paper itself notes that outcome variance alone determines neither gradient magnitude nor direction.
  3. "Optimization budget reallocation" is somewhat misleading, since the method mainly reallocates objective weights rather than computation. At the prompt level, all rollouts must already be generated before outcome variance can be computed, after which low-variance prompts are simply assigned zero optimization weight. Thus, ReAlloc discards optimization contribution after paying the rollout cost, rather than reallocating the actual sampling budget.
  4. The numerical improvements reported in Sec. 4.2 appear to be computed relative to the vanilla GRPO/OPD baselines, although the preceding sentences jointly discuss improvements over stronger baselines such as FORK, TIP, and BRTS. This makes the reported gains potentially misleading. For example, on Qwen3-4B AMC23, the reported +8.44 Avg@8 gain is relative to vanilla GRPO (57.50→65.94), whereas the gain over the strongest competing baseline FORK is only +4.69. Similarly, under OPD on AIME25, the reported +5.42 gain is relative to vanilla OPD, while the gain over BRTS is +3.75. This paper should explicitly state the reference baseline for each reported improvement and preferably also report gains over the strongest baseline.
Questions for authors

Please refer to the Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer TzPn (1/2)

Dear Reviewer TzPn,

Thank you for your valuable comments and review! We sincerely appreciate your positive assessment of ReAlloc's effectiveness, minimal computational overhead and ease of implementation. We will respond point to point.

W1: ReAlloc appears to be limited to group-based training paradigms, so the phrase “for LLM post-training” in the title is somewhat overclaim.

Thank you for your valuable comment. We respectfully clarify that ReAlloc is not limited to group-based training paradigms. We conducted experiments with single-rollout PPO and OPD, where only one response is sampled per prompt (K=1). L-P is disabled because outcome variance cannot be estimated from a single response. Accordingly, we evaluate four configurations: vanilla, L-T, L-R and L-R + L-T. Notably, without L-P, L-R is adapted to perform batch-global rollout weighting. We report Avg@8 and Pass@8 on Qwen3-4B-Base.

As shown in Table R1 and Table R2 below, the results support the applicability of ReAlloc beyond group-based optimization, demonstrating that the rollout and token level reallocation mechanisms remain effective even without group-based sampling. Specifically, L-R + L-T improves the average Avg@8 by 3.02 points under PPO and 3.69 points under OPD, with corresponding gains of 5.72 and 7.40 points in Pass@8, respectively. We have incorporated the experiments and discussion into Sec. 4.6 (Line 482) of the revised manuscript.

Table R1: Results of ReAlloc under single-rollout PPO and OPD on Qwen3-4B-Base, evaluated by Avg@8.

L-RL-TAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
PPO (K=1)
--12.509.588.759.7950.312.9271.9823.69
-✓13.339.589.1710.0055.633.3374.9525.14
✓-12.9210.008.7510.0057.194.5877.6525.87
✓✓13.7510.429.1710.2159.065.0079.3526.71
OPD (K=1)
--17.9212.0811.2511.0453.135.0077.5026.85
-✓20.0012.5014.1712.2955.638.3380.0028.99
✓-20.8312.5013.7512.7157.197.9280.6029.36
✓✓21.2513.3314.5814.3859.389.5881.2530.54

Table R2: Results of ReAlloc under single-rollout PPO and OPD on Qwen3-4B-Base, evaluated by Pass@8.

L-RL-TAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
PPO (K=1)
--23.3320.0010.0015.0070.0010.0087.6033.70
-✓26.6726.6713.3316.6772.5013.3387.8036.71
✓-26.6723.3313.3318.3375.0013.3388.2036.88
✓✓30.0026.6716.6720.0077.5016.6788.4039.42
OPD (K=1)
--26.6716.6713.3320.0082.5013.3386.0036.93
-✓30.0016.6716.6723.3387.5013.3386.2039.10
✓-30.0020.0020.0025.0085.0016.6786.8040.50
✓✓36.6723.3326.6726.6790.0020.0087.0044.33
AuthorsOpenReview ↗

Response to Reviewer TzPn (2/2)

W2: Theoretical Motivation of Reallocation Signals The theoretical motivation does not fully justify the specific reallocation signals used by ReAlloc...The first-order analysis only shows that shifting weight toward units with higher true gradient alignment can help, but it does not establish that outcome variance, rollout-centroid distance, or local entropy prominence reliably approximate that quantity.

Thank you for your comment! Average aggregation is a standard practice in LLM post-training. Our analysis of how aggregation weights shape the parameter update shows that average allocation is locally suboptimal, and motivates the search for informative reallocation signals.

  • For a fixed batch and local gradients gug_u, let G(q)=∑uquguG(q)=\sum_u q_u g_u, au=⟨∇F(θ),gu⟩a_u=\langle\nabla F(\theta),g_u\rangle, where FF is a differentiable target objective. Starting from the baseline’s effective weights q0q^0, transferring mass 0<δ<qv00<\delta<q_v^0 from a unit vv to a unit uu with au>ava_u>a_v produces weights q′q' satisfying $ F\bigl(\theta+\eta G(q')\bigr)-F\bigl(\theta+\eta G(q^0)\bigr) =\eta\delta(a_u-a_v)+o(\eta)>0 $, for sufficiently small positive step sizes η\eta.

Motivated by this opportunity to improve aggregation, we reallocates weight at three levels. The selection of three signals is supported by empirical investigations in Sec. 2.3–2.5, and their effectiveness is further validated through empirical experiments under both GRPO and OPD. Table R3 below reports average scores over seven benchmarks on Qwen3-4B-Base. Compared with average aggregation, ReAlloc improves Avg@8/Pass@8 by 4.92/8.89 points under GRPO and 2.85/7.23 points under OPD. Removing any level lowers both metrics under both algorithms, showing that all three levels contribute to the best observed performance.

Table R3: Comparison of standard average aggregation and hierarchical reallocation under GRPO and OPD on Qwen3-4B-Base. Results report average scores across seven benchmarks.

MethodGRPO Avg@8GRPO Pass@8OPD Avg@8OPD Pass@8
average aggregation29.1143.2631.2546.62
Full ReAlloc34.0352.1534.1053.85
↳ w/ L-P only30.7547.0732.3147.51
↳ w/ L-R only30.8046.7132.6847.57
↳ w/ L-T only31.6047.5032.3547.99
↳ w/o L-P32.9547.8733.0050.60
↳ w/o L-R32.7348.1132.8249.69
↳ w/o L-T32.8649.9032.9851.11

W3: Terminological Precision "Optimization budget reallocation" is somewhat misleading, since the method mainly reallocates objective weights rather than computation...ReAlloc discards optimization contribution after paying the rollout cost, rather than reallocating the actual sampling budget.

Thank you for your constructive comment. Following your suggestion, we have revised the writing throughout the paper, using "objective weight reallocation" instead of "optimization budget reallocation" to more accurately characterize our method. We emphasize that the core contribution of ReAlloc lies in hierarchical signal-guided objective aggregation rather than computational budget savings. All related descriptions have been updated accordingly in the revised manuscript.

W4: Writing of Performance Improvements The numerical improvements reported in Sec. 4.2 appear to be computed relative to the vanilla GRPO/OPD baselines...This makes the reported gains potentially misleading...This paper should explicitly state the reference baseline for each reported improvement and preferably also report gains over the strongest baseline.

We sincerely appreciate your thoughtful review. Following your suggestion, we have clarified that the reported performance gains are measured relative to vanilla GRPO and vanilla OPD. We have also added the corresponding improvements over the strongest baselines. Such changes have been incorporated into Sec. 4.2 of the revised manuscript:

  • Under GRPO, ReAlloc outperforms vanilla GRPO as well as DAPO and FORK. On Qwen3-4B-Base, ReAlloc improves Avg@8 by 8.44 points on AMC23 and 5.00 points on AIME26 over vanilla GRPO, corresponding to gains of 4.69 and 4.17 points over FORK, respectively. It also increases Pass@8 on AIME26 by 13.33 points over vanilla GRPO and 6.67 points over FORK. The average Avg@8 improves by 4.92 points over vanilla GRPO and 2.54 points over FORK.
  • Under OPD, ReAlloc similarly improves over vanilla OPD, TIP and BRTS. On Qwen3-4B-Base, it achieves gains of 5.42 points over vanilla OPD and 3.75 points over BRTS in AIME25 Avg@8. It also increases Pass@8 from 23.33 to 46.67, corresponding to gains of 23.34 points over vanilla OPD and 10.00 points over BRTS. The average Avg@8 improves by 2.85 points over vanilla OPD and 1.82 points over BRTS.

We hope these revisions and our detailed clarifications help to address your concerns. We appreciate your valuable suggestions, which have enhanced the clarity of our manuscript. Thanks again for your constructive comments.

Dear Reviewer TzPn,

Thank you for your time and consideration. As the discussion period is drawing to a close, we would sincerely appreciate it if you could let us know whether our rebuttal and clarifications have addressed your concerns.

Many thanks,
Authors of #30028

Dear Reviewer TzPn,

Thank you for acknowledging that most of your concerns have been addressed. We apologize for the late response, as we conducted extensive experiments. We sincerely appreciate your time, understanding and consideration.

Best regards,
Authors of #30028

Reviewer_QBghOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper views the aggregation of per-token objectives into a batch objective as a hierarchical weight-allocation problem over prompts, rollouts and tokens. Specifically, the paper proposes 3 levels of signals: prompt level, rollout level and token level. The paper trains Qwen3-8B-Base, Qwen3-4B-Base and DeepSeek-R1-Distill-1.5B on DAPO-Math-17K and evaluate on seven math benchmarks. The proposed method improves over GRPO / DAPO / FORK and over OPD / TIP / BRTS on every model.

Critical strengths
  1. The proposed method is pluggable and applicable across current post-training algorithms.

  2. There are solid empirical experiment results across benchmarks, models and algorithms.

  3. The method is analysis-driven. Section 2 with Appendix A measures each signal on the actual training setup (Qwen3-4B-Base, DAPO-Math-17K, both GRPO and OPD).

Critical weaknesses
  1. Prompt-level variance filtering is DAPO dynamic sampling / difficulty curricula (CoDaPO, PACED) with a different cut; S_t is a local-peak variant of FORK. For the rollout level, Sec. 2.4 / App. A.5 show that high-d_IC failures are repetition, garbage or truncation, yet no comparison is made to simply down-weighting rollouts by repetition ratio or length cap. If such a filter recovers most of the L-R gain, the centroid/temperature machinery is unnecessary. L-P also keeps 50% of prompts and renormalizes α_i, so part of its gain may be an implicit learning-rate increase rather than selection; a batch-refilled (DAPO-style) control is needed.

  2. The Sec. 2 analysis motivates the signals but does not diagnose the baselines. It never involves DAPO / FORK / TIP, so it does not show that they lose accuracy because of mis-weighting. Fig. 2(b,c) does not state whether zero-variance prompts are excluded from the low tertile; under GRPO these have zero advantage, so the low group's small gain may be the trivial effect DAPO already removes. The rollout analysis amounts to "degenerate text has redundant gradients", which is expected and detectable without hidden states. The token analysis shows S_t finds reasoning transitions better than H_t but not that this is why training improves. The analysis also fixes only the direction of each signal, not the functional form (hard cut, rank-softmax, window), which comes from the test-set sweep above.

Questions for authors
  1. Could you compare L-R against simple degeneration filters: repetition ratio, truncation at max length, and length-normalized weights? What fraction of the top-20% d_IC rollouts are truncated or repetitive?

  2. How often is the fallback "no correct rollout in the batch → original weights" triggered, and does performance degrade early in training when most groups have no correct rollout?

  3. Prop. 1 / Eq. 6 only say that non-uniform weights can help; it would strengthen the paper to show a case where ReAlloc's weights are provably or empirically aligned with ⟨∇F, g_u⟩, e.g., by correlating a_u with each signal on a held-out batch.

6 comments
AuthorsOpenReview ↗

Response to Reviewer QBgh (1/5)

Dear Reviewer QBgh,

Thank you for your valuable comments and positive recognition of our work! We sincerely appreciate your positive assessment of ReAlloc's plug-and-play compatibility, its consistent empirical improvements and its analysis-driven design. We address each point in detail below.

W1.1: Prompt-level variance filtering is DAPO dynamic sampling / difficulty curricula (CoDaPO, PACED) with a different cut; S_t is a local-peak variant of FORK.

Thank you for your comment. We respectfully clarify that ReAlloc’s contribution lies in hierarchically reallocating objective weights while preserving the underlying local objectives. Below we restate the necessity and contribution of each level, and we have revised Sec. 4.7 (Line 509) and Appendix A.1 (Line 735) of the PDF accordingly.

  • Problem. Existing prioritization methods typically focus on a single granularity, overlooking the inherent hierarchical structure and complementary learning signals in post-training, thereby limiting the effective allocation of optimization weight.

  • Challenges. To address this problem, we identify four challenges, each addressed by a corresponding design in our method. Table R1 shows that ReAlloc improves over average aggregation and removing any level reduces performance under both GRPO and OPD.

  • Comparison. As shown in Table R2, we compare ReAlloc with prior methods in terms of their coverage, prioritized levels and improvements over vanilla baselines. We also conduct additional experiments comparing ReAlloc with CoDaPO and PACED. Table R3 reports the detailed benchmark results. ReAlloc outperforms all four methods across all benchmarks.

These results demonstrate the benefits of jointly reallocating optimization weights across prompts, rollouts and tokens, beyond prioritization at a single granularity.

Table R1: Challenge, our core design and experimental evidence. Scores are Avg@8 averaged over seven benchmarks on Qwen3-4B-Base.

ChallengeCore DesignExperimental Evidence
Single granularity prioritization leaves complementary signals unusedHierarchical objective aggregation: jointly reallocates weights across prompts, rollouts and tokens while preserving the underlying local objectiveVanilla → ReAlloc: GRPO: 29.11 → 34.03; OPD: 31.25 → 34.10
Average prompt weights overlook learning potentialOutcome variance-guided prompt reallocation: prioritizes prompts with mixed successful and unsuccessful outcomesReAlloc without prompt reallocation: GRPO: 34.03 → 32.95; OPD: 34.10 → 33.00
Average rollout weights overlook reasoning structureCentroid distance-guided rollout reallocation: assigns greater weight to incorrect rollouts closer to the correct centroidReAlloc without rollout reallocation: GRPO: 34.03 → 32.73; OPD: 34.10 → 32.82
Average token weights overlook reasoning transitionsLocal entropy prominence-guided token reallocation: emphasizes tokens whose entropy stands out from their local contextReAlloc without token reallocation: GRPO: 34.03 → 32.86; OPD: 34.10 → 32.98

Table R2: Comparison with prior methods. Scores are Avg@8 averaged over seven benchmarks on Qwen3-4B-Base.

MethodAlgorithm CoveragePrompt LevelRollout LevelToken LevelExperimental Comparison
CoDaPOGRPO✓--Vanilla GRPO: +1.95
FORKGRPO--✓Vanilla GRPO: +2.38
PACEDOPD✓--Vanilla OPD: +0.80
TIPOPD--✓Vanilla OPD: +0.67
ReAllocGRPO, OPD✓✓✓Vanilla GRPO: +4.92; Vanilla OPD: +2.85

Table R3: Detailed comparison between ReAlloc and prior methods on Qwen3-4B-Base. Scores are Avg@8.

AlgorithmMethodAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
GRPOVanilla19.1714.5813.7512.7157.507.0879.0029.11
CoDaPO20.8314.5818.3315.0059.698.7580.2531.06
FORK22.0817.5014.5815.8361.257.0882.1331.49
ReAlloc23.7519.5818.7517.5065.949.1783.5034.03
OPDVanilla22.0815.8315.4213.9660.949.1781.3831.25
TIP22.9217.0815.0014.5861.5610.4281.9031.92
PACED22.0818.3316.2514.5861.889.1782.0832.05
ReAlloc25.4221.2518.3316.4663.7510.8382.6334.10
AuthorsOpenReview ↗

Response to Reviewer QBgh (2/5)

W1.2: Batch-Refilled Control of Prompt Level L-P also keeps 50% of prompts and renormalizes α_i, so part of its gain may be an implicit learning-rate increase rather than selection; a batch-refilled (DAPO-style) control is needed.

Thank you for this constructive comment. Following your suggestion, we introduce two controls:

  • Random 50%, which randomly retains half of the prompts with the same weight renormalization as L-P.
  • L-P + Refill, which applies L-P selection with additional sampling to control for the potential implicit learning-rate increase caused by renormalization.

As shown in Table R4, Random 50% performs comparably to Vanilla, while L-P achieves higher scores. Importantly, L-P + Refill outperforms L-P under both GRPO and OPD, suggesting that the gains of L-P are not driven by the potential implicit learning-rate increase from weight renormalization. We have revised these points in Sec. 4.7 (Line 509).

Table R4: Prompt level controls on Qwen3-4B-Base, evaluated using Avg@8.

AlgorithmMethodAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
GRPOVanilla19.1714.5813.7512.7157.507.0879.0029.11
Random 50%18.7514.5812.9212.9257.816.6778.8328.92
L-P20.8314.1717.9214.5859.388.7579.6330.75
L-P + Refill21.6715.0018.3314.7958.759.1779.7331.06
OPDVanilla22.0815.8315.4213.9660.949.1781.3831.25
Random 50%21.2516.2514.5813.3361.259.1781.4831.04
L-P22.5018.3316.2515.0062.509.1782.4532.31
L-P + Refill22.5018.7516.6714.7963.1310.0082.5832.63
AuthorsOpenReview ↗

Response to Reviewer QBgh (3/5)

W1.3+Q1: Necessity of Centroid-Based Rollout Weighting For the rollout level, Sec. 2.4 / App. A.5 show that high-d_IC failures are repetition, garbage or truncation, yet no comparison is made to simply down-weighting rollouts by repetition ratio or length cap. If such a filter recovers most of the L-R gain, the centroid/temperature machinery is unnecessary. Could you compare L-R against simple degeneration filters: repetition ratio, truncation at max length, and length-normalized weights? What fraction of the top-20% d_IC rollouts are truncated or repetitive?

Thank you for your thoughtful comment. We respectfully clarify that the centroid-based weighting mechanism is necessary for the effectiveness of L-R. Following your suggestion, we compare L-R with three alternatives under both GRPO and OPD. We have revised these points in Sec. 4.7 (Line 516).

Setting: For a fair comparison, all methods modify only the weights of incorrect rollouts, while keeping other training settings unchanged. For the first two baselines, we retain the same rank-based softmax weighting as L-R, changing only the ranking criterion.

  • Repetition-Based: We measure the repeated 4-gram ratio of each incorrect rollout and rank rollouts in ascending order.
  • Truncation-Based: We prioritize normally terminated incorrect rollouts over those truncated at the maximum generation length, with random ordering within each group.
  • Length-Based: We directly assign weights proportional to 1/T1/T, normalized across incorrect rollouts.

We further examine incorrect rollouts at three training stages (15%, 50%, 85%), grouping them by dICd^{IC} into the highest (Top-20%), middle (Mid-20%) and lowest (Bot-20%) distance groups, with repetition defined as a repeated 4-gram ratio > 0.15.

Analysis: As shown in Table R5, repetition-based weighting performs best, but recovers only 33.1% and 36.4% of the L-R gains under GRPO and OPD. Truncation-based weighting provides negligible or negative gains, while length-based weighting underperforms the vanilla baselines. As shown in Table R6, among the Top-20% incorrect rollouts, only 0.6%, 0.5% and 0.0% are truncated, while 65.5%, 30.7% and 2.5% are repetitive at the three stages. These findings indicate that dICd^{IC} reflects trajectory characteristics beyond surface-level repetition, truncation and length, supporting the value of centroid-based rollout weighting.

Table R5: Comparison of L-R with simple degeneration-based rollout weighting methods on Qwen3-4B-Base, evaluated using Avg@8.

AlgorithmMethodAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
GRPOVanilla19.1714.5813.7512.7157.507.0879.0029.11
Repetition-Based19.5815.0014.5812.9257.818.3379.4829.67
Truncation-Based18.7515.0013.3312.5057.817.9279.3529.24
Length-Based17.5014.1712.0812.5057.506.2578.9328.42
L-R (Ours)18.7513.3315.4215.8362.507.5082.3030.80
OPDVanilla22.0815.8315.4213.9660.949.1781.3831.25
Repetition-Based22.5016.2516.2513.9660.9410.0082.5031.77
Truncation-Based22.9215.8315.0013.7560.318.7581.4031.14
Length-Based21.2515.4215.0013.7560.318.3380.4830.65
L-R (Ours)24.5817.9217.0813.9662.5010.4282.3332.68

Table R6: Fractions of truncated and repetitive incorrect rollouts across dICd^{IC} percentile groups at different training stages (15%, 50% and 85% of total steps). Top–20% corresponds to the largest distances.

Training StageTop–20% TruncatedTop–20% RepetitiveMid-20% TruncatedMid-20% RepetitiveBot-20% TruncatedBot-20% Repetitive
Step 15%0.6%65.5%0.0%31.7%0.0%3.6%
Step 50%0.5%30.7%0.0%6.6%0.0%0.6%
Step 85%0.0%2.5%0.0%0.0%0.0%0.0%
Reviewer_VtLdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReAlloc is a weighting scheme for LLM post-training that decides where the optimization effort should go, and it does so at three levels. At the prompt level it keeps prompts whose rollouts come out mixed, some correct and some not; at the rollout level it up-weights the incorrect rollouts that sit close to a batch-wide centroid of the correct ones; at the token level it up-weights tokens whose entropy stands out against their local neighborhood. None of this touches the local loss itself, only how the loss is aggregated. The prompt-level choice is motivated through a score-function identity, the other two through observations about rollout structure, gradient diversity, and tokens that mark reasoning transitions. Empirically, three models are trained on DAPO-Math-17K and evaluated on seven math benchmarks with Avg@8 and Pass@8. Averages improve under both algorithms, and the ablations, which remove one level at a time, suggest the three levels contribute somewhat independently. Step-time measurements at fixed batch size indicate the extra cost is small. So the contribution is a fairly practical and reusable recipe for selective training. The evidence, however, stays within math reasoning, and the paper neither claims nor shows that its particular budget split is the right one.

Critical strengths
  1. The three-level design is genuinely reusable. Since prompt, rollout and token weights only change loss aggregation, the same machinery drops into GRPO and OPD without rewriting either objective.

  2. The gains are consistent and look complementary: averages improve on three models across seven benchmarks, and the leave-one-out ablations in Tables 1-4 favor the full hierarchy rather than any single level.

  3. The overhead is measured rather than assumed, 1.35% extra step time for GRPO and 0.28% for OPD at fixed batch size (Table 5). That matters for whether anyone would actually turn this on.

Critical weaknesses
  1. I could not tell how much of this is new. There is no comparison with CoDaPO or PACED, and no token-weighting control that matches the selected-token fraction or the effective concentration of weight. The centroid-similarity and gradient-rank analyses are suggestive, but they do not show that what gets aligned is reasoning.

  2. Several details are underspecified for reproduction: whether ranks are raw or normalized, how the representation is built, how entropy neighborhoods and boundaries are defined, and what happens when a score is zero. The OPD objective in the appendix also does not obviously match the top-k setting reported in Table 7.

  3. Robustness evidence is thin. Tables 1-4 report no training-seed variability and no uncertainty, so the smaller margins are hard to read as real.

Questions for authors

These are the points that would move my score.

  1. Under the same rollout and update budget, and the same validation protocol, how does ReAlloc compare with CoDaPO or PACED? For the token level specifically: once the selected-token fraction, or the effective concentration of weight, is matched, does local prominence still beat plain entropy magnitude? One controlled comparison here would do more for the paper than another benchmark.

  2. A few implementation questions. Are the batch-global rollout ranks raw integers or normalized? Which layer is the representation taken from, how many segments, and what token-boundary convention? What is the fallback when a score is zero, and are the proxy weights detached during policy updates? Please also reconcile the sampled-token OPD objective in Appendix A.2 with the top-k=16 implementation in Table 7.

  3. Do you have independent-seed runs and paired uncertainty estimates for the main tables, particularly for the smaller DeepSeek-R1-1.5B margins? Were hyperparameters chosen on a held-out validation split rather than on the reported benchmarks? A note on the baseline tuning budget and on how checkpoints were selected would help as well.

  4. Does the rollout-level benefit survive if you control for response length and repetition, or if the batch-global centroid is replaced by a prompt-conditioned reference? Either can be run on the existing setup, and it would separate reasoning structure from surface-level similarity.

6 comments
AuthorsOpenReview ↗

Response to Reviewer VtLd (1/5)

Dear Reviewer VtLd,

Thank you for your thoughtful review and valuable questions! We greatly appreciate your recognition of our genuinely reusable design. Moreover, we are encouraged that you find ReAlloc’s gains consistent and complementary across levels, with sufficiently low overhead that anyone would actually turn it on. We address your questions point-to-point in the following.

W1.1+Q1: Baseline Comparison and Token Level Analysis There is no comparison with CoDaPO or PACED, and no token-weighting control...how does ReAlloc compare with CoDaPO or PACED? For the token level specifically: once the selected-token fraction, or the effective concentration of weight, is matched, does local prominence still beat plain entropy magnitude?

We sincerely appreciate your valuable and constructive comments. Following your advice, we conducted two experiments: (1) comparisons with CoDaPO and PACED under matched training budgets and (2) token level controls for selected-token fraction and effective weight concentration. Experiments are conducted on Qwen3-4B-Base.

1. Comparison with CoDaPO and PACED.

Setting: We follow the original experimental settings of CoDaPO and PACED, while aligning relevant training configurations for fair comparison. We ensure that all methods use the same rollout and update budgets and follow our original evaluation protocol.

Analysis: As shown in Table R1, ReAlloc consistently outperforms both CoDaPO and PACED across all benchmarks. While CoDaPO and PACED primarily focus on prompt level prioritization, ReAlloc further reallocates weights at the rollout level and token level. The results demonstrate that hierarchical reallocation provides additional benefits beyond prompt level optimization alone. These results have been incorporated into Sec. 4.7 (Line 513).

Table R1: Comparison with CoDaPO and PACED, evaluated using Avg@8.

AlgorithmMethodAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
GRPOVanilla19.1714.5813.7512.7157.507.0879.0029.11
CoDaPO20.8314.5818.3315.0059.698.7580.2531.06
ReAlloc23.7519.5818.7517.5065.949.1783.5034.03
OPDVanilla22.0815.8315.4213.9660.949.1781.3831.25
PACED22.0818.3316.2514.5861.889.1782.0832.05
ReAlloc25.4221.2518.3316.4663.7510.8382.6334.10

2. Token level controls: selected-token fraction and effective weight concentration.

Setting: We conduct two controlled comparisons between entropy magnitude (HtH_t) and local entropy prominence (StS_t), with only token level reallocation enabled. First, following FORK, we perform batch-wise token selection, selecting the top 20% of response tokens ranked by either HtH_t or StS_t and assigning average weights. Second, to match effective weight concentration, we independently rank tokens by HtH_t and StS_t within each rollout and assign the same set of StS_t-derived weights according to their respective rankings. This ensures identical weight distributions. All experiments use the same configurations on Qwen3-4B-Base.

Analysis: StS_t consistently outperforms HtH_t under both GRPO and OPD, whether matching the selected-token fraction or the effective weight concentration. These results demonstrate that local entropy prominence provides a more effective token level optimization signal than entropy magnitude alone, and that its benefits cannot be attributed merely to token selection budgets or weight concentration.

Table R2: Comparison of HtH_t and StS_t under matched token selection budgets, evaluated using Avg@8.

AlgorithmToken selectionAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
GRPOAll19.1714.5813.7512.7157.507.0879.0029.11
Top-20% HtH_t22.0817.5014.5815.8361.257.0882.1331.49
Top-20% StS_t22.5017.9215.0016.4662.197.5082.5032.01
OPDAll22.0815.8315.4213.9660.949.1781.3831.25
Top-20% HtH_t22.5016.2514.5814.1761.568.7580.7531.22
Top-20% StS_t22.9217.0815.8314.3861.888.7582.0031.83

Table R3: Comparison of HtH_t and StS_t under matched effective weight concentration, evaluated using Avg@8.

AlgorithmToken weightingAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
GRPOAverage19.1714.5813.7512.7157.507.0879.0029.11
Rank-matched HtH_t21.6717.5014.1716.0461.256.6779.6530.99
Original StS_t22.5018.3315.0016.2561.887.0880.1831.60
OPDAverage22.0815.8315.4213.9660.949.1781.3831.25
Rank-matched HtH_t22.0819.5815.8314.1759.389.5881.4031.72
Original StS_t22.9220.0016.6714.3860.6310.4281.4332.35
AuthorsOpenReview ↗

Response to Reviewer VtLd (2/5)

W1.2+Q4: Rollout Level Analysis The centroid-similarity and gradient-rank analyses are suggestive...Does the rollout-level benefit survive if you control for response length and repetition, or if the batch-global centroid is replaced by a prompt-conditioned reference?

Thank you for this valuable suggestion. Following your suggestion, we conducted two experiments: (1) controlling for response length and repetition and (2) replacing the batch-global centroid with a prompt-conditioned reference. Experiments are conducted on Qwen3-4B-Base under both GRPO and OPD.

1. Rollout level controls: response length and repetition.

Setting: For length control, we set length bounds to the 5th–95th percentiles of correct rollouts’ response lengths in the current batch. The length constraint is disabled when fewer than two correct rollouts are available. For repetition control, we measure repetition using the repeated 4-gram ratio threshold of 0.15. Incorrect rollouts violating either constraint are rejected and resampled up to five times. We also apply the same constraints to the baselines and retrain them.

Analysis: As shown in Table R4, under identical length and repetition constraints, L-R only consistently outperforms the controlled vanilla baselines. Full ReAlloc achieves larger gains over the same controlled baselines. The results demonstrate that our rollout level signal captures reasoning structure beyond surface-level characteristics.

Table R4: Results on Qwen3-4B-Base with identical response length and repetition controls applied to all methods, evaluated using Avg@8.

AlgorithmMethodAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
GRPOVanilla20.0015.4215.0013.7557.817.9279.8029.96
L-R only20.0014.1715.8316.0463.757.9282.0031.39
ReAlloc24.1720.0018.7518.3366.569.5884.2534.52
OPDVanilla22.5016.2516.6714.7960.009.5882.0031.68
L-R only24.5818.3317.9214.7963.4410.4282.3833.12
ReAlloc25.4220.8319.1717.0864.6910.4283.0034.37

2. Prompt-conditioned reference.

Setting: We construct a separate centroid for each prompt using its correct rollouts. We compute the distance of each incorrect rollout to its corresponding prompt-specific centroid, while retaining the original batch-global ranking and weight assignment procedure. For prompts with no correct rollouts, we fall back to average weighting.

Analysis: As shown in Table R5, replacing the batch-global centroid with prompt-conditioned centroids consistently leads to lower performance under both GRPO and OPD. This degradation arises because a large number of prompts have only incorrect sampled rollouts. For these prompts, no prompt-conditioned centroid is available, forcing it to fall back to average weighting and thereby limiting the effectiveness of rollout level reallocation.

We further measure the proportion of incorrect rollouts that can receive distance-based weights under the two centroid configurations in the L-R only setting. Table R6 reports these proportions at different training stages. At 15% of training, 58.18%/75.13% fall back to average weighting under prompt-conditioned centroid for GRPO/OPD, and only 41.82%/24.87% of incorrect rollouts can be assigned weights. In contrast, the batch-global centroid enables weighting for all incorrect rollouts.

Table R5: Results of prompt-conditioned centroids rollout level reallocation under GRPO and OPD on Qwen3-4B-Base, evaluated by Avg@8.

AlgorithmMethodCentroidAIME24AIME25AIME26AIMOAMC23HMMT25MATH500Avg.
GRPOVanilla-19.1714.5813.7512.7157.507.0879.0029.11
ReAllocPrompt22.0817.9217.0816.6764.069.1782.5032.78
ReAllocBatch23.7519.5818.7517.5065.949.1783.5034.03
L-R onlyPrompt18.7514.1715.0013.1360.316.6780.8329.84
L-R onlyBatch18.7513.3315.4215.8362.507.5082.3030.80
OPDVanilla-22.0815.8315.4213.9660.949.1781.3831.25
ReAllocPrompt24.5818.3317.5015.8362.199.5882.4532.92
ReAllocBatch25.4221.2518.3316.4663.7510.8382.6334.10
L-R onlyPrompt23.7516.2516.6714.1761.889.1781.5031.91
L-R onlyBatch24.5817.9217.0813.9662.5010.4282.3332.68

Table R6: Proportion of incorrect rollouts eligible for rollout level weighting at different training stages.

AlgorithmCentroidStep 15%Step 30%Step 45%Step 60%Step 75%Step 90%
GRPOPrompt41.82%42.70%45.45%46.82%47.37%52.76%
Batch100%100%100%100%100%100%
OPDPrompt24.87%33.97%34.75%41.09%49.74%59.69%
Batch100%100%100%100%100%100%
AuthorsOpenReview ↗

Response to Reviewer VtLd (3/5)

W2+Q2: Implementation Details Several details are underspecified for reproduction...Are the batch-global rollout ranks raw integers or normalized? Which layer is the representation taken from, how many segments, and what token-boundary convention? What is the fallback when a score is zero, and are the proxy weights detached during policy updates? Please also reconcile the sampled-token OPD objective in Appendix A.2 with the top-k=16 implementation in Table 7.

Thank you for the thoughtful questions. Below are our detailed settings:

1. Batch-global rollout rank. We use normalized ranks to prevent the optimization weights from becoming excessively concentrated on a small number of rollouts in certain cases. Normalized ranks allow for meaningful differentiation among rollouts while preventing overly skewed weight distributions. We have clarified this in Sec. 3 (Line 309).

2. Rollout representation construction. We construct representations using the policy model's final layer hidden states. Specifically, we use the hidden states at the response token positions themselves, considering only valid response tokens while excluding prompt and padding tokens. We use 1616 segments. For a response containing TT valid tokens, token t∈{1,…,T}t \in \{1,\ldots,T\} is assigned to segment ⌊16(t−1)/T⌋\lfloor 16(t-1)/T \rfloor. For responses shorter than 16 tokens, empty segments are filled with zero vectors. We have added a clarification in Sec. 3 (Line 306).

3. Entropy neighborhoods and fallback. Near response boundaries, we average over the available neighbors using their actual count, rather than extending into padding positions. If either side has no available neighbor, we set St=0S_t=0. In practice, we apply epsilon smoothing during token level weight normalization: γi,j,t(w)=Si,j,t(w)+ϵ∑t=1Ti,j(Si,j,t(w)+ϵ),ϵ=10−6\gamma_{i,j,t}^{(w)}=\frac{S_{i,j,t}^{(w)}+\epsilon}{\sum_{t=1}^{T_{i,j}}(S_{i,j,t}^{(w)}+\epsilon)}, \epsilon=10^{-6}. Thus, individual zero-score tokens retain a small weight, while responses with all-zero scores fall back to average weighting. We have revised the token level equation in Sec. 3 (Line 314) and added a corresponding explanation.

4. Proxy weight detachment. All proxy weights are detached from the computation graph during policy updates and treated as constants during backpropagation. We have clarified this in Sec. 3 (Line 316).

5. OPD objective and top-k setting. Our experiments use the top-k=1616 setting reported in Table 7. Specifically, we compute distillation rewards at each generation position for the student’s top 16 tokens. The single-sample formulation in the original Appendix A.2 described the general sampled-token formulation of OPD, rather than the specific implementation adopted in our experiments. To avoid confusion, we have revised the single-sample description and added clarification of the top-k setting in the updated Appendix A.2 (Line 761).

Comments on the paper

1 comment

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely appreciate your commitment to evaluating our paper despite your busy schedules. We are glad to see that the paper is considered to have the following strengths:

  • [Reviewer TzPn] Novel hierarchical formulation of objective aggregation.
  • [Reviewer QBgh] Analysis-driven design and empirical motivation.
  • [Reviewers VtLd, QBgh] Genuinely reusable design and plug-and-play compatibility.
  • [Reviewers VtLd, QBgh, TzPn] Consistent empirical improvements across models and benchmarks.
  • [Reviewers VtLd, TzPn] Complementary benefits of hierarchical reallocation.
  • [Reviewers VtLd, TzPn] Minimal computational overhead and ease of implementation.

The reviewers also raised several concerns regarding the justification of level-specific signals (Reviewers QBgh, TzPn), comparisons with rollout-level controls and alternatives (Reviewers VtLd, QBgh), comparisons with prior work (Reviewers VtLd, QBgh), generalizability beyond group-based training (Reviewer TzPn), statistical robustness (Reviewer VtLd), technical details (Reviewer VtLd, TzPn).

We have carefully addressed these concerns through additional experiments, detailed analyses and clarifications in the revised manuscript:

  • Justification of Level-Specific Signals: We conducted prompt-level controls with batch refilling, token-level comparisons under matched selection fractions and weight distributions, and additional gradient-alignment analyses.

  • Rollout-Level Controls and Alternatives: We compared centroid-based weighting against simple rule-based alternatives, controlled for response length and repetition, and evaluated batch-global versus prompt-conditioned centroids.

  • Comparisons with Prior Work: We added comparisons with CoDaPO and PACED, further demonstrating the advantages of hierarchical reallocation over existing methods.

  • Generality Beyond Group-Based Training: We extended ReAlloc to single-rollout PPO and OPD, demonstrating its applicability beyond group-based optimization.

  • Statistical Robustness: We conducted independent experiments with three training seeds across three models and reported statistical significance analyses.

  • Technical and Presentation Clarifications: We clarified implementation details, hyperparameter and checkpoint selection. We also refined the terminology to objective weight reallocation.

We are particularly grateful to Reviewer TzPn for acknowledging that our responses and additional experiments have addressed their concerns. We believe the added analyses, new experiments and expanded explanations have substantially strengthened the manuscript.

Finally, we thank all reviewers again for their thoughtful reviews and for spending time writing the reviews.

Best regards,
Authors of Paper #30028

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.