acceptodds
Under review as a conference paper at ICLR 2027

SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning

Abstract

Maximum Likelihood Reinforcement Learning (MaxRL) targets prompt-wise log-success and has shown strong performance on reasoning tasks. Under finite rollout budgets, however, the estimator used by MaxRL attenuates each prompt's likelihood gradient by a factor that depends on its success probability and rollout count. Under uniform rollout allocation, the common rollout count fails to compensate for success-dependent attenuation, leaving low-success prompts more strongly attenuated and distorting their relative contributions to the expected aggregate gradient. We introduce SERA (Scale-Equalized Rollout Allocation), which redistributes a fixed rollout budget to approximately equalize these finite-rollout scaling factors. Building on our theoretical analysis of how finite rollouts distort prompt-wise likelihood gradients, we formulate the allocation as a fixed-budget max-min problem, derive a waterline solution to its continuous relaxation, and introduce a multiplicity correction to remove the additional prompt weighting induced by heterogeneous rollout counts. Experiments show stronger alignment with exact likelihood gradients in a controlled ImageNet setting and improved multi-sample solution coverage over MaxRL on maze navigation and mathematical reasoning under matched training rollout budgets.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Rollout Allocation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GnFnOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper points out the attenuation effect of original MaxRL's truncated objective and proposes SERA, a rollout allocation method according to a max-min solution on weight function. The paper follows MaxRL's experimental setups and presents consistent gains. It also observes enhanced alignments to logp objective on ImageNet and clearly discusses the difference with related works, such as Depth-Breadth and Reinforce-Ada.

Critical strengths
  1. This paper identifies an interesting theoretical issue of original MaxRL, and provides a principled solution.
  2. This paper provides enough experiments and analysis to justify the effectiveness of SERA.
Critical weaknesses

SERA introduces additional hyperparameters, such as N_{max}=4N_0, u_0=0.05, \lambda=0.75. Therefore it might make implementation a little bit harder (potentially, less general) compared with a fixed allocation.

Questions for authors
  1. Does the authors have some explanations or insights on how to choose these hyperparameters during SERA-MaxRL training? How sensitive is SERA-MaxRL training to these choices?
  2. Would it be possible to extend the gradient-alignment analysis in the ImageNet experiment to sequential tasks such as maze navigation or mathematical reasoning?
3 comments

We thank the reviewer for the constructive feedback and insightful questions.

W. Q1. Hyperparameter choice and sensitivity.

We chose Nmax⁡=4N0N_{\max}=4N_0 as a conservative default that balances rollout concentration and prompt coverage. It allows a difficult prompt to receive up to four times the average rollout count while ensuring that at least one quarter of the prompts in each batch receive rollouts. Only groups with both a success and a failure yield nonzero centered advantages. A low threshold (u0=0.05u_0=0.05) on the estimated chance of such a group at the assigned count lets rarely solved prompts qualify. We discount past outcomes to smooth noisy success estimates while tracking policy changes. For a history half-life of hh epochs, choose λ=2−1/h\lambda=2^{-1/h}. Our default λ=0.75\lambda=0.75 gives 2.4 epochs. A shorter half-life gives more weight to recent observations when prompt difficulty changes rapidly.

Rollout cap Nmax⁡N_{\max}. We compared caps of 64 and 128 on ImageNet and GSM8K (SmolLM2). Both settings used 256 prompts per training batch and N0=16N_0=16 rollouts per prompt on average. For SmolLM2, the cap comparison reports online validation with 32 responses per question and bootstrap Best@32.

Task (step)Metrics (%)Nmax⁡=64N_{\max}=64Nmax⁡=128N_{\max}=128
ImageNet (100,000)Pass@1 / Pass@3212.367 / 16.25521.896 / 31.857
SmolLM2 (2,000)Pass@1 / Best@3232.02 / 58.0329.43 / 60.32

On ImageNet, increasing Nmax⁡N_{\max} from 64 to 128 raised Pass@1 and Pass@32 by 9.53 and 15.60 percentage points, respectively. Both caps exceeded the corresponding uniform MaxRL scores of 4.43% and 6.47%. On SmolLM2, the larger cap raised online Best@32 by 2.29 points but lowered Pass@1 by 2.59 points. Both caps exceeded MaxRL’s Best@32 of 53.89%.

Observability threshold u0u_0 and history discount λ\lambda. Given time and compute constraints, we limited the SmolLM2 sensitivity analysis to five combinations of u0u_0 and λ\lambda at training step 500, with Nmax⁡=64N_{\max}=64. We varied u0u_0 at fixed λ=0.50\lambda=0.50 and compared the two discount factors only at u0=0.05u_0=0.05. Each training update used 128 prompts and 2,048 rollouts (N0=16N_0=16 on average). We evaluated all five settings on GSM8K-Platinum, sampling 32 responses per question. We report mean response accuracy as Pass@1 and estimate Best@32 by bootstrap resampling within each question.

u0u_0λ\lambdaPass@1 (%)Best@32 (%)
0.050.7518.3158.36
0.050.5018.6862.58
0.100.5018.6456.34
0.200.5019.3855.89
0.300.5021.9057.36

At step 500 on SmolLM2, Pass@1 was relatively insensitive to reducing λ\lambda from 0.75 to 0.50 at fixed u0=0.05u_0=0.05, increasing by only 0.37 percentage points. Best@32 increased by 4.22 points in the same comparison and ranged from 55.89% to 62.58% across the thresholds tested at fixed λ=0.50\lambda=0.50. In practice, a lower u0u_0 keeps a prompt eligible for allocation even when its estimated chance of producing both a success and a failure is small. A shorter history half-life (smaller λ\lambda) gives more weight to recent outcomes when prompt difficulty changes quickly, while a longer half-life smooths noisy estimates over more epochs.

Q2. Gradient alignment on sequential tasks.

We chose ImageNet for the original gradient-alignment analysis because the model directly gives the probability pqp_q of the correct class for each image. We can therefore compute ∇θlog⁡pq\nabla_\theta\log p_q exactly and average it over the batch to obtain the reference gradient used in Figure 2(d), as detailed in Appendix D.2.1 (line 1434). In maze navigation or mathematical reasoning, pqp_q is the sum of the probabilities of all trajectories that solve a given problem. Computing that sum exactly is generally infeasible.

We appreciate the reviewer’s constructive suggestion to examine gradient alignment on a sequential task. We extended the analysis to Maze using an approximate reference gradient. At each of 12 matched training steps (250–3,000), we froze each method’s own model and evaluated both on the same batch of 256 training mazes. For each model, the ideal batch gradient is g⋆=1256∑q=1256∇θlog⁡pqg^\star=\frac{1}{256}\sum_{q=1}^{256}\nabla_\theta\log p_q, where pqp_q is the model’s probability of solving maze qq. For each maze, we used search and guided sampling to identify successful paths, summed their contributions to pqp_q and ∇θpq\nabla_\theta p_q exactly, and estimated the remaining contributions by importance sampling. We estimated ∇θlog⁡pq\nabla_\theta\log p_q by dividing the estimated ∇θpq\nabla_\theta p_q by the estimated pqp_q, then averaged these vectors over all 256 mazes. This per-maze ratio can have finite-sample bias. We constructed two independent batch reference estimates in this way, using 3,136 reference trajectory samples per maze in total. At each checkpoint, we computed the cosine similarity between the average of the two reference vectors and one independently sampled RL gradient obtained using that method’s rollout allocation. The Maze row reports the arithmetic mean of these 12 checkpoint-level cosines for each method.

TaskReference gradientSteps evaluatedMaxRL mean cosineSERA mean cosine
ImageNetExact1000.4320.627
MazeStratified trajectory estimate120.5770.728

These results provide additional evidence that SERA's improved gradient alignment extends beyond the one-step ImageNet setting to sequential decision-making. Although the Maze reference gradient is estimated rather than exact, SERA achieves a higher mean cosine similarity than MaxRL (0.728 vs. 0.577) across the evaluated checkpoints, with better alignment at 8 of 12 checkpoints. This supports the applicability of our finite-rollout gradient-scaling analysis to sequential tasks, while acknowledging the uncertainty introduced by approximate reference estimation.

Thank you for the response. I have read the rebuttal carefully, and I will maintain my current score.

Reviewer_omqqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the MaxRL method's one weakness: under fixed budgets, MaxRL would attenuate the data point's likelihood gradient a factor, which depends on its success probability and rollout count. The common rollout could would leave the low-success prompts more attenuated. Therefore, this paper proposes SERA, aiming to redistribute the fixed rollout budget to equalize the scaling factor. It formulates the allocation as a fixed-budget max-min optimization problem. The empirical experiments also show the better performances than other baselines.

Critical strengths
  1. The paper is well-written and the motivation is natural and worthy researching.
  2. The problem formulation is also interesting and novel, which introduces an elegant derivation and solutions
  3. The empirical experiments are sufficient and covered diversely, on ImageNet, Maze navigation, and math reasoning, wheres the improvements are significant.
Critical weaknesses
  1. I am wondering the baseline with dividing the factor k(pq,N)k(p_q,N), which is only analyzed theoretically instead of the experiments demonstration.
  2. I also find the improvements are more obvious on Pass@K when KK is large, like 512 or 1024. When KK is in small number, the improvement is limited.
Questions for authors

Please refer to weakness.

2 comments

We thank the reviewer for the constructive feedback and insightful questions.

W1. Experimental results for the direct division by the attenuation factor.

Following the reviewer’s valuable suggestion, we tested direct division by the attenuation factor on ImageNet and added the results to Appendix D.2.1 (line 1458) of the revised PDF (Figure 15). We kept MaxRL’s uniform allocation of 128 rollouts per image and divided each image’s centered gradient estimate by κi=1−(1−pi)127\kappa_i=1-(1-p_i)^{127}, using the model’s exact probability pip_i of the correct class. Equation (32) in Appendix B.5 (line 1011) of the original submission shows that this correction removes attenuation in expectation and multiplies the per-image gradient covariance by 1/κi21/\kappa_i^2. For images with low pip_i, all-incorrect groups still give zero gradient, and the correction can greatly amplify the rare gradients from mixed-reward groups. All three ImageNet runs used the same training budget of 128 rollouts per image on average.

MethodValidation Pass@1
MaxRL31.69%
MaxRL with direct division by κi\kappa_i0.10%
SERA45.98%

Direct division produced zero gradients on 74,947 out of 75,123 updates (approximately 99.77%). Occasional large spikes in the pre-clipping gradient norm were followed by sharp increases in validation cross-entropy (Figure 15b–c). Together, these observations provide empirical evidence that direct division by the attenuation factor does not necessarily lead to stable or effective training, even when the factor is known exactly.

W2. Why are the gains smaller at low KK?

We thank the reviewer for highlighting this observation. Like MaxRL, SERA targets prompt-wise log-success, an objective closely related to solution coverage across inference budgets. Specifically, for pq∈(0,1)p_q\in(0,1), the identity ∇θlog⁡pq=∑k=1∞k−1∇θPass@k(q)\nabla_\theta\log p_q=\sum_{k=1}^{\infty}k^{-1}\nabla_\theta\mathrm{Pass@}k(q) connects the log-success objective to Pass@K optimization across different sampling budgets.

The larger gains at high K can be explained by SERA's emphasis on low-success prompts. Under uniform rollout allocation, these prompts experience stronger gradient attenuation, and SERA reallocates rollouts to better preserve their contributions to the expected gradient. For a prompt with success probability pqp_q, we have Pass@K(q)=1−(1−pq)K\mathrm{Pass@}K(q)=1-(1-p_q)^K, and therefore ∂Pass@K(q)/∂pq=K(1−pq)K−1\partial\mathrm{Pass@}K(q)/\partial p_q=K(1-p_q)^{K-1}. For low-success prompts, this sensitivity can be substantially larger than one before coverage saturates. Consequently, even a small improvement in single-sample success probability can lead to a much larger improvement in multi-sample solution coverage.

This pattern is consistent with our experiments. On SmolLM2 with token-mean normalization, SERA improves Pass@1 by 0.41 percentage points while increasing Pass@256 by 5.13 points. On Qwen2.5-Math-1.5B, Pass@1 remains within 0.22 points of MaxRL, whereas Pass@512 improves by 7.00 points on BeyondAIME and 6.67 points on AIME 2025. Importantly, SERA does not inherently trade off Pass@1 for higher Pass@K; rather, its improved treatment of low-success prompts can yield more pronounced benefits at larger, non-saturated inference budgets.

Thanks, I would like to keep my positive score

Reviewer_jCgsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SERA, a scale-equalized rollout allocation strategy for MaxRL. The authors identify that finite-rollout estimation introduces prompt-dependent gradient attenuation, which can distort the aggregate optimization direction. SERA addresses this issue by adaptively allocating rollouts under a fixed budget to approximately equalize gradient scaling factors. Experiments demonstrate improved gradient alignment and multi-sample solution coverage across several tasks.

Critical strengths
  1. Interesting and well-motivated insight. The paper identifies an important limitation of uniform rollout allocation and clearly explains how unequal gradient scaling can distort the overall optimization direction.
  2. Strong connection between theory and method. The proposed allocation strategy follows naturally from the theoretical analysis, with a principled optimization formulation and a well-derived solution. The overall method is also relatively simple and practical.
  3. Comprehensive and convincing experiments. The evaluation covers multiple tasks and model scales under matched rollout budgets. In particular, the gradient alignment experiment directly supports the theoretical motivation and strengthens the empirical evidence.
Critical weaknesses
  1. Limited isolation of the core contribution. Although the theoretical formulation is meaningful, the experiments do not fully distinguish the benefits of scale equalization from those of simply allocating more rollouts to difficult prompts. More controlled analysis would help clarify the method's distinct contribution.
  2. Dependence on historical success estimates. The method relies on reward statistics from previous epochs, which may become outdated as the policy evolves. This could limit its applicability to settings with infrequent prompt repetition or rapidly changing policies.
  3. Limited analysis of practical training efficiency. Although the experiments use matched rollout budgets, heterogeneous rollout counts may affect token consumption and execution efficiency. Additional measurements of actual training time and computational cost would strengthen the practical evaluation.
Questions for authors
  1. The proposed allocation maximizes the minimum gradient scaling factor across prompts. Since this criterion addresses expected gradient scaling but does not directly account for sampling variance, could the authors further explain the trade-off between scale equalization and gradient estimation variance?
  2. How sensitive is SERA to inaccurate or outdated success estimates? Would the method remain effective when prompts are rarely revisited or the policy changes rapidly during training?
3 comments

We thank the reviewer for the constructive feedback and insightful questions.

W1. What does scale equalization add beyond allocating more rollouts to difficult prompts?

Thanks for the valuable suggestion. A closely related difficulty-based strategy is inverse-success allocation, which assigns rollout counts proportional to 1/pq1/p_q (cf. Yang et al., 2025b). To isolate the contribution of our allocation criterion, we compare SERA against this baseline on ImageNet, retaining all prompts and using the same total rollout budget. After 50,000 training steps, SERA achieves 32.16% Pass@1 and 84.68% Pass@128, outperforming inverse-success allocation by 4.45 and 2.80 percentage points, respectively. These results suggest that SERA's gains are not solely attributable to assigning more rollouts to difficult prompts.

Allocation (N0=128N_0=128)Pass@1Pass@128
MaxRL23.92%47.43%
Inverse-success27.71%81.88%
SERA32.16%84.68%

Under finite rollout budgets, the expected MaxRL gradient for each prompt is scaled by κq=1−(1−pq)Nq−1\kappa_q=1-(1-p_q)^{N_q-1}. Unlike inverse-success allocation, which assigns Nq∝1/pqN_q\propto 1/p_q, SERA directly optimizes these finite-rollout scaling factors under a fixed budget. Specifically, the interior solution to our continuous max–min formulation is Nq⋆=1+c/[−log⁡(1−pq)]N_q^\star=1+c/[-\log(1-p_q)], which equalizes κq\kappa_q across prompts. When pqp_q is small and Nq⋆N_q^\star is large, this reduces to Nq⋆≈c/pqN_q^\star\approx c/p_q. Thus, inverse-success allocation approximates SERA in the low-success regime, but does not generally achieve scale equalization.

To illustrate this distinction, we added Figure 5 to Appendix B.3 (line 982). In a four-prompt example with a fixed total budget, inverse-success and SERA assign nearly identical rollout counts at pq=0.10p_q=0.10, but inverse-success assigns 16.2% fewer rollouts at pq=0.40p_q=0.40. Consequently, its minimum gradient scaling factor is only 0.440, whereas SERA equalizes the scaling factors at 0.546 across all four prompts. This illustrates how directly optimizing finite-rollout gradient fidelity differs from simply allocating rollouts inversely proportional to prompt success probabilities.

Q1. What is the trade-off between scale equalization and gradient estimation variance?

We thank the reviewer for raising this important question. Scale equalization and variance reduction address distinct aspects of the gradient estimator. SERA aims to preserve the relative contributions of prompt-wise log-success gradients in expectation, rather than directly minimizing sampling variance. Improving gradient fidelity does not necessarily reduce variance, and a trade-off may therefore arise.

To investigate this trade-off, we followed the setup in MaxRL's Appendix M.2 (Policy Gradient Variance Analysis) and examined how reallocating rollouts to raise the minimum expected gradient scale affects the sampling variance of the MaxRL gradient estimator. We added the results for six rollout budgets to Appendix D.2.6 (line 1800) of the revised PDF (Figure 20). Within each task, we compare SERA and MaxRL on the same frozen weights and candidate batches, giving both methods the same total number of rollouts per batch at each budget. We use the ImageNet checkpoint and the final SERA checkpoints for Maze and SmolLM2.

For each method, we fix its allocation and resample the rollouts. Let GG be the full batch gradient from one such sample and μ=Erollout[G]\mu=\mathbb{E}_{\mathrm{rollout}}[G] its mean under that allocation. We measure V=tr⁡Cov⁡rollout(G)=Erollout∥G−μ∥22V=\operatorname{tr}\operatorname{Cov}_{\mathrm{rollout}}(G)=\mathbb{E}_{\mathrm{rollout}}\lVert G-\mu\rVert_2^2. Let G1,…,GRG_1,\ldots,G_R be the batch gradients from RR independently sampled sets of rollouts. The trace of their sample covariance matrix (MaxRL, Appendix M.2) is

V^=1R−1∑r=1R∥Gr−Gˉ∥22,Gˉ=1R∑r=1RGr.\widehat{V}=\frac{1}{R-1}\sum_{r=1}^R\lVert G_r-\bar{G}\rVert_2^2, \qquad \bar{G}=\frac{1}{R}\sum_{r=1}^R G_r.

On ImageNet, we average the squared deviations of 64 batch gradients from each method’s analytic mean on each of 16 batches of 256 images. On Maze, we use the sample covariance trace with R=64R=64 gradients on each of eight batches of 256 mazes. On a fixed batch of 64 GSM8K prompts, we compute eight batch gradients by resampling from each of eight independently generated response pools. We estimate V=E∥G∥22−∥μ∥22V=\mathbb{E}\lVert G\rVert_2^2-\lVert \mu\rVert_2^2 using inner products between mean gradients from distinct pools for the second term, accounting for response reuse within each pool.

To see how allocation affects VV, write G=B−1∑q∈Ag^qG=B^{-1}\sum_{q\in\mathcal{A}}\widehat{g}_q, where BB is the number of candidate prompts and A\mathcal{A} is the retained set. Independent rollouts across prompts give V=B−2∑q∈A(E∥g^q∥22−∥mq∥22)V=B^{-2}\sum_{q\in\mathcal{A}}(\mathbb{E}\lVert \widehat{g}_q\rVert_2^2-\lVert m_q\rVert_2^2). Equation (3) gives the mean mq=κ(pq,Nq)∇θlog⁡pqm_q=\kappa(p_q,N_q)\nabla_\theta\log p_q. Only mixed-reward groups can produce a nonzero centered update. Giving a low-success prompt more rollouts raises κ\kappa and makes mixed groups more likely. The net change in VV also depends on the sizes of those updates and on the variance changes at prompts that give up rollouts.

Figure 20 (line 1846) shows that gradient variance generally decreases with the rollout budget for both methods. At matched budgets, SERA achieves lower variance at most budgets on ImageNet and Maze, while exhibiting higher variance at several budgets on SmolLM2. Overall, these results suggest that SERA can improve the minimum expected gradient scaling factor without systematically incurring a substantial increase in gradient variance. Although variance reduction is not explicitly optimized by our allocation criterion, the empirical results indicate that improving gradient fidelity does not necessarily come at the cost of substantially higher sampling variance.

W2. Q2. Sensitivity to historical success estimates.

We thank the reviewer for raising this insightful question. Success-probability estimates based on historical rollouts can be noisy when observations are limited and become stale as the policy evolves. This is a common challenge for success-dependent rollout allocation methods (e.g., Yang et al., 2025b; Li et al., 2025; Xiong et al., 2025; Nguyen et al., 2026), as discussed in Appendix A (line 794). To assess its practical impact, we conducted an additional ImageNet experiment comparing SERA using historical estimates with SERA using exact current success probabilities and uniform MaxRL. The results are included in Appendix D.2.1 (line 1507, Table 4).

At the same rollout budget (N0=128N_0=128), we compared MaxRL with two adaptive runs that reserved two rollouts per image and allocated the remainder using the same auxiliary rule. One run used the frozen estimates and the other used the current policy’s exact correct-class probabilities. The policy underwent 5,005 updates per epoch while these estimates stayed fixed. In epoch 20, the frozen estimates had a mean absolute error of 0.1410.141 relative to the current probabilities. Even with this lag and estimation error, the historical run outperformed uniform MaxRL at step 100,000 by 8.2 percentage points on Pass@1 and 14.8 points on Pass@128. Using exact probabilities improved both scores further.

ImageNet allocationPass@1Pass@128
MaxRL36.59%53.22%
SERA, historical p^q\hat{p}_q44.75%68.06%
SERA, exact pqp_q55.69%87.98%

When prompts are rarely revisited, historical estimates may fail to reflect changes in their current difficulty. For example, a prompt that becomes substantially harder may temporarily receive too few rollouts. To mitigate this issue, SERA uses discounted historical statistics (Appendix C.1, line 1049), with an evidence discount factor of λ=0.75\lambda=0.75, corresponding to a half-life of approximately 2.4 epochs. If a previously observed prompt receives no new rollouts, its historical evidence gradually decays and its estimate approaches the prior mean of 0.5, allowing it to be reconsidered for allocation.

For previously unseen prompts, SERA uses a conservative fallback: if any prompt in a batch lacks historical observations, the entire batch receives uniform rollout allocation. Consequently, a stream consisting entirely of new prompts would use uniform allocation throughout. We acknowledge that our current history-based implementation is therefore most applicable to settings with recurring prompts. The ImageNet experiment demonstrates that historical allocation can remain beneficial despite substantial estimation lag, but does not establish robustness to arbitrary policy changes or one-pass prompt streams. Extending SERA with more general success-probability estimators is a promising direction for addressing these settings.

W3. Practical training efficiency.

Following the reviewer’s constructive suggestion, we added a wall-clock comparison of SERA and MaxRL to Appendix D.2.7 (line 1832) of the revised PDF (Table 11). We measured time per step for Qwen2.5-Math-1.5B over steps 501–600 on eight 80 GB A100 GPUs. Both methods generated 4,096 responses per step.

MethodStep time (s/step)Generation (s/step)Mean response length (tokens)Valid tokens (M/step)tˉ/Lˉ\bar t/\bar L (ms)
MaxRL112.2343.00750.83.58149.48
SERA150.6257.881,067.75.01141.08

The last column divides mean step time (tˉ\bar t) by mean response length (Lˉ\bar L). Compared with MaxRL, SERA generated responses that were 42.2% longer on average and took 34.2% longer per training step. However, when normalized by the number of valid tokens processed, the two methods exhibit comparable throughput: 31.35 seconds per million valid tokens for MaxRL versus 30.06 seconds for SERA. This suggests that heterogeneous rollout allocation does not introduce a substantial per-token throughput penalty in our evaluated setting. SERA and MaxRL showed comparable computational efficiency as measured by step time normalized by mean response length.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief introduction to the revised PDF

We thank all reviewers for their time and valuable feedback. Following their constructive suggestions, we added three figures and two tables to the appendices.

  1. Figure 5 in Appendix B.3 (line 999) compares rollout counts and gradient scaling under SERA and inverse-success allocation.
  2. Figure 15 in Appendix D.2.1 (line 1475) evaluates an ImageNet baseline that divides each image’s gradient estimate by its exact scaling factor.
  3. Table 4 in Appendix D.2.1 (line 1512) compares uniform MaxRL with allocation based on historical or exact success probabilities on ImageNet.
  4. Figure 20 in Appendix D.2.6 (line 1846) compares gradient variance across rollout budgets for SERA and MaxRL on ImageNet, Maze, and GSM8K.
  5. Table 11 in Appendix D.2.7 (line 1849) reports training time for Qwen2.5-Math-1.5B trained with SERA and MaxRL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.