- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper points out the attenuation effect of original MaxRL's truncated objective and proposes SERA, a rollout allocation method according to a max-min solution on weight function. The paper follows MaxRL's experimental setups and presents consistent gains. It also observes enhanced alignments to logp objective on ImageNet and clearly discusses the difference with related works, such as Depth-Breadth and Reinforce-Ada.
Critical strengths
- This paper identifies an interesting theoretical issue of original MaxRL, and provides a principled solution.
- This paper provides enough experiments and analysis to justify the effectiveness of SERA.
Critical weaknesses
SERA introduces additional hyperparameters, such as N_{max}=4N_0, u_0=0.05, \lambda=0.75. Therefore it might make implementation a little bit harder (potentially, less general) compared with a fixed allocation.
Questions for authors
- Does the authors have some explanations or insights on how to choose these hyperparameters during SERA-MaxRL training? How sensitive is SERA-MaxRL training to these choices?
- Would it be possible to extend the gradient-alignment analysis in the ImageNet experiment to sequential tasks such as maze navigation or mathematical reasoning?
3 comments
Q2. Gradient alignment on sequential tasks.
We chose ImageNet for the original gradient-alignment analysis because the model directly gives the probability of the correct class for each image. We can therefore compute exactly and average it over the batch to obtain the reference gradient used in Figure 2(d), as detailed in Appendix D.2.1 (line 1434). In maze navigation or mathematical reasoning, is the sum of the probabilities of all trajectories that solve a given problem. Computing that sum exactly is generally infeasible.
We appreciate the reviewer’s constructive suggestion to examine gradient alignment on a sequential task. We extended the analysis to Maze using an approximate reference gradient. At each of 12 matched training steps (250–3,000), we froze each method’s own model and evaluated both on the same batch of 256 training mazes. For each model, the ideal batch gradient is , where is the model’s probability of solving maze . For each maze, we used search and guided sampling to identify successful paths, summed their contributions to and exactly, and estimated the remaining contributions by importance sampling. We estimated by dividing the estimated by the estimated , then averaged these vectors over all 256 mazes. This per-maze ratio can have finite-sample bias. We constructed two independent batch reference estimates in this way, using 3,136 reference trajectory samples per maze in total. At each checkpoint, we computed the cosine similarity between the average of the two reference vectors and one independently sampled RL gradient obtained using that method’s rollout allocation. The Maze row reports the arithmetic mean of these 12 checkpoint-level cosines for each method.
| Task | Reference gradient | Steps evaluated | MaxRL mean cosine | SERA mean cosine |
|---|---|---|---|---|
| ImageNet | Exact | 100 | 0.432 | 0.627 |
| Maze | Stratified trajectory estimate | 12 | 0.577 | 0.728 |
These results provide additional evidence that SERA's improved gradient alignment extends beyond the one-step ImageNet setting to sequential decision-making. Although the Maze reference gradient is estimated rather than exact, SERA achieves a higher mean cosine similarity than MaxRL (0.728 vs. 0.577) across the evaluated checkpoints, with better alignment at 8 of 12 checkpoints. This supports the applicability of our finite-rollout gradient-scaling analysis to sequential tasks, while acknowledging the uncertainty introduced by approximate reference estimation.
Thank you for the response. I have read the rebuttal carefully, and I will maintain my current score.
We thank the reviewer for the constructive feedback and insightful questions.
W. Q1. Hyperparameter choice and sensitivity.
We chose Nmax=4N0 as a conservative default that balances rollout concentration and prompt coverage. It allows a difficult prompt to receive up to four times the average rollout count while ensuring that at least one quarter of the prompts in each batch receive rollouts. Only groups with both a success and a failure yield nonzero centered advantages. A low threshold (u0=0.05) on the estimated chance of such a group at the assigned count lets rarely solved prompts qualify. We discount past outcomes to smooth noisy success estimates while tracking policy changes. For a history half-life of h epochs, choose λ=2−1/h. Our default λ=0.75 gives 2.4 epochs. A shorter half-life gives more weight to recent observations when prompt difficulty changes rapidly.
Rollout cap Nmax. We compared caps of 64 and 128 on ImageNet and GSM8K (SmolLM2). Both settings used 256 prompts per training batch and N0=16 rollouts per prompt on average. For SmolLM2, the cap comparison reports online validation with 32 responses per question and bootstrap Best@32.
On ImageNet, increasing Nmax from 64 to 128 raised Pass@1 and Pass@32 by 9.53 and 15.60 percentage points, respectively. Both caps exceeded the corresponding uniform MaxRL scores of 4.43% and 6.47%. On SmolLM2, the larger cap raised online Best@32 by 2.29 points but lowered Pass@1 by 2.59 points. Both caps exceeded MaxRL’s Best@32 of 53.89%.
Observability threshold u0 and history discount λ. Given time and compute constraints, we limited the SmolLM2 sensitivity analysis to five combinations of u0 and λ at training step 500, with Nmax=64. We varied u0 at fixed λ=0.50 and compared the two discount factors only at u0=0.05. Each training update used 128 prompts and 2,048 rollouts (N0=16 on average). We evaluated all five settings on GSM8K-Platinum, sampling 32 responses per question. We report mean response accuracy as Pass@1 and estimate Best@32 by bootstrap resampling within each question.
At step 500 on SmolLM2, Pass@1 was relatively insensitive to reducing λ from 0.75 to 0.50 at fixed u0=0.05, increasing by only 0.37 percentage points. Best@32 increased by 4.22 points in the same comparison and ranged from 55.89% to 62.58% across the thresholds tested at fixed λ=0.50. In practice, a lower u0 keeps a prompt eligible for allocation even when its estimated chance of producing both a success and a failure is small. A shorter history half-life (smaller λ) gives more weight to recent outcomes when prompt difficulty changes quickly, while a longer half-life smooths noisy estimates over more epochs.