- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the theoretical properties of EGGROLL, an evolution strategies (ES) method for efficient LLM post-training that uses low-rank Gaussian perturbations instead of dense perturbations. The authors derive the exact finite-rank EGGROLL population field as a resolvent-filtered gradient and show that it can be nonconservative and locally destabilize an optimum. The authors prove exactness on quadratics, identify the leading finite-rank correction, and show that the surplus finite-population variance relative to dense Gaussian ES decays with matrix width. The authors characterize the optimization dynamics and sampling variance of EGGROLL and propose LOO-ROLL, a more computationally efficient estimator based on leave-one-out baselines. The authors conductt experiments across 14 LLM post-training settings, using Qwen3 and SmolLM2 models up to 14B parameters, demonstrate approximately 50% lower estimator MSE at equal evaluation cost. Under matched wall-clock time, LOO-ROLL improves performance significantly in 11 of 14 individual paired comparisons, with 9 remaining significant after multiple-testing correction.
Critical strengths
Following are the strengths of this paper:
- The theoretical analysis directly informs the algorithmic design, with the variance derivations providing a clear rationale for why rank-one perturbations remain competitive with full-rank Gaussian smoothing despite their restricted geometric support. By targeting an explicit forward-pass inefficiency in EGGROLL, LOO-ROLL delivers an elegant, forward-only formulation that integrates cleanly into existing pipelines without disrupting the underlying low-rank structure.
- The empirical evaluation is thorough and methodologically sound, testing across two model families, multiple model scales, and varied downstream objectives such as GSM8K, MATH-500, Countdown, and next-token prediction. Reporting matched-iteration, matched-evaluation, and matched-wall-clock baselines effectively isolates estimator quality from training budget effects, while the inclusion of five paired seeds, formal hypothesis testing with Holm-Bonferroni corrections, and controlled numerical variance validation lends solid credibility to the gains.
- The work is transparent and reproducible, supported by comprehensive proofs, concrete training configurations, compute budgets, and extensive appendix diagnostics.
Critical weaknesses
Following are the weaknesses of the paper:
- The core algorithmic contribution of LOO-ROLL is somewhat incremental, as leave-one-out baselines and single-evaluation score-function estimators are well-established variance reduction techniques in reinforcement learning and evolutionary optimization. While the exact variance derivation tailored to low-rank ES is informative, the algorithmic advance primarily amounts to porting an existing design pattern into EGGROLL rather than introducing a distinct optimization paradigm.
- The reported performance gains appear to be driven by evaluation throughput rather than intrinsically superior optimization per sample. Under equal-evaluation budgets, LOO-ROLL yields negligible improvements over EGGROLL, with its downstream advantages surfacing predominantly under equal-wall-time regimes where it completes roughly 2 to 2.6 times as many updates. Disentangling true gradient estimator improvements from raw wall-clock throughput and comparing against alternative compute reallocation baselines is necessary to substantiate the algorithmic merit of the estimator itself.
- The baseline comparisons are narrowly scoped to standard EGGROLL. Despite referencing related zeroth-order and evolutionary techniques like MeZO, standard REINFORCE-style estimators, and classic ES control variates, the paper lacks head-to-head empirical comparisons against these alternatives under matched compute budgets to contextualize LOO-ROLL's broader competitiveness.
Questions for authors
Following are the questions for the authors:
- Because leave-one-out estimators couple updates across distinct population members rather than canceling shared prompt noise via antithetic pairs, how robust is LOO-ROLL to prompt heterogeneity, noisy task verifiers, and high reward variance compared to standard antithetic sampling?
- How does population-level score standardization alter the expected update fields and theoretical unbiasedness of LOO-ROLL relative to antithetic EGGROLL, particularly in regimes with small population sizes or skewed reward distributions?
- How does LOO-ROLL compare against other single-evaluation ES estimators or standard zeroth-order baselines (such as MeZO or classical one-point control variates) when evaluated under strictly matched wall-clock and parameter-update budgets?
- Given that the transformer-block updates maintain around 0.98 cosine similarity with backpropagated gradients despite theoretical nonconservative instability, could the authors quantify the frequency and magnitude of update misalignment across different layers, training phases, and perturbation radii to demonstrate the real-world impact of these dynamics?
3 comments
Thank you for the explanation. But the authors acknowledge the lack of novelty, LOO-ROLL provides little or no advantage at equal evaluation cost and can underperform EGGROLL at smaller population sizes, and the robustness to genuinely noisy verifiers and high-variance rewards remains untested. Thus, I will keep my score.
Thank you for the reply. We would like to clarify three points.
Novelty. We did not intend to concede a lack of novelty. We said that the leave-one-out baseline is a standard control variate, and we do not claim it as new. The contribution of the paper is the analysis. This covers the finite-rank mean field and its pathologies (Sec. 2 and 3), the exact finite-population variance (Sec. 4), and the exact MSE comparison of antithetic and one-evaluation estimators for an arbitrary objective (Prop. 5.2). LOO-ROLL is the estimator that this analysis motivates. Sec. 4 shows that rank cannot reduce the dominant variance factor , which leaves the allocation of evaluations as the remaining lever.
Equal evaluation cost. The equal-cost results follow exactly the behavior that Prop. 5.2 predicts. At equal cost the MSE ratio is about , plus a baseline term of order . Where the fitness is nearly linear in the perturbation (), Prop. 5.2 predicts a halving of the MSE, which we measure. Greedy-decoded GSM8K rewards have between 0.77 and 0.97, and the measured ratios of the actual updates match the prediction to within 0.05 (Table 1). The term accounts for the behavior at small . The practical value of the analysis is that it tells which estimator to use before training, from fitness values that EGGROLL already records.
Noisy verifiers and high-variance rewards. Prop. 5.2 holds for any objective, including stochastic ones. Verifier noise that is independent across evaluations adds the same variance to the odd and the even part of the fitness. Such noise therefore moves toward 1, and the criterion covers this regime without change.
Response to Reviewer TxHr
We thank the reviewer for the detailed review.
W1. LOO-ROLL is incremental. We agree that the leave-one-out baseline is standard, and we do not claim it as novel. We propose it because our analysis identifies evaluation allocation, not rank, as the lever for EGGROLL's sampling error (please see G1 in the general response). The contributions around it are an exact criterion for choosing between antithetic and one-evaluation estimators, validated in three regimes, and the allocation finding explained below.
W2. Separate the estimator from throughput, and compare other reallocations. We ran additional experiments to exactly test this: an equal-wall-time sweep over evaluations per update, for both estimators, on Qwen3-1.7B GSM8K (please see G1 in the general response, as well as App. H.4 in the revised manuscript).
The revised paper therefore presents LOO-ROLL as the analysis-motivated way to reallocate evaluations, together with the criterion that tells when it beats a smaller antithetic population.
W3 / Q3. MeZO, single-evaluation ES and control variates at matched time.
Q1. Prompt heterogeneity, noisy verifiers, high reward variance. All population members share the prompt batch, and the trainer centers each prompt's fitness, so prompt offsets cancel in both estimators. Decoding is greedy, so the verifier is deterministic. With fixed and with resampled prompts, e/o is between 0.77 and 0.97, and the two estimators tie (App. H.1).
Q2. Standardization, small populations, skewed rewards. Standardization changes the expected update by O(1/N) for both estimators (Prop. G.3, Cor. G.4). Measured on rebuilt updates with populations of 16 to 256, it leaves the MSE ratio at 1.000 and the cosines unchanged (App. H.5). At small populations, LOO-ROLL degrades first, as the baseline term in Prop. 5.2 predicts.
Q4. Misalignment across layers, phases and radii. Please see G3 in the general response. It is below detection across 8 matrices in layers 0 to 27, five radii and two training phases. A new small MLP experiment we have added shows where it does appear (near optima) and that it follows the predicted 1/r law (App. H.6).
We hope that this addresses your concerns and would welcome additional discussion on any points that might remain,