acceptodds
Under review as a conference paper at ICLR 2027

EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale

Abstract

EGGROLL (Sarkar et al., 2026) makes evolution strategies (ES) practical for LLMs by replacing dense Gaussian weight perturbations with low-rank Gaussian products, often of rank one. This choice is computationally attractive but geometrically severe: Each rank-one perturbation lies in a zero-volume subset of the ambient matrix space, despite having identity covariance. We characterize the EGGROLL update mean field at finite rank and nonzero perturbation radius as a resolvent applied to the gradient of the perturbation-smoothed objective. This transformation can make the mean field nonconservative and reverse the local stability of an optimum. EGGROLL nevertheless recovers the gradient exactly on quadratic objectives at every rank and radius. In finite populations, the additional sampling variance of rank-one perturbations relative to dense Gaussian ES decays inversely with matrix width under a local affine model, and is only at width . Finally, we introduce LOO-ROLL, a leave-one-out estimator that replaces EGGROLL's two antithetic evaluations per direction by one. At equal evaluation cost, LOO-ROLL halves estimator MSE in transformer blocks. Across fourteen post-training settings up to 14B parameters, matched-time comparisons with EGGROLL yield eleven improvements in individual paired tests and no significant loss. At 1.7B, 8B, and 14B parameters, matched-time gains are , , and percentage points on GSM8K and , , and points on MATH-500.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evolution strategies, zeroth-order optimization, LLM post-training, low-rank perturbations, gradient-free learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TxHrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the theoretical properties of EGGROLL, an evolution strategies (ES) method for efficient LLM post-training that uses low-rank Gaussian perturbations instead of dense perturbations. The authors derive the exact finite-rank EGGROLL population field as a resolvent-filtered gradient and show that it can be nonconservative and locally destabilize an optimum. The authors prove exactness on quadratics, identify the leading finite-rank correction, and show that the surplus finite-population variance relative to dense Gaussian ES decays with matrix width. The authors characterize the optimization dynamics and sampling variance of EGGROLL and propose LOO-ROLL, a more computationally efficient estimator based on leave-one-out baselines. The authors conductt experiments across 14 LLM post-training settings, using Qwen3 and SmolLM2 models up to 14B parameters, demonstrate approximately 50% lower estimator MSE at equal evaluation cost. Under matched wall-clock time, LOO-ROLL improves performance significantly in 11 of 14 individual paired comparisons, with 9 remaining significant after multiple-testing correction.

Critical strengths

Following are the strengths of this paper:

  • The theoretical analysis directly informs the algorithmic design, with the variance derivations providing a clear rationale for why rank-one perturbations remain competitive with full-rank Gaussian smoothing despite their restricted geometric support. By targeting an explicit forward-pass inefficiency in EGGROLL, LOO-ROLL delivers an elegant, forward-only formulation that integrates cleanly into existing pipelines without disrupting the underlying low-rank structure.
  • The empirical evaluation is thorough and methodologically sound, testing across two model families, multiple model scales, and varied downstream objectives such as GSM8K, MATH-500, Countdown, and next-token prediction. Reporting matched-iteration, matched-evaluation, and matched-wall-clock baselines effectively isolates estimator quality from training budget effects, while the inclusion of five paired seeds, formal hypothesis testing with Holm-Bonferroni corrections, and controlled numerical variance validation lends solid credibility to the gains.
  • The work is transparent and reproducible, supported by comprehensive proofs, concrete training configurations, compute budgets, and extensive appendix diagnostics.
Critical weaknesses

Following are the weaknesses of the paper:

  • The core algorithmic contribution of LOO-ROLL is somewhat incremental, as leave-one-out baselines and single-evaluation score-function estimators are well-established variance reduction techniques in reinforcement learning and evolutionary optimization. While the exact variance derivation tailored to low-rank ES is informative, the algorithmic advance primarily amounts to porting an existing design pattern into EGGROLL rather than introducing a distinct optimization paradigm.
  • The reported performance gains appear to be driven by evaluation throughput rather than intrinsically superior optimization per sample. Under equal-evaluation budgets, LOO-ROLL yields negligible improvements over EGGROLL, with its downstream advantages surfacing predominantly under equal-wall-time regimes where it completes roughly 2 to 2.6 times as many updates. Disentangling true gradient estimator improvements from raw wall-clock throughput and comparing against alternative compute reallocation baselines is necessary to substantiate the algorithmic merit of the estimator itself.
  • The baseline comparisons are narrowly scoped to standard EGGROLL. Despite referencing related zeroth-order and evolutionary techniques like MeZO, standard REINFORCE-style estimators, and classic ES control variates, the paper lacks head-to-head empirical comparisons against these alternatives under matched compute budgets to contextualize LOO-ROLL's broader competitiveness.
Questions for authors

Following are the questions for the authors:

  • Because leave-one-out estimators couple updates across distinct population members rather than canceling shared prompt noise via antithetic pairs, how robust is LOO-ROLL to prompt heterogeneity, noisy task verifiers, and high reward variance compared to standard antithetic sampling?
  • How does population-level score standardization alter the expected update fields and theoretical unbiasedness of LOO-ROLL relative to antithetic EGGROLL, particularly in regimes with small population sizes or skewed reward distributions?
  • How does LOO-ROLL compare against other single-evaluation ES estimators or standard zeroth-order baselines (such as MeZO or classical one-point control variates) when evaluated under strictly matched wall-clock and parameter-update budgets?
  • Given that the transformer-block updates maintain around 0.98 cosine similarity with backpropagated gradients despite theoretical nonconservative instability, could the authors quantify the frequency and magnitude of update misalignment across different layers, training phases, and perturbation radii to demonstrate the real-world impact of these dynamics?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TxHr

We thank the reviewer for the detailed review.

W1. LOO-ROLL is incremental. We agree that the leave-one-out baseline is standard, and we do not claim it as novel. We propose it because our analysis identifies evaluation allocation, not rank, as the lever for EGGROLL's sampling error (please see G1 in the general response). The contributions around it are an exact criterion for choosing between antithetic and one-evaluation estimators, validated in three regimes, and the allocation finding explained below.

W2. Separate the estimator from throughput, and compare other reallocations. We ran additional experiments to exactly test this: an equal-wall-time sweep over evaluations per update, for both estimators, on Qwen3-1.7B GSM8K (please see G1 in the general response, as well as App. H.4 in the revised manuscript).

  • Throughput explains the gains. With 256 evaluations per update, accuracy reaches 80.8. Using 32 to 64 evaluations per update and more updates reaches about 85.0.
  • The estimator matters little at equal cost. LOO-ROLL and EGGROLL tie from 64 evaluations per update upward, and EGGROLL is ahead at 16 and 32. The update-space MSE ratio is 1.000 after standardization.
  • Prop. 5.2 predicts both effects. The even part of GSM8K rewards is as large as the odd part, which explains the tie. The baseline term grows as 1/M21/M^2 for small populations, which explains why LOO-ROLL falls behind there.

The revised paper therefore presents LOO-ROLL as the analysis-motivated way to reallocate evaluations, together with the criterion that tells when it beats a smaller antithetic population.

W3 / Q3. MeZO, single-evaluation ES and control variates at matched time.

  • MeZO on GSM8K (1.7B, 6.2 h, learning-rate sweep, 3 seeds): 83.6, against about 85.0 for EGGROLL or LOO-ROLL at 32 to 64 evaluations per update.
  • MeZO on next-token prediction: it wins at 0.6B and 1.7B, where each evaluation is one cheap forward pass. EGGROLL with 8 pairs wins at 8B (−14.3% NLL against −12.5%).
  • Single-evaluation ES: LOO-ROLL is one-sided ES with a population baseline, and it is included in every comparison.
  • Control variates: a reference-point control variate gave no gain (App. E.1).
  • Tuning: none of the ES runs were specifically tuned. The settings for the hyperparameters follow the EGGROLL authors' publicly available launch scripts (App. F.2).

Q1. Prompt heterogeneity, noisy verifiers, high reward variance. All population members share the prompt batch, and the trainer centers each prompt's fitness, so prompt offsets cancel in both estimators. Decoding is greedy, so the verifier is deterministic. With fixed and with resampled prompts, e/oe/o is between 0.77 and 0.97, and the two estimators tie (App. H.1).

Q2. Standardization, small populations, skewed rewards. Standardization changes the expected update by O(1/N)O(1/N) for both estimators (Prop. G.3, Cor. G.4). Measured on rebuilt updates with populations of 16 to 256, it leaves the MSE ratio at 1.000 and the cosines unchanged (App. H.5). At small populations, LOO-ROLL degrades first, as the baseline term in Prop. 5.2 predicts.

Q4. Misalignment across layers, phases and radii. Please see G3 in the general response. It is below detection across 8 matrices in layers 0 to 27, five radii and two training phases. A new small MLP experiment we have added shows where it does appear (near optima) and that it follows the predicted 1/r1/r law (App. H.6).

We hope that this addresses your concerns and would welcome additional discussion on any points that might remain,

Thank you for the explanation. But the authors acknowledge the lack of novelty, LOO-ROLL provides little or no advantage at equal evaluation cost and can underperform EGGROLL at smaller population sizes, and the robustness to genuinely noisy verifiers and high-variance rewards remains untested. Thus, I will keep my score.

Thank you for the reply. We would like to clarify three points.

Novelty. We did not intend to concede a lack of novelty. We said that the leave-one-out baseline is a standard control variate, and we do not claim it as new. The contribution of the paper is the analysis. This covers the finite-rank mean field and its pathologies (Sec. 2 and 3), the exact finite-population variance (Sec. 4), and the exact MSE comparison of antithetic and one-evaluation estimators for an arbitrary objective (Prop. 5.2). LOO-ROLL is the estimator that this analysis motivates. Sec. 4 shows that rank cannot reduce the dominant variance factor mn+1mn+1, which leaves the allocation of evaluations as the remaining lever.

Equal evaluation cost. The equal-cost results follow exactly the behavior that Prop. 5.2 predicts. At equal cost the MSE ratio is about 12(1+e/o)\tfrac12(1+e/o), plus a baseline term of order 1/M21/M^2. Where the fitness is nearly linear in the perturbation (e≈0e \approx 0), Prop. 5.2 predicts a halving of the MSE, which we measure. Greedy-decoded GSM8K rewards have e/oe/o between 0.77 and 0.97, and the measured ratios of the actual updates match the prediction to within 0.05 (Table 1). The 1/M21/M^2 term accounts for the behavior at small MM. The practical value of the analysis is that it tells which estimator to use before training, from fitness values that EGGROLL already records.

Noisy verifiers and high-variance rewards. Prop. 5.2 holds for any objective, including stochastic ones. Verifier noise that is independent across evaluations adds the same variance to the odd and the even part of the fitness. Such noise therefore moves e/oe/o toward 1, and the criterion covers this regime without change.

Reviewer_o442OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how low-rank perturbations affect evolution strategies and introduces LOO-ROLL, a one-evaluation alternative to EGGROLL’s antithetic estimator. The analysis characterizes the expected update at finite rank, shows when it can fail to follow the gradient of any objective, and derives the additional sampling variance caused by low-rank perturbations. Experiments examine these predictions on synthetic objectives and transformer blocks, followed by language-model post-training comparisons under matched evaluation and time budgets.

Critical strengths
  1. The finite-rank analysis is the strongest contribution. Showing that identity covariance does not guarantee a conservative expected update clarifies an important limitation of low-rank perturbations. The exact result on quadratic objectives and the width-dependent variance formula also help explain why rank-one perturbations can nevertheless work well.

  2. The matched-time experiments demonstrate practical value. Spending the saved evaluations on additional updates improves GSM8K and MATH performance across several model sizes, including 14B models, while retaining EGGROLL’s existing computational structure.

Critical weaknesses
  1. The optimizer contribution is less clearly distinguished from prior work. I see the finite-rank analysis as the main contribution. The optimizer change is more incremental: it applies an established leave-one-out baseline to EGGROLL’s perturbation population.

  2. The near-twofold MSE reduction has not been established for the standardized reward estimator used in full training. Proposition 5.2 relies on a locally affine objective, while Appendix G examines changes to the expected update rather than demonstrating the same variance advantage for GSM8K or MATH rewards.

  3. Evidence that the nonconservative mean field harms optimization comes from constructed examples. The transformer-block audit finds little rank dependence in the mean direction, and higher rank brings no consistent reward advantage. The analysis therefore identifies a possible failure mode without showing that it materially affects the reported training runs.

  4. I found the appendix tables difficult to read. Entries such as .99970 omit the leading zero; writing 0.99970 instead would make the numerical results easier to follow.

Questions for authors
  1. Beyond its application to EGGROLL’s low-rank perturbations, what distinguishes LOO-ROLL’s estimator construction from the leave-one-out methods discussed in Appendix A?

  2. At fixed training checkpoints, what variance reduction remains after applying the reward standardization used in the GSM8K and MATH experiments?

  3. Do any evaluated training settings show a rank-dependent distortion of the mean direction that affects optimization, separately from finite-population sampling variance?

1 comment
AuthorsOpenReview ↗

Response to Reviewer o442

We thank the author for the careful review. We agree that the finite-rank analysis is the main contribution. Your questions have led to new results and additions, which we summarize here.

W1 / Q1. What distinguishes LOO-ROLL from the leave-one-out methods in App. A? The control variate is the same. In RLOO, the baseline averages rewards of other responses to the same prompt and weights them by the policy score. In LOO-ROLL, it averages the fitness of other perturbed models and weights it by the Gaussian-product direction. We do not claim the estimator as new (App. A). We propose it as the remedy that our analysis motivates, since the sampling error is dominated by a factor that rank cannot reduce, so the remaining lever is how evaluations are spent. The new results are an exact MSE comparison with antithetic sampling for any objective (Prop. 5.2) and its validation in three regimes (please see G2 in the general response). The audit in G1 of the general response also shows that the gains come from reallocating evaluations, and that LOO-ROLL matches tuned antithetic EGGROLL at 64 or more evaluations per update.

W2 / Q2. What variance reduction remains after standardization, at fixed training checkpoints? We have run experiments to measure this directly, and we see none. At fixed GSM8K checkpoints of Qwen3-1.7B and 8B (base, trained EGGROLL, trained LOO-ROLL), we drew 24 fresh populations per state with learning rate 0 and rebuilt the actual updates of 16 weight matrices from the recorded seeds (App. H.1, H.5).

  • At equal evaluations, the MSE ratio of LOO-ROLL to EGGROLL is 0.86 to 1.13 before standardization and 1.000 (0.997 to 1.003) after.
  • Both estimators have the same cosine with the reference update (about 0.14 to 0.17).
  • The reason is that greedy-decoded rewards have an even part almost as large as the odd part (e/oe/o = 0.77 to 0.97), so the cost of antithetic pairs and their benefit cancel out (Prop. 5.2).

The near-50% reduction holds only where e/o≈0e/o \approx 0, as in transformer sub-blocks. We have therefore corrected every claim.

W3 / Q3. Does any setting show a rank-dependent distortion that affects optimization? We do not observe this in LLM post-training. Across 240 audit runs (8 matrices, layers 0 to 27, five radii, base and trained states), the squared bias of the rank-one mean stays below 0.003 ∥G∥20.003\,\lVert G \rVert^2, and its cosine with the gradient (0.930) matches sampling noise (0.928). Rank 8 never beats rank 1 in reward. The theory does predict, however, where distortion appears: near optima, where the gradient is small compared with σ2/r\sigma^2/r. We have included a new small MLP experiment which confirms this. The distortion follows the 1/r1/r law and raises the final loss of rank-one ES 10 to 30 times (App. H.6). We present Prop. 3.2 as a proof of possibility only.

W4. Leading zeros. We have fixed the presentation in every table.

We hope these additions and clarifications address the soundness concerns. We would welcome a discussion on any point that remains open.

Reviewer_yziVOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work provides a rigorous theoretical analysis of EGGROLL. The authors show that the finite-rank perturbations are not spherically symmetric and EGGROLL damps singular directions unequally and can tilt updates away from the local optimum. The analysis reveals that the excess sampling variance of the rank-one perturbation relative to dense Gaussian ES decays inversely with matrix width, which justifies a lot of the design decisions in the prior work this paper examines. These insights are used to propose a more sample-efficient estimator called LOO-ROLL (Leave-One-Out Low-Rank Evolution Strategies), which replaces the antithetic sampling with unpaired, fully independent perturbations and halves the estimator MSE in the transformer blocks, while matching EGGROLL’s performance in half the wall time (and significantly outperforming it at equal wall time) with no significant loss.

Critical strengths
  • Rigorous Analysis

  • Simple algorithmic improvement with no overhead

Critical weaknesses
  • The pathology does not really apply to any practical post-training setting.

  • The Leave-one-out replacement is a common technique for variance reduction so the contribution has to be stated more clearly and with this fact in mind.

  • The theory should be delivered more intuitively. The point of this line of work is to make it easier and more accessible to train large models and it defeats the purpose if an improvement claim comes in format that is difficult to parse.

Questions for authors

Refer to the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer yziV

We thank the reviewer for the positive review.

W1. The pathology does not apply to practical post-training. We agree for LLM post-training at the radii in use, and we now show why (please see G3 in the general response). The finite-rank correction scales as σ2/r\sigma^2/r, so it is significant only where the gradient is comparatively small. Our LLM audit finds it below detection. A new MLP experiment shows the same correction near an optimum, where it follows the predicted 1/r1/r law and makes rank-one ES stall at 10 to 30 times the final loss of dense ES (App. H.6). The theory therefore explains both why rank one is safe in LLM post-training and when it stops being safe. We present Prop. 3.2 as a proof of possibility.

W2. Leave-one-out is a common technique. We agree that the baseline is common, and we do not claim it as novel (please see G1 in the general response). We propose it because the analysis identifies evaluation allocation, not rank, as the lever for the sampling error. The contributions around it are:

  1. an exact, testable criterion for choosing between antithetic and one-evaluation estimators (Prop. 5.2), which predicted three measured regimes (please see G2 in the general response)
  2. the finding that matched-time accuracy depends mainly on the number of evaluations per update. Spending fewer evaluations per update yields up to +14 points at matched time, LOO-ROLL and a smaller antithetic population can both achieve this.

W3. The theory should be more intuitive. We added a plain-language interpretation paragraph after each theory section to help readability.

We hope that these changes address all of your concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers. All three found the finite-rank analysis rigorous, and all three asked whether the pathologies matter in practice and what LOO-ROLL really adds. We ran the requested experiments with the EGGROLL code of Sarkar et al., using the same seeds and evaluators. The revised PDF contains every result (Sec. 6, App. F.2, App. H). The supplementary material now includes the code for these experiments and a PDF (changes_from_submission.pdf) that marks every removal in red and every addition in blue relative to the submission.

G1. Where do the gains come from, and what does LOO-ROLL add? (all reviewers)

We ran an equal-wall-time sweep over the number of evaluations per update on Qwen3-1.7B GSM8K (App. H.4, Table 23, base model 66.2):

evaluations per updateEGGROLLLOO-ROLL
1682.781.2
3285.084.3
6484.984.8
12883.483.7

The matched-time gains come from cheaper updates. At a fixed time budget, 32 to 64 evaluations per update work best on these tasks, and spending the remaining evaluations on more updates yields about +4 points at 1.7B and +14 points at 8B. At equal cost, LOO-ROLL and EGGROLL perform the same from 64 evaluations per update upward, and LOO-ROLL falls behind below that. We do not claim the leave-one-out baseline as new, and App. A cites its sources. We propose it because our analysis points to it. Sec. 4 shows that the sampling error of EGGROLL is dominated by the factor mn+1mn+1 that every Gaussian ES method shares, so increasing the rank cannot reduce it, and the remaining lever is how evaluations are spent. LOO-ROLL is the principled one-evaluation alternative to antithetic pairs. It keeps the same mean field (Prop. 5.1), and Prop. 5.2 states exactly when it is the more accurate choice. A smaller antithetic population is a second way to reallocate evaluations, and the criterion of G2 tells which of the two to prefer.

G2. Which estimator should we use? An exact criterion. (o442, TxHr)

For any objective, Prop. 5.2 gives the exact MSE of both estimators. We can split each fitness change into a part that changes sign with the perturbation (odd, variance oo) and a part that does not (even, variance ee). Antithetic pairs cancel the even part at twice the cost. At equal cost the LOO-to-antithetic MSE ratio is about 12(1+e/o)\tfrac12(1+e/o), and the baseline adds a term of order 1/M21/M^2 for MM directions. The criterion predicted all three regimes we measured:

  • Transformer sub-blocks, smooth loss, e/o≈0e/o \approx 0: LOO-ROLL halves the MSE.
  • GSM8K rewards, e/oe/o = 0.77 to 0.97: the two tie, and the predicted ratio matches the measured one to within 0.05 (new Table 1). The measured MSE ratio of the actual updates is 0.86 to 1.13, it is 1.000 after standardization, and the cosines with the reference update are equal.
  • Whole-model next-token loss, e/oe/o = 2.2 to 7.7: antithetic pairs win (8B: −14.3% NLL against −12.5%).

oo and ee come for free from the fitness values that EGGROLL already records.

G3. Do the finite-rank pathologies matter in practice? (all reviewers)

Prop. 3.4 says the finite-rank correction has size about σ2/r\sigma^2/r. This is significant only when the gradient is comparatively small.

  • LLMs. A 240-run audit over 8 matrices in layers 0 to 27, five radii and a trained checkpoint finds no rank-dependent bias beyond sampling noise (App. H.2).
  • Small MLP (new App. H.6). Near the optimum of a ReLU or tanh MLP, the rank-rr field differs from dense ES by 124%, 68%, 36% and 20% of its norm for rr = 1, 2, 4, 8, which is the predicted 1/r1/r law. Rank-one ES then stalls at 10 to 30 times the final loss of dense ES.
  • Nonconservativity stays at the noise floor in the MLP. We keep Prop. 3.2 as a proof that it can occur, but we do not claim that it occurs in practice in LLM applications.

G4. MeZO and tuning. (TxHr)

  • GSM8K at 1.7B, matched time: MeZO reaches 83.6, below EGGROLL and LOO-ROLL at 32 to 64 evaluations per update (about 85.0).
  • Next-token prediction: MeZO wins at 0.6B and 1.7B, where each evaluation is a cheap forward pass. EGGROLL with 8 pairs wins at 8B.
  • Tuning: neither EGGROLL nor LOO-ROLL was tuned. Both use the authors' code and launch-script settings (App. F.2).

G5. Presentation. We added a plain-language interpretation paragraph to the end of Sec. 2, 3 and 4 to help readability and fixed the number format in all tables.

Dear Reviewers,

As the rebuttal deadline is approaching, please review the authors' responses and participate in the discussion. Kindly indicate whether your concerns have been addressed and update your scores if appropriate.

Thank you for your time and effort!

Best regards,
--AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.