acceptodds
Under review as a conference paper at ICLR 2027

The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective

Abstract

Weight-perturbation evolution strategies (ES) can fine-tune billion-parameter language models with surprisingly small populations (e.g., ), contradicting classical zeroth-order curse-of-dimensionality intuition. A second, seemingly separate phenomenon also appears under fixed hyperparameters, when the fine-tuning reward often rises, peaks, and then degrades, not just with ES but also with GRPO. Both effects admit a common geometric account: fine-tuning landscapes are *low-dimensional in curvature*, with a few stiff directions embedded in a large, nearly flat bulk. The separation of curvature scales produces (i) heterogeneous time scales that yield rise-then-decay under fixed stochasticity, as captured by a minimal quadratic stochastic-ascent model, while the concentration of curvature produces (ii) degenerate improving updates, where many random perturbations share similar components along the stiff directions. For isotropic perturbations, directions without curvature add nothing to the cost of a random perturbation, so the probability of sampling an improving one does not depend on the ambient dimension , and the scaling question reduces to how the curvature-active dimension grows with . Probing the fine-tuning reward landscapes of GSM8K, ARC-C, and WinoGrande with ES across Qwen2.5-Instruct models (0.5B-7B) reveals that reward-improving perturbations remain accessible with small populations at every scale, and direct Hessian-spectrum measurements on the same models yield effective curvature dimensions on the order of tens, growing sub-linearly with over the tested scales and occupying a vanishing fraction of parameter space. These results reconcile ES scalability with non-monotonic training dynamics under a single geometric account, and they open the door to a broad class of fine-tuning algorithms, beyond policy-gradient RL, that worst-case zeroth-order theory had ruled out as infeasible at scale.

Primary area:
optimization
Keywords:
Evolution strategies, Zeroth-order optimization, LLM fine-tuning, Loss landscape geometry, Non-monotonic training dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DH1QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies two phenomena in LLM fine-tuning: why a small number of random parameter perturbations can find reward improvements in a high-dimensional parameter space, and why training rewards can rise and then decline under fixed hyperparameters. The authors propose a local geometric explanation based on the interaction between curvature and optimization noise. The theoretical analysis models stochastic updates on a local quadratic objective, deriving the dynamics of mean convergence and variance accumulation along different curvature directions and the conditions under which expected reward peaks. For a single random perturbation, the paper further shows that, under a quadratic approximation with strictly low-rank curvature, the improvement probability can be expressed through low-dimensional projections, the norm of the orthogonal gradient component, and the relevant curvature quantities, without explicit dependence on the total parameter dimension. The experiments include ES and GRPO training trajectories, random-perturbation distributions for four Qwen model sizes on GSM8K, ARC-Challenge, and WinoGrande, and curvature measurements using zeroth-order Hessian-vector products and stochastic Lanczos quadrature. The authors report that, at suitable perturbation scales among those examined, small candidate populations can find reward improvements, the gains from increasing the population diminish, and the corresponding curves do not shift systematically to the right as model size increases. The reported curvature effective ranks range from approximately 6.7 to 25.9.

Critical strengths
  1. A concrete and testable explanation for non-monotonic reward trajectories Section 4 and Appendix A decompose expected reward changes into mean improvement and variance accumulation, showing how directional timescales, initialization, and noise strength jointly determine whether a peak occurs. This goes beyond describing training curves: non-monotonic expected reward can arise even with a fixed objective and stable updates. Local quadratic models and the associated stochastic optimization tools are established, but the paper applies them to specific observations in LLM fine-tuning and identifies conditions that produce the phenomenon. As a candidate mechanism, this analysis helps organize the empirical observations and provides a concrete basis for further validation. The paper is clearly organized, with a coherent progression from observations to analysis; the figures and appendices make the main ideas and derivations easy to follow.
  2. Independent empirical insight into perturbation gains under limited budgets Section 5, Figure 6, and Appendix G examine improvement probabilities, reward-change distributions, and best-of-N gains across four model sizes and three tasks. Within each task, the experiments use the same prompt pool and decoding settings, and report both raw gains and gains normalized by remaining headroom. These choices provide a reasonably clear basis for comparing the evaluated model conditions. Prior work has established that few random directions can be useful in high-dimensional LLM optimization and that effective curvature can be more relevant than total parameter dimension. Here, the experiments characterize how the level and distribution of gains vary with model size, task, and perturbation scale, beyond the generic observation of diminishing returns from more candidates. Together with the transient analysis, these empirical findings support my weak-accept recommendation. Their value does not depend on accepting the precise effective-rank estimates or treating the proposed explanation as a verified unified mechanism.
Critical weaknesses
  1. The connection from local theory to actual training remains incomplete The paper aims to use curvature and noise to unify the explanation of non-monotonic training rewards and the effectiveness of small perturbation populations. Section 4 and Appendix A support a local mechanism that produces reward peaks, and Figures 1 and 10 show compatible training phenomena. However, agreement in curve shape does not establish that this mechanism primarily drives the observed decline. Noise in actual ES and GRPO depends on the state and algorithmic settings, and the analysis does not sufficiently connect these quantities to the simplified model. Two statements connecting the results also require narrowing. First, non-monotonic rewards do not require low-dimensional curvature, so a peak alone does not identify low-dimensional curvature. Second, Section 5 treats improvement by at least one candidate as a necessary condition for ES to make progress. This is not generally true: relative candidate rewards can produce a beneficial aggregated update even when every candidate performs worse than the current point. Candidate improvement probability therefore characterizes the accessibility of improvements through random search, but is not directly a necessary condition for a successful ES update. These issues primarily limit the strength of the unified explanation. The possible mechanism demonstrated by the local model and the direct distributional observations remain valuable, but do not justify stronger attribution of the actual training dynamics to this mechanism.
  2. The quantitative curvature claim needs further clarification and calibration Section 5 and Appendix H use effective-rank measurements as direct evidence for the geometric explanation. The authors report five random seeds and check Lanczos depth and prompt count, which provide support for the measurements. Nevertheless, increasing the prompt count changes the effective rank by approximately 21–43%. The baseline construction, reuse of random perturbations, and effective-rank calculation for a signed spectrum also require clearer specification. These details affect the relationship between finite-sample estimates and spectral concentration in the target Hessian. The eigenvalue magnitudes of approximately 1012 in Figure 17 also need to be explained in relation to the reward range, smoothing scale, and parameter coordinates. The cross-entropy Hessian measurements in Appendix H.4 provide complementary observations for a different objective, but do not directly calibrate the reward-Hessian estimator. The established limitations are insufficient support for the precise quantitative conclusion and missing implementation details that affect its interpretation. Whether estimation bias is present, and its magnitude, remain open questions. This limits confidence in the direct geometric measurements; it does not establish that the estimator artificially creates low effective rank, nor does it automatically invalidate the distribution experiments.
Questions for authors
  1. Applicability of the theoretical analysis Section 4 and Appendix A give conditions for a reward peak. Beyond agreement in curve shape, is there existing evidence connecting the reward decline in actual ES or GRPO training to the proposed variance-accumulation process? Please clarify which aspects of the local model’s explanation of actual training have direct support and which remain hypotheses.
  2. Reliability of the curvature estimation method Please specify the effective-rank calculation, the construction of the baseline in Eq. (55), and how perturbation samples are reused across matrix-vector products. Please also explain the eigenvalue scale in Figure 17 in relation to the reward range and Gaussian smoothing scale. Beyond the existing checks of random seeds, Lanczos depth, and prompt count, are there calibration results assessing the effect of finite perturbation samples on the effective-rank estimate, to support the quantitative claim that the effective rank is in the tens?
  3. Robustness of the experimental results For the distribution experiments, please provide or locate the prompt-bootstrap and prompt-subset checks mentioned in Appendix G.6, particularly for settings with small gains. These results would help establish how far the observations on the fixed prompt pools generalize. For the training trajectories, please clarify how the z-score normalization described in the main text and the baseline centering described in the appendix are used in the actual updates. Please also report the number of independent runs and the meaning of the shaded regions in the main curves, so that the consistency of the rise-then-decline observation can be assessed.
3 comments

We thank the reviewer for the careful and supportive review.

Summary of the response. As the reviewer notes, the value of the paper does not depend on the precise effective-rank estimates. The response confirms this point. Every calibration question about the curvature estimator has an exact answer, and none of these answers changes the conclusions of the paper (W2 / Q2). The experimental results hold under resampling of the candidates and the prompts, along the training trajectory, and on a second task (Q3). Hence the evidence that small populations suffice as models grow is now stronger. This result matters beyond ES. Low-dimensional curvature is a property of the fine-tuning landscape itself, and hence opens up a broad class of algorithms that the classical worst-case analysis of zeroth-order optimization had ruled out at scale, with interesting tradeoffs to be studied in the future. These points are illustrated below in the order of the review.

W1 / Q1: Connection from the local theory to actual training.

"Please clarify which aspects of the local model’s explanation of actual training have direct support and which remain hypotheses."

The local model shows that the mechanism (heterogeneous Hessian spectrum) is sufficient to produce rise-then-decay, not that it is the only cause. We have revised Section 4 accordingly to separate what is observed from what the hypothesis is. In short,

Observed in actual training.

  • (i) The reward rises and then decays under fixed hyperparameters, in ES (on GSM8K and Countdown) and in GRPO (Figs. 1, 10).
  • (ii) The decay changes with the population size as the local model predicts, i.e. larger populations delay the reward peak and reduce the decay.

Hypothesis. The noise sits along the flat directions of the landscape, and the decay in GRPO has the same cause as the decay in ES. Appendix C extends the analysis to general anisotropic noise, including the noise of GRPO.

Hence, beyond agreement in curve shape, the population-size dependence (ii) supports the mechanism in actual training, and GRPO shows that the pattern is consistent across algorithms.

On the two statements raised in the review:

"a peak alone does not identify low-dimensional curvature."

Indeed, rise-then-decay is evidence for heterogeneous curvature, i.e. separated curvature scales. Low-dimensional curvature is tested separately in Section 5, by the best-of-NN experiments and the curvature measurements (see W2 / Q2 below).

"Section 5 treats improvement by at least one candidate as a necessary condition for ES to make progress."

Actually ES can move uphill even when every candidate is worse than the current point, as recently shown by Dailey et al. (2026, arXiv:2611.04113), which the revised Section 5 now cites. Best-of-NN measures how accessible improvements are to random search, which is the quantity that the scaling argument needs.

We have revised Sections 4 and 5 accordingly to make both statements precise. Neither statement affects the results of the paper, which rest on heterogeneous curvature for rise-then-decay and on accessibility for scaling.

(Continued in Part 2/2.)

(Continued from Part 1/2.)

W2 / Q2: The curvature claim and the reliability of the estimator.

"Please specify the effective-rank calculation, the construction of the baseline in Eq. (55), and how perturbation samples are reused across matrix-vector products."

Each question has an exact answer, given below, and none of these answers changes the conclusions of the paper.

  • Baseline in Eq. (55). The baseline is the mean of the NN perturbed rewards, so the −v-v term in Eq. (55) cancels exactly.
  • Reuse of perturbations. The same N=30N=30 perturbations serve every matrix-vector product, so the estimated matrix has rank of at most NN.
  • Effective rank of a signed spectrum. It is computed from the magnitudes of the eigenvalues, ∑i∣λi∣/max⁡i∣λi∣\sum_i|\lambda_i|/\max_i|\lambda_i|, with no thresholding (see also the response to Reviewer HZLX, W1).
  • The 101210^{12} eigenvalue scale in Fig. 17. For a reward in [0,1][0,1] and a smoothing scale σ = 10⁻³, the true Hessian eigenvalues are at most 0.5/σ2≈4.8×1050.5/\sigma^2\approx 4.8\times10^5. The 101210^{12} scale comes from the rank-NN structure of the estimator, as each of its eigenvalues is roughly a reward difference times D/(Nσ2)D/(N\sigma^2).
  • Calibration for finite perturbation samples. The estimated effective rank is bounded by NN. This bound also explains why the estimate changes by 21–43% with the prompt count but does not change with the Lanczos depth.

We have revised the paper accordingly (Section 5, Appendix H.1) to state this rank bound, and to treat the zeroth-order estimate as a qualitative indicator of concentrated curvature rather than as a measurement of the curvature-active dimension kk. The claims never rested on this number, but instead on two kinds of evidence for low-dimensional curvature:

  • Best-of-NN is the operational measure of accessibility. It tests directly, without any curvature estimate, whether a small population still finds reward improvements as models grow. The result is robust to resampling and holds along the training trajectory (see the response to Reviewer HZLX, W2 and Q2).
  • The curvature measurements explain why improvements stay accessible. The cross-entropy Hessian is measured independently of the reward estimator. It uses exact Hessian-vector products, so it has no rank limit, and it puts the curvature-active part 6–8 orders of magnitude below the parameter count DD. Gradient concentration, measured up to 14B, agrees.

Hence the conclusions do not depend on the rank ceiling of the zeroth-order estimator.

Q3: Robustness of the experimental results.

"please provide or locate the prompt-bootstrap and prompt-subset checks mentioned in Appendix G.6"

The best-of-NN results are robust to resampling, and rise-then-decay is reproduced across population sizes, algorithms and tasks.

  • Bootstrap and subset checks (new App. G.6). We ran the checks on the 36 of the 60 settings in Fig. 6 where the best of 30 candidates improves on the base model. When the 240 candidates are resampled, the improvement stays significant in 35 of the 36 settings. When the 320 prompts are resampled, it stays significant in 33 of the 36, and the three exceptions are all at the smallest model (0.5B). When the best candidate is selected on half of the prompts and tested on the other half, it still improves in 24 of the 36 settings, including the 7B model on every task. For the settings with small gains, most improving candidates gain at least two prompts, i.e. more than a single-answer flip.
  • Normalization. All ES training runs use the z-score update of Eq. (2). Baseline centering is used only in the curvature estimator of Eq. (55) and not in the training updates, as the revised Appendices F and G.12 now state.
  • Runs and shading. Fig. 1 shows one run per population size (ES) or group size (GRPO). The shaded band is the variation of the reward across population members (ES) or group samples (GRPO), as the revised caption of Fig. 1 now states. Showing one run per setting does not mean that the phenomenon is inconsistent across runs. Rise-then-decay recurs at every population size and in both algorithms; Countdown provides independent confirmation on a second task, where ES shows the same pattern at every population size in both training and held-out test reward (App. F, Fig. 10).

Hence the results are robust across resampling, runs and tasks, as the revised caption of Fig. 1 and the new Appendix G.6 make clear.

We thank the reviewer for the precise questions. We hope that the response and the revision address all of the concerns raised.

Thank you for your detailed response, which has addressed my concerns. As my original rating was already positive, I will maintain my current rating.

Reviewer_HZLXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why small-population weight-perturbation evolution strategies (ES) can fine-tune billion-parameter LLMs and why training reward under fixed stochasticity often exhibits a rise–then–decay pattern in both ES. The authors hypothesize fine-tuning landscapes are low-dimensional in curvature, with a small number of stiff directions embedded in a largely flat parameter space. This geometric structure keeps reward-improving perturbations accessible as model size increases and yields non-monotonic training dynamics. The paper substantiates this hypothesis through a quadratic stochastic-ascent analysis, a dimension-independent accessibility result under low-rank curvature, and empirical measurements of reward-improving perturbations and Hessian spectra across model scales.

Critical strengths
  1. This paper addresses the scalability of zeroth-order methods for LLM fine-tuning and explains the often-observed non-monotonic training dynamics under stochastic updates.

  2. The findings have clear practical relevance. By showing that reward-improving perturbations remain accessible with a small fixed population across model scales and tasks, the paper challenges pessimistic worst-case intuitions and broadens the design space for backpropagation-free fine-tuning.

  3. The empirical evaluation examines the central hypothesis from several complementary angles, including Hessian-spectrum measurements, best-of-N accessibility, surrogate-loss curvature, gradient concentration, and curvature changes along an ES trajectory.

Critical weaknesses
  1. Several quantities central to the empirical claims are insufficiently defined. In particular, the manuscript does not provide explicit formulas for the reported effective rank or participation ratio, nor does it specify how positive, negative, and near-zero eigenvalues are handled.

  2. The static accessibility experiments use fixed candidate pools and prompt sets; while practical, this design may entangle landscape geometry with finite-sample evaluation variance and reward discretization effects.

  3. DeComFL(ICLR 2025) establishes dimension-independent convergence rates for zeroth-order optimization under a k-effective-rank Hessian assumption, whereas Proposition 5.1 shows that the probability of sampling an improving perturbation is independent of the ambient dimension under exact rank-k curvature. It would be helpful to clarify the relationship between these two results.

Questions for authors
  1. Can Proposition 5.1 be extended to an approximate low-rank setting, yielding bounds on N as a function of an “\epsilon-effective rank” (e.g., tail-trace beyond top-k), thereby connecting the theory more tightly to measured spectra?
  2. The best-of-N accessibility experiments are conducted around the base checkpoints. Does the accessibility of improving perturbations remain comparable after ES moves away from initialization, particularly near the reward peak? A best-of-N evaluation at several checkpoints along one representative trajectory would help clarify whether this property is local to initialization or persists throughout training.
4 comments

We thank the reviewer for the detailed and constructive review.

Summary of the response. Thanks to the reviewer's suggestions, we now report three extended results in the revised paper. (i) Proposition 5.1 holds under approximate low rank, which ties the theory to the measured spectra (Q1). (ii) Reward-improving perturbations stay accessible along the training trajectory, including at the reward peak (Q2). (iii) The best-of-N gains exceed finite-sample variance and reward discretization (W2). Hence the central claim, i.e. that a small fixed population suffices as models grow, now rests on firmer theory and evidence. This result matters beyond ES. Low-dimensional curvature is a property of the fine-tuning landscape itself, and hence opens up a broad class of algorithms that the classical worst-case analysis of zeroth-order optimization had ruled out at scale, with interesting tradeoffs to be studied in the future. These points are further illustrated below, together with answers to the other concerns.

W1: Definitions of the spectral statistics.

"the manuscript does not provide explicit formulas for the reported effective rank or participation ratio"

Both statistics are computed from the magnitudes of the eigenvalues, with no thresholding. For Hessian eigenvalues {λi}\lbrace\lambda_i\rbrace:

  • effective rank =∑i∣λi∣/max⁡i∣λi∣= \sum_i|\lambda_i| / \max_i|\lambda_i|;
  • participation ratio =(∑i∣λi∣)2/∑iλi2= (\sum_i|\lambda_i|)^2/\sum_i\lambda_i^2.

Since both statistics use magnitudes, positive and negative curvature both count. Near-zero eigenvalues are not thresholded, i.e. they are kept and contribute little. For the cross-entropy Hessian HlossH_{\text{loss}} the positive and negative parts of the spectrum are also reported separately. Fig. 17 shows these statistics divided by the parameter count DD. We have revised the paper accordingly to define both statistics (Section 5, Appendix H.1).

W2: Accessibility measures.

"this design may entangle landscape geometry with finite-sample evaluation variance and reward discretization effects."

The best-of-N improvements exceed both finite-sample variance and reward discretization, at every model size. Fig. 6 covers 60 settings (three tasks, four model sizes, five perturbation scales). We ran the checks below on the 36 settings where the best of 30 candidates improves on the base model. The other 24 are large-perturbation settings where perturbations mostly degrade performance.

  • Candidate-pool variance. When the 240 candidate perturbations are resampled, the best-of-30 gain stays statistically significant (95% lower bound above zero) in 35 of the 36 settings.
  • Prompt variance. When the 320 evaluation prompts are resampled, the gain stays significant in 33 of the 36 settings. The three exceptions are all at the smallest model (0.5B).
  • Selection bias. When the best candidate is selected on one half of the prompts and evaluated on the other, unseen half, it still improves on the base model in 24 of the 36 settings, including the 7B model on every task.
  • Discretization. Most improving candidates gain at least two prompts, so the gains are not single-answer flips.

Hence the improvements reflect the landscape and exceed finite-sample variance and discretization. They hold at every model size, with no trend of getting harder as models grow. The full per-task tables are in the new Appendix G.6.

A related clarification concerns what is measured. The claim is that improving perturbations remain accessible with a small population as models grow, without guaranteeing the size of the improvement. The improvement size depends on the pretrained checkpoint, so the claim is tested with measures that factor the checkpoint out, i.e. gains normalized by the remaining headroom and the saturation population N90N_{90}. We have revised Section 5 accordingly to make this distinction clear.

(Continued in Part 2/2.)

(Continued from Part 1/2.)

W3: Relation to DeComFL.

"It would be helpful to clarify the relationship between these two results."

The two results are complementary. DeComFL concerns the average progress per step, while Proposition 5.1 concerns the chance that a random perturbation improves the reward. DeComFL shows that the average progress per step of zeroth-order SGD depends on the effective rank of the Hessian rather than on DD. Proposition 5.1 is about the chance that a random perturbation improves, which determines whether a small population can find an improvement. Both results come from the same fact, i.e. that a random perturbation pays a curvature cost whose average is tr(C)\mathrm{tr}(C), where CC is the curvature matrix. The extension in Q1 below brings Proposition 5.1 to the same approximate-low-rank setting as DeComFL, and recovers the effective-rank condition of DeComFL as a special case. Hence the two results are complementary, as per-step convergence rates do not capture accessibility. We have revised the paper accordingly to cite DeComFL and state this relationship (Section 2, Appendix I).

Q1: Proposition 5.1 under approximate low rank.

"Can Proposition 5.1 be extended to an approximate low-rank setting, yielding bounds on N as a function of an “\epsilon-effective rank” [...]?"

Yes. In the revision, Proposition 5.1 is extended to approximate low rank, and the dimension DD only enters through the total curvature in the tail of the spectrum. The curvature is split into its top-kk part and the remaining tail C⊥C_\perp. A random perturbation pays an extra tail cost 12σ2ε⊤C⊥ε\tfrac12\sigma^2\varepsilon^\top C_\perp\varepsilon, which concentrates around its mean 12σ2τ\tfrac12\sigma^2\tau, where τ=tr(C⊥)\tau = \mathrm{tr}(C_\perp) is the total tail curvature. Hence the tail acts as an extra margin that the improvement must clear,

pimp≈pk(12σ2τ),p_{\mathrm{imp}} \approx p_k\big(\tfrac12\sigma^2\tau\big),

where pimpp_{\mathrm{imp}} is the probability that a random perturbation improves the reward, and pk(s)p_k(s) is the DD-free improvement probability of Proposition 5.1 with margin ss.

Therefore, DD only enters through the tail curvature τ\tau. If τ\tau is small, accessibility stays dimension-free, and if τ\tau grows with DD, the classical curse of dimensionality returns. This extension gives three things the reviewer asked for:

  • the "ϵ\epsilon-effective rank", defined as the smallest kk whose tail cost is below ϵ\epsilon times the signal;
  • a bound on the population size NN in terms of this rank;
  • the effective-rank condition of DeComFL as a special case, which connects to W3 above.

We have revised the paper accordingly. Section 5 states the extension, and the new Appendix I gives the two-sided bound with its explicit fluctuation terms (Proposition I.1).

Q2: Best-of-N away from initialization.

"Does the accessibility of improving perturbations remain comparable after ES moves away from initialization, particularly near the reward peak?"

Yes. Improving perturbations remain accessible at every checkpoint along the trajectory, including at the reward peak. We ran the best-of-N probe every 100 iterations along the 1.5B GSM8K ES run of Fig. 1, with the same setup as Fig. 6. The accuracy of the current model, R0R_0, peaks at iteration 300 and then decays. The table below is for perturbation scale σ = 10⁻⁴, and σ = 3×10⁻⁴ gives similar results.

IterationR0R_0Fraction of perturbations that improveBest-of-30 gain [lower 95%]
00.8000.870.036 [0.023]
300 (peak)0.8470.310.014 [0.008]
5000.8250.610.021 [0.011]
800 (decay)0.7840.900.038 [0.027]
10000.7940.520.019 [0.013]

The best-of-30 gain is significantly above zero at every checkpoint, so accessibility is not local to initialization. The trend along the run also matches the rise-then-decay mechanism: improvements are hardest to find at the peak, where the easy gains are used up, and easier again during the decay, where the accumulated drift can be undone by a random perturbation. The full trajectory, with all 11 checkpoints and three perturbation scales, is in the new Appendix G.7 (Table 3).

We thank the reviewer for the specific and actionable suggestions. We hope that the response and the revision address all of the concerns raised.

Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.

Thank you again for your time and consideration!

We would like to thank the reviewer again for the helpful and constructive feedback! As the rebuttal period is coming to an end, we are happy to address any remaining concerns the reviewer might have.

Thank you!

Reviewer_hbQAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a variance–curvature perspective to explain why small-population evolution strategies can fine-tune large language models and why stochastic fine-tuning can exhibit rise–then–decay behavior. It combines a local quadratic model, curvature-spectrum estimates, and random-perturbation probes to argue that optimization is governed by a small, sub-linearly growing set of curvature-active directions.

Critical strengths
  1. The attempt to connect optimization scalability and non-monotonic training dynamics through a common geometric framework is interesting and conceptually useful.
  2. The paper goes beyond qualitative speculation by providing analytical results, curvature measurements, and perturbation experiments across several model sizes.
Critical weaknesses
  1. The framework’s explanatory power for practical training remains insufficiently established. The quadratic model shows that heterogeneous curvature and persistent noise can produce rise–then–decay, but this does not establish that the mechanism drives the observed neural-training behavior. Although the appendix discusses anisotropic policy-gradient noise, its relationship to actual GRPO updates is not empirically validated.

  2. The practical implications remain largely speculative. Noise scheduling, adaptive step sizes, and curvature-based stopping are proposed without intervention experiments. Demonstrating that a theory-guided intervention outperforms a conventional schedule would substantially strengthen the framework’s value beyond reproducing qualitative phenomena.

  3. The curvature measurements require further validation. If the centered-reward HVP estimator reuses a fixed set of N=30 perturbations, the estimated operator is itself rank-limited by the perturbation span. If perturbations are resampled for each matvec, the reliability of standard Lanczos spectral estimation under these stochastic products needs justification. The reported checks vary Lanczos depth and prompt count, but not the HVP sampling budget, leaving potential estimator-induced low dimensionality unresolved.

  4. The scaling claim relies on four sizes from one model family, with reward-curvature measurements restricted to GSM8K and small prompt subsets. Effective-rank estimates also change by approximately 21–43% when the prompt subset is doubled. This evidence supports a local empirical trend, but is insufficient to establish that sub-linear curvature growth generalizes across model families, tasks, or larger scales.

Questions for authors

Can the proposed curvature–noise statistics predict the onset of reward degradation, or guide an intervention that improves training under a matched compute budget?

5 comments

We thank the reviewer for the careful review.

Summary of the response. The main message of the paper does not hinge on the toy model or on the zeroth-order curvature estimator. In real training runs, the reward decay responds to the noise level as the proposed mechanism predicts (W1, W2, Q). The scaling conclusion, i.e. that a small fixed population suffices as models grow, is supported by the best-of-N experiments on three tasks and four model sizes + two other curvature measurements that have no sampling budget (W3, W4). This result also matters beyond ES. Low-dimensional curvature is a property of the fine-tuning landscape itself, and hence opens up a broad class of algorithms that the classical worst-case analysis of zeroth-order optimization had ruled out at scale, with interesting tradeoffs to be studied in the future. These points are illustrated below in the order of the review.

W1: Explanatory power of the quadratic model.

"this does not establish that the mechanism drives the observed neural-training behavior."

The quadratic model is used to show that the mechanism is sufficient to produce rise-then-decay, which is the scope of the claim in the paper. Two ingredients, heterogeneous curvature + persistent noise, are enough on their own to reproduce the observed rise-then-decay. The paper does not claim that these two ingredients are the only cause of the decay in LLM fine-tuning, and other contributing effects are discussed in Appendices C–E.

Two observations in real training support the mechanism beyond the shape of the reward curve:

  • The decay changes with the population size in the direction that the model predicts, i.e. larger populations delay the reward peak and reduce the decay (Fig. 1).
  • GRPO shows the same rise-then-decay pattern under a very different, first-order method, which suggests that the pattern comes from the landscape rather than from the algorithm. Appendix C extends the analysis to general anisotropic noise, which covers the noise of GRPO.

Hence the scope of the quadratic model matches the claim made in the paper. We have revised Section 4 accordingly to make this scope clearer.

W2: Practical implications.

"The practical implications remain largely speculative."

The paper already contains evidence that these implications hold, since each of them is a way of lowering the noise, and the experiments show that lower noise reduces the decay. Fig. 1 shows that the decay is controlled by the noise level in real ES training. Larger populations, which give less noisy updates, delay the reward peak and reduce the decay after it. The same holds on a second task, Countdown (Fig. 10), and the toy model reproduces this dependence on the noise level (Fig. 3). Each suggested technique follows directly from this dependence:

  • Early stopping keeps the model at the reward peak.
  • Increasing the population size NN, or annealing the learning rate α\alpha or the perturbation scale σ\sigma after the peak all lower the noise, which Figs. 1 and 3 show reduces the decay.

Hence the experiments in the paper support these implications, and in practice these techniques also eliminated the decay. We have revised Section 6 accordingly to tie these implications to this evidence.

(Continued in Part 2/2.)

(Continued from Part 1/2.)

W3: Curvature measurements and the HVP sampling budget.

"The reported checks vary Lanczos depth and prompt count, but not the HVP sampling budget, leaving potential estimator-induced low dimensionality unresolved."

The conclusion that curvature is low-dimensional does not depend on the zeroth-order estimator, since two other curvature measurements with no sampling budget agree with it. Each curvature measurement method has its limitations, which is why the paper uses three complementary ones. On the three points raised about the zeroth-order Hessian-vector-product (HVP) estimator:

  • Fixed vs. resampled perturbations. The same N=30N=30 antithetic perturbations are reused for every matrix-vector product, so the measured operator has rank at most NN.
  • Lanczos under resampled products. Since the perturbations are fixed, every Lanczos step applies the same deterministic matrix. Standard stochastic Lanczos quadrature therefore applies.
  • The HVP sampling budget. Varying NN only moves the rank ceiling of the estimator, so a budget sweep cannot settle whether the true curvature is low-dimensional.

We have revised the paper accordingly (Section 5, Appendix H.1) to state this rank bound, and to treat the zeroth-order effective-rank estimate as a qualitative indicator of concentrated curvature rather than as a value of the curvature-active dimension kk. The two other measurements have no sampling budget, and both show that curvature is concentrated:

  • Autodiff cross-entropy Hessian. It uses exact Hessian-vector products, so it has no rank limit. Its effective rank is 6–8 orders of magnitude below the parameter count DD for the 0.5B–3B models.
  • Gradient concentration, measured up to 14B. 90% of the gradient sits in 4–11% of the parameters, and this fraction shrinks by ~13× from 0.5B to 14B.

Hence estimator-induced low dimensionality is not what drives the conclusion. Further, the theory does not need a precise value of kk. Following a suggestion of Reviewer HZLX, Proposition 5.1 is extended in the revision so that it only requires the total curvature in the tail of the spectrum to be small (see the response to Reviewer HZLX, Q1).

W4: Scope of the scaling claim.

"The scaling claim relies on four sizes from one model family, with reward-curvature measurements restricted to GSM8K and small prompt subsets."

The scaling claim rests on the best-of-NN experiments and on gradient concentration, which cover three tasks and model sizes up to 14B. The 21–43% change with the prompt subset and the restriction to GSM8K both apply only to the zeroth-order effective-rank estimate, and the scaling claim does not rest on that estimate (see W3 above). The claim is that a small population keeps finding reward-improving perturbations as models grow. It is tested directly by best-of-NN on three tasks (GSM8K, ARC-C, WinoGrande) and four model sizes (0.5B–7B), and the result is robust to resampling the prompts and the candidate perturbations (see the response to Reviewer HZLX, W2). Gradient concentration, measured up to 14B, agrees.

On model families, small-population ES has also been shown to fine-tune the Llama family successfully (Qiu et al., 2026), which is consistent with the effect not being specific to Qwen. Hence the scaling claim does not hinge on the GSM8K-only estimate, and rests on substantial evidence across tasks and scales. We have revised Section 6 accordingly to make the tested range explicit.

Q: Predicting the onset of degradation / intervention.

"Can the proposed curvature–noise statistics predict the onset of reward degradation, or guide an intervention that improves training under a matched compute budget?"

The framework guides an intervention at a matched compute budget, i.e. lowering the noise after the reward peak. Annealing the learning rate α\alpha or the perturbation scale σ\sigma after the peak lowers the noise without any additional reward evaluations, and Figs. 1 and 3 show that lower noise reduces the decay.

On predicting the onset, the paper does not claim that the noise statistics predict when the decay begins, since the onset depends on quantities that cannot be measured during training. The model instead explains why a reward peak exists and how the peak moves with the noise level, since the decay comes from noise accumulating once the curvature-driven gains are used up. We have revised Section 6 accordingly to make these points clear.

We thank the reviewer for the constructive feedback. We hope that the response and the revision address all of the concerns raised.

Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.