- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies two phenomena in LLM fine-tuning: why a small number of random parameter perturbations can find reward improvements in a high-dimensional parameter space, and why training rewards can rise and then decline under fixed hyperparameters. The authors propose a local geometric explanation based on the interaction between curvature and optimization noise. The theoretical analysis models stochastic updates on a local quadratic objective, deriving the dynamics of mean convergence and variance accumulation along different curvature directions and the conditions under which expected reward peaks. For a single random perturbation, the paper further shows that, under a quadratic approximation with strictly low-rank curvature, the improvement probability can be expressed through low-dimensional projections, the norm of the orthogonal gradient component, and the relevant curvature quantities, without explicit dependence on the total parameter dimension. The experiments include ES and GRPO training trajectories, random-perturbation distributions for four Qwen model sizes on GSM8K, ARC-Challenge, and WinoGrande, and curvature measurements using zeroth-order Hessian-vector products and stochastic Lanczos quadrature. The authors report that, at suitable perturbation scales among those examined, small candidate populations can find reward improvements, the gains from increasing the population diminish, and the corresponding curves do not shift systematically to the right as model size increases. The reported curvature effective ranks range from approximately 6.7 to 25.9.
Critical strengths
- A concrete and testable explanation for non-monotonic reward trajectories Section 4 and Appendix A decompose expected reward changes into mean improvement and variance accumulation, showing how directional timescales, initialization, and noise strength jointly determine whether a peak occurs. This goes beyond describing training curves: non-monotonic expected reward can arise even with a fixed objective and stable updates. Local quadratic models and the associated stochastic optimization tools are established, but the paper applies them to specific observations in LLM fine-tuning and identifies conditions that produce the phenomenon. As a candidate mechanism, this analysis helps organize the empirical observations and provides a concrete basis for further validation. The paper is clearly organized, with a coherent progression from observations to analysis; the figures and appendices make the main ideas and derivations easy to follow.
- Independent empirical insight into perturbation gains under limited budgets Section 5, Figure 6, and Appendix G examine improvement probabilities, reward-change distributions, and best-of-N gains across four model sizes and three tasks. Within each task, the experiments use the same prompt pool and decoding settings, and report both raw gains and gains normalized by remaining headroom. These choices provide a reasonably clear basis for comparing the evaluated model conditions. Prior work has established that few random directions can be useful in high-dimensional LLM optimization and that effective curvature can be more relevant than total parameter dimension. Here, the experiments characterize how the level and distribution of gains vary with model size, task, and perturbation scale, beyond the generic observation of diminishing returns from more candidates. Together with the transient analysis, these empirical findings support my weak-accept recommendation. Their value does not depend on accepting the precise effective-rank estimates or treating the proposed explanation as a verified unified mechanism.
Critical weaknesses
- The connection from local theory to actual training remains incomplete The paper aims to use curvature and noise to unify the explanation of non-monotonic training rewards and the effectiveness of small perturbation populations. Section 4 and Appendix A support a local mechanism that produces reward peaks, and Figures 1 and 10 show compatible training phenomena. However, agreement in curve shape does not establish that this mechanism primarily drives the observed decline. Noise in actual ES and GRPO depends on the state and algorithmic settings, and the analysis does not sufficiently connect these quantities to the simplified model. Two statements connecting the results also require narrowing. First, non-monotonic rewards do not require low-dimensional curvature, so a peak alone does not identify low-dimensional curvature. Second, Section 5 treats improvement by at least one candidate as a necessary condition for ES to make progress. This is not generally true: relative candidate rewards can produce a beneficial aggregated update even when every candidate performs worse than the current point. Candidate improvement probability therefore characterizes the accessibility of improvements through random search, but is not directly a necessary condition for a successful ES update. These issues primarily limit the strength of the unified explanation. The possible mechanism demonstrated by the local model and the direct distributional observations remain valuable, but do not justify stronger attribution of the actual training dynamics to this mechanism.
- The quantitative curvature claim needs further clarification and calibration Section 5 and Appendix H use effective-rank measurements as direct evidence for the geometric explanation. The authors report five random seeds and check Lanczos depth and prompt count, which provide support for the measurements. Nevertheless, increasing the prompt count changes the effective rank by approximately 21–43%. The baseline construction, reuse of random perturbations, and effective-rank calculation for a signed spectrum also require clearer specification. These details affect the relationship between finite-sample estimates and spectral concentration in the target Hessian. The eigenvalue magnitudes of approximately 1012 in Figure 17 also need to be explained in relation to the reward range, smoothing scale, and parameter coordinates. The cross-entropy Hessian measurements in Appendix H.4 provide complementary observations for a different objective, but do not directly calibrate the reward-Hessian estimator. The established limitations are insufficient support for the precise quantitative conclusion and missing implementation details that affect its interpretation. Whether estimation bias is present, and its magnitude, remain open questions. This limits confidence in the direct geometric measurements; it does not establish that the estimator artificially creates low effective rank, nor does it automatically invalidate the distribution experiments.
Questions for authors
- Applicability of the theoretical analysis Section 4 and Appendix A give conditions for a reward peak. Beyond agreement in curve shape, is there existing evidence connecting the reward decline in actual ES or GRPO training to the proposed variance-accumulation process? Please clarify which aspects of the local model’s explanation of actual training have direct support and which remain hypotheses.
- Reliability of the curvature estimation method Please specify the effective-rank calculation, the construction of the baseline in Eq. (55), and how perturbation samples are reused across matrix-vector products. Please also explain the eigenvalue scale in Figure 17 in relation to the reward range and Gaussian smoothing scale. Beyond the existing checks of random seeds, Lanczos depth, and prompt count, are there calibration results assessing the effect of finite perturbation samples on the effective-rank estimate, to support the quantitative claim that the effective rank is in the tens?
- Robustness of the experimental results For the distribution experiments, please provide or locate the prompt-bootstrap and prompt-subset checks mentioned in Appendix G.6, particularly for settings with small gains. These results would help establish how far the observations on the fixed prompt pools generalize. For the training trajectories, please clarify how the z-score normalization described in the main text and the baseline centering described in the appendix are used in the actual updates. Please also report the number of independent runs and the meaning of the shaded regions in the main curves, so that the consistency of the rise-then-decline observation can be assessed.
3 comments
(Continued from Part 1/2.)
W2 / Q2: The curvature claim and the reliability of the estimator.
"Please specify the effective-rank calculation, the construction of the baseline in Eq. (55), and how perturbation samples are reused across matrix-vector products."
Each question has an exact answer, given below, and none of these answers changes the conclusions of the paper.
- Baseline in Eq. (55). The baseline is the mean of the perturbed rewards, so the term in Eq. (55) cancels exactly.
- Reuse of perturbations. The same perturbations serve every matrix-vector product, so the estimated matrix has rank of at most .
- Effective rank of a signed spectrum. It is computed from the magnitudes of the eigenvalues, , with no thresholding (see also the response to Reviewer HZLX, W1).
- The eigenvalue scale in Fig. 17. For a reward in and a smoothing scale σ = 10⁻³, the true Hessian eigenvalues are at most . The scale comes from the rank- structure of the estimator, as each of its eigenvalues is roughly a reward difference times .
- Calibration for finite perturbation samples. The estimated effective rank is bounded by . This bound also explains why the estimate changes by 21–43% with the prompt count but does not change with the Lanczos depth.
We have revised the paper accordingly (Section 5, Appendix H.1) to state this rank bound, and to treat the zeroth-order estimate as a qualitative indicator of concentrated curvature rather than as a measurement of the curvature-active dimension . The claims never rested on this number, but instead on two kinds of evidence for low-dimensional curvature:
- Best-of- is the operational measure of accessibility. It tests directly, without any curvature estimate, whether a small population still finds reward improvements as models grow. The result is robust to resampling and holds along the training trajectory (see the response to Reviewer HZLX, W2 and Q2).
- The curvature measurements explain why improvements stay accessible. The cross-entropy Hessian is measured independently of the reward estimator. It uses exact Hessian-vector products, so it has no rank limit, and it puts the curvature-active part 6–8 orders of magnitude below the parameter count . Gradient concentration, measured up to 14B, agrees.
Hence the conclusions do not depend on the rank ceiling of the zeroth-order estimator.
Q3: Robustness of the experimental results.
"please provide or locate the prompt-bootstrap and prompt-subset checks mentioned in Appendix G.6"
The best-of- results are robust to resampling, and rise-then-decay is reproduced across population sizes, algorithms and tasks.
- Bootstrap and subset checks (new App. G.6). We ran the checks on the 36 of the 60 settings in Fig. 6 where the best of 30 candidates improves on the base model. When the 240 candidates are resampled, the improvement stays significant in 35 of the 36 settings. When the 320 prompts are resampled, it stays significant in 33 of the 36, and the three exceptions are all at the smallest model (0.5B). When the best candidate is selected on half of the prompts and tested on the other half, it still improves in 24 of the 36 settings, including the 7B model on every task. For the settings with small gains, most improving candidates gain at least two prompts, i.e. more than a single-answer flip.
- Normalization. All ES training runs use the z-score update of Eq. (2). Baseline centering is used only in the curvature estimator of Eq. (55) and not in the training updates, as the revised Appendices F and G.12 now state.
- Runs and shading. Fig. 1 shows one run per population size (ES) or group size (GRPO). The shaded band is the variation of the reward across population members (ES) or group samples (GRPO), as the revised caption of Fig. 1 now states. Showing one run per setting does not mean that the phenomenon is inconsistent across runs. Rise-then-decay recurs at every population size and in both algorithms; Countdown provides independent confirmation on a second task, where ES shows the same pattern at every population size in both training and held-out test reward (App. F, Fig. 10).
Hence the results are robust across resampling, runs and tasks, as the revised caption of Fig. 1 and the new Appendix G.6 make clear.
We thank the reviewer for the precise questions. We hope that the response and the revision address all of the concerns raised.
Thank you for your detailed response, which has addressed my concerns. As my original rating was already positive, I will maintain my current rating.
We thank the reviewer for the careful and supportive review.
Summary of the response. As the reviewer notes, the value of the paper does not depend on the precise effective-rank estimates. The response confirms this point. Every calibration question about the curvature estimator has an exact answer, and none of these answers changes the conclusions of the paper (W2 / Q2). The experimental results hold under resampling of the candidates and the prompts, along the training trajectory, and on a second task (Q3). Hence the evidence that small populations suffice as models grow is now stronger. This result matters beyond ES. Low-dimensional curvature is a property of the fine-tuning landscape itself, and hence opens up a broad class of algorithms that the classical worst-case analysis of zeroth-order optimization had ruled out at scale, with interesting tradeoffs to be studied in the future. These points are illustrated below in the order of the review.
W1 / Q1: Connection from the local theory to actual training.
The local model shows that the mechanism (heterogeneous Hessian spectrum) is sufficient to produce rise-then-decay, not that it is the only cause. We have revised Section 4 accordingly to separate what is observed from what the hypothesis is. In short,
Observed in actual training.
Hypothesis. The noise sits along the flat directions of the landscape, and the decay in GRPO has the same cause as the decay in ES. Appendix C extends the analysis to general anisotropic noise, including the noise of GRPO.
Hence, beyond agreement in curve shape, the population-size dependence (ii) supports the mechanism in actual training, and GRPO shows that the pattern is consistent across algorithms.
On the two statements raised in the review:
Indeed, rise-then-decay is evidence for heterogeneous curvature, i.e. separated curvature scales. Low-dimensional curvature is tested separately in Section 5, by the best-of-N experiments and the curvature measurements (see W2 / Q2 below).
Actually ES can move uphill even when every candidate is worse than the current point, as recently shown by Dailey et al. (2026, arXiv:2611.04113), which the revised Section 5 now cites. Best-of-N measures how accessible improvements are to random search, which is the quantity that the scaling argument needs.
We have revised Sections 4 and 5 accordingly to make both statements precise. Neither statement affects the results of the paper, which rest on heterogeneous curvature for rise-then-decay and on accessibility for scaling.
(Continued in Part 2/2.)