- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates when Evolution Strategies (ES) can rival Policy Gradient (PG) methods for LLM fine-tuning. It proposes a unified variance-based framework in which ES is governed by the reward-relevant effective parameter dimension, while PG is governed by the effective horizon, determined jointly by credit assignment and weight sharing. The authors validate these relationships using controlled synthetic tasks, analyzing gradient variance and training performance across different dimensions and horizons. They further estimate practical proxies for both quantities across 27 NLP, mathematical reasoning, and agentic benchmarks, and compare the resulting task characteristics with ES–PG training outcomes on real LLMs.
Critical strengths
- The paper offers a useful theoretical perspective on the ES–PG trade-off by distinguishing effective dimension and effective horizon from nominal parameter count and sequence length. The analysis of credit assignment and weight sharing provides further insight into the sources of PG's horizon-dependent variance.
- The controlled synthetic experiments systematically examine the proposed mechanisms across different dimensions, horizons, and credit-assignment settings. The observed variance and training-performance trends, together with additional autoregressive-policy experiments, provide meaningful support for the theoretical analysis.
Critical weaknesses
- The real-task proxies are not directly validated against the theoretical quantities. In Section 5, effective dimension is estimated from gold-target gradients, while effective horizon relies on token-level score energy, omitting the credit-assignment and gradient-overlap factors central to the analysis. Although these choices are motivated, no real-LLM analysis establishes whether the resulting proxies reliably reflect reward-relevant gradient variance, limiting the evidence connecting the theoretical framework to observed ES–PG performance.
- The real-agent experiments provide limited support for the predicted ES advantage in long-horizon fine-tuning. Although ES outperforms PG on ALFWorld (0.28 vs. 0.22), WebShop does not show the advantage suggested by the task atlas, while ScienceWorld and Terminal-Bench remain inconclusive. Moreover, evaluations involve only 8–20 training iterations and small test sets, including six Terminal-Bench tasks, with best-observed checkpoints reported. These results leave the practical ES–PG trade-off insufficiently validated.
Questions for authors
- Could the authors provide evidence that the effective-horizon proxy tracks PG gradient variance or signal-to-noise trends on representative real LLM tasks, and clarify how the gold-target-gradient dimension proxy relates to ES's reward-relevant parameter directions? How might the omitted credit-assignment and gradient-overlap factors affect these relationships?
- Given the budget-dependent ES–PG crossover observed in Figure 5, could the authors examine whether the relative performance remains consistent under longer, matched training budgets on representative agentic tasks?
4 comments
(Continued from Part 1/2.)
W2 / Q2: Agentic evidence and training budgets.
"The real-agent experiments provide limited support for the predicted ES advantage in long-horizon fine-tuning."
In our setup, ALFWorld and WebShop turn out to be short-horizon tasks, so PG being ahead there is consistent with the theory. Terminal-Bench is the long-horizon task, and there ES is ahead.
ALFWorld and WebShop. For the revision, we re-measured the horizon of each task in its training setup. The ALFWorld and WebShop agents emit one short action per step without reasoning, so their horizons are short. In our controlled experiments, ES becomes cheaper than PG only beyond a certain horizon, which we call the crossover, and this crossover grows with the number of skills. The horizons of ALFWorld and WebShop are below this crossover. These horizons are even shorter than those of the math reasoning tasks, where prior work reports ES and PG roughly tied (Qiu et al., 2026). This also explains the WebShop result the reviewer pointed out, as with the revised proxies WebShop sits on the PG side of the task atlas, in agreement with the outcome.
| ALFWorld | WebShop | Math reasoning | Terminal-Bench | |
|---|---|---|---|---|
| T_eff | 125 | 59 | 224 to 522 | about 7,300 |
| Outcome | PG ahead | PG ahead | close | ES ahead |
"whether the relative performance remains consistent under longer, matched training budgets"
Yes, the relative performance is consistent. With matched budgets and runs three times as long, PG outperforms ES on both tasks and the performance gap grows.
Both methods got the same games or goals and the same number of episodes, runs were three times as long (30 and 60 updates), and we report final checkpoints on fixed held-out sets (96 games, 256 goals) over three seeds.
| Base | PG | ES | |
|---|---|---|---|
| ALFWorld | 0.16 | 0.49 ± 0.08 | 0.23 ± 0.04 |
| WebShop | 0.40 | 0.61 ± 0.04 | 0.53 ± 0.01 |
PG was ahead by 0.13 and 0.05 at the earlier length (same fixed games) and by 0.26 and 0.08 at the longer one. The submission's ALFWorld numbers (ES 0.28 vs. PG 0.22) were from short runs with best checkpoints, and the revision reports the longer runs above. Hence the relative performance does not depend on the training budget.
In the revised paper. Sec. 5, with the scores in Table 8 and the setup in App. D.2. The measured horizons are in Table 6, and the Terminal-Bench horizon is in App. D.2.
ScienceWorld. The reruns focused on tasks where both methods can learn. On ScienceWorld the base model does not yet solve the tasks in our setup.
Terminal-Bench. This task is the real test of the long-horizon prediction. The six-task result in the submission used Qwen2.5-7B, which solves very few of the tasks, so there was little to compare. With Qwen3.5-4B in the public Tmax setup (Ivison et al., 2026), the agent reasons before acting, and the horizon is about 60 times that of ALFWorld. Robustness checks of this horizon estimate are in our response to Reviewer FFjh. At such a horizon the theory predicts that ES should lead. Indeed, ES is ahead of the released PG checkpoint.
| Terminal-Bench 2.1, Qwen3.5-4B (T_eff ≈ 7,300) | Success rate |
|---|---|
| Base | 12.1% ± 2.2 |
| PG (released Tmax checkpoint) | 18.2% ± 2.2 |
| ES | 21.1% ± 1.5 |
In more detail, each method contributes one checkpoint evaluated five times, the released PG checkpoint from Tmax and an ES run in the same setup. Each method was trained to its own final checkpoint rather than to a matched budget, so this compares attainable performance. Training at this horizon is extremely expensive, and therefore this is the only long-horizon point we have so far. As such, it is a significant data point.
In the revised paper. Sec. 5 (paragraph "Terminal-Bench at a longer horizon"), with the setup, the horizon measurement and the scores in App. D.2 (Table 10). The ES run is from concurrent work, which the paper cites anonymously.
With these additional experiments, the results are now significantly stronger and the paper is better as a result. We thank the reviewer for the constructive feedback and hope we have addressed all of the reviewer's concerns.
References.
- Ivison, H., Yin, J. O., Shao, R., Xiao, T., Lambert, N., and Hajishirzi, H. (2026). Tmax: A Simple Recipe for Terminal Agents. arXiv:2606.23321.
- Qiu, X., Gan, Y., Hayes, C. F., Liang, Q., Xu, Y., Dailey, R., Meyerson, E., Hodjat, B., and Miikkulainen, R. (2026). Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning. ICML.
Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.
Thank you again for your time and consideration!
Response to the rebuttal
Thanks for the detailed response. The clarifications and additional experiments have addressed most of my concerns, and I keep my overall positive assessment.
We thank the reviewer for the careful review.
Summary of the response. The reviewer's two concerns are whether the real-task proxies are validated and whether the agentic evidence is conclusive. In response, the paper has been revised in two ways, and we point to the relevant places below.
Together, these results reinforce the thesis of the paper and give solid support for the link between the theory and real ES vs. PG outcomes.
W1 / Q1: Are the real-task proxies validated?
We ran this test for the revision. Tasks with a longer measured horizon indeed had noisier PG updates.
What we measured. The effective horizon Teff is meant to capture how noisy PG's gradient is on a task. To test this on the real model, we computed PG's gradient for each rollout and measured how noisy one PG update was relative to its signal (the variance of the update divided by the squared norm of its mean).
Result. Across 20 tasks, the tasks with a longer Teff were the ones with noisier updates (rank correlation +0.83). Hence the horizon proxy does track the signal-to-noise trend the reviewer asks about.
How accurate is the proxy? We checked the value of Teff in two ways.
Hence Teff is accurate to within about a factor of two. This is sufficient for our analysis, as the horizons of the tasks differ by far more than that (from about 2 to about 7,300), so the ordering of the tasks does not change.
In the revision, the proxy includes these two factors. The proxy in the submission was based on token-level score energy alone. The proxy in the revision was measured from PG's gradients in the actual training setup of each task, so it includes how credit is assigned to the tokens and how much their gradients overlap. On the synthetic task, where the credit assignment can be changed, the horizon cost disappeared when credit was given to each token separately, as the theory says. Hence the proxy now covers the factors the reviewer points out.
In the revision, the dimension is measured directly from the reward.
The earlier proxy. It used gradients toward the human-written reference answers. These gradients also reflect differences in writing style between the model and the references, in addition to the reward.
The new measurement. ES pays for the number of independent skills, i.e. the number of independent ways in which the parameters must change to increase the reward. We measured this number directly. For each example, we computed the gradient of the reward from the model's own rollouts, and counted how many independent directions these gradients span across examples.
Hence the dimension is now measured directly. All counts are far below the number of parameters, which supports the claim that ES pays for skills and not for parameters.
Together, these checks confirm that the two proxies are valid on real models. The horizon proxy tracks PG's noise, is accurate to within a factor of two and includes the credit and overlap factors, and the dimension is measured directly.
In the revised paper. Sec. 5 and App. D.1, where Teff is defined in Eq. 12, with its values for the 20 tasks in Table 6 and the skill counts in Table 5.
(Continued in Part 2/2.)