acceptodds
Under review as a conference paper at ICLR 2027

Dimension Is Free, Horizon Is Not: When Evolution Strategies Rival Policy Gradients for LLM Fine-Tuning

Abstract

Policy gradient (PG) methods such as GRPO are the dominant approach to LLM fine-tuning, but they suffer from imprecise credit assignment, a limitation that becomes especially pronounced in long-horizon agentic tasks. Recent work has established Evolution Strategies (ES) as a scalable alternative to PG. Because ES evaluates each perturbed model by its total return, it avoids per-token credit assignment, which makes it a natural candidate for long-horizon settings. We analyze ES and PG within a single framework, as two estimators of the same gradient that explore in different currencies. ES perturbs parameters and therefore pays a dimension tax, whereas PG samples actions and therefore pays a horizon tax. We show that the taxes actually incurred are effective quantities. For ES, the tax is , the number of reward-relevant parameter directions. For PG, it is , the number of token positions that contribute reward-relevant variance, which increases only when reward-irrelevant tokens share both credit and weights with reward-carrying ones. To disentangle the two, we construct a synthetic task in which and can be controlled independently. On this task, PG outperforms ES at high and short , whereas ES outperforms PG when is long relative to , and this region expands with the training budget of ES. We then estimate proxies for and on 27 NLP, mathematical reasoning, and agentic tasks with Qwen2.5-7B-Instruct and relate the resulting atlas to head-to-head ES–PG comparisons. Our results support ES as a scalable alternative for LLM fine-tuning, particularly for long-horizon tasks with sparse credit.

Primary area:
reinforcement learning
Keywords:
evolution strategies, policy gradient, RL fine-tuning of language models, gradient estimator variance, credit assignment, effective horizon, intrinsic dimensionality, zeroth-order optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SeKDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates when Evolution Strategies (ES) can rival Policy Gradient (PG) methods for LLM fine-tuning. It proposes a unified variance-based framework in which ES is governed by the reward-relevant effective parameter dimension, while PG is governed by the effective horizon, determined jointly by credit assignment and weight sharing. The authors validate these relationships using controlled synthetic tasks, analyzing gradient variance and training performance across different dimensions and horizons. They further estimate practical proxies for both quantities across 27 NLP, mathematical reasoning, and agentic benchmarks, and compare the resulting task characteristics with ES–PG training outcomes on real LLMs.

Critical strengths
  1. The paper offers a useful theoretical perspective on the ES–PG trade-off by distinguishing effective dimension and effective horizon from nominal parameter count and sequence length. The analysis of credit assignment and weight sharing provides further insight into the sources of PG's horizon-dependent variance.
  2. The controlled synthetic experiments systematically examine the proposed mechanisms across different dimensions, horizons, and credit-assignment settings. The observed variance and training-performance trends, together with additional autoregressive-policy experiments, provide meaningful support for the theoretical analysis.
Critical weaknesses
  1. The real-task proxies are not directly validated against the theoretical quantities. In Section 5, effective dimension is estimated from gold-target gradients, while effective horizon relies on token-level score energy, omitting the credit-assignment and gradient-overlap factors central to the analysis. Although these choices are motivated, no real-LLM analysis establishes whether the resulting proxies reliably reflect reward-relevant gradient variance, limiting the evidence connecting the theoretical framework to observed ES–PG performance.
  2. The real-agent experiments provide limited support for the predicted ES advantage in long-horizon fine-tuning. Although ES outperforms PG on ALFWorld (0.28 vs. 0.22), WebShop does not show the advantage suggested by the task atlas, while ScienceWorld and Terminal-Bench remain inconclusive. Moreover, evaluations involve only 8–20 training iterations and small test sets, including six Terminal-Bench tasks, with best-observed checkpoints reported. These results leave the practical ES–PG trade-off insufficiently validated.
Questions for authors
  1. Could the authors provide evidence that the effective-horizon proxy tracks PG gradient variance or signal-to-noise trends on representative real LLM tasks, and clarify how the gold-target-gradient dimension proxy relates to ES's reward-relevant parameter directions? How might the omitted credit-assignment and gradient-overlap factors affect these relationships?
  2. Given the budget-dependent ES–PG crossover observed in Figure 5, could the authors examine whether the relative performance remains consistent under longer, matched training budgets on representative agentic tasks?
4 comments

We thank the reviewer for the careful review.

Summary of the response. The reviewer's two concerns are whether the real-task proxies are validated and whether the agentic evidence is conclusive. In response, the paper has been revised in two ways, and we point to the relevant places below.

  • Validated proxies (W1, Q1). The horizon estimate tracks PG's measured gradient noise on real models (rank correlation +0.83) and is accurate to within about a factor of two. In the revision, the dimension is measured directly from reward gradients.
  • Conclusive agentic evidence (W2, Q2). Matched runs three times as long show PG ahead on ALFWorld and WebShop, consistent with their short measured horizons, and ES ahead on long-horizon Terminal-Bench.

Together, these results reinforce the thesis of the paper and give solid support for the link between the theory and real ES vs. PG outcomes.

W1 / Q1: Are the real-task proxies validated?

"Could the authors provide evidence that the effective-horizon proxy tracks PG gradient variance or signal-to-noise trends on representative real LLM tasks"

We ran this test for the revision. Tasks with a longer measured horizon indeed had noisier PG updates.

What we measured. The effective horizon TeffT_{\rm{eff}} is meant to capture how noisy PG's gradient is on a task. To test this on the real model, we computed PG's gradient for each rollout and measured how noisy one PG update was relative to its signal (the variance of the update divided by the squared norm of its mean).

Result. Across 20 tasks, the tasks with a longer TeffT_{\rm{eff}} were the ones with noisier updates (rank correlation +0.83). Hence the horizon proxy does track the signal-to-noise trend the reviewer asks about.

How accurate is the proxy? We checked the value of TeffT_{\rm{eff}} in two ways.

  • On the synthetic task, the true horizon is known. The proxy recovered it closely, with a median ratio of 0.97 between the two (for tasks with at least four skills).
  • On real tasks, the true horizon is not known, so we checked how much TeffT_{\rm{eff}} changed when it was computed from a different set of examples. It changed by less than a factor of 2.1 on every task except SQuAD (2.4), WebQuestions (2.3) and Terminal-Bench (2.6).

Hence TeffT_{\rm{eff}} is accurate to within about a factor of two. This is sufficient for our analysis, as the horizons of the tasks differ by far more than that (from about 2 to about 7,300), so the ordering of the tasks does not change.

"How might the omitted credit-assignment and gradient-overlap factors affect these relationships?"

In the revision, the proxy includes these two factors. The proxy in the submission was based on token-level score energy alone. The proxy in the revision was measured from PG's gradients in the actual training setup of each task, so it includes how credit is assigned to the tokens and how much their gradients overlap. On the synthetic task, where the credit assignment can be changed, the horizon cost disappeared when credit was given to each token separately, as the theory says. Hence the proxy now covers the factors the reviewer points out.

"clarify how the gold-target-gradient dimension proxy relates to ES's reward-relevant parameter directions"

In the revision, the dimension is measured directly from the reward.

The earlier proxy. It used gradients toward the human-written reference answers. These gradients also reflect differences in writing style between the model and the references, in addition to the reward.

The new measurement. ES pays for the number of independent skills, i.e. the number of independent ways in which the parameters must change to increase the reward. We measured this number directly. For each example, we computed the gradient of the reward from the model's own rollouts, and counted how many independent directions these gradients span across examples.

  • On the synthetic task, where the number of skills is known, this count recovered it.
  • On real tasks, it gave 12 to 385 skills on five language tasks, about 20 on WebShop and about 100 on ALFWorld.

Hence the dimension is now measured directly. All counts are far below the number of parameters, which supports the claim that ES pays for skills and not for parameters.

Together, these checks confirm that the two proxies are valid on real models. The horizon proxy tracks PG's noise, is accurate to within a factor of two and includes the credit and overlap factors, and the dimension is measured directly.

In the revised paper. Sec. 5 and App. D.1, where TeffT_{\rm{eff}} is defined in Eq. 12, with its values for the 20 tasks in Table 6 and the skill counts in Table 5.

(Continued in Part 2/2.)

(Continued from Part 1/2.)

W2 / Q2: Agentic evidence and training budgets.

"The real-agent experiments provide limited support for the predicted ES advantage in long-horizon fine-tuning."

In our setup, ALFWorld and WebShop turn out to be short-horizon tasks, so PG being ahead there is consistent with the theory. Terminal-Bench is the long-horizon task, and there ES is ahead.

ALFWorld and WebShop. For the revision, we re-measured the horizon of each task in its training setup. The ALFWorld and WebShop agents emit one short action per step without reasoning, so their horizons are short. In our controlled experiments, ES becomes cheaper than PG only beyond a certain horizon, which we call the crossover, and this crossover grows with the number of skills. The horizons of ALFWorld and WebShop are below this crossover. These horizons are even shorter than those of the math reasoning tasks, where prior work reports ES and PG roughly tied (Qiu et al., 2026). This also explains the WebShop result the reviewer pointed out, as with the revised proxies WebShop sits on the PG side of the task atlas, in agreement with the outcome.

ALFWorldWebShopMath reasoningTerminal-Bench
T_eff12559224 to 522about 7,300
OutcomePG aheadPG aheadcloseES ahead

"whether the relative performance remains consistent under longer, matched training budgets"

Yes, the relative performance is consistent. With matched budgets and runs three times as long, PG outperforms ES on both tasks and the performance gap grows.

Both methods got the same games or goals and the same number of episodes, runs were three times as long (30 and 60 updates), and we report final checkpoints on fixed held-out sets (96 games, 256 goals) over three seeds.

BasePGES
ALFWorld0.160.49 ± 0.080.23 ± 0.04
WebShop0.400.61 ± 0.040.53 ± 0.01

PG was ahead by 0.13 and 0.05 at the earlier length (same fixed games) and by 0.26 and 0.08 at the longer one. The submission's ALFWorld numbers (ES 0.28 vs. PG 0.22) were from short runs with best checkpoints, and the revision reports the longer runs above. Hence the relative performance does not depend on the training budget.

In the revised paper. Sec. 5, with the scores in Table 8 and the setup in App. D.2. The measured horizons are in Table 6, and the Terminal-Bench horizon is in App. D.2.

ScienceWorld. The reruns focused on tasks where both methods can learn. On ScienceWorld the base model does not yet solve the tasks in our setup.

Terminal-Bench. This task is the real test of the long-horizon prediction. The six-task result in the submission used Qwen2.5-7B, which solves very few of the tasks, so there was little to compare. With Qwen3.5-4B in the public Tmax setup (Ivison et al., 2026), the agent reasons before acting, and the horizon is about 60 times that of ALFWorld. Robustness checks of this horizon estimate are in our response to Reviewer FFjh. At such a horizon the theory predicts that ES should lead. Indeed, ES is ahead of the released PG checkpoint.

Terminal-Bench 2.1, Qwen3.5-4B (T_eff ≈ 7,300)Success rate
Base12.1% ± 2.2
PG (released Tmax checkpoint)18.2% ± 2.2
ES21.1% ± 1.5

In more detail, each method contributes one checkpoint evaluated five times, the released PG checkpoint from Tmax and an ES run in the same setup. Each method was trained to its own final checkpoint rather than to a matched budget, so this compares attainable performance. Training at this horizon is extremely expensive, and therefore this is the only long-horizon point we have so far. As such, it is a significant data point.

In the revised paper. Sec. 5 (paragraph "Terminal-Bench at a longer horizon"), with the setup, the horizon measurement and the scores in App. D.2 (Table 10). The ES run is from concurrent work, which the paper cites anonymously.

With these additional experiments, the results are now significantly stronger and the paper is better as a result. We thank the reviewer for the constructive feedback and hope we have addressed all of the reviewer's concerns.

References.

  • Ivison, H., Yin, J. O., Shao, R., Xiao, T., Lambert, N., and Hajishirzi, H. (2026). Tmax: A Simple Recipe for Terminal Agents. arXiv:2606.23321.
  • Qiu, X., Gan, Y., Hayes, C. F., Liang, Q., Xu, Y., Dailey, R., Meyerson, E., Hodjat, B., and Miikkulainen, R. (2026). Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning. ICML.

Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.

Thank you again for your time and consideration!

Reviewer_SeKDOpenReview ↗

Response to the rebuttal

Thanks for the detailed response. The clarifications and additional experiments have addressed most of my concerns, and I keep my overall positive assessment.

Reviewer_FFjhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

The paper measures the performances of policy gradient (PG) with the effective token that is reward relevant and effective, and measures the performances of evolutionary strategies (ES) with effective gradient directions that the reward induces. It constructs an examples where these two factors can be separated and recommends ES and PG in respective regime. However I have concerns for the experiments when they scale to real datasets.

Critical strengths
  1. Nice example separating the effect of effective dimension and effective horizon on which PG and ES clearly wins respectively.
  2. upfront about limitations
Critical weaknesses
  1. Feels like the benchmark of ES against PG is not fair as ES is full training and PG is lora and ES uses more rollouts than PG. Even so the evidence for recommending ES is not obvious, t-bench is the only task where ES outperforms PG observably, and the authors excluded that point themselves.

  2. Training for 20 iterations on hard tasks like t-bench is inconclusive for the advantage of ES over PG. Longer trainings are needed. At the early training, the reads for fig 6 can be noisy. Along with point 1, the recommendation for ES for long horizon fine tunning is not justified.

  3. The PG side of the story is well known and the ES side is expected from Liang et.al. 2026, see their fig 4 and page 5 which says 'what matters is not the ambient parameter count but how many directions are meaningfully curvature-active'. Making the theoretical contribution incremental. The author can highlight more their contribution against Liang et.al.

    Qiyao Liang, Jinyeop Song, Yizhou Liu, Jeff Gore, Ila Fiete, Risto Miikkulainen, and Xin Qiu. The Blessing of Dimensionality in LLM Fine-tuning: A Variance–Curvature Perspective. arXiv preprint arXiv:2602.00170, 2026. https://arxiv.org/abs/2602.00170

Questions for authors

per weakness 1, 2, 3.

6 comments

We thank the reviewer for the careful review.

Summary of the response. The reviewer's main concern is whether the real-task comparisons are fair and long enough to justify recommending ES. In response, the paper has been revised in three ways, and we point to the relevant places below.

  • Fair comparisons (W1). Both methods now get the same examples and the same number of episodes on 14 tasks, and PG is also trained on all weights, like ES.
  • Longer training (W2). The agentic runs are three times as long, and budget checks show that both methods are trained long enough.
  • A justified recommendation (W1, W2). ES is recommended only for long horizons, and on Terminal-Bench with a model that can learn the task, ES is ahead of PG.

Together, these results reinforce the thesis of the paper and give it solid support on real tasks. PG is ahead at short horizons, the two methods are close on math, and ES is ahead at the long horizon. A task's horizon and skill count therefore give a useful guide to which method to use in which task. We also clarify our contribution beyond Liang et al. (W3).

W1: Fairness of the comparison.

"the benchmark of ES against PG is not fair as ES is full training and PG is lora and ES uses more rollouts than PG."

The revision compares the two methods on equal terms. With equal rollouts and with PG trained on all weights, the conclusions are the same.

  • Equal rollouts. Both PG and ES got the same examples and the same number of episodes in the comparisons (4,320 per language run, 5,760 on ALFWorld, 7,680 on WebShop), with step sizes tuned per method and final held-out scores over three seeds on Qwen2.5-7B-Instruct.
  • Full-weight PG. For a comparison with LoRA, we also trained PG on all weights on the 12 language tasks. This full-weight version was within 0.03 of LoRA PG everywhere except AG News, where it did slightly better. The agentic runs train all weights for both methods.
TaskFull-weight PGLoRA PGESBase
SQuAD0.82 ± 0.010.82 ± 0.020.38 ± 0.100.28
TriviaQA0.54 ± 0.020.57 ± 0.010.10 ± 0.020.10
WebQuestions0.28 ± 0.010.30 ± 0.020.11 ± 0.000.11
XSum0.34 ± 0.000.35 ± 0.010.27 ± 0.010.26
Translation0.58 ± 0.010.58 ± 0.000.52 ± 0.000.52
IMDB0.98 ± 0.030.96 ± 0.030.91 ± 0.010.88
AG News0.88 ± 0.010.82 ± 0.070.80 ± 0.010.78

On the other five language tasks (HellaSwag, BoolQ, CSQA, ARC, Winogrande), all methods stayed within 0.03 of base, partially because many of these tasks have little headroom for improvement.

Hence the comparisons are now fair, and they reinforce the theory and its conclusions. On all 14 tasks trained at matched budgets, PG leads or ties with either parameterization. This is expected from the theory, as these tasks all have a short effective horizon TeffT_{\rm{eff}} (roughly, the number of generated tokens that add noise to PG's update), at most 35 for language and 125 for agentic tasks, where PG should be cheaper.

In the revised paper. Sec. 5 (paragraph "Training comparisons at equal samples"), with Tables 7 and 8 in App. D.2. TeffT_{\rm{eff}} is defined in Eq. 12 (App. D.1), with its values for the 20 tasks in Table 6.

W2: Short training.

"Longer trainings are needed. At the early training, the reads for fig 6 can be noisy."

We trained three times longer and evaluated on fixed held-out games. The ordering of the two methods stays the same, and both methods have sufficient budget.

In the revision, the agentic runs are three times as long (30 and 60 updates instead of 10 and 20), with final checkpoints on fixed held-out games. The resulting performance differences are now clearer:

BasePGES
ALFWorld0.160.49 ± 0.080.23 ± 0.04
WebShop0.400.61 ± 0.040.53 ± 0.01
  • Longer runs. Before, PG was already ahead by 0.13 and 0.05, so training longer widens its lead further.
  • PG budget. With 500 updates on the language tasks, PG ended within 0.04 of its 60-update scores.
  • ES budget. ES did not improve with 500 updates at the default population. With a population of 30, it plateaued on WebShop (0.51 ± 0.03 at the same episodes).

Hence the ordering of the two methods on these tasks holds with longer training.

In the revised paper. App. D.2, paragraphs on training length and on the agentic tasks.

(Continued in Part 2/2.)

(Continued from Part 1/2.)

W1 and W2: ES advantage on long-horizon tasks.

"t-bench is the only task where ES outperforms PG observably, and the authors excluded that point themselves."

We now have a Terminal-Bench comparison at a long horizon, and there ES is ahead of PG. This is the regime for which we recommend ES.

The theory suggests that ES should lead when the task needs few independent skills, the horizon is beyond the crossover (the horizon beyond which ES becomes cheaper than PG in our controlled experiments), and the reward gives a learning signal. Terminal-Bench is such a task. In the submission, however, it was run with Qwen2.5-7B, which solves very few of the tasks, so that point could not yet test the theory.

New experiments with Qwen3.5-4B in the public Tmax setup (Ivison et al., 2026) were run to verify this part of the theory. This better model was able to solve some of the tasks, and since it reasons before acting, the horizon is much longer. The Terminal-Bench horizons TeffT_{\rm{eff}} were as follows (the last column recomputes the revised estimate on a separate, held-out set of reference tasks as a robustness check):

Terminal-Bench horizon (T_eff)SubmissionRevisedRevised, held-out references
Qwen2.5-7B, our harness98.53181,060
Qwen3.5-4B, Tmax setupnot measuredabout 7,300about 19,000

TeffT_{\rm{eff}} is measured as in Eq. 12 of the revised paper (App. D.1), from the model's rollouts and a set of solved reference tasks, so it depends on which tasks are used. Recomputing it for Qwen3.5-4B on random resamples of these tasks gives 2,500 to 10,300. Hence the horizon is long under every variant of the estimate. With such a long horizon, ES indeed beat the released PG checkpoint:

Terminal-Bench 2.1, Qwen3.5-4B (T_eff ≈ 7,300)Success rate
Base12.1% ± 2.2
PG (released Tmax checkpoint)18.2% ± 2.2
ES21.1% ± 1.5

In more detail, each method contributes one checkpoint evaluated five times, the released PG checkpoint from Tmax and an ES run in the same setup. Each method was trained to its own final checkpoint rather than to a matched budget, so this setup compares attainable performance. This new experiment thus completes the experimental validation of the theory.

In the revised paper. Sec. 5 (paragraph on Terminal-Bench) and App. D.2 (Table 10). The ES run is from concurrent work, cited there anonymously.

Hence the recommendation is narrower and better supported than in the submission. PG remains the better choice on the 14 short-horizon tasks, the two methods are reported to be close on math reasoning at intermediate horizons (224 to 522; Qiu et al., 2026), and ES is ahead at the longest horizon.

W3: Relation to Liang et al. (2026).

"The author can highlight more their contribution against Liang et.al."

Liang et al. explained why ES can scale at all. Our paper explains when ES should be preferred over PG, which requires the PG side and a comparison of the two.

Liang et al. explained why ES can scale to billion-dimensional spaces from a curvature perspective: its cost depends on a few curvature-active directions and not on the parameter count itself. This paper tells the other half of the story, i.e. the effect of the horizon, which is what PG pays for. The contributions beyond Liang et al. thus are:

  • What PG pays for. PG pays for an effective horizon and not for the raw output length. A long output by itself does not make PG expensive, as the horizon only grows when reward-irrelevant tokens share both weights and credit with the rewarded ones. This insight is new and explains why PG is not as effective in such tasks as ES.
  • A crossover. On a controlled task with both methods tuned, ES wins beyond a crossover horizon that grows with the number of skills, and PG wins below it. This result cannot be obtained from an analysis of ES alone as in the Liang et al. paper.
  • Measurements on real tasks. We measure the effective horizon on 20 real tasks, where it tracks the measured noise of PG's update (rank correlation +0.83), and the number of skills on seven. This places each benchmark relative to the crossover. This result makes it possible to decide which method should be used in which task.

In the revised paper. The related work (Sec. 6) now states this distinction.

With these additional experiments, the results are now significantly stronger and the paper is better as a result. We thank the reviewer for the constructive feedback and hope we have addressed all of the reviewer's concerns.

References.

  • Ivison, H., Yin, J. O., Shao, R., Xiao, T., Lambert, N., and Hajishirzi, H. (2026). Tmax: A Simple Recipe for Terminal Agents. arXiv:2606.23321.
  • Qiu, X., Gan, Y., Hayes, C. F., Liang, Q., Xu, Y., Dailey, R., Meyerson, E., Hodjat, B., and Miikkulainen, R. (2026). Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning. ICML.

Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.

Thank you again for your time and consideration!

Reviewer_n1izOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether and when Evolution Strategies (ES) can outperform Policy Gradients (PG) in LLM fine-tuning. It introduces a unified framework based on effective dimension and effective horizon, supported by theoretical analysis. The paper uses controlled synthetic experiments to validate their framework.

Critical strengths
  1. The paper provides an insightful perspective on the trade-offs between parameter-space and action-space exploration.
  2. The synthetic experiments are well designed and insightful.
Critical weaknesses
  1. The main experiments compare full-weight ES against LoRA-based Dr.GRPO. This difference in parameterization introduces confounding factors and makes it difficult to isolate the effects of the methods.
  2. The real-world agentic experiments provide inconclusive evidence for the claimed advantages of ES. ES shows a modest improvement on ALFWorld, while WebShop and ScienceWorld has similar performance than PG.
Questions for authors

Please see weaknesses

3 comments

We thank the reviewer for the careful review.

Summary of the response. The reviewer raises two points about the real-task experiments, the comparison of LoRA PG with full-weight ES and the agentic evidence. In response, the paper has been revised as follows, and we point to the relevant places below.

  • Fair comparisons (W1). Both methods are now trained on the same examples and episodes on 14 tasks, and a full-weight PG control gives the same result as LoRA PG.
  • Conclusive agentic results (W2). Longer runs with fixed held-out games show PG ahead on both ALFWorld and WebShop, and the measured horizons and skill counts explain why.
  • A long-horizon test. In contrast, ES has an advantage on Terminal-Bench, where the horizon is about 60 times longer.

Together, these results reinforce the thesis of the paper and give it solid support on real tasks. PG is ahead where the measured horizon is short, and ES is ahead where it is long. The effective horizon and the number of independent skills therefore give a useful guide to which method to use in which task.

W1: LoRA PG vs. full-weight ES.

"The main experiments compare full-weight ES against LoRA-based Dr.GRPO. This difference in parameterization introduces confounding factors"

Full-weight PG was added as a control on all 12 language tasks, and it performs the same as LoRA PG. Hence the comparison holds when both methods train all weights.

Setup. All new runs used Qwen2.5-7B-Instruct. Both methods saw the same examples and the same number of episodes, step sizes were tuned per method, and the final held-out scores are reported over three seeds.

Result. Full-weight PG was within 0.03 of LoRA PG on every language task except AG News, where full weights helped slightly.

TaskFull-weight PGLoRA PGESBase
SQuAD0.82 ± 0.010.82 ± 0.020.38 ± 0.100.28
TriviaQA0.54 ± 0.020.57 ± 0.010.10 ± 0.020.10
WebQuestions0.28 ± 0.010.30 ± 0.020.11 ± 0.000.11
XSum0.34 ± 0.000.35 ± 0.010.27 ± 0.010.26
Translation0.58 ± 0.010.58 ± 0.000.52 ± 0.000.52
IMDB0.98 ± 0.030.96 ± 0.030.91 ± 0.010.88
AG News0.88 ± 0.010.82 ± 0.070.80 ± 0.010.78

On the other five language tasks (HellaSwag, BoolQ, CSQA, ARC, Winogrande), all methods stayed within 0.03 of base, partially because many of these tasks have little headroom for improvement. In addition, the agentic runs in Part 2 train all weights for both methods, so both methods use the same parameterization there as well.

In the revised paper. Sec. 5 (paragraph "Training comparisons at equal samples") and Table 7 in App. D.2.

(Continued in Part 2/2.)

(Continued from Part 1/2.)

W2: Agentic evidence.

"The real-world agentic experiments provide inconclusive evidence for the claimed advantages of ES."

With longer matched runs, the agentic results are now conclusive. PG is ahead on ALFWorld and WebShop, which is consistent with their short measured horizons, and ES is ahead on Terminal-Bench, where the horizon is long.

ALFWorld and WebShop. The results in these domains in the submission came from short runs whose evaluation redrew the games at every checkpoint. We now use fixed held-out sets (96 games, 256 goals), trained three times longer, and tested performance on the final checkpoints over three seeds. PG is ahead on both tasks:

BasePGES
ALFWorld0.160.49 ± 0.080.23 ± 0.04
WebShop0.400.61 ± 0.040.53 ± 0.01

PG being ahead on both tasks is what the theory indeed predicts: ES should only lead when the horizon is long and the task needs few independent skills. In our controlled experiments, ES becomes cheaper than PG only beyond a certain horizon, which we call the crossover, and this crossover grows with the number of skills. Instead, the ALFWorld and WebShop agents emit one short action per step, so their effective horizon TeffT_{\rm{eff}} (roughly, the number of generated tokens that add noise to PG's update) is short and below this crossover. The skill count also explains why PG's lead is larger on ALFWorld even though its horizon is longer: ALFWorld needs about 100 independent skills whereas WebShop needs only about 20.

T_effIndependent skillsES − PG
ALFWorld125about 100−0.26
WebShop59about 20−0.08

In the revised paper. Sec. 5, with the scores in Table 8, the setup in App. D.2 and the skill counts in Table 5. TeffT_{\rm{eff}} is defined in Eq. 12 (App. D.1), with its values for the 20 tasks in Table 6.

ScienceWorld. The reruns focused on tasks where both methods can learn. On ScienceWorld the base model does not yet solve the tasks in the current setup.

Terminal-Bench. In the submission, Qwen2.5-7B in our previous harness solved very few of the tasks, which left little to compare. We therefore moved to the public Tmax setup (Ivison et al., 2026) with Qwen3.5-4B, where the agent reasons before each action. The horizon is then much longer, roughly 60 times that of ALFWorld:

ALFWorldWebShopTerminal-Bench
T_eff12559about 7,300
OutcomePG aheadPG aheadES ahead

TeffT_{\rm{eff}} is measured in the same way on all three tasks. For Terminal-Bench it is about 19,000 when estimated on a separate set of reference tasks, and further robustness checks of this estimate are in our response to Reviewer FFjh. In such a long-horizon setting, ES indeed beats the released PG checkpoint:

Terminal-Bench 2.1, Qwen3.5-4B (T_eff ≈ 7,300)Success rate
Base12.1% ± 2.2
PG (released Tmax checkpoint)18.2% ± 2.2
ES21.1% ± 1.5

In more detail, each method contributes one checkpoint evaluated five times, the released PG checkpoint from Tmax and an ES run in the same setup. Each method was trained to its own final checkpoint rather than to a matched budget, so this compares attainable performance. This new experiment thus confirmed the prediction of the theory.

In the revised paper. Sec. 5 (paragraph "Terminal-Bench at a longer horizon"), with the setup, the horizon measurement and the scores in App. D.2 (Table 10). The ES run is from concurrent work, which the paper cites anonymously.

With these additional experiments, the results are now significantly stronger and the paper is better as a result. We thank the reviewer for suggesting them! We hope we have sufficiently addressed all of the reviewer's concerns.

Reference. Ivison, H., Yin, J. O., Shao, R., Xiao, T., Lambert, N., and Hajishirzi, H. (2026). Tmax: A Simple Recipe for Terminal Agents. arXiv:2606.23321.

Thank you again for your thoughtful review! As the discussion period is coming to a close, we wanted to kindly follow up on our response and see if you might have any further thoughts or questions. We'd be happy to clarify anything or discuss any remaining concerns.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.