acceptodds
Under review as a conference paper at ICLR 2027

Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Memory Requirements

Abstract

Adapting long-horizon LLM agents to specific tasks requires efficient and flexible optimization under limited GPU resources. Although Reinforcement Learning (RL) has shown promise in single-turn LLM fine-tuning, this setting exposes two challenges: backpropagation makes full-parameter adaptation increasingly expensive for larger LLMs and longer contexts, while sparse rewards and branching interactions complicate long-horizon credit assignment. We argue that evolution strategies (ES) can often be a better choice for such \textbf{task-specific adaptation, offering 1) Model Scalability through forward-only full-parameter optimization, 2) Flexibility to integrate parameter updates with prompt-space optimization, and 3) Long-Horizon Scalability by avoiding explicit turn-level credit decomposition. Based on these insights, we propose Agentic ESOpt, a full-parameter ES framework supporting both train-time fine-tuning and on-the-fly adaptation during agentic test-time compute, together with a cosine perturbation schedule for balancing exploration and adaptation. Agentic ESOpt outperforms the strongest Agentic RL baseline by 13.19 percentage points in our longest-horizon Sudoku experiment, improves over Agentic GRPO on multi-turn tool-usage Math and DocVQA, enables full-parameter fine-tuning on a MoE LLM Qwen3-30B-A3B with 480GB GPUs, and improves existing automatic heuristic design pipelines in 28 of 36 comparisons. Code is available at https://anonymous.4open.science/status/Agentic-ESOpt-anonymous-197C.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agentic Fine-tuning, Efficient Fine-tuning, Evolutionary Strategies, Long-Horizon Agents, Agentic Adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4PuWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates evolution strategies (ES) for task-specific adaptation of long-horizon LLM agents under limited GPU resources. The authors propose Agentic ESOpt, a forward-only, full-parameter optimization framework that incorporates a cosine perturbation schedule and can be combined with external optimization components such as heuristic search and skill optimization. The paper evaluates the approach on controlled multi-turn Sudoku, Math reasoning, DocVQA, WebArena-Lite, and automatic heuristic design (AHD). The results demonstrate promising long-horizon adaptation performance, reduced GPU-memory requirements, and compatibility with existing agent optimization pipelines. The paper presents empirical findings regarding long-horizon adaptation, demonstrates scalability to larger models under constrained GPU resources, and explores a flexible interface for integrating parameter adaptation into existing agent optimization workflows.

Critical strengths
  1. The paper explores a problem that is both practically important and clearly defined: how to efficiently tune long-horizon agents under GPU resource constraints.

  2. The Sudoku experiments provide compelling empirical evidence of horizon-dependent behavior. In particular, the combination of external structures and cosine decay strikes me as a particularly novel aspect of the approach.

  3. The experiments go beyond simple controlled environments to encompass multi-turn tool use, full-parameter adaptation of a 30B MoE model, and integration with external optimization pipelines.

Critical weaknesses
  1. Limited analysis of the behaviors learned through ES adaptation. There is no direct examination of what Agentic ESOpt actually learns during adaptation. For example, does it improve tool-use decisions, reduce redundant interactions, or recover more effectively from mistakes? I would recommend to analyze:
  • Parameter distribution before and after training
  • The number of Turns before and after training
  • etc
  1. The consine schedule introduces more hyperparameters although relatively good effect. Is there any solution or intuition for the hyperparameter tuning issues associated with the cosine schedule?
Questions for authors
  1. Why is task-specific adaptation the primary target of Agentic ESOpt? How do you envision the boundary between settings where ES is preferable and those where conventional RL post-training remains more suitable?

  2. The paper provides useful estimates of training FLOPs and GPU-memory. Could you clarify which components are included in the FLOPs accounting, and how these estimates relate to the measured wall-clock results?

  3. Math and DocVQA use only half the FLOPs, how will Agentic ESOpt performs when enlarging the total FLOPs of Agentic ESOpt to the same level as RL.

  4. How sensitive are the results to perturbation scale and decoding hyperparameters? For the population-size experiments and the zero-terminal-noise ablation, were the perturbation scale and generation temperature independently tuned for each configuration, or held fixed to isolate the variables under study? Clarifying this would help interpret the reported scaling and ablation results.

  5. Continual learning is crucial for agentic learning; does the method proposed in this paper raise any concerns regarding continual learning?

  6. Following up on "weakness 1": How do the parameters and turns change after agentic ESOPT training?

  7. Following up on "weakness 2": Is there a solution for the hyperparameter tuning issues associated with the cosine schedule?

  8. The discussion of related work on automated heuristic design (AHD) focuses primarily on EoH. A more comprehensive review of recent advances in automated algorithm design and learning-based optimization would help better position the proposed approach within the broader research landscape. Relevant references include, but are not limited to:

  • Automated Algorithm Design for Black-Box Optimization: Progress and Challenges
  • B2Opt: Learning to Optimize Black-box Optimization with Little Budget
  • A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving
  1. Figure 1 serves as the main motivational figure, but its current presentation appears visually cluttered. In particular, panel (c) is somewhat unclear, especially with regard to the depiction of the function landscape in the background. The authors are encouraged to refine and improve the figure for better visual clarity and overall presentation.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 4PuW (1/3)

Thank you for your time and effort in reviewing our work. We are glad to know that you find the problem practically important and clearly defined, the Sudoku experiments compelling evidence of horizon-dependent behavior, and the combination of external structures and cosine decay novel.

We address your concerns as follows, and all revisions are marked in blue in the revised PDF.

Weakness 1 & Question 6. What Agentic ESOpt Learns: There is no direct examination of what Agentic ESOpt actually learns during adaptation. For example, does it improve tool-use decisions, reduce redundant interactions, or recover more effectively from mistakes? I would recommend to analyze: Parameter distribution before and after training; The number of Turns before and after training

Thanks for your suggestion. We analyze what Agentic ESOpt learns through three research questions on its behavior and its parameters.

  1. RQ1: Does Agentic ESOpt reduce redundant interactions? Yes, it uses fewer turns, mainly by cutting the turns of failed trajectories. On Math (Qwen3.5-4B, the runs of Table 2, 130 test problems; Detailed in Appendix D.3.3):
BehaviorBase LLMAgentic ESOptΔ\Delta
Accuracy (%)61.3575.38+14.03
Average turns20.686.30−14.38
Average turns of failed trajectories34.399.23−25.16
Trajectories reaching the 50-turn limit (%)5.581.15−4.43
Turns without a valid action per trajectory16.371.96−14.41
Trajectories without a final answer (%)19.810.58−19.23

The same holds during training (Figure 7): on Sudoku (H∗=15H^*=15), Agentic ESOpt stays near the 15-turn minimum (about 15.3 turns at step 80, versus 18.5 for Agentic GRPO), and on DocVQA its average turns fall from about 30 to about 15, while those of Agentic GRPO rise to about 46.

  1. RQ2: Does Agentic ESOpt improve tool-use decisions? Yes, it wastes far fewer turns without a valid action and almost always ends with a final answer. The removed turns are almost all turns without a valid action, while the number of valid actions per trajectory stays about the same (4.31 versus 4.34).

  2. RQ3: How do the parameters change after training? The update is spread almost evenly over layers and modules, and stays small. We compare the weights of Qwen3.5-4B before and after training (update RMS over all coordinates):

Qwen3.5-4BMathDocVQA
Max / min update RMS across the 32 layers1.00421.0040
Last- versus first-quarter layers+0.27%+0.28%
Max / min update RMS across the 12 projection types1.01561.0164
Full-rank update matrices after removing small entries100%100%

Normalization weights change less than the projections. On Qwen3.5-27B for WebArena-Lite, 96.26% of the parameter changes are within σ=1.5×10−3\sigma=1.5\times10^{-3} and 99.42% within 2×10−32\times10^{-3} (Table 6). Overall, Agentic ESOpt mainly learns to remove redundant interactions, through a small update spread over all parameters.

AuthorsOpenReview ↗

Response to Reviewer 4PuW (2/3)

Weakness 2 & Question 7. Hyperparameters of the Cosine Schedule: The consine schedule introduces more hyperparameters although relatively good effect. Is there any solution or intuition for the hyperparameter tuning issues associated with the cosine schedule?

Thanks for your question. We would like to clarify that Math and DocVQA share one hand-set schedule on both backbones (Table 29), and during the rebuttal period we propose an extension of the cosine schedule that sets σ0\sigma_0 and σT\sigma_T from rollout rewards, as an alternative to setting them by hand (Details in Sec. 3.1 and Appendix D.7.1):

  1. Mechanism. A permutation test on rollout rewards checks whether the perturbed agents are ranked beyond rollout noise. Before training, σ0\sigma_0 is the largest scanned radius that carries this ranking signal without collapsing the reward, and σT=σ0/2\sigma_T=\sigma_0/2. During training, σ\sigma decays from σ0\sigma_0 to σT\sigma_T with the cosine schedule, and the decay can be held.

  2. Result. On both Math and DocVQA, the extension selects σ0=10−3\sigma_0=10^{-3}, the hand-set value of the paper. As shown in the table below, its results match those of the hand-set schedule:

Qwen3.5-4BDAPO Mean@4AIME 2026 Mean@4DocVQA ANLS Mean@4DocVQA Accuracy Mean@4
Base LLM (Table 2)63.055.80.387540.3
Agentic ESOpt, hand-set (10−3→5×10−410^{-3}\rightarrow5\times10^{-4}; Table 2)76.870.80.504352.5
Automatic extension76.2570.000.504752.50
  1. Motivation and intuition. Lemma 1 shows that a larger σ\sigma regularizes more but biases the objective more, so σ\sigma decays. Over three training seeds at H∗=15H^*=15, the cosine schedule reaches 54.86 ± 2.19, versus 44.79 ± 2.19 and 46.88 ± 3.00 with a fixed σ\sigma (Table 12). In practice, σ\sigma should be large enough that the perturbations change the ranking of the population beyond rollout noise, and small enough that the perturbed agents still solve the task.

Question 1. Scope of Task-Specific Adaptation: Why is task-specific adaptation the primary target of Agentic ESOpt? How do you envision the boundary between settings where ES is preferable and those where conventional RL post-training remains more suitable?

Thanks for your question. We target task-specific adaptation because it brings together the conditions where ES helps: long interactions, terminal rewards, limited GPU memory, and one environment with a modest number of training tasks (e.g., 400 Math problems and 50 DocVQA questions).

ES is preferable for long horizons with terminal rewards, where action-level credit assignment is hard (Table 1, H∗=15H^*=15), under tight GPU memory, since it needs only inference-level memory (Tables 1 and 9), and when external structures such as skills or heuristics are optimized together with the LLM (Tables 3, 17, and 20). RL post-training remains more suitable for short horizons or dense turn-level rewards, for broad post-training across many tasks and large datasets, and when environment evaluation dominates the cost, since ES evaluates more trajectories (Appendix A.1).

Question 2. FLOPs Accounting and Wall-Clock: Could you clarify which components are included in the FLOPs accounting, and how these estimates relate to the measured wall-clock results?

Thanks for your question. FLOPs count the model-side training computation only (Appendix C.5). For PP parameters and LL processed tokens, a forward pass costs about 2PL2PL and a backward pass about 4PL4PL. Agentic ESOpt costs one policy forward per trajectory (2PL2PL); GRPO adds a reference forward and a policy backward (8PL8PL in total); PPO also adds a critic forward and backward (14PL14PL in total). Environment execution is excluded. For Table 7, LL is the measured number of processed tokens, so longer trajectories increase the measured FLOPs.

The wall-clock time in Table 7 is measured on the same four H100 GPUs, and the two agree in direction. On Sudoku, G=32G=32 matches the model-side budget of eight-rollout GRPO by design (2×32=8×82\times32=8\times8). As shown below, the measured FLOPs are comparable, and Agentic ESOpt finishes training faster at every horizon:

MethodH∗=5H^*=5H∗=10H^*=10H∗=15H^*=15
Agentic GRPO3.2 EFLOPs / 5.4 h7.6 EFLOPs / 13.1 h10.9 EFLOPs / 19.0 h
Agentic ESOpt (G=32G=32)3.1 EFLOPs / 3.1 h6.3 EFLOPs / 5.8 h9.4 EFLOPs / 9.4 h
AuthorsOpenReview ↗

Response to Reviewer 4PuW (3/3)

Question 3. Matching the FLOPs of RL on Math and DocVQA: Math and DocVQA use only half the FLOPs, how will Agentic ESOpt performs when enlarging the total FLOPs of Agentic ESOpt to the same level as RL.

Thanks for your question. In Table 2, Agentic ESOpt with G=16G=16 already outperforms Agentic GRPO and GiGPO at about half of their budgeted model-side FLOPs. We further run Agentic ESOpt with G=32G=32, which doubles these FLOPs and matches Agentic GRPO on Math. As shown in the table below (Qwen3.5-4B, Mean@4), it still outperforms both baselines:

MethodDAPOAIME 2026DocVQA ANLS
Agentic GRPO68.858.30.4627
Agentic GiGPO73.365.00.4846
Agentic ESOpt (G=16G=16)76.870.80.5043
Agentic ESOpt (G=32G=32)75.2575.830.5472

Question 4. Sensitivity to Perturbation Scale and Decoding: For the population-size experiments and the zero-terminal-noise ablation, were the perturbation scale and generation temperature independently tuned for each configuration, or held fixed to isolate the variables under study?

Thanks for your question. Both were held fixed as a preliminary setting to isolate the variables under study. The population-size runs (Table 5 and Appendix D.1.4) use σ=α=5×10−4\sigma=\alpha=5\times10^{-4} and temperature 0.7 for every GG, and the σT=0\sigma_T=0 ablation in Table 1 uses the same σ0\sigma_0 and decoding as the full method. We did not tune the perturbation scale or the temperature for each GG, and we leave this tuning to future work.

Question 5. Continual Learning: Continual learning is crucial for agentic learning; does the method proposed in this paper raise any concerns regarding continual learning?

Thanks for your question. We agree that continual learning is crucial for agentic learning. During the rebuttal period, we evaluate each Qwen3.5-4B Agentic ESOpt checkpoint of Table 2 on the task it was not trained on (Detailed in Appendix D.4.5):

ModelDAPOAIMEANLSAcc.
Base LLM63.055.80.387540.3
Agentic ESOpt (trained on Math)76.870.80.360537.8
Agentic ESOpt (trained on DocVQA)65.057.50.504352.5

On the task it was not trained on, each checkpoint stays close to the base LLM, and we observe no catastrophic forgetting. In addition, the ES update is concentrated in magnitude: on Qwen3.5-27B, 96.26% of the parameter changes stay within the perturbation scale σ\sigma (Table 6). Measuring retention over longer task sequences is our future work, which we have added to Appendix A.2 in the revised manuscript.

Question 8. Related Work on Automated Algorithm Design: Automated Algorithm Design for Black-Box Optimization: Progress and Challenges; B2Opt: Learning to Optimize Black-box Optimization with Little Budget; A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving.

Thanks for the suggestion. These works study automated algorithm design, learning to optimize black-box problems, and LLMs for evolutionary optimization, which relate to the heuristic-design setting in Sec. 4.3. We cite and discuss B2Opt [1] and the survey on LLMs for evolutionary optimization [2] in Appendix B.4.

Question 9. Figure 1: Figure 1 ... appears visually cluttered. In particular, panel (c) is somewhat unclear, especially with regard to the depiction of the function landscape in the background.

Thanks for pointing this out. We will simplify Figure 1.


We sincerely hope our responses have addressed all your concerns. If you have any further questions, please feel free to follow up, and we are glad to address them.

References

[1] Li, Xiaobin, et al. "B2Opt: Learning to Optimize Black-box Optimization with Little Budget." AAAI 2025.

[2] Zhang, Yisong, et al. "A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving." arXiv:2509.08269, 2025.

Reviewer_3BaBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Agentic ESOpt, an evolution-strategy-based approach for fine-tuning long-horizon LLM agents with limited GPU memory. Rather than using backpropagation, the method evaluates perturbed model parameters through agent-environment interactions and updates the parameters using trajectory-level rewards. The authors also introduce a cosine schedule for the perturbation scale and show how the approach can be combined with prompt and skill optimization. Experiments cover Sudoku, multi-turn Math reasoning, DocVQA, WebArena-Lite, and automatic heuristic design. The results show that Agentic ESOpt performs particularly well in the longest Sudoku setting, improves over the tested agentic RL baselines on Math and DocVQA, and enables full-parameter adaptation of a 30B MoE model using four 80GB GPUs.

Critical strengths
  1. Strong experimental results, particularly on longer-horizon tasks: The Sudoku experiments are one of the strongest parts of the paper. What I find interesting is that ESOpt is not consistently the best method across all settings, this provides some evidence for the authors' argument that ES becomes more competitive as the task horizon increases. I also appreciate that Appendix D.2 reports results from independent training runs across different seeds.
  2. Low GPU memory requirements and flexibility with existing methods: One practical advantage of ESOpt is that it can update all model parameters without storing backward activations or optimizer states. The Qwen3-30B-A3B experiment demonstrates that full-parameter fine-tuning is feasible with four 80GB GPUs. The authors also evaluate ESOpt on WebArena-Lite and combine it with existing heuristic design methods. I find these experiments useful because they show how the approach can be applied beyond standard fine-tuning.
Critical weaknesses
  1. The explanation for why ESOpt performs better at longer horizons is not fully established: The Sudoku results are promising, and I think increasing the number of masked cells is a reasonable way to evaluate longer-horizon tasks. However, I am less convinced that avoiding action-level credit assignment is the main reason for ESOpt's advantage. Section 3.2 and Appendix C.3 provide a theoretical argument, but the analysis relies on simplifying assumptions and does not directly compare the variance of the complete optimizers used in the experiments. The authors also include advantage-sign analysis and trajectory-length results in Appendix D.1.1, which I found helpful. To their credit, they acknowledge that these diagnostics are not causal evidence. My concern is therefore about how strongly the results support the proposed explanation. The experiments show that ESOpt can perform better at longer horizons, but they do not clearly isolate credit assignment as the reason for this improvement.

  2. The Math and DocVQA results could provide more evidence of stability across training runs: The improvements reported in Table 2 are promising, but I would have liked to see results from multiple independent training runs, particularly for Math and DocVQA. The authors evaluate each question with four samples and provide additional sampling results in Figure 9, but these do not show how much the results might change with different training seeds. This is especially relevant for AIME 2026, where the evaluation set contains only 30 problems.

  3. The results show some variability, including one run where GRPO outperforms ESOpt, but the average still favors ESOpt. The ablation without perturbation decay also helps show the contribution of the cosine schedule. Beyond Sudoku, the improvements on Math and DocVQA are encouraging, although the evidence for consistency across training runs is more limited there. Appendix C.7 also provides a useful memory breakdown, although the accounting does not represent the complete runtime memory peak.

Questions for authors

In addition to defending the critical weaknesses:

  1. Table 1 suggests that ESOpt becomes more effective than GRPO as the task horizon increases. However, Appendix D.2 only reports independent training runs for H* = 15, where GRPO still outperforms ESOpt in one of the three runs. Could the authors provide results across multiple training seeds for H* = 5 and H* = 10 as well? This would help establish whether the observed crossover is consistent or partly due to training variability.

  2. Were the Math and DocVQA experiments repeated with different training seeds? Table 2 reports results using four evaluation samples, but I could not find corresponding training-seed variability. If multiple runs were conducted, could the authors report these results? Otherwise, it would be helpful to clarify how stable the improvements are, particularly on the smaller AIME 2026 evaluation set.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3BaB (1/2)

Thank you for your time and effort in reviewing our work. We are glad to know that you find the long-horizon Sudoku results strong, the low GPU memory requirement practical, and the combination with existing methods useful.

We address your concerns as follows, and all revisions are marked in blue in the revised PDF.

Weakness 1. Why Agentic ESOpt Performs Better at Longer Horizons: My concern is therefore about how strongly the results support the proposed explanation. The experiments show that ESOpt can perform better at longer horizons, but they do not clearly isolate credit assignment as the reason for this improvement.

Thanks for your comment. To better isolate credit assignment, we introduce a controlled experiment that keeps the puzzles fixed and triples the number of actions to credit (Detailed in Appendix D.2.1). We keep the H∗=10H^*=10 puzzles and split each cell-filling action into three turns, e.g., the single action set 1 5 2 becomes row 1, col 5, value 2. The puzzles and their solutions stay the same, while the number of actions that need credit assignment triples. As shown below (averaged over three training runs), this lowers Agentic GRPO‡ from 67.01 to 53.82, whereas Agentic ESOpt, which scores each whole trajectory under one parameter perturbation instead of crediting each action, rises from 61.81 to 68.40:

H∗=10H^*=10 puzzles10 turns (original)30 turns (split)
Agentic GRPO‡67.01 ± 1.3453.82 ± 3.00
Agentic ESOpt61.81 ± 2.6468.40 ± 2.28

This experiment complements the analysis in Appendix D.1.1, which shows that, as the horizon grows, Agentic GRPO assigns negative advantage to more correct actions and Agentic GiGPO assigns positive advantage to more incorrect actions. Together, they are consistent with action-level credit assignment being the main reason for the advantage of Agentic ESOpt at longer horizons.

Question 1. Training Seeds at H∗=5H^*=5 and H∗=10H^*=10: Could the authors provide results across multiple training seeds for H* = 5 and H* = 10 as well? This would help establish whether the observed crossover is consistent or partly due to training variability.

Thanks for your suggestion. We have added two training seeds for Agentic Turn-PPO, Agentic GRPO‡ (the strongest GRPO configuration in Table 1), and Agentic ESOpt at H∗=5H^*=5 and H∗=10H^*=10 (Detailed in Appendix D.2). Each run is mean ± std over three evaluations of its final checkpoint, and "Avg." averages the mean and std of the three runs, as in Table 1:

H∗H^*MethodRun 1Run 2Run 3Avg.
5Agentic Turn-PPO90.63 ± 0.0089.58 ± 0.0095.83 ± 1.4792.01 ± 0.49
5Agentic GRPO‡85.42 ± 1.4786.46 ± 1.4785.42 ± 2.9585.76 ± 1.96
5Agentic ESOpt89.58 ± 2.9585.42 ± 3.9093.75 ± 1.4789.58 ± 2.77
10Agentic Turn-PPO56.25 ± 0.0061.46 ± 1.4761.46 ± 0.0059.72 ± 0.49
10Agentic GRPO‡67.71 ± 1.4771.88 ± 2.5561.46 ± 0.0067.01 ± 1.34
10Agentic ESOpt62.50 ± 2.5562.50 ± 3.9060.42 ± 1.4761.81 ± 2.64
15Agentic GRPO‡40.63 ± 1.4731.25 ± 2.9553.13 ± 2.5541.67 ± 2.32
15Agentic ESOpt53.13 ± 2.5564.58 ± 2.5546.88 ± 1.4754.86 ± 2.19

Run 1 at H∗=5H^ *=5 and H∗=10H^*=10 is the run in Table 1, and at H∗=15H^*=15 Table 1 reports the average of the three runs. The ranking at each horizon is the same as in Table 1: Agentic Turn-PPO is the best at H∗=5H^*=5, Agentic GRPO‡ at H∗=10H^*=10, and Agentic ESOpt at H∗=15H^*=15, where Agentic ESOpt has its largest margin over both baselines. At H∗=15H^*=15, the mean of Agentic ESOpt (54.86) exceeds the best run of Agentic GRPO‡ (53.13), and its worst run (46.88) exceeds the mean of Agentic GRPO‡ (41.67).

AuthorsOpenReview ↗

Response to Reviewer 3BaB (2/2)

Weakness 2 & Question 2. Training Seeds on Math and DocVQA: Were the Math and DocVQA experiments repeated with different training seeds? ... Otherwise, it would be helpful to clarify how stable the improvements are, particularly on the smaller AIME 2026 evaluation set.

Thanks for your suggestion. The results in Table 2 vary with both the training seed and the evaluation samples, and we report both below.

  1. Training seeds. We have retrained Agentic GiGPO, the strongest baseline in Table 2, and Agentic ESOpt with a new training seed, and evaluated both with the protocol of Table 2. With both training seeds, Agentic ESOpt outperforms Agentic GiGPO on all eight metrics of Table 2 (five shown below).
Qwen3.5-4BDAPO Mean@4AIME 2026 Mean@4AIME 2026 Pass@4DocVQA ANLS Mean@4DocVQA Accuracy Mean@4
Agentic GiGPO, run of Table 273.365.083.30.484649.8
Agentic GiGPO, new training seed73.863.380.00.470849.0
Agentic ESOpt, run of Table 276.870.896.70.504352.5
Agentic ESOpt, new training seed76.371.793.30.512353.0
  1. Evaluation samples. To measure how much the results change with the evaluation samples alone, we evaluate the base LLM, Agentic GiGPO, and Agentic ESOpt of Table 2 with three further evaluation seeds of four samples each. The table below reports the mean ± std over the run of Table 2 and the three new seeds. Averaged over the four evaluation runs, Agentic ESOpt remains the best on all eight metrics, and its Mean@4 margin over Agentic GiGPO exceeds the standard deviation of both methods on every benchmark.
MethodDAPO Mean@4DAPO Pass@4AIME Mean@4AIME Pass@4ANLS Mean@4ANLS Max@4Acc. Mean@4Acc. Pass@4
Base LLM65.06±3.5985.25±0.4357.71±2.3184.17±2.770.3901±0.01090.6174±0.013640.26±1.7963.00±1.87
Agentic GiGPO73.70±1.2385.25±1.3066.04±3.3086.67±4.080.4714±0.01360.5894±0.021748.88±1.0464.00±1.87
Agentic ESOpt75.58±1.2886.75±1.3070.62±2.4594.17±1.460.5028±0.00730.6470±0.014652.31±0.3266.75±1.64

Weakness 3. Runtime Memory Peak: Appendix C.7 also provides a useful memory breakdown, although the accounting does not represent the complete runtime memory peak.

Thanks for your comment. Yes, Appendix C.7 analyzes the minimal memory requirement of each method, i.e., the minimal possible runtime memory peak (e.g., without KV cache or CUDA graphs). In our actual runs, every method fully occupies the four GPUs. The minimal requirement therefore indicates which hardware is available to run each method: full-parameter Agentic ESOpt needs only inference-level memory and runs Qwen3-30B-A3B on four 80GB GPUs, whereas full-parameter GRPO needs 427.45GB for its static state alone (Table 9).


We sincerely hope our responses have addressed all your concerns. If you have any further questions, please feel free to follow up, and we are glad to address them.

AuthorsOpenReview ↗

Reminder to Reviewer 3BaB

Dear Reviewer 3BaB,

We are eager to engage in further discussions with you! In our earlier responses, we have actively addressed your concerns by providing clarifications of:

  1. New experiment isolating credit assignment: We keep the Sudoku puzzles fixed and triple the number of actions to credit, which lowers Agentic GRPO‡ but raises Agentic ESOpt.
  2. Training seeds on Sudoku: With three training runs at each horizon, the ranking is the same as in Table 1.
  3. Training seeds on Math and DocVQA: With a new training seed, Agentic ESOpt again outperforms Agentic GiGPO on all eight metrics of Table 2.
  4. Evaluation variance: Averaged over four evaluation runs, Agentic ESOpt remains the best on all eight metrics.

We sincerely hope that our responses have effectively addressed your concerns and may encourage a more favorable reconsideration of our paper.

Best regards,

Submission 1510 Authors

Reviewer_FSgwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Agentic ESOpt, an evolution strategies (ES) based framework for full-parameter optimization of long-horizon LLM agents under limited GPU resources. Unlike conventional agentic reinforcement learning methods such as PPO and GRPO, Agentic ESOpt estimates parameter updates through reward-weighted random perturbations without backpropagation or explicit turn-level credit assignment. The framework further introduces a cosine perturbation schedule and supports joint optimization of model parameters and external agent components, including prompts, skills, and memory.

Experiments on Sudoku, mathematical reasoning, DocVQA, WebArena-Lite, and automatic heuristic design demonstrate competitive performance against agentic RL baselines. In particular, Agentic ESOpt achieves a 13.19 percentage point improvement over the strongest GRPO baseline in the longest-horizon Sudoku setting and enables full-parameter optimization of a 30B MoE model using four 80GB GPUs.

Critical strengths
  1. The paper addresses the challenge of adapting long-horizon LLM agents under limited GPU resources. Its forward-only optimization enables memory-efficient full-parameter fine-tuning, including for a 30B MoE model.

  2. Agentic ESOpt demonstrates competitive performance across Sudoku, mathematical reasoning, DocVQA, and web-based agent tasks, with particularly notable improvements in long-horizon settings.

  3. The framework can be integrated with existing heuristic and skill optimization methods, achieving additional improvements with EoH, Trace2Skill, and SkillOpt.

Critical weaknesses
  1. The core ES formulation for RL optimization have been explored in prior work (Salimans et al., 2017; Qiu et al., 2026). The cosine perturbation schedule appears relatively incremental, making the overall algorithmic novelty somewhat limited.

  2. The theoretical analysis does not fully establish that ES has lower estimation variance as trajectories grow longer. Moreover, increasing the number of masked cells in Sudoku simultaneously increases task difficulty, making it difficult to isolate the effect of trajectory length.

  3. Although the paper demonstrates substantial GPU memory savings and approximately matches FLOPs, it lacks comprehensive comparisons of wall-clock training time, environment interaction costs, and sample efficiency.

  4. The experiments demonstrate improvements when combining ESOpt with existing heuristic and skill optimization methods, but do not sufficiently isolate the benefits of joint optimization compared with optimizing each component independently.

Questions for authors

Please see the Critical Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer FSgw (1/3)

Thank you for your time and effort in reviewing our work. We are glad to know that you find the forward-only optimization memory-efficient for full-parameter fine-tuning and the framework compatible with existing heuristic and skill optimization methods.

We address your concerns as follows, and all revisions are marked in blue in the revised PDF.

Weakness 1. Novelty of the Cosine Schedule: The cosine perturbation schedule appears relatively incremental, making the overall algorithmic novelty somewhat limited.

Thanks for your comment. During the rebuttal period, we propose an extension of the cosine schedule that sets σ0\sigma_0 and σT\sigma_T automatically from rollout rewards, as an alternative to setting them by hand (Details in Sec. 3.1 and Appendix D.7.1). The idea is to start from the largest radius at which the perturbed agents are still ranked beyond rollout noise, and to keep decaying it only while this ranking signal persists:

  1. Ranking-signal test. Let ri,br_{i,b} be the reward of perturbed agent ii on problem bb and rˉi\bar r_i its mean over problems (rˉ\bar r averages rˉi\bar r_i over agents). We compare the spread V=1G−1∑i(rˉi−rˉ)2V=\frac{1}{G-1}\sum_{i}(\bar r_i-\bar r)^2 with its distribution under permutations of the rewards across agents within each problem; a radius carries a ranking signal if p<0.05p<0.05.

  2. Setting σ0\sigma_0 and σT\sigma_T. Before training, we scan an ascending grid of σ\sigma on 32 validation problems, set σ0\sigma_0 to the largest radius that carries a ranking signal without collapsing the reward, and set σT=σ0/2\sigma_T=\sigma_0/2.

  3. Gated cosine decay. During training, σ\sigma decays from σ0\sigma_0 to σT\sigma_T with the cosine schedule of Sec. 3.1, and the decay can be held.

On both Math and DocVQA, the extension selects σ0=10−3\sigma_0=10^{-3} from grid search. As shown in the table below, its results match those of the hand-set schedule:

Qwen3.5-4BDAPO Mean@4AIME 2026 Mean@4DocVQA ANLS Mean@4DocVQA Accuracy Mean@4
Base LLM (Table 2)63.055.80.387540.3
Agentic ESOpt, hand-set (10−3→5×10−410^{-3}\rightarrow5\times10^{-4}; Table 2)76.870.80.504352.5
Automatic extension76.370.00.504752.50

This extension establishes a cosine schedule based on rollout rewards and adjusts its decay during training according to a ranking signal. With additional validation rollouts, it replaces the manual choice of σ0\sigma_0 and σT\sigma_T with a calibration from rollout rewards.

Moreover, the cosine schedule itself is simple but necessary: on Sudoku at H∗=15H^*=15, Agentic ESOpt drops from 54.86 to 46.88 without the decay and to 28.13 with σT=0\sigma_T=0.

Weakness 1 (continued). Relation to ES Prior Work: The core ES formulation for RL optimization have been explored in prior work (Salimans et al., 2017; Qiu et al., 2026).

We agree that the ES formulation has been explored in prior work: Salimans et al. [1] apply ES to RL policies, and Qiu et al. [2] apply it to single-turn LLM fine-tuning. Building on these works, our main contribution is to show that ES gains an advantage over Agentic RL in multi-turn agent fine-tuning as the horizon grows, revealing a horizon-dependent crossover with Agentic RL and enabling full-parameter fine-tuning of a 30B MoE LLM on four 80GB GPUs.

Beyond the estimator, Agentic ESOpt replaces the fixed radius of prior LLM ES [2] with a cosine perturbation schedule, which moves from broad exploration to finer local adaptation and is necessary on Sudoku. It also enables on-the-fly parameter–context co-optimization with inference-level GPU memory: existing approaches that jointly update the policy and prompt-space components need specialized designs that are hard to scale with limited GPUs, whereas Agentic ESOpt needs only forward passes and scalar rewards and can update the parameters inside test-time heuristic search and skill optimization.

AuthorsOpenReview ↗

Response to Reviewer FSgw (2/3)

Weakness 2. Trajectory Length versus Task Difficulty: ... increasing the number of masked cells in Sudoku simultaneously increases task difficulty, making it difficult to isolate the effect of trajectory length.

Thanks for your comment. We agree that more masked cells also make the puzzles harder.

To further separate trajectory length from task difficulty, we have added an experiment that keeps the puzzles fixed and lengthens the trajectory (Detailed in Appendix D.2.1). We keep the H∗=10H^*=10 puzzles and split each cell-filling action into three turns. For example, filling the digit 2 into row 1, column 5 is the single action set 1 5 2 in the original interface of Table 1, and the three actions row 1, col 5, value 2 in the split interface. The minimum number of turns thus grows from 10 to 30, while the puzzle difficulty stays the same.

As shown below (averaged over three training runs), tripling the trajectory length lowers Agentic GRPO‡ from 67.01 to 53.82 but raises Agentic ESOpt from 61.81 to 68.40:

H∗=10H^*=10 puzzles10 turns (original)30 turns (split)
Agentic GRPO‡67.01 ± 1.3453.82 ± 3.00
Agentic ESOpt61.81 ± 2.6468.40 ± 2.28

Weakness 2 (continued). Theoretical Analysis: The theoretical analysis does not fully establish that ES has lower estimation variance as trajectories grow longer.

Thanks for pointing this out. Whether ES has lower variance than policy gradients depends on the horizon: the policy-gradient score variance accumulates across turns, while the ES perturbation term does not depend on the horizon, so ES can have lower variance at longer horizons (Appendix C.3). Where this crossover lies differs from problem to problem, so we cannot guarantee lower variance than RL in general; the analysis isolates how the horizon enters. Empirically, Agentic ESOpt overtakes the RL baselines on Sudoku at H∗=15H^*=15 (Table 1), which is consistent with ES having lower variance at this horizon.

AuthorsOpenReview ↗

Response to Reviewer FSgw (3/3)

Weakness 3. Wall-Clock, Interaction Cost, and Sample Efficiency: Although the paper demonstrates substantial GPU memory savings and approximately matches FLOPs, it lacks comprehensive comparisons of wall-clock training time, environment interaction costs, and sample efficiency.

Thanks for your valuable suggestion. We present a comprehensive comparison of wall-clock training time, environment interaction costs, and sample efficiency on Sudoku and on the ReAct-style Math and DocVQA tasks in Tables 7 and 8 of Appendix C.5.

  1. Wall-clock training time. Each pair runs on the same four GPUs. As shown in the table below, Agentic ESOpt does not increase the wall-clock training time in any setting:
Setting (baseline)BaselineAgentic ESOpt
Sudoku H∗=5H^*=5 / 10 / 15, 4B (GRPO)5.4 / 13.1 / 19.0 h3.1 / 5.8 / 9.4 h
Math, 4B (GiGPO)31.9 h31.3 h
DocVQA, 4B (GiGPO)48.6 h14.5 h
Math, 30B (GRPO+LoRA)19.4 h9.8 h
DocVQA, 30B (GRPO+LoRA)15.5 h8.1 h
  1. Sample efficiency. As a limitation, Agentic ESOpt usually consumes more training trajectories than policy-gradient-based methods. The table below compares the number of training trajectories of Agentic ESOpt with the best-performing baseline on the same training problems. Each ES trajectory needs only a forward pass, about a quarter of the model-side FLOPs of a GRPO trajectory of equal length, so these additional trajectories do not increase the wall-clock time above.
Setting (baseline)BaselineAgentic ESOpt
Sudoku H∗=15H^*=15, 4B (GRPO‡)25,600102,400
Math, 4B (GiGPO)3,2006,400
DocVQA, 4B (GiGPO)5,12010,240
Math, 30B (GRPO+LoRA)3,2006,400
DocVQA, 30B (GRPO+LoRA)2,56010,240
  1. Environment interaction cost. Measured in assistant turns, Agentic ESOpt also uses more interactions in total:
Setting (baseline)BaselineAgentic ESOpt
Math, 4B (GiGPO)24.0k73.0k
DocVQA, 4B (GiGPO)158.7k213.7k
Math, 30B (GRPO+LoRA)14.7k51.4k
DocVQA, 30B (GRPO+LoRA)21.8k53.8k

Weakness 4. Joint versus Independent Optimization: The experiments demonstrate improvements when combining ESOpt with existing heuristic and skill optimization methods, but do not sufficiently isolate the benefits of joint optimization compared with optimizing each component independently.

Thanks for your insightful comment. We have added an analysis of joint optimization to the revised manuscript in Appendix D.4.3, comparing each component alone, Agentic ESOpt alone, and both:

  1. Compared to component only: Adding Agentic ESOpt improves every external component, including SkillOpt on WebArena-Lite (28.89 to 34.75).

  2. Compared to Agentic ESOpt only: This paper mainly demonstrates that Agentic ESOpt can make these combinations straightforward with inference-level GPU memory. In practice, the combination may outperform Agentic ESOpt alone with EoH and Trace2Skill, but not with SkillOpt (34.75 versus 36.16), whose joint run uses at most 10 ES updates versus 70. Designing component optimizers that fit ES and make the combination significantly outperform ES alone remains future work.

SettingComponent onlyAgentic ESOpt onlyBoth
TSP N=50N=50, EoH (lower is better)6.5456.5496.463
Math DAPO / AIME Mean@4, Trace2Skill64.8 / 50.876.8 / 70.877.3 / 71.7
WebArena-Lite, Trace2Skill33.9436.1636.36
WebArena-Lite, SkillOpt28.8936.1634.75

We sincerely hope our responses have addressed all your concerns. If you have any further questions, please feel free to follow up, and we are glad to address them.

References

[1] Salimans, Tim, et al. "Evolution Strategies as a Scalable Alternative to Reinforcement Learning." arXiv:1703.03864, 2017.

[2] Qiu, Xin, et al. "Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning." ICML 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

Dear Area Chair,

Thank you for handling our paper. As the discussion ends, we would like to summarize it for you.

1. Contributions of Agentic ESOpt.

Long-horizon LLM agents are hard to adapt with agentic RL under limited GPU resources, because RL training needs high GPU memory and credit assignment becomes difficult across long horizons. Our contributions are:

  • Algorithm: cosine schedule. Agentic ESOpt is a forward-only, full-parameter evolution strategies (ES) framework for long-horizon LLM agents. It updates the LLM from the trajectory-level rewards of perturbed agents, without backpropagation or action-level credit assignment. Instead of the fixed perturbation radius typically used in prior LLM ES, it decays the radius with a cosine schedule, moving from broad exploration to finer local adaptation.
  • Algorithm: combination with test-time optimization. Since ES needs only forward passes and scalar rewards, Agentic ESOpt updates the LLM parameters on the fly inside test-time heuristic search and skill optimization, with inference-level GPU memory. This improves EoH on all six constructive heuristic-design test sets and skill optimization on WebArena-Lite.
  • Finding: long-horizon advantage of ES. We identify that ES gains an advantage over agentic RL in multi-turn agent fine-tuning as the horizon grows: on Sudoku, Agentic ESOpt overtakes agentic RL at the longest horizon (54.86 versus 41.67 for the strongest GRPO at H∗=15H^*=15). This matches our analysis that the policy-gradient score variance accumulates across turns, while the ES perturbation term does not depend on the horizon (App. C.3).
  • Results. On Sudoku, Agentic ESOpt overtakes agentic RL as the horizon grows (54.86 versus 41.67 for the strongest GRPO at H∗=15H^*=15). On multi-turn Math and DocVQA, it outperforms the agentic RL baselines on every metric for Qwen3.5-4B and Qwen3-30B-A3B (Table 2).
  • Scalability. It fine-tunes all parameters of Qwen3-30B-A3B on four 80GB GPUs.

The reviewers recognized its long-horizon results, memory efficiency and novelty:

  • Long-horizon results: the Sudoku experiments give strong evidence of horizon-dependent behavior, with the most notable gains in the longest-horizon setting (3BaB, 4PuW, FSgw).
  • Memory efficiency: forward-only optimization enables full-parameter fine-tuning of a 30B MoE model on four 80GB GPUs (FSgw, 3BaB).
  • Novelty and flexibility: the combination of external structures and cosine decay is a novel aspect of the approach (4PuW), and the framework integrates with existing heuristic and skill optimization methods (FSgw, 3BaB).

2. Rebuttal Summary.

We took every concern seriously and answered each one with new evidence; Reviewer 4PuW raised the rating from 3 to 4, and Reviewer 3BaB confirmed that "the additional evidence has addressed my concerns" and kept the positive score.

  • Cosine schedule: The cosine schedule beats a fixed radius at every Sudoku horizon (Table 1), and its automatic extension performs on par with the hand-set schedule on Math and DocVQA (App. D.7.1).
  • Trajectory length and credit assignment: We keep the H∗=10H^*=10 Sudoku puzzles fixed and split each action into three turns. Tripling the trajectory length lowers Agentic GRPO‡ from 67.01 to 53.82 but raises Agentic ESOpt from 61.81 to 68.40 (App. D.2.1).
  • Training seeds: With three training runs at each Sudoku horizon, the ranking is the same as in Table 1 (App. D.2). With a new training seed on Math and DocVQA, Agentic ESOpt again outperforms Agentic GiGPO on all eight metrics of Table 2.
  • Joint optimization: Adding Agentic ESOpt improves every external component, e.g., SkillOpt on WebArena-Lite from 28.89 to 34.75 (App. D.4.3).
  • Learned behavior: Agentic ESOpt cuts the average number of turns on Math from 20.68 to 6.30, mostly turns without a valid action (App. D.3.3).

However, Reviewer FSgw gave the only negative rating (2), with confidence 2 ("little or no direct research experience on this topic"), and described the soundness concerns as "minor or fixable". FSgw has not responded to our rebuttal or to our follow-up, so this rating does not take our responses into account.

Overall, two of the three reviewers rate the paper positively and support it after the rebuttal, and the concerns of Reviewer FSgw are answered with new experiments in the revised paper. We would be grateful if you could take these responses into account in your recommendation. We are confident that this paper will contribute to ICLR 2027 and the whole community.

Thank you again for your time and consideration.

Best Regards,

Paper1510 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.