- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates evolution strategies (ES) for task-specific adaptation of long-horizon LLM agents under limited GPU resources. The authors propose Agentic ESOpt, a forward-only, full-parameter optimization framework that incorporates a cosine perturbation schedule and can be combined with external optimization components such as heuristic search and skill optimization. The paper evaluates the approach on controlled multi-turn Sudoku, Math reasoning, DocVQA, WebArena-Lite, and automatic heuristic design (AHD). The results demonstrate promising long-horizon adaptation performance, reduced GPU-memory requirements, and compatibility with existing agent optimization pipelines. The paper presents empirical findings regarding long-horizon adaptation, demonstrates scalability to larger models under constrained GPU resources, and explores a flexible interface for integrating parameter adaptation into existing agent optimization workflows.
Critical strengths
-
The paper explores a problem that is both practically important and clearly defined: how to efficiently tune long-horizon agents under GPU resource constraints.
-
The Sudoku experiments provide compelling empirical evidence of horizon-dependent behavior. In particular, the combination of external structures and cosine decay strikes me as a particularly novel aspect of the approach.
-
The experiments go beyond simple controlled environments to encompass multi-turn tool use, full-parameter adaptation of a 30B MoE model, and integration with external optimization pipelines.
Critical weaknesses
- Limited analysis of the behaviors learned through ES adaptation. There is no direct examination of what Agentic ESOpt actually learns during adaptation. For example, does it improve tool-use decisions, reduce redundant interactions, or recover more effectively from mistakes? I would recommend to analyze:
- Parameter distribution before and after training
- The number of Turns before and after training
- etc
- The consine schedule introduces more hyperparameters although relatively good effect. Is there any solution or intuition for the hyperparameter tuning issues associated with the cosine schedule?
Questions for authors
-
Why is task-specific adaptation the primary target of Agentic ESOpt? How do you envision the boundary between settings where ES is preferable and those where conventional RL post-training remains more suitable?
-
The paper provides useful estimates of training FLOPs and GPU-memory. Could you clarify which components are included in the FLOPs accounting, and how these estimates relate to the measured wall-clock results?
-
Math and DocVQA use only half the FLOPs, how will Agentic ESOpt performs when enlarging the total FLOPs of Agentic ESOpt to the same level as RL.
-
How sensitive are the results to perturbation scale and decoding hyperparameters? For the population-size experiments and the zero-terminal-noise ablation, were the perturbation scale and generation temperature independently tuned for each configuration, or held fixed to isolate the variables under study? Clarifying this would help interpret the reported scaling and ablation results.
-
Continual learning is crucial for agentic learning; does the method proposed in this paper raise any concerns regarding continual learning?
-
Following up on "weakness 1": How do the parameters and turns change after agentic ESOPT training?
-
Following up on "weakness 2": Is there a solution for the hyperparameter tuning issues associated with the cosine schedule?
-
The discussion of related work on automated heuristic design (AHD) focuses primarily on EoH. A more comprehensive review of recent advances in automated algorithm design and learning-based optimization would help better position the proposed approach within the broader research landscape. Relevant references include, but are not limited to:
- Automated Algorithm Design for Black-Box Optimization: Progress and Challenges
- B2Opt: Learning to Optimize Black-box Optimization with Little Budget
- A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving
- Figure 1 serves as the main motivational figure, but its current presentation appears visually cluttered. In particular, panel (c) is somewhat unclear, especially with regard to the depiction of the function landscape in the background. The authors are encouraged to refine and improve the figure for better visual clarity and overall presentation.
5 comments
Response to Reviewer 4PuW (2/3)
Weakness 2 & Question 7. Hyperparameters of the Cosine Schedule: The consine schedule introduces more hyperparameters although relatively good effect. Is there any solution or intuition for the hyperparameter tuning issues associated with the cosine schedule?
Thanks for your question. We would like to clarify that Math and DocVQA share one hand-set schedule on both backbones (Table 29), and during the rebuttal period we propose an extension of the cosine schedule that sets and from rollout rewards, as an alternative to setting them by hand (Details in Sec. 3.1 and Appendix D.7.1):
-
Mechanism. A permutation test on rollout rewards checks whether the perturbed agents are ranked beyond rollout noise. Before training, is the largest scanned radius that carries this ranking signal without collapsing the reward, and . During training, decays from to with the cosine schedule, and the decay can be held.
-
Result. On both Math and DocVQA, the extension selects , the hand-set value of the paper. As shown in the table below, its results match those of the hand-set schedule:
| Qwen3.5-4B | DAPO Mean@4 | AIME 2026 Mean@4 | DocVQA ANLS Mean@4 | DocVQA Accuracy Mean@4 |
|---|---|---|---|---|
| Base LLM (Table 2) | 63.0 | 55.8 | 0.3875 | 40.3 |
| Agentic ESOpt, hand-set (; Table 2) | 76.8 | 70.8 | 0.5043 | 52.5 |
| Automatic extension | 76.25 | 70.00 | 0.5047 | 52.50 |
- Motivation and intuition. Lemma 1 shows that a larger regularizes more but biases the objective more, so decays. Over three training seeds at , the cosine schedule reaches 54.86 ± 2.19, versus 44.79 ± 2.19 and 46.88 ± 3.00 with a fixed (Table 12). In practice, should be large enough that the perturbations change the ranking of the population beyond rollout noise, and small enough that the perturbed agents still solve the task.
Question 1. Scope of Task-Specific Adaptation: Why is task-specific adaptation the primary target of Agentic ESOpt? How do you envision the boundary between settings where ES is preferable and those where conventional RL post-training remains more suitable?
Thanks for your question. We target task-specific adaptation because it brings together the conditions where ES helps: long interactions, terminal rewards, limited GPU memory, and one environment with a modest number of training tasks (e.g., 400 Math problems and 50 DocVQA questions).
ES is preferable for long horizons with terminal rewards, where action-level credit assignment is hard (Table 1, ), under tight GPU memory, since it needs only inference-level memory (Tables 1 and 9), and when external structures such as skills or heuristics are optimized together with the LLM (Tables 3, 17, and 20). RL post-training remains more suitable for short horizons or dense turn-level rewards, for broad post-training across many tasks and large datasets, and when environment evaluation dominates the cost, since ES evaluates more trajectories (Appendix A.1).
Question 2. FLOPs Accounting and Wall-Clock: Could you clarify which components are included in the FLOPs accounting, and how these estimates relate to the measured wall-clock results?
Thanks for your question. FLOPs count the model-side training computation only (Appendix C.5). For parameters and processed tokens, a forward pass costs about and a backward pass about . Agentic ESOpt costs one policy forward per trajectory (); GRPO adds a reference forward and a policy backward ( in total); PPO also adds a critic forward and backward ( in total). Environment execution is excluded. For Table 7, is the measured number of processed tokens, so longer trajectories increase the measured FLOPs.
The wall-clock time in Table 7 is measured on the same four H100 GPUs, and the two agree in direction. On Sudoku, matches the model-side budget of eight-rollout GRPO by design (). As shown below, the measured FLOPs are comparable, and Agentic ESOpt finishes training faster at every horizon:
| Method | |||
|---|---|---|---|
| Agentic GRPO | 3.2 EFLOPs / 5.4 h | 7.6 EFLOPs / 13.1 h | 10.9 EFLOPs / 19.0 h |
| Agentic ESOpt () | 3.1 EFLOPs / 3.1 h | 6.3 EFLOPs / 5.8 h | 9.4 EFLOPs / 9.4 h |
Response to Reviewer 4PuW (3/3)
Question 3. Matching the FLOPs of RL on Math and DocVQA: Math and DocVQA use only half the FLOPs, how will Agentic ESOpt performs when enlarging the total FLOPs of Agentic ESOpt to the same level as RL.
Thanks for your question. In Table 2, Agentic ESOpt with already outperforms Agentic GRPO and GiGPO at about half of their budgeted model-side FLOPs. We further run Agentic ESOpt with , which doubles these FLOPs and matches Agentic GRPO on Math. As shown in the table below (Qwen3.5-4B, Mean@4), it still outperforms both baselines:
| Method | DAPO | AIME 2026 | DocVQA ANLS |
|---|---|---|---|
| Agentic GRPO | 68.8 | 58.3 | 0.4627 |
| Agentic GiGPO | 73.3 | 65.0 | 0.4846 |
| Agentic ESOpt () | 76.8 | 70.8 | 0.5043 |
| Agentic ESOpt () | 75.25 | 75.83 | 0.5472 |
Question 4. Sensitivity to Perturbation Scale and Decoding: For the population-size experiments and the zero-terminal-noise ablation, were the perturbation scale and generation temperature independently tuned for each configuration, or held fixed to isolate the variables under study?
Thanks for your question. Both were held fixed as a preliminary setting to isolate the variables under study. The population-size runs (Table 5 and Appendix D.1.4) use and temperature 0.7 for every , and the ablation in Table 1 uses the same and decoding as the full method. We did not tune the perturbation scale or the temperature for each , and we leave this tuning to future work.
Question 5. Continual Learning: Continual learning is crucial for agentic learning; does the method proposed in this paper raise any concerns regarding continual learning?
Thanks for your question. We agree that continual learning is crucial for agentic learning. During the rebuttal period, we evaluate each Qwen3.5-4B Agentic ESOpt checkpoint of Table 2 on the task it was not trained on (Detailed in Appendix D.4.5):
| Model | DAPO | AIME | ANLS | Acc. |
|---|---|---|---|---|
| Base LLM | 63.0 | 55.8 | 0.3875 | 40.3 |
| Agentic ESOpt (trained on Math) | 76.8 | 70.8 | 0.3605 | 37.8 |
| Agentic ESOpt (trained on DocVQA) | 65.0 | 57.5 | 0.5043 | 52.5 |
On the task it was not trained on, each checkpoint stays close to the base LLM, and we observe no catastrophic forgetting. In addition, the ES update is concentrated in magnitude: on Qwen3.5-27B, 96.26% of the parameter changes stay within the perturbation scale (Table 6). Measuring retention over longer task sequences is our future work, which we have added to Appendix A.2 in the revised manuscript.
Question 8. Related Work on Automated Algorithm Design: Automated Algorithm Design for Black-Box Optimization: Progress and Challenges; B2Opt: Learning to Optimize Black-box Optimization with Little Budget; A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving.
Thanks for the suggestion. These works study automated algorithm design, learning to optimize black-box problems, and LLMs for evolutionary optimization, which relate to the heuristic-design setting in Sec. 4.3. We cite and discuss B2Opt [1] and the survey on LLMs for evolutionary optimization [2] in Appendix B.4.
Question 9. Figure 1: Figure 1 ... appears visually cluttered. In particular, panel (c) is somewhat unclear, especially with regard to the depiction of the function landscape in the background.
Thanks for pointing this out. We will simplify Figure 1.
We sincerely hope our responses have addressed all your concerns. If you have any further questions, please feel free to follow up, and we are glad to address them.
References
[1] Li, Xiaobin, et al. "B2Opt: Learning to Optimize Black-box Optimization with Little Budget." AAAI 2025.
[2] Zhang, Yisong, et al. "A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving." arXiv:2509.08269, 2025.
Response to Reviewer 4PuW (1/3)
Thank you for your time and effort in reviewing our work. We are glad to know that you find the problem practically important and clearly defined, the Sudoku experiments compelling evidence of horizon-dependent behavior, and the combination of external structures and cosine decay novel.
We address your concerns as follows, and all revisions are marked in blue in the revised PDF.
Thanks for your suggestion. We analyze what Agentic ESOpt learns through three research questions on its behavior and its parameters.
Appendix D.3.3):The same holds during training (
Figure 7): on Sudoku (H∗=15), Agentic ESOpt stays near the 15-turn minimum (about 15.3 turns at step 80, versus 18.5 for Agentic GRPO), and on DocVQA its average turns fall from about 30 to about 15, while those of Agentic GRPO rise to about 46.RQ2: Does Agentic ESOpt improve tool-use decisions? Yes, it wastes far fewer turns without a valid action and almost always ends with a final answer. The removed turns are almost all turns without a valid action, while the number of valid actions per trajectory stays about the same (4.31 versus 4.34).
RQ3: How do the parameters change after training? The update is spread almost evenly over layers and modules, and stays small. We compare the weights of Qwen3.5-4B before and after training (update RMS over all coordinates):
Normalization weights change less than the projections. On Qwen3.5-27B for WebArena-Lite, 96.26% of the parameter changes are within σ=1.5×10−3 and 99.42% within 2×10−3 (Table 6). Overall, Agentic ESOpt mainly learns to remove redundant interactions, through a small update spread over all parameters.