acceptodds
Withdrawn from ICLR 2027

Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-evolving methods improve the capabilities of LLM agents by sampling trajectories from the underlying LLMs and learning from these trajectories. However, these methods struggle to learn beyond the inherent capability boundary of the agents, since successful trajectories can be rare on difficult examples for further improvements.In this paper, we propose a zeroth-order self-evolution framework that enables agents to learn beyond their capability boundary by perturbing LLM parameters to adapt to difficult examples without any trajectory annotations.Specifically, we perturb LoRA parameters of LLMs, run the agent, compute the losses under the perturbed and original parameters, and use the loss difference to estimate gradients and further update the LoRA parameters. We sample trajectories using the updated LLMs for supervised fine-tuning to break through the capability boundary of the agents, forming a closed self-evolution loop.We introduce a parallel perturbation inference mechanism and an adaptive lookup mechanism to reduce time consumption in zeroth-order optimization, with an answer perplexity loss that provides smooth and stable zeroth-order loss values. Experiments on multiple deep research benchmarks show that our method obtains substantially more successful trajectories and consistently outperforms strong baselines, especially on difficult examples.

Primary area:
optimization
Keywords:
Deep research, LLM Agent, Self-Evolving, Zeroth-order Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5unqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to use answer supervision to improve hard-problem trajectory discovery for LLM agents when annotated reasoning trajectories are unavailable. The method applies zeroth-order optimization to instance-specific LoRAs, using trajectory-conditioned answer NLL as feedback, and uses the generated trajectories for subsequent SFT.

Critical strengths
  1. For each training question, an independent LoRA is perturbed and updated via zeroth-order optimization to regenerate tool-use trajectories, which are then distilled into a single agent through SFT. Rather than simply sampling more from a fixed policy, the method changes the policy so that successful trajectories become easier to find.

  2. The NLL of the gold answer, conditioned on the generated trajectory, serves as a scalar ZO signal in place of sparse correctness rewards. This requires only question-answer pairs and distinguishes trajectories that binary rewards cannot.

  3. Parallel evaluation of multiple LoRA perturbations and tool-result caching via exact or semantic matching reduce the cost of model evaluation and repeated search and browsing.

Critical weaknesses

W1. The novelty is not clearly delineated from related work.

  • Evolution Strategies at Scale [1] studies parameter-space perturbation with response-level feedback for LLM fine-tuning
  • MeZO [2] LOZO [3] establish forward-only and low-rank ZO optimization and likelihood-based evaluation of tool usefulness [6].

The novelty therefore lies in integrating these ideas for instance-specific, answer-supervised trajectory discovery and distillation, not in the individual principles.

Please clarify this and provide matched-budget, end-to-end comparisons. The parallel LoRA inference contribution should also be positioned against S-LoRA [4] and Punica [5], distinguishing shared backbone weights from shared backbone computation.

[1] Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning. 2025

[2] Fine-Tuning Language Models with Just Forward Passes. NeurIPS 2023.

[3] Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures. ICLR 2025.

[4] S-LoRA: Serving Thousands of Concurrent LoRA Adapters. 2023

[5] Punica: Multi-Tenant LoRA Serving. MLSys 2024.

[6] Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023.

W2. Parallel multi-LoRA inference should be positioned against S-LoRA and Punica. Sharing backbone weights, batching heterogeneous adapters, and maintaining independent request states are well established in existing systems. The paper neither adequately cites this work nor shows that its approach goes beyond standard multi-LoRA batching.

In addition, the Agentic ESOpt work directly studies ES-based parameter optimization for long-horizon agents. It should be discussed as related work. Overall, the contribution lies in the combination rather than in new ZO principles, and its value depends on mechanistic validation and full-pipeline comparisons.

W3. Full backbone forward sharing is not justified (Eq. (12)). WxWx can be shared only within a single layer with identical input. In a multi-layer Transformer, each adapter changes the layer output, so later inputs, attention, KV caches, generated tokens, and tool calls all diverge across branches. Shared weights do not imply shared activations or a single backbone forward. If the implementation only batches separate branches, the speedup and novelty claims should be revised. If it reuses unperturbed activations, it approximates rather than computes Eq. (7), and the error and its effect on the gradient estimator must be analyzed.

Please compare against independently run perturbed models under fixed inputs, decoding randomness, and tool outputs, reporting per-token logit errors and generation agreement.

W4. Missing a hybrid-gradient baseline for the same objective. Please add a baseline combining the FO answer-loss gradient with a policy gradient. Otherwise, the advantage of ZO may come from covering both terms rather than from a better zeroth-order estimator.

W5. Table 2 compares 67 trajectories from the initial policy with the larger set generated by the proposed method, so the gains may come from more data, broader question coverage, or higher trajectory quality.

Please compare under matched trajectory and token counts, and pass data from random perturbation, answer-conditioned search, and RL through the same SFT protocol.

Questions for authors

See Weakness

Reviewer_idHNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes using perturbations to identify variations of a base model in reasoning tasks, using LoRA perturbations and a perplexity loss. Experiments show that the method increases reasoning ability of Qwen 4-8B models.

Critical strengths

Test-time scaling has become an important way to improve performance of LLMs. Zeroth order methods are a new and promising approach to TTS. This paper focuses on long-horizon reasoning in particular. The two ideas of making it practical, i.e. focusing on LoRA and using a perplexity loss are compelling and worth exploring.

Critical weaknesses

The results should be put in the context of alternatives; otherwise it is hard to evaluate what they contribute.

Questions for authors

Evolution Strategies appears to be a powerful zeroth order method (see e.g. https://arxiv.org/abs/2509.24372, https://arxiv.org/abs/2511.16652). It actually makes full-parameter optimization possible, i.e. suggests that it is not necessary to focus perturbations to LoRA only. In order to understand how well the method in this paper works, it is important to compare to ES. It could be applied to LoRA as well--that's one comparison. Another one is full-parameter optimization. How well does your method work on full parameters? Third, how does the full-parameter ES compare with LoRA-only optimization (yours or ES)? Without those comparisons, it is hard to say how well your method really works.

A similar comparison is needed for the perplexity loss. What would happen if you didn't have it? In principle, zeroth-order methods should be useful because they don't care about horizon length---you get the reward in the end regardless of how long the reasoning chain is. So what does the perplexity applied to the trajectory actually buy you? That's an easy experiment to run, and if it works, would make an intriguing result.

The pass@n results should be extended to larger n. Performance is still increasing, and in light of other recent results (e.g. https://arxiv.org/abs/2608.12679) there's more to discover. That is, RL may saturate, and the base model may exceed it. It would be interesting to see whether your method stays above both as n increases to 128 and beyond.

The list of contributions in the intro is instead a summary of your work. They are activites you performed: they may or may not contribute to science. Here you actually have to think about what they contribute, and list the contributions instead. For instance, introducing a method is not a contribution. Even if that method improves performance, it is not quite a contribution. If you identify a problem with the current state of the art, design a method to solve that problem, and demonstrate that it does, then it is a contribution. That's how you need to phrase the contributions.

Reviewer_YkJ5OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Zeroth-Order Optimization (ZO)-based self-evolution framework to address the limited availability of successful trajectories for challenging tasks in existing self-evolving LLM agents. The proposed method perturbs instance-specific LoRA parameters for each problem and estimates gradients based on the difference in answer loss between the original and perturbed models. By exploring the model's parameter space without requiring intermediate trajectory annotations, the framework discovers additional successful trajectories, which are subsequently used for supervised fine-tuning (SFT) to improve agent performance. The framework further incorporates parallel perturbation inference to reduce the computational overhead of multiple perturbations, an adaptive lookup mechanism to avoid redundant tool calls, and an answer perplexity loss to provide a stable optimization signal. Experiments on GAIA and WebWalkerQA demonstrate the effectiveness of the proposed framework in improving agent performance and discovering successful trajectories.

Critical strengths

1. A Novel Approach to Improving Trajectory Discovery on Challenging Tasks.

This paper proposes ZO-based instance-specific LoRA adaptation to address the limited discovery of successful trajectories in existing self-evolving LLM agents. Unlike conventional sampling and trajectory refinement methods, the proposed approach directly explores the model's parameter space. The comparisons with various sampling and optimization baselines in Tables 3–5 provide empirical support for its effectiveness.

2. Comprehensive Experimental Validation and Analysis.

The proposed method demonstrates performance improvements across different models on GAIA and WebWalkerQA, particularly on challenging tasks. The Pass@N, matched-budget, optimization trajectory, and hyperparameter sensitivity analyses provide further evidence of its effectiveness. Additionally, the theoretical discussion of capability boundaries in Appendix A.2 strengthens the overall empirical findings.

Critical weaknesses

1. Insufficient Validation of Parallel Perturbation Inference

  • Section 3.3 assumes shared backbone computation across perturbation branches, but its feasibility remains unclear when autoregressive trajectories diverge in generated tokens, tool calls, and observations.
  • The sequential execution time in Table 6 is estimated rather than directly measured, making it difficult to distinguish the benefits of computation sharing from conventional batching or parallelization.
  • Further empirical validation is needed to substantiate the claimed computational efficiency.

2. Unclear Computational Budget and Attribution of ZO-Based Improvements

  • The accounting of optimization costs within the rollout budget is unclear, particularly for the N=1N=1 setting in Table 3, where both baseline and perturbed loss evaluations may be required.
  • Differences in the quantity and composition of SFT trajectories make it difficult to isolate the contributions of trajectory quality versus quantity.
  • Given prior work on LOZO and agentic RL, more rigorous comparisons are needed to establish the specific contribution of ZO-based trajectory discovery.
Questions for authors

Q1. How Is Backbone Computation Shared After Trajectory Divergence in Parallel Perturbation Inference?

  • Section 3.3 assumes shared backbone computation, but its feasibility is unclear when perturbation branches generate different tokens or receive different tool observations.
  • Could the authors clarify how computation sharing is implemented after trajectory divergence?
  • Could the speedup in Table 6 be decomposed into contributions from backbone sharing, batching, and parallel execution?

Q2. Does the Matched Rollout Budget in Table 3 Include All ZO Optimization Rollouts?

  • ZO requires both baseline and perturbed loss evaluations, raising concerns about how these costs are accounted for under the matched rollout budget.
  • Could the authors clarify how the ZO result for N=1N=1 is obtained?
  • Does the reported budget include all optimization rollouts or only the final sampling from the optimized policy?

Q3. How Strongly Does Answer Perplexity Loss Correlate with Trajectory Correctness?

  • Section 3.5 and Figure 4 demonstrate optimization convergence, but the relationship between decreasing answer perplexity loss and successful trajectory discovery remains unclear.
  • Could the authors compare trajectory discovery success rates across different continuous surrogate losses under matched optimization budgets?
  • Could the authors report the correlation between answer NLL and actual trajectory correctness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.