- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to use answer supervision to improve hard-problem trajectory discovery for LLM agents when annotated reasoning trajectories are unavailable. The method applies zeroth-order optimization to instance-specific LoRAs, using trajectory-conditioned answer NLL as feedback, and uses the generated trajectories for subsequent SFT.
Critical strengths
-
For each training question, an independent LoRA is perturbed and updated via zeroth-order optimization to regenerate tool-use trajectories, which are then distilled into a single agent through SFT. Rather than simply sampling more from a fixed policy, the method changes the policy so that successful trajectories become easier to find.
-
The NLL of the gold answer, conditioned on the generated trajectory, serves as a scalar ZO signal in place of sparse correctness rewards. This requires only question-answer pairs and distinguishes trajectories that binary rewards cannot.
-
Parallel evaluation of multiple LoRA perturbations and tool-result caching via exact or semantic matching reduce the cost of model evaluation and repeated search and browsing.
Critical weaknesses
W1. The novelty is not clearly delineated from related work.
- Evolution Strategies at Scale [1] studies parameter-space perturbation with response-level feedback for LLM fine-tuning
- MeZO [2] LOZO [3] establish forward-only and low-rank ZO optimization and likelihood-based evaluation of tool usefulness [6].
The novelty therefore lies in integrating these ideas for instance-specific, answer-supervised trajectory discovery and distillation, not in the individual principles.
Please clarify this and provide matched-budget, end-to-end comparisons. The parallel LoRA inference contribution should also be positioned against S-LoRA [4] and Punica [5], distinguishing shared backbone weights from shared backbone computation.
[1] Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning. 2025
[2] Fine-Tuning Language Models with Just Forward Passes. NeurIPS 2023.
[3] Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures. ICLR 2025.
[4] S-LoRA: Serving Thousands of Concurrent LoRA Adapters. 2023
[5] Punica: Multi-Tenant LoRA Serving. MLSys 2024.
[6] Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023.
W2. Parallel multi-LoRA inference should be positioned against S-LoRA and Punica. Sharing backbone weights, batching heterogeneous adapters, and maintaining independent request states are well established in existing systems. The paper neither adequately cites this work nor shows that its approach goes beyond standard multi-LoRA batching.
In addition, the Agentic ESOpt work directly studies ES-based parameter optimization for long-horizon agents. It should be discussed as related work. Overall, the contribution lies in the combination rather than in new ZO principles, and its value depends on mechanistic validation and full-pipeline comparisons.
W3. Full backbone forward sharing is not justified (Eq. (12)). can be shared only within a single layer with identical input. In a multi-layer Transformer, each adapter changes the layer output, so later inputs, attention, KV caches, generated tokens, and tool calls all diverge across branches. Shared weights do not imply shared activations or a single backbone forward. If the implementation only batches separate branches, the speedup and novelty claims should be revised. If it reuses unperturbed activations, it approximates rather than computes Eq. (7), and the error and its effect on the gradient estimator must be analyzed.
Please compare against independently run perturbed models under fixed inputs, decoding randomness, and tool outputs, reporting per-token logit errors and generation agreement.
W4. Missing a hybrid-gradient baseline for the same objective. Please add a baseline combining the FO answer-loss gradient with a policy gradient. Otherwise, the advantage of ZO may come from covering both terms rather than from a better zeroth-order estimator.
W5. Table 2 compares 67 trajectories from the initial policy with the larger set generated by the proposed method, so the gains may come from more data, broader question coverage, or higher trajectory quality.
Please compare under matched trajectory and token counts, and pass data from random perturbation, answer-conditioned search, and RL through the same SFT protocol.
Questions for authors
See Weakness