acceptodds
Under review as a conference paper at ICLR 2027

Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

Abstract

Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data. A major obstacle in large-scale fine-tuning is the memory overhead of backpropagation, which requires storing activations, gradients, and optimizer states. Zeroth-order (ZO) optimization offers a memory-efficient alternative, but its performance is highly sensitive to the stepsize and smoothing parameter, often requiring costly task-specific tuning. Parameter-free (PF) optimization addresses this issue by adapting algorithmic parameters without prior knowledge of problem-dependent constants. Moreover, large-scale fine-tuning can benefit from geometry-aware updates that account for the heterogeneous structure of parameter blocks, which can be modeled through methods that exploit linear minimization oracle (LMO). In this work, we study PF adaptation for LMO-based ZO optimization and introduce , a method that unifies gradient-free training, adaptive tuning, and non-Euclidean update geometry. We establish convergence guarantees and validate the method on large-scale LLM fine-tuning task with model.

Primary area:
optimization
Keywords:
convex optimization, stochastic optimization, zeroth order, fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vjP1OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies parameter free adaptation for linear minimization oracle on zero-order optimization. Motivated by the fact that ZO methods needs careful hyper parameter selection, the proposed method uses the local-smoothness stepsize idea and applies to ZO setting. Theoretical analysis are provided on convergence. Experimental results are provided on Opt-1.3B on SST-2 dataset with LoRA.

Critical strengths

The idea of reusing the same perturbation at adjacent updates is interesting and is a valid contribution. This helps with analysis and cancels some terms nicely.

Critical weaknesses

The experiment results are very thin and limited. Just one model over one dataset and test case, with LoRA rather than full parameter FT. There's no error bars and it's a single seed. More importantly, no training curve are shown for those methods. It's unclear how other methods work under same 35 wall clock hours. The training configurtions are also unclear for those baselines compared. There's also no empirically results and discussions on training stability and remaining hyper-parameters.

Questions for authors
  1. More experiments and details need to be provided. See weakness.

  2. What hyper parameters are eliminated, preserved, and introduced? How does the algorithm work with different hype-paramters? How sensitive is it?

  3. Can you compare with MeZO? The algorithm seems requiring more compute (function evaluations) than MeZO. How does that affect memory usage and wall clock runtime?

Reviewer_L82vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AdaNAGED, a parameter-free zeroth-order optimization method incorporating linear minimization oracles (LMOs). It adaptively estimates local smoothness to adjust the stepsize and smoothing parameter without requiring prior knowledge of problem-dependent constants. The authors establish convergence guarantees and extend the framework to SignSGD-like and Muon-like updates. Experiments on OPT-1.3B fine-tuning demonstrate competitive accuracy with reduced memory consumption and hyperparameter tuning costs.

Critical strengths
  1. The paper addresses the practical challenge of hyperparameter sensitivity in zeroth-order optimization. Its integration of parameter-free adaptation with LMO-based updates is interesting and supports different optimization geometries within a unified framework.
  2. The theoretical analysis attempts to establish convergence guarantees for both exact and approximate LMO updates. Experimental results also suggest competitive accuracy and improved memory efficiency compared with existing baselines.
Critical weaknesses
  1. The theoretical analysis raises concerns regarding the zeroth-order estimator, particularly the missing dimension-dependent normalization in its claimed unbiasedness relationship. Inconsistencies between the local smoothness bounds in the main text and appendix further weaken confidence in the convergence guarantees.
  2. Experimentally, evaluation on only OPT-1.3B and SST-2 is insufficient to demonstrate generalizability. The absence of repeated trials, ablation studies, and clearly specified hyperparameter search budgets makes the reported improvements difficult to assess.
Questions for authors
  1. Could the authors clarify why the proposed zeroth-order estimator does not require dimension-dependent normalization and how this affects the convergence analysis?
  2. How can the different local smoothness bounds in Lemma 3.1 and Appendix C be reconciled?
  3. Could the authors clarify the hyperparameter tuning budgets used for the baselines and provide comparisons under matched computational budgets?
  4. Could the authors provide results across multiple random seeds and additional tasks to demonstrate the robustness and generalizability of AdaNAGED?
Reviewer_EVEoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to achieve parameter adaptation in zeroth-order optimization based on Linear Minimization Oracles (LMOs), aiming to reduce the reliance on manual hyperparameter tuning in large language model fine-tuning. The proposed method combines zeroth-order gradient estimation, parameter-free stepsize adaptation, and non-Euclidean geometry-aware updates.

The theoretical analysis studies the convergence properties of the algorithm under convexity, smoothness, and Lipschitz continuity assumptions, and further considers the use of Newton-Schulz iterations to approximate the matrix polar decomposition. Experiments are conducted on the SST-2 fine-tuning task using OPT-1.3B. The results show that the proposed method achieves performance comparable to tuned ZO-SignSGD, ZO-Muon, and ZO-AdaMM baselines, while reducing the additional computational cost associated with hyperparameter search.

Critical strengths

Zeroth-order optimization can effectively reduce the memory overhead associated with backpropagation during large language model fine-tuning. However, its practical performance is often sensitive to the choice of stepsize and perturbation radius, requiring task-specific hyperparameter tuning. This paper addresses an important problem with potential value in reducing the practical cost of large language model fine-tuning, making it a research direction worthy of further investigation.

Critical weaknesses
  1. Appendix B treats the proposed single-direction finite-difference estimator as an unbiased estimator of the gradient of the smoothed objective. However, under the unit-sphere random perturbation distribution and the definition of the smoothed objective adopted in the paper, this identity does not generally hold as stated.

  2. The experimental evaluation is limited to the SST-2 fine-tuning task using OPT-1.3B. Although AdaNAGED achieves performance comparable to tuned zeroth-order optimization baselines, its accuracy improvement over ZO-Muon is relatively small. Furthermore, the limited experimental scope is insufficient to establish the generalizability of the proposed method.

Questions for authors
  1. Could the authors further explain how the estimation error term in Appendix C, which is independent of T, is consistent with the O(1/ε²) convergence complexity claimed in Theorem 3.2? Additionally, how can the unbiasedness relationship between the finite-difference estimator and the gradient of the smoothed objective in Appendix B be rigorously established under the adopted unit-sphere perturbation distribution?

  2. Could the authors further validate the applicability and effectiveness of AdaNAGED across a broader range of tasks and model architectures?

  3. Could the authors provide more detailed analyses or visualizations of convergence behavior, such as convergence time and convergence curves? It would also be helpful to include a more comprehensive evaluation of computational overhead, memory consumption, and downstream task performance to better assess the practical efficiency and effectiveness of the proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.