acceptodds
Under review as a conference paper at ICLR 2027

Stabilizing Zeroth-Order Fine-Tuning of Mixture-of-Experts via One-Sided Estimation and Expert Replay

Abstract

Zeroth-order (ZO) optimization enables memory-efficient fine-tuning through forward loss queries. In sparse mixture-of-experts (MoE) models, however, parameter perturbations can change the experts selected for a token. This path inconsistency mixes continuous parameter effects with discrete routing changes in finite-difference estimates. We propose ZOMoE, a ZO fine-tuning algorithm that combines expert replay with one-sided estimation. At each update, an unperturbed forward pass records the selected expert indices. Perturbed queries reuse these indices while recomputing routing weights and other continuous computations. Expert selections are refreshed at the next update, allowing routing to adapt throughout training. The recording pass also supplies a shared loss baseline, so replay adds no forward queries to the one-sided estimator. Mathematically, we isolate the term introduced by expert switching in the finite-difference estimate and show that expert replay removes this term. Across two sparse MoE backbones and six downstream tasks, ZOMoE achieves the highest average accuracy among the evaluated ZO methods, accelerates loss descent, and adds negligible memory or runtime overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
zeroth-order optimization, mixture-of-experts, large language models, memory-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iPfJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why zeroth-order (ZO) fine-tuning struggles on sparse mixture-of-experts (MoE) language models, hypothesizing that the difficulty stems in part from “path inconsistency”: parameter perturbations can change which experts a token is routed to, so the loss queries in a finite-difference estimate may evaluate different expert subnetworks and become incomparable. The authors document this perturbation-induced expert switching on two MoE backbones and propose ZOMoE, which freezes the router weights, records expert indices in an unperturbed forward pass, replays them inside all perturbed queries of each update while keeping continuous routing responses adaptive, refreshes the selections every update, and reuses the recording pass as the shared baseline of a one-sided estimator at no extra query cost. They accompany the method with a stationarity guarantee under smoothness, minibatch-noise, and a controlled expert-reselection assumption, and evaluate it on two backbones across classification and question-answering tasks, reporting the best average accuracy among ZO baselines with negligible memory and runtime overhead.

Critical strengths
  1. Clear technical motivation. The paper identifies perturbation-induced expert switching as a source of inconsistency in zeroth-order estimates and formally decomposes its contribution.

  2. Efficient intervention with promising results. Expert replay introduces negligible measured memory and throughput overhead, while the reported average accuracy improves over FZOO on both evaluated backbones.

Critical weaknesses
  1. Insufficient evidence for robust gains attributable to replay. The main comparison confounds expert replay with router freezing, and it is unclear whether the FZOO baseline received the same coefficient clipping and hyperparameter grid. All experiments use a single random seed without uncertainty estimates. Although the ablations partially isolate replay’s contribution, its benefit varies across tasks and perturbation scales, and on several small test sets the reported differences correspond to only a few examples. The robustness and magnitude of the claimed improvements therefore cannot be assessed.

  2. The convergence analysis does not cover the experimental setting and explain the empirical gains. The guarantee requires a perturbation scale far smaller than those used in the experiments, so it does not apply to the reported runs. It relies on Assumption 4.3 to control changes in the replay-conditioned objective when expert selections are refreshed, and the paper does not establish that this assumption holds in the evaluated MoE models. Moreover, the analysis provides no comparison with the same update without replay, so it does not support the claim that replay is responsible for the observed improvements.

Questions for authors
  1. Can the authors repeat the main comparisons (the main results table and the router-freezing/replay ablation) across multiple random seeds and report uncertainty estimates, to establish whether the gains attributable to replay are robust?

  2. The convergence guarantee requires a perturbation scale far smaller than those used in the experiments. Can the authors clarify whether any part of the theory applies to the reported settings? Can they also empirically check Assumption 4.3 on the evaluated backbones, for example by measuring the loss change caused by expert reselection between consecutive updates?

  3. Can the authors measure, during the actual fine-tuning runs, how often expert selections change under perturbation and how large the switching term is relative to the parameter-response term? This would show whether the observed gains come from removing the switching term, as the paper claims.

1 comment

Dear Reviewer iPfJ,

We have replied to each of your three follow-up comments (W1, W2 and Q2) in its thread, with the requested clarifications and new measurements. We believe these replies resolve the points you raised, and we kindly ask you to consider raising your score.

Best regards, The Authors

Reviewer_ZpnMOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies instability in zeroth-order fine-tuning of sparse MoE language models caused by perturbation-dependent changes in expert selection. ZOMoE records expert indices during an unperturbed forward pass and reuses them for the perturbed queries within an update, while recomputing continuous quantities such as hidden states and routing weights. The recording pass also provides the shared baseline for one-sided estimation. Expert selections are refreshed between updates. The paper provides an estimator decomposition, a conditional convergence analysis, and experiments on two MoE backbones with classification and question-answering tasks.

Critical strengths

The paper identifies a clear optimization issue and proposes a direct solution. Fixing expert indices within each finite-difference estimate separates continuous parameter effects from discrete changes in the executed subnetwork. Reusing the recording pass as the one-sided loss baseline is a sensible design that avoids an additional forward query for replay.

The controls are a strength. Table 2 separates expert replay from router freezing, so the improvement is not attributed to replay solely by comparing two substantially different end-to-end configurations. The appendix also examines replay with central differences. The matched memory and throughput measurements support the claim that replay itself adds little overhead in the evaluated implementation.

Critical weaknesses
  1. The related-work discussion should include more recent optimization and adaptation methods. Please update the discussion to include AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments, LOZO+: Provably Efficient Zeroth-Order Fine-Tuning via Greedy Low-Rank Subspace Selection, and CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning, all published at ICML 2026. These provide relevant context for improving ZO convergence and perturbation quality. Other relevant additions are Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs (ICML 2026), Fine-tuning Quantized Neural Networks with Zeroth-order Optimization (ICLR 2026), and LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts (ICLR 2026). Please distinguish optimization methods from broader routing-based adaptation work and explain which mechanisms could complement replay. This is a discussion request, not a requirement to experimentally compare every cited method.

  2. A stronger contemporary ZO comparison and a practical PEFT reference would strengthen the empirical positioning. I would prioritize an experimental comparison with AGZO: Activation-guided zerothorder optimization for LLM fine-tuning, which is already cited but not evaluated. Its activation-guided perturbations provide a relevant alternative approach to improving estimator quality. A representative comparison on one backbone and a small task subset would be sufficient to make this more informative; reproducing every recent optimizer across the entire suite is unnecessary. For the broader memory-efficiency claim, a first-order LoRA or QLoRA experiment would provide useful practical context. Full-parameter AdamW running out of memory does not establish that first-order adaptation is generally infeasible. Because PEFT and full-parameter ZO optimize different parameter sets, this should be presented as a separate quality–memory trade-off rather than an identical optimization comparison.

Questions for authors

Does ZOMoE retain its advantage when the central-difference baselines receive a comparable perturbation-scale search?

How does it compare with AGZO, and what quality–memory trade-off does it offer relative to first-order PEFT?

Can you provide repeated-run results for the principal comparisons and empirical evidence about the size of the loss changes caused by expert reselection?

See weakness.

Reviewer_Fxk3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the ZO optimization problem of sparse MoE models. It observes that parameter perturbations can change the experts selected for a token, causing finite-difference estimates to mix continuous parameter effects with discrete expert-switching effects. To address this, the authors propose ZOMoE, which uses expert replay to maintain consistent expert selections within each update and combines it with one-sided estimation to construct the update direction. Extensive experiments on two sparse MoE backbones across multiple downstream tasks show the effectiveness of ZOMoE.

Critical strengths
  1. The analysis is insightful. The authors identify a potential conflict between discrete routing in sparse MoEs and ZO optimization, and use a mathematical decomposition to distinguish continuous parameter effects from expert-switching effects.

  2. The proposed method is straightforward and introduces little additional overhead.

Critical weaknesses
  1. The problem studied in this paper mainly arises from applying ZO optimization to sparse MoEs. Without a clearer explanation of its advantages over existing memory-efficient fine-tuning methods, such as comparisons with LoRA and QLoRA in terms of performance and computational cost, the practical significance of this problem remains unclear.

  2. The novelty may be limited. The proposed method mainly adds expert replay to an existing ZO optimization framework, and the overall design is relatively straightforward.

Questions for authors
  1. I am curious about how much the expert-switching term in Eq. (5) affects gradient estimation in practice. A quantitative analysis of its impact would help better support the paper's motivation.

  2. Could the authors include LoRA and QLoRA as additional baselines and compare final performance, wall-clock training time, and computational cost under similar GPU memory budgets? This would help better assess the practical value of ZOMoE.

  3. Although the paper shows that expert replay removes the expert-switching term from ZO estimates and provides a convergence guarantee under certain assumptions, its theoretical advantages over methods without replay remain unclear. Could the authors further clarify what additional insights expert replay brings to the convergence analysis?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for reviewer engagement before the discussion deadline (Nov 18 AoE)

Dear Area Chair,

With the author–reviewer discussion closing on November 18 (AoE), we have not yet received any reply from the reviewers, including to the reminders we posted in each reviewer's thread. Since our responses consist mainly of new experiments that the reviewers themselves requested, we believe a brief look could resolve their main concerns. Could you kindly send a reminder to Reviewers iPfJ and Fxk3, asking them to check whether the new results address their concerns? As reviewers can still update their assessments during the reviewer–AC discussion, we would also be grateful if, should they not respond before the deadline, you could raise these points with them in that phase.

Thank you very much for your time and help.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.