- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why zeroth-order (ZO) fine-tuning struggles on sparse mixture-of-experts (MoE) language models, hypothesizing that the difficulty stems in part from “path inconsistency”: parameter perturbations can change which experts a token is routed to, so the loss queries in a finite-difference estimate may evaluate different expert subnetworks and become incomparable. The authors document this perturbation-induced expert switching on two MoE backbones and propose ZOMoE, which freezes the router weights, records expert indices in an unperturbed forward pass, replays them inside all perturbed queries of each update while keeping continuous routing responses adaptive, refreshes the selections every update, and reuses the recording pass as the shared baseline of a one-sided estimator at no extra query cost. They accompany the method with a stationarity guarantee under smoothness, minibatch-noise, and a controlled expert-reselection assumption, and evaluate it on two backbones across classification and question-answering tasks, reporting the best average accuracy among ZO baselines with negligible memory and runtime overhead.
Critical strengths
-
Clear technical motivation. The paper identifies perturbation-induced expert switching as a source of inconsistency in zeroth-order estimates and formally decomposes its contribution.
-
Efficient intervention with promising results. Expert replay introduces negligible measured memory and throughput overhead, while the reported average accuracy improves over FZOO on both evaluated backbones.
Critical weaknesses
-
Insufficient evidence for robust gains attributable to replay. The main comparison confounds expert replay with router freezing, and it is unclear whether the FZOO baseline received the same coefficient clipping and hyperparameter grid. All experiments use a single random seed without uncertainty estimates. Although the ablations partially isolate replay’s contribution, its benefit varies across tasks and perturbation scales, and on several small test sets the reported differences correspond to only a few examples. The robustness and magnitude of the claimed improvements therefore cannot be assessed.
-
The convergence analysis does not cover the experimental setting and explain the empirical gains. The guarantee requires a perturbation scale far smaller than those used in the experiments, so it does not apply to the reported runs. It relies on Assumption 4.3 to control changes in the replay-conditioned objective when expert selections are refreshed, and the paper does not establish that this assumption holds in the evaluated MoE models. Moreover, the analysis provides no comparison with the same update without replay, so it does not support the claim that replay is responsible for the observed improvements.
Questions for authors
-
Can the authors repeat the main comparisons (the main results table and the router-freezing/replay ablation) across multiple random seeds and report uncertainty estimates, to establish whether the gains attributable to replay are robust?
-
The convergence guarantee requires a perturbation scale far smaller than those used in the experiments. Can the authors clarify whether any part of the theory applies to the reported settings? Can they also empirically check Assumption 4.3 on the evaluated backbones, for example by measuring the loss change caused by expert reselection between consecutive updates?
-
Can the authors measure, during the actual fine-tuning runs, how often expert selections change under perturbation and how large the switching term is relative to the parameter-response term? This would show whether the observed gains come from removing the switching term, as the paper claims.
Dear Reviewer iPfJ,
We have replied to each of your three follow-up comments (W1, W2 and Q2) in its thread, with the requested clarifications and new measurements. We believe these replies resolve the points you raised, and we kindly ask you to consider raising your score.
Best regards, The Authors