acceptodds
Under review as a conference paper at ICLR 2027

MuPACE: Learning to Pace Muon for LoRA Fine-Tuning

Abstract

Learning rate schedules are typically fixed before training begins, yet when fine-tuning large language models with low-rank adapters (LoRA), the appropriate scale varies substantially with the model, task, and adapter configuration. Practitioners re-tune a schedule for every new problem, yet a tuned schedule still cannot react to the training dynamics it encounters. We instead amortize learning rate selection across LoRA fine-tuning problems. We leave the base optimizer, Muon, untouched, so that its normalized update fixes the direction, and meta-learn only the global step size. Our method MuPACE is a small recurrent policy, meta-trained across a distribution of fine-tuning runs, that conditions on the observed trajectory and emits bounded adjustments to an interpretable analytic schedule. On 64 fine-tuning problems, MuPACE outperforms standard fixed schedules at a strong shared default on 64–78% of problems and is on par with the same schedules with per-task tuned peaks. Meta-trained only on models with up to 16 layers, MuPACE transfers one-shot to unseen tasks on models as deep as 64 layers, such as Qwen3-32B and Qwen3.8-27B, where a single run remains competitive with swept schedules.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LoRA Fine-Tuning, Learned Optimization, Learning-Rate Scheduling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ji3JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MuPACE, a meta-learned learning-rate scheduler for LoRA fine-tuning with the Muon optimizer. The method uses a small recurrent policy conditioned on static task descriptors and training dynamics. MuPACE is meta-trained over a distribution of LoRA fine-tuning episodes involving nanochat models and Lots-of-LoRAs subtasks. The evaluation covers 64 in-distribution configurations and out-of-distribution transfer to Qwen3-32B and Qwen3.8-27B.

Critical strengths
  1. MuPACE avoids the complexity and potential instability of learning a complete optimizer.
  2. Many evaluation results on amortized hyperparameter selection and out-of-distribution transfer.
Critical weaknesses
  • The paper does not sufficiently isolate trajectory-dependent learning from static schedule calibration
  • The one-shot comparison depends on online validation feedback, and the cost–quality trade-off requires a more precise interpretation.
Questions for authors
  • Does MuPACE actually need trajectory feedback to outperform fixed schedules?
  • How much does online validation feedback contribute, and what does it cost?
  • How much of the large-model transfer benefit comes from the prescribed depth scaling?
  • What are the limits of the learned controller's transfer across training horizons?
Reviewer_tb47OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MuPACE, a meta-learned learning-rate scheduler for LoRA fine-tuning with Muon. Rather than learning the optimizer’s update direction, a small recurrent policy uses task descriptors and optimization statistics to produce bounded adjustments to an analytic schedule. The method combines schedule-shape normalization, depth-dependent learning-rate scaling, and scale augmentation during meta-training. Evaluation covers 64 in-distribution fine-tuning configurations and transfer from models with at most 16 layers to two 64-layer models on eight unseen datasets. The reported results show improvements over default schedules and performance close to, although generally below, per-task tuned schedules while requiring a single target-task run.

Critical strengths
  • A practical and well-motivated approach to reducing learning-rate search. MuPACE preserves Muon’s update rule and learns only a global learning-rate schedule, providing a focused way to amortize tuning across LoRA fine-tuning problems. The reported results show improvements over default schedules and performance close to per-task tuned baselines without target-task learning-rate search. This tradeoff between performance and tuning effort is the paper’s strongest practical contribution.

  • Promising generalization beyond the meta-training setting. The learned scheduler transfers from smaller models to substantially larger models and unseen downstream datasets. Such transfer addresses an important challenge in learned optimization: whether a policy remains useful beyond the problems on which it was trained. These results support the potential usefulness of a reusable scheduler for fine-tuning.

Critical weaknesses
  • The role of trajectory-dependent adaptation is unclear. The paper does not compare MuPACE with a trained policy that uses only time and static task information. It also lacks a comparison with a policy that sets the full schedule at the start of training. These comparisons would show whether feedback during training improves performance. Without them, the gains may come from learning a better initial schedule. This leaves the paper’s main contribution unclear.

  • The evidence for preserving downstream performance is limited. MuPACE generally performs slightly worse than per-task tuned baselines. The paper does not report repeated-seed results or confidence intervals for these gaps. It is therefore unclear how consistent the results are. A lack of statistical significance does not show that the methods perform equally well. The large-model experiments also use short training runs and report cross-entropy loss. They do not show whether MuPACE preserves math accuracy or code-generation success. This makes its practical benefit harder to assess.

Questions for authors

Can you provide a matched comparison against a meta-trained policy that uses only time and static task descriptors, or predicts the schedule parameters only at initialization?

2 comments

Thank authors for the response. They basically addressed my concerns, so I will maintain my positive score.

We sincerely thank you for your time and thoughtful consideration of our rebuttal.

If there are any remaining concerns or clarifications we could provide that might help you further reassess your score, we would be more than happy to address them.

Reviewer_j5s5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MuPACE to reduce the cost of learning-rate search for LoRA fine-tuning with Muon. The method preserves Muon's update direction and uses a lightweight LSTM controller to adjust an analytical learning-rate curve based on model configuration, training progress, losses, and gradient statistics. It incorporates bounded outputs, curve RMS normalization, and depth scaling. After meta-training across tasks, the controller is deployed directly on new fine-tuning configurations.

The experiments show that MuPACE generally outperforms the specified default schedules on in-distribution configurations, with typical single-run performance approaching that of schedules tuned separately for each task. The results also demonstrate transfer to larger models and unseen datasets. However, the current evidence does not sufficiently isolate the incremental benefit of dynamic feedback or fully quantify the wall-clock savings relative to the baselines under comparable evaluation protocols.

Critical strengths

1.The research problem is clearly defined, and the method is well aligned with its objective.

2.The transfer results across model scales and datasets are valuable.

3.The paper includes several relevant ablations and reports results relatively transparently.

Critical weaknesses

1.The current experiments do not sufficiently isolate the incremental benefit of dynamic feedback. A comparison with a comparably meta-trained controller using only training progress and static model features is needed to distinguish the contribution of real-time feedback from that of a general schedule prior.

2.The number of search trials alone does not adequately capture actual computational savings. Full wall-clock costs should be reported under comparable validation and evaluation protocols, including the additional overhead of controller inference and statistics collection, while distinguishing between using a pretrained controller and performing meta-training again.

3.The current results support typical performance close to that of tuned schedules, but a lack of statistical significance does not establish equivalence or non-inferiority. Confidence intervals for paired performance differences, quantiles reflecting poorer performance, and results across multiple random seeds are needed to assess single-run reliability and the risk of performance degradation.

4.Key HGD hyperparameters and details of the out-of-distribution evaluation, including data splits, sample sizes, and evaluation frequency, require further specification to improve reproducibility. Including a simple feedback-based schedule that uses the same validation signal and is configured only on the source distribution would also help assess the added value of the learned controller.

Questions for authors

1.Could the authors provide a separately meta-trained controller that uses only training progress and static model and LoRA features, under a comparable meta-training setup, to quantify the incremental benefit of loss and gradient statistics?

2.What are the per-run and total wall-clock costs of MuPACE and the baselines under matched validation and evaluation protocols, including controller inference and statistics collection? Under the same wall-clock budget, how does MuPACE compare with the schedules in Appendix G.1 that require no target-task search?

3.Could the authors report confidence intervals for paired test-loss differences, quantiles reflecting poorer performance, and results across multiple random seeds for representative configurations? What performance gap relative to tuned schedules would they consider acceptable for replacing per-task search?

4.Could the authors specify HGD's initial learning rate, hypergradient step size, and any clipping or positivity constraints, together with the OOD data splits, sample sizes, and evaluation frequency? How would a simple feedback-based schedule perform when given the same validation signal and configured only on the source distribution?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We thank the reviewers for their thoughtful feedback and for recognizing that our “method is well aligned with its objective” (j5s5). We appreciate their acknowledgement that MuPACE offers a “practical and well-motivated approach to reducing learning-rate search” (tb47), demonstrates “promising generalization beyond the meta-training setting” (tb47), and avoids “the complexity and potential instability of learning a complete optimizer” (ji3J). We are also encouraged by their positive assessment of the “several relevant ablations” (j5s5), as well as the benefits of “amortized hyperparameter selection” (ji3J) and “out-of-distribution transfer” (ji3J).

Following our rebuttal, tb47 confirmed that their concerns were addressed and maintained their positive score, while j5s5 stated that their concerns were addressed and raised their rating to 3. Several of their concerns overlap with those raised by ji3J, who has not yet provided a follow-up assessment.

Concerns addressed (confirmed by tb47 and j5s5)

  • Contribution of dynamic feedback (all reviewers): We separately meta-trained controllers using only static+time and only dynamic+time features. Full MuPACE outperforms the static+time controller on 53/64 in-distribution tasks and all 8 Qwen3-32B tasks. Fixing the schedule at initialization did not close this gap (tb47). This suggests that trajectory feedback is the main source of MuPACE’s gains, particularly when transferring to larger models.

  • Computational cost (j5s5, ji3J): MuPACE adds ~8.6% runtime per run (582 s vs. 536 s). Under matched wall-clock budgets of 300 s and 600 s, it outperforms all four depth-scaled fixed schedules on 7/8 Qwen3-32B tasks, despite completing fewer updates.

  • Feedback-based baseline (j5s5, ji3J): We added ReduceLROnPlateau, using the same online validation signal and feedback frequency as MuPACE. Despite receiving per-task learning-rate tuning, ReduceLROnPlateau is outperformed by MuPACE on 47/64 in-distribution tasks, 6/8 Qwen3-32B tasks, and 5/8 Qwen3.8-27B tasks.

  • Longer training horizons and downstream performance (tb47, ji3J): We extended training to 1,000 steps (3.33× the meta-training horizon) on XSum, where MuPACE continues improving and remains competitive with tuned baselines. On GSM8K, MuPACE preserves accuracy (89.18% vs. 87.94% for the best baseline).

  • Experimental clarifications (j5s5): We clarified HGD hyperparameters and details on the experimental setup to improve reproducibility.

  • Reliability and performance relative to tuned baselines (j5s5, tb47): In-distribution, paired bootstrap CIs and degradation quantiles show a median penalty of 0.24–0.90% versus tuned schedules, though about 10% at the 90th percentile. Three seeds on Qwen3-32B give consistent trends (std 0.008–0.023).

Concerns specific to ji3J, addressed in our response but not yet acknowledged:

  • Validation feedback (ji3J): Zeroing the validation-loss features at inference increases median Qwen3-32B test loss by 0.85%, suggesting a modest benefit.

  • Depth scaling (ji3J): We clarified the existing ablation results, showing that while depth scaling improves transfer, MuPACE outperforms depth-scaled fixed schedules in all but one comparison across both Qwen3 models.

We hope these analyses, together with the shared concerns above, resolve ji3J’s questions.

The rebuttal motivated us to conduct and report also additional experiments:

  • Additional analysis, update frequency: Updating the controller every 2–3 steps costs little in-distribution (+0.32–0.42%) but more on Qwen3-32B (+1.31–1.45%). Reducing overhead this way is therefore a setting-dependent trade-off.

We thank the reviewers for their constructive suggestions, which we believe have strengthened the paper. Supplementary analyses (bootstrap intervals, validation-feature ablation, initialization-only schedule, GSM8K accuracy) are reported in our responses, and we are happy to add them to the paper if requested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.