- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MuPACE, a meta-learned learning-rate scheduler for LoRA fine-tuning with the Muon optimizer. The method uses a small recurrent policy conditioned on static task descriptors and training dynamics. MuPACE is meta-trained over a distribution of LoRA fine-tuning episodes involving nanochat models and Lots-of-LoRAs subtasks. The evaluation covers 64 in-distribution configurations and out-of-distribution transfer to Qwen3-32B and Qwen3.8-27B.
Critical strengths
- MuPACE avoids the complexity and potential instability of learning a complete optimizer.
- Many evaluation results on amortized hyperparameter selection and out-of-distribution transfer.
Critical weaknesses
- The paper does not sufficiently isolate trajectory-dependent learning from static schedule calibration
- The one-shot comparison depends on online validation feedback, and the cost–quality trade-off requires a more precise interpretation.
Questions for authors
- Does MuPACE actually need trajectory feedback to outperform fixed schedules?
- How much does online validation feedback contribute, and what does it cost?
- How much of the large-model transfer benefit comes from the prescribed depth scaling?
- What are the limits of the learned controller's transfer across training horizons?
Thank authors for the response. They basically addressed my concerns, so I will maintain my positive score.