acceptodds
Under review as a conference paper at ICLR 2027

No-Momentum Muon Sublates the Edge of Stability in LLM Pretraining

Abstract

Muon is increasingly used for language-model pretraining, yet its large-step dynamics are not captured by the classical edge-of-stability (EoS) picture of gradient descent (GD). In GD, loss neutrality, equal-magnitude update reversal, and marginal stability meet at a single learning-rate-dependent edge. We show that Muon breaks this coupling. For stochastic no-momentum Muon, we derive a coherence-corrected conditional loss-neutral boundary , while temporal alignment follows a separate geometry. Controlled experiments show that loss balance and temporal alignment respond differently to learning rate and batch size. Across our language model experiments, the 130M Llama-like LLM runs exhibit loss-boundary tracking with weak negative alignment, whereas the studied 1B LLM configuration shows stronger partial cancellation; in both settings, directions remain far from coherent reversal while training continues to improve. These results support a split EoS picture for Muon: a stochastic loss-neutral edge survives, but it is not accompanied by a universal temporal-direction signature.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Edge of Stability, Muon, Gradient Descent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V8hTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the classical Edge of Stability theory extends to no-momentum Muon, focusing on the relationship between the loss-neutral boundary and temporal orthogonality. The authors derive a coherence-corrected loss-neutral condition for stochastic Muon and theoretically demonstrate that the same effective curvature does not necessarily imply the same alignment of consecutive update directions. Through experiments on MLPs, CNNs, Transformers, and LLMs up to 1B parameters, they show that Muon can approach the loss-neutral boundary without exhibiting complete update-direction reversal, and that learning rate and batch size affect these phenomena differently. Based on these findings, the authors propose a "split Edge of Stability" interpretation for Muon.

Critical strengths
  1. I find the theoretical result that the same effective curvature does not necessarily imply the same alignment of consecutive update directions particularly interesting. The matrix quadratic example clearly illustrates why the classical curvature-based relationship between loss neutrality and directional behavior does not directly extend to Muon. This provides a useful perspective on Muon's optimization dynamics that is not captured by the classical EoS framework.

  2. The authors conduct extensive experiments across different model architectures and scales, including LLM pretraining. These experiments examine how learning rate and batch size affect loss neutrality and temporal alignment, providing empirical support for the theoretical findings.

  3. The learning-rate perturbation experiments are particularly convincing. By branching from the same checkpoint, the authors show that the conditional loss change can switch signs while consecutive update directions remain negatively aligned. This provides more direct evidence for the separation between T1 and T2 than simply observing their trajectories. I also find the frozen-state experiments useful for distinguishing the effects of minibatch randomness from the negative alignment induced by training dynamics.

Critical weaknesses
  1. I am not fully convinced that the theoretical results sufficiently support the proposed "split Edge of Stability" interpretation. The loss-neutral boundary in Theorem 2.1 appears to follow directly from the definition of the finite-step curvature remainder, leaving its theoretical novelty unclear. The authors also acknowledge that ckpol=0c_k^{\mathrm{pol}}=0 is not itself a stability criterion. While the matrix quadratic example shows that loss neutrality and temporal alignment can be decoupled, this alone does not establish a new understanding of stability.

  2. The paper focuses exclusively on no-momentum Muon, whereas Muon implementations used in LLM pretraining typically incorporate momentum. Since momentum affects the alignment of consecutive updates, the results may not carry over to practical Muon implementations. While isolating the effects of polar normalization is reasonable, the absence of momentum experiments limits the generalizability of the findings.

Questions for authors
  1. What additional theoretical insight does Theorem 2.1 provide beyond reformulating the one-step loss change? In particular, is there a theoretical explanation for why Muon training tends to approach the loss-neutral boundary?

  2. Given that ckpol=0c_k^{\mathrm{pol}}=0 is not itself a stability criterion, how does the separation between T1 and T2 justify the proposed "split EoS" interpretation? What does this separation reveal about stability beyond the distinct behaviors of the two diagnostics?

  3. Do the authors expect the separation between T1 and T2 to persist when momentum is introduced? Preliminary experiments with momentum would help establish whether the findings extend to practical Muon training.

  4. The 1B pretraining experiment examines temporal alignment but does not measure the T1 boundary. Is there evidence that the separation between loss neutrality and temporal alignment persists at this scale? Since the 130M and 1B configurations differ in both model size and batch size, how much of the observed difference in alignment can be attributed to batch size?

5 comments

We thank Reviewer V8hT for your constructive comments and positive support.

W1/Q1. Additional theoretical insight of Theorem 2.1

Theorem 2.1 serves as a loss identity, and we added Proposition 2.1 in the main text, which shows the attraction that effective curvature s^k,bM\hat s^M_{k,b} cannot remain below 2/η2/\eta.

W1/Q2. Connections between EoS and T1, generalized sharpness, T2

According to your suggestions, we have included the generalized-sharpness measurements for EoS and conducted experiments for comparions on 130M runs, see Figure 1 in the updated version.

Our 130M runs show that normalized generalized sharpness approaches 2/η2/\eta, which provides empirical evidence of EoS in LLM Muon pretraining, following the framework of Islamov et al (2026). However, effective curvature (T1, tracks the smaller boundary 2ρ/η2\rho/\eta) differs such boundary and arrives even eariler.

This is one part of what our finding: effective curvature separates from normalized generalized sharpness, while loss balance does not uniquely determine update alignment (T2). This may suggest to imrpove the WSD learning-rate schedules in practice as there is still some space between T1 and T2.

In summary, before proving dynamics of Muon at EoS, this paper firstly confirms a phenomenon: Muon exhibits generalized-sharpness EoS signatures in LLM pretraining regimes, then identifies two non-interchangeable indicators/diagnostics of this behavior in different charecterizations. Furthermore, our work suggests the possibility of improving learning-rate schedules in practice.

W2/Q3. Is the separation between T1 and T2 persist with momentum?

Yes. We added a 130M run with momentum 0.950.95 (Section3.1 figure6).Loss balance (T1) persists, while update alignment (T2) is positive (median cosine (0.712)), unlike the negative alignment observed without momentum. Thus, the need to examine T1 and T2 separatly extends to momentum Muon.

Q4. Loss balance (T1) and batch-size effects at 1B

We added T1 measurements for the 1B model at batch sizes 128, 256, and 512 (Figure 1(d–f), Section 4.3, and Appendix D.6).

Across all three runs, loss-neutrality balance persists. Larger batches show more negative consecutive-update alignment(T2). This comparison isolates batch-size variation at a fixed model scale.


Reference Islamov et al. (2026). Non-Euclidean Gradient Descent Operates at the Edge of Stability. ICML'26.

Dear Reviewer V8hT,

As the discussion period draws to a close, we would appreciate it if you could let us know whether our response addresses your concerns. We would be happy to clarify any remaining points. Thank you for taking the time to review our response.

Kind regards, The Authors

Thank you for the detailed response and the additional theoretical and experimental results. I appreciate the authors' efforts in addressing my concerns.

I have one remaining question regarding W1/Q2. The new generalized-sharpness measurements provide more direct evidence that Muon exhibits EoS behavior, which addresses part of my original concern. However, if generalized sharpness already characterizes EoS, what additional insight into stability does the separation between T1 and T2 provide? In other words, what makes the proposed "split EoS" a new characterization of EoS, rather than simply two additional diagnostics of Muon's optimization dynamics?

I would appreciate further clarification on this conceptual distinction. Otherwise, I am satisfied with the authors' responses.

Reviewer_NPv5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is about whether Muon exhibits the edge-of-stability dynamics same as the gradient descent. In traditional EoS dynamics, the stability boundary is ηλ=2\eta \lambda = 2, and the authors show that Muon introduces an additional factor related to stochasticity and temporal alignment. Also, the authors observe that the loss change may not correspond to strong correlation between two consecutive update directions. The authors show that the similar phenomenon can be observed in controlled MLP, CNN, and transformer settings.

Critical strengths

The research question of exploring Muon-specific edge-of-stability phenomenon on LLM pre-training regime is interesting.

Critical weaknesses

The paper is poorly written in the sense that non-standard terms and claims are introduced without proper explanations, as if they are generated by AI. To name few, in Line 13, the terms "loss neutrality," "equal-magnitude update reversal" are not properly explained. Relatedly, claim like "Existing geometry-aware analyses of Spectral GD and normalized Spectral GD are primarily deterministic or controlled" in Line 42 requires more explanations as the sentence itself is too generic and ambiguous.

In addition, I think the paper's logic of observing "the loss change may not correspond to strong correlation between two consecutive update directions" and making claims about Muon's EoS is too far-fetching. Specifically, the loss neutrality in the paper means zero expected loss change. However, it does not directly connected to dynamical stability across successive optimization steps.

Questions for authors

What are the bottom lines of the paper? Honestly, it is hard for me to draw any meaningful or surprising conclusions from the paper's observation. To me, EoS is interesting because it shows neural net training can be stable beyond the classical wisdom about the stable learning rate (1/eta) and the local instability does not necessarily lead to global instability. This leads to many interesting questions related to modern optimization theory, implicit regularization, and learning dynamics. I am wondering more general picture of the paper's observations on long term research direction. Damian (2022) and Arora (2022) might be useful for authors to capture the point.

Reference

  • Damian, A., Nichani, E., & Lee, J. D. (2022). Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594.

  • Arora, S., Li, Z., & Panigrahi, A. (2022). Understanding gradient descent on the edge of stability in deep learning. In ICML

3 comments

We thank Reviewer NPv5 for your constructive comments

W1.1 Improve writing on non-standard terms and claims

Thanks for your suggestions on writing. We have improved the readability by 1) introducing definitions, notations; 2) unifying the expression. The major revisions have been highlighted in red.

For example, we changed the terminonality "loss neutrality" as "lose balance" related to T1 for clarity, and term T2 as "update alignment".

W1.2 Connections between EoS and T1, generalized sharpness, T2

For complete comparisons, we have included the generalized-sharpness measurements from Islamov et al. (2026) for EoS.

Our findings are: normalized generalized sharpness approaches 2/η2/\eta, which provides empirical evidence of EoS in LLM Muon pretraining. However, effective curvature (T1, tracks the smaller boundary 2ρ/η2\rho/\eta) differs such boundary and arrives even eariler, see Figure 1.

This is one part of what our main findings: effective curvature separates from normalized generalized sharpness, while loss balance does not uniquely determine update alignment (T2). This may suggest to imrpove the WSD learning-rate schedules in practice as there is still some space between T1 and T2.

Q1. Bottom lines of this paper and insights for future theory/practice

We polish the findings/insights of this paper according to your feedback.

This paper confirms a phenomenon: Muon exhibits generalized-sharpness EoS signatures in LLM pretraining regimes, identifies two non-interchangeable indicators/diagnostics of this behavior, and suggests the possibility of improving learning-rate schedules in practice.

The conclusion motivate concrete theory questions on how curvature and update geometry evolve together, as well as improving learning-rate schedules supported by Figure 8.

We appreciate Damian et al. and Arora et al. work on theoretical understanding of EoS and acknowledge that our work does not belong to such style. Before proving something, we need to confirm a phenomenon and provide several charecterizations of such pheomenon, which is the target of this work. We leave the theoretical understanding of Muon at EoS as a future work.

We thank for Reviewer NPv5 again for this feedback, which helps us to sharp the conclusion and value of this work.


Reference Islamov et al. (2026). Non-Euclidean Gradient Descent Operates at the Edge of Stability. ICML'26.

Dear Reviewer NPv5,

As the discussion period draws to a close, we would appreciate it if you could let us know whether our response addresses your concerns. We would be happy to clarify any remaining points. Thank you for taking the time to review our response.

Kind regards, The Authors

Thank you for the clarifications. The explicit ±10% tolerance, paired Fieller confidence sets, and treatment of near-zero denominators address my statistical concerns. I also appreciate the clear distinction between the curvature-visit proposition and empirical boundary tracking, as well as the separation between the generalized-sharpness diagnostic and the implemented NS-5 training updates.

Together with the additional experiments in the earlier response, these clarifications substantially address my main concerns and support revising my recommendation to weak acceptance.

The remaining limitations should remain explicit: numerical search stability does not certify a global maximum, and pointwise conditional confidence sets do not establish simultaneous trajectory coverage or robustness across training seeds. I view these as limitations on the interpretation of the results rather than reasons to retain my original recommendation.

Reviewer_ANSwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether the edge-of-stability picture extends to stochastic Muon without momentum. It defines a batch-to-population coherence coefficient and a finite-step effective curvature, yielding an exact conditional loss identity with a loss-neutral boundary of 2ρb/η2\rho_b/\eta. It separately measures the cosine between consecutive update directions. A matrix quadratic example demonstrates that the same effective loss curvature can accompany opposite signs of temporal alignment. Experiments on controlled networks and language models investigate these diagnostics, finding weak negative alignment in the 130M configurations and stronger partial cancellation in the studied 1B configuration. The 1B experiment measures alignment and loss, but not the conditional loss boundary.

Critical strengths
  1. The distinction between conditional loss balance and temporal alignment is useful and concretely demonstrated. The matrix quadratic in Section 2.3 provides an explicit example in which equal effective curvature produces different alignment signs. The 22M checkpoint interventions also show that conditional loss increments can change sign while temporal alignment remains negative. These results caution against interpreting a negative update cosine as sufficient evidence of loss instability.

  2. The frozen-state comparison in Appendix B.7 strengthens the interpretation of the alignment measurements. It distinguishes independent-minibatch dispersion at a fixed state from the negative temporal bias observed during training, and reproduces this difference across two seeds. The paper also documents several important limitations of its measurements, including the scope of the reference-loss probes and the absence of T1 measurements at 1B scale.

Critical weaknesses
  1. The connection between the proposed loss-neutral diagnostic and an edge of dynamical stability remains insufficiently established. Equation (5) defines effective curvature using the finite-step loss increment, so Theorem 2.1 follows by rearrangement. This identity is valid, but does not itself explain attraction to the boundary, establish marginal stability, or distinguish boundary tracking from small expected increments during slowly progressing training. Proposition A.1 provides a conditional full-batch curvature-visit result, rather than persistent stochastic boundary tracking. Consequently, the main theoretical result offers less explanatory support for the EoS interpretation than the framing suggests. A matched GD/SGD comparison would also help establish how much of the observed separation is specific to polar normalization rather than stochasticity or multidimensional dynamics.

  2. The evidence for the loss-edge interpretation in language-model training remains limited by the measurement scope. The 130M conditional probes use a fixed reference objective of 32 sequences, eight draws per probe batch size, and Muon-only virtual updates with auxiliary parameters held fixed. Appendix D provides useful complete-update checks, but these are realized increments rather than conditional complete-update expectations, and the reported intervals exclude reference-sample uncertainty. Moreover, the 130M learning-rate branches have no conditional T1 probes (Appendix C.2), and the 1B run has no T1 measurements. The contribution statement on page 3 nevertheless attributes conditional loss-sign changes at fixed checkpoints to both 22M and 130M. These limitations and the inconsistent attribution weaken the claimed breadth of the split-edge evidence.

Questions for authors
  1. What operational criterion distinguishes meaningful attraction to the proposed loss-neutral boundary from a regime with small expected loss increments? Please quantify the relative discrepancy between curvature and boundary, with uncertainty accounting for their shared alignment term, and clarify what supports interpreting this as an EoS phenomenon beyond the exact loss identity.

  2. How sensitive are the 130M conclusions to the size and choice of the reference set? Can additional checkpoint measurements establish whether near-neutral conditional increments persist across independent reference sets and for complete Muon-plus-AdamW updates?

  3. Page 3 attributes conditional loss-sign changes under learning-rate interventions to both 22M and 130M, whereas Appendix C.2 states that no conditional T1 probes are available for the 130M branches. Is there additional evidence, or should this claim be restricted to 22M?

  4. Do matched stochastic GD/SGD controls exhibit a similar separation between loss balance and temporal alignment? Such evidence would materially affect the assessment of which aspects of the proposed split-edge picture are specific to Muon.

5 comments

We thank Reviewer ANSw for your constructive comments.

W1.1/Q1. The connection between loss balance (T1) and EoS

According to your suggestions, we have included the generalized-sharpness measurements for EoS and conducted experiments for comparions on 130M runs, see Figure 1 in the updated version.

Our 130M runs show that normalized generalized sharpness approaches 2/η2/\eta, which provides empirical evidence of EoS in LLM Muon pretraining, following the framework of Islamov et al (2026). However, effective curvature (T1, tracks the smaller boundary 2ρ/η2\rho/\eta) differs such boundary and arrives even eariler.

This is one part of what our finding: effective curvature separates from normalized generalized sharpness, while loss balance does not uniquely determine update alignment (T2). This may suggest to imrpove the WSD learning-rate schedules in practice as there is still some space between T1 and generalized sharpness.

In summary, before proving dynamics of Muon at EoS, this paper firstly confirms a phenomenon: Muon exhibits generalized-sharpness EoS signatures in LLM pretraining regimes, then identifies two non-interchangeable indicators/diagnostics of this behavior in different charecterizations. Furthermore, our work suggests the possibility of improving learning-rate schedules in practice.

W1.2/Q1. Theorem 2.1 and Proposition A.1 only provides a full-batch result, rather than a stochastic one.

We agree that Theorem 2.1 follows by rearrangement, so we rename it. Instead, we move Proposition A.1 under the stochstic version to the main text. We demonstrate that effective curvature cannot stay below the boundary, see Proposition 2.1.

W1.3/Q1. Quantify the discrepancy/uncertainty between curvature and boundary

Yes. We define the relative curvature–boundary discrepancy as:

r^k,b=s^k,bM−2ρ^k,b/ηk2ρ^k,b/ηk\hat r_{k,b} = \frac{\hat s^M_{k,b}-2\hat\rho_{k,b}/\eta_k} {2\hat\rho_{k,b}/\eta_k} where s^k,bM\hat s^M_{k,b} is the effective curvature and 2ρ^k,b/ηk2\hat\rho_{k,b}/\eta_k is the boundary, combined is T1.

Across four original 130M runs under different learning rates, the median r^k,b\hat r_{k,b} is near zero. Entire paired 95% confidence intervals lie within a descriptive ±10%\pm10\% tolerance at 44–56 of 60 checkpoints per run. (See Appendices D.2 and D.3 for details).

W1.4 / Q4. Is split-edge picture specific to Muon? How about GD/SGD

Theoratically, we compared GD on the same matrix quadratic model of section2.3: GD’s effective curvature determines the update-alignment sign, whereas Muon can exhibit opposite alignment signs at identical effective curvature (Appendix E). This isolates an effect of polar normalization.

Empiracaly, our added 130M SGD experiments also exhibit loss balance (T1) with negative, varying update alignment (Appendix E). We therefore do not claim that this empirical pattern is unique to Muon, but that won't affect our question: which signatures of EoS persist or change under LLM pretraining with Muon.

W2.1/Q2.1. Is T1 robust to reference sets with larger size and more runs?

Yes. We use three independently sampled reference sets for each size 32,64,12832,64,128, with 32 draws at each checkpoint (Appendix D.3).

The median r^k,b\hat r_{k,b} are −0.0458,−0.0415,−0.0427-0.0458,-0.0415,-0.0427 across the three reference sizes. Thus, loss balance (T1) persists across the tested reference sets.

Using the first 8 and 16 draws, r^k,b\hat r_{k,b} relative to 32 draws by absolute amounts of 0.00360.0036 and 0.00190.0019. Median 95% interval widths decrease from 0.02460.0246 to 0.01580.0158 and 0.01130.0113 for 8, 16, and 32 draws.These intervals quantify conditional Monte Carlo uncertainty.

W2.2/Q2.2: Does T1 persist for complete Muon-plus-AdamW updates?

Yes. We measured T1 in Muon-plus-AdamW updates. Loss balance persists, see Appendix D.3.

W2.2/Q3. Additional T1 measurements for 130M perturbation branches and 1B training.

We included these measurements on 130M runs (Section 4.2 and Appendix C) and 1B runs (Section 4.3 and Appendix D.6), respecitvely.

For 130M, halving and doubling the learning rate produce conditional loss increments of −0.01900-0.01900 and 0.133430.13343. After certain updates, T1 shows an empirically recovery toward loss-neutrality balance.

Across 1B at batch sizes 128,256,512128,256,512, all three runs show loss balance (T1).


Reference

Islamov et al. (2026). Non-Euclidean Gradient Descent Operates at the Edge of Stability. ICML'26.

Thank you for the substantial additions. The generalized-sharpness measurements, reference-set robustness checks, SGD controls, and additional T1 probes directly address several of my original concerns. I also appreciate the distinction between a descriptive characterization of Muon and a phenomenon unique to Muon.

Two points require clarification:

The response states that paired confidence intervals lie within a “±10” relative-discrepancy tolerance. Does this mean ±10%, i.e., ±0.1? Please specify how the ratio confidence intervals are constructed and how small or near-zero coherence denominators are handled.

Please distinguish the stochastic curvature-visit proposition from attraction to or persistent tracking of the boundary. For the new generalized-sharpness and complete-update measurements, please make their definitions, sampling procedures, and uncertainty explicit, including their relationship to the implemented NS-5 directions and auxiliary AdamW updates.

Thank you for these clarification requests.

1. Relative tolerance and near-zero denominators.

Yes, the tolerance is ±10%, i.e., [−0.1,0.1][-0.1,0.1].

The ratio confidence intervals are constructed by paired 95% Fieller confidence set (Appendix D.2): C0.95={q:(dˉ−qaˉ)2≤t0.975,m−12m(sd2−2qsda+q2sa2)}.\mathcal{C}_{0.95}=\left\lbrace q:(\bar{d}-q\bar{a})^2\leq\frac{t_{0.975,m-1}^{2}}{m}\left(s_d^2-2q s_{da}+q^2s_a^2\right)\right\rbrace. where di=LR(θ+Δθi)−LR(θ),ai=−⟨∇LR(θ),Δθi⟩,r^=dˉaˉd_i=L_R(\theta+\Delta\theta_i)-L_R(\theta), a_i=-\langle\nabla L_R(\theta),\Delta\theta_i\rangle, \widehat r=\frac{\bar d}{\bar a} and the sample variances and covariance use divisor m−1m-1.

We agree that near-zero coherence can make the ratio unreliable. We do not clip or regularize the denominator. If it cannot be reliably distinguished from zero, the Fieller set can be unbounded or disconnected; such a result cannot satisfy the full-set containment criterion and is not classified as near-balance. Also, at all checkpoints summarized in Table 5, the denominators are significantly positive at the pointwise 95% level, and all Fieller sets are bounded.

2. Scope of Proposition 2.1.

We agree: the proposition proves that effective curvature cannot eventually remain a fixed distance below the boundary, which does not establish attraction, convergence to the boundary or persistent tracking. Near-balance behavior and post-intervention recovery are empirical observations.

We have clarified this distinction in the updated version. We also clarify that the proposition concerns exact-polar no-momentum Muon.

3. Generalized sharpness and complete-update measurements.

Generalized sharpness. We have added the measurement descriptions in Appendix D.3.

With auxiliary parameters ZZ fixed, normalized generalized sharpness is defined as

SR=max⁡max⁡j∥Dj∥op=1⟨D,∇W2LR(W,Z)[D]⟩NR,NR=∑j∥∇WjLR(W,Z)∥∗.S_R=\frac{\displaystyle\max_{\max_j\lVert D_j\rVert_{\mathrm{op}}=1}\langle D,\nabla_W^2L_R(W,Z)[D]\rangle}{N_R},\qquad N_R=\sum_j\lVert\nabla_{W_j}L_R(W,Z)\rVert_{\ast}.

We estimate this quantity using Hessian–vector products and a Frank–Wolfe-type search with SVD-polar oracles, using 3 starts × 50 iterations. Increasing the search budget to 5 starts × 100 iterations changed the estimates by less than 1% at the checked checkpoints. We also compared three independently sampled reference sets at each size of 1024, 2048, and 4096, with each set held fixed across checkpoints. The observed generalized-sharpness pattern persisted across these reference sets. These checks assess sensitivity to the numerical search budget and reference-set selection.

The diagnostic’s SVD-polar search is distinct from the NS-5 directions used during training.

Complete-update measurements. See Appendix D.4 for details.

Each draw starts from the same parameters and optimizer state. We construct the virtual displacement

Δθi=(ΔWiMuon,ΔZiAdamW),\Delta\theta_i=\left(\Delta W_i^{\mathrm{Muon}},\Delta Z_i^{\mathrm{AdamW}}\right),

where ΔWiMuon\Delta W_i^{\mathrm{Muon}} follows the implemented NS-5 Muon update, including its learning rate and scaling conventions. Both components use the same sampled batch’s pre-update gradients. The AdamW component includes moment updates, the step counter, bias correction, the auxiliary learning rate, and weight decay. Virtual optimizer states are discarded between draws.

We also construct the paired Fieller confidence set for r^full=dˉaˉ.\widehat r_{\mathrm{full}}=\frac{\bar d}{\bar a}., which quantifies the uncertainty.


We again appreciate reviewer ANSw’s time and constructive feedback, and we are happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.