acceptodds
Under review as a conference paper at ICLR 2027

Augmenting Matrix Orthogonalization with Sample-Space Geometry

Abstract

Orthogonalized optimizers such as Muon exploit the matrix structure of parameter updates to improve neural network training. Their normalization acts on the row and column directions of aggregated updates, leaving the co-update directions shared across individual training examples without explicit spectral control. We propose augmenting matrix-based orthogonalization with this complementary sample-space geometry. By organizing per-sample updates into a matrix, we identify shared update directions through a sample-space Gram matrix, enabling spectral shaping across vector, matrix, and heterogeneous parameter blocks. To incorporate these directions without disproportionately amplifying weak components, we introduce Selective Orthogonalization: a smooth transformation that normalizes strong singular directions while preserving the scale of vanishingly weak ones. We prove that, for a fixed positive transition scale, its relative amplification approaches one as a singular value vanishes, and derive an equivalent parameter-side spectral formulation. The resulting approach supports a hybrid optimizer that alternates Selective and Muon updates across iterations, bringing sample-space and parameter-matrix geometry into the same training process. Standalone Selective experiments establish the behavior of the sample-space component: controlled studies validate weak-direction scale preservation across five parameter structures, while image-classification experiments demonstrate substantially reduced weak-direction amplification and competitive accuracy across datasets, architectures, and batch sizes. Together, these results motivate per-sample update geometry as a complementary source of structure for orthogonalized optimization.

Primary area:
optimization
Keywords:
deep learning optimization, orthogonalized optimizers, spectral optimization, gradient geometry, matrix preconditioning, sample-space methods, Muon

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dbkQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a change to Muon-style optimizers. Instead of only normalizing the combined (averaged) update, it also looks at how the individual training examples in a batch push the parameters, and rebalances those per-example directions.

They also combine it with Muon in a hybrid optimizer that alternates between the two methods. Experiments on small synthetic tasks and short CIFAR-10/100 training runs show that Selective avoids the blow-up and does slightly better than Hard, and that the hybrid beats plain Muon

Critical strengths

Strength:

  1. Clear and verifiable math. All results seems to be correct, and the two equivalent views (T(S) = U g(Σ) Vᵀ = S h(C)) are cleanly presented. The B×B implementation is practical for small B.
  2. It works on any parameter shape. It applies to vectors, matrices and heterogeneous blocks, which Muon does not.
  3. Some real gains. On CIFAR-100 Airbench, both Hard and Selective strongly beat Muon and AdamW.
  4. Diagnostics match the motivation. Weak-direction amplification is measured during training, layer by layer and across five parameter structures, and Hard really does blow up the tail.
Critical weaknesses

Weakness: Providing more experimental results, under different hyper-parameters setting and on more relevant tasks is pertinent to proving the efficacy of the proposed method. The experiments are small (8-epoch CIFAR), have no error bars, give minimum tuning protocol for the real-data baselines, include no wall-clock comparison, and leave out the obvious damped, truncated, identity and SGD baselines.

  1. Statistics: multiple seeds with standard deviations, and the native Airbench94 setup (batch 1024, ~94%) as a reference point.
  2. Equal wall-clock comparisons and time-to-accuracy. (compared to standard Muon & SGD)
  3. Scale-up: transformer or LM training (e.g., NanoGPT speedrun-scale) and ImageNet-scale vision, where Muon is actually used.
  4. Ablations of the hybrid: alternation schedule and the shared-versus-separate momentum
Questions for authors

Questions:

  1. Do we have the wall-clock time per epoch and the peak memory for Selective versus Muon and SGD?
  2. What is the possible reason for the hybrid underperform Selective alone on all Airbench settings?
3 comments

Thank you for recognizing the mathematical clarity, parameter-shape generality, and weak-spectrum diagnostics, and for requesting a more complete assessment of experimental reproducibility and computational cost. We distinguish the behavior of standalone Selective from the role of Selective--Muon, whose objective is to augment Muon's aggregated-matrix orthogonalization with sample-space geometry.

To W1 — Baselines, tuning, and controlled comparisons:

The experiments separate two comparisons with different purposes. (1) Hard and Selective act on the same per-sample update matrix, isolating the effect of the weak-direction spectral map. (2) Muon and Selective--Muon compare aggregated matrix orthogonalization with its shared-momentum, sample-space augmentation. Table 2 also includes AdamW as a conventional reference. In Section 5.1, retuning Hard's global RMS does not recover Selective's weak-spectrum validation losses across the tested range. Figure 2(c) reports sensitivity to Selective's transition parameter τ\tau. These are the controls included in the study. Damped or truncated spectral transforms, identity updates, and a separately tuned SGD baseline are not part of the reported real-data comparison.

To W2 — Repeated runs, uncertainty, and the Airbench94 reference:

Table 2 presents TTA-2 accuracy as mean ±\pm sample standard deviation over four seeds for all five optimizers across three settings and three physical batch sizes. The experiments use eight epochs and B∈{64,128,256}B\in\{64,128,256\}, with T(B)=8⌊50,000/B⌋T(B)=8\lfloor50{,}000/B\rfloor under the drop-last convention also used in Keller Jordan's publicly released Airbench94 Muon implementation. These settings examine batch-size sensitivity under a common training-horizon convention. They are distinct from the reviewer-requested batch-1024, approximately 94%-accuracy Airbench94 reference and should be read as the specified experimental configurations rather than a reproduction of that reference result.

To W3 / Q1 — Step time, peak memory, and computational cost:

Appendix E.2.1 (Table 4) provides measured RTX 5090 GPU step times and peak allocated memory. At batch 256, AdamW requires 6.896 ms/step and 3.692 GiB, Muon 5.630 ms/step and 1.846 GiB, Hard 466.214 ms/step and 10.464 GiB, and Selective 467.785 ms/step and 10.464 GiB. Hybrid records 67.866 / 133.428 / 357.404 ms/step for batches 64 / 128 / 256, with reported memory peaks of 1.593 / 3.945 / 6.290 GiB. These measurements characterize the implemented methods, with the respective timing protocols specified in the appendix. For Selective at batch 256, exact per-sample gradient extraction accounts for 79.2% of measured runtime. The small B×BB\times B Gram representation reduces the dimension of the spectral problem, while the extraction of per-sample updates remains the main implementation cost. The accuracy comparison is based on eight epochs, not a separately measured time-to-target or equal-wall-clock experiment. AdamW is not presented as a measured SGD runtime baseline.

To W4 — Broader training scales:

We appreciate the suggestion to examine Transformers and larger-scale vision tasks. The spectral mechanism itself is not tied to the CIFAR architecture or training scale. Hard polar normalization flattens nonzero singular values, while Selective has unit relative amplification in the vanishing-singular-value limit for fixed ϵ>0\epsilon>0. The sample Gram and parameter-side second-moment matrices share the same nonzero spectrum for any parameter-block dimension. Our five controlled parameter structures and the CIFAR-10/100 experiments test these mathematical properties across different update geometries. They establish the reported spectral behavior and image-classification performance, without relying on a large-scale model to define the mechanism.

To W5 — Hybrid schedule and shared momentum:

Selective--Muon combines two distinct spectral operations within a single momentum trajectory, rather than maintaining independent optimizer states. Section 3(5) and Appendix E.2.1 (Algorithm 2) specify

$ M_t=0.6M_{t-1}+G_t,\qquad Z_t=G_t+0.6M_t. $

Even iterations apply sample-space Selective to ZtZ_t. Odd iterations sum the rows, restore the original parameter-matrix geometry, and apply Muon's Newton--Schulz transformation. Both branches extract per-sample gradients, and switching never resets the shared buffer. The evaluated 1:1 schedule is an implementation choice, not a restriction of the shared-state construction. Other fixed or adaptive branch-selection rules can use the same state. Determining the optimum schedule for a given training setting is a direction for future study.

To Q2 — Why does Hybrid trail standalone Selective on Airbench?

The hybrid is designed to augment Muon with sample-space geometry, not to reproduce an all-Selective trajectory. Its shared-state 1:1 schedule alternates between sample-space Selective and matrix-space Muon, with branch-specific scaling. Accordingly, the two algorithms follow different optimization trajectories.

The following comparisons use Table 2 four-seed mean TTA-2 accuracy. Differences are in percentage points (pp).

SettingBatch sizeHybrid −- SelectiveHybrid −- Muon
CIFAR-10 / Airbench9464−0.73-0.73+0.63
128−0.47-0.47+0.29
256−0.20-0.20+0.14
CIFAR-10 / ResNet-1864+0.08+0.25
128+0.50+0.10
256+0.77+0.99
CIFAR-100 / Airbench-style64−4.24-4.24+3.94
128−2.66-2.66+2.67
256−0.52-0.52+1.87

Hybrid improves over Muon in all nine settings and achieves the highest reported mean accuracy among all five methods at every CIFAR-10/ResNet-18 batch size. On the two Airbench-style architectures, standalone Selective instead achieves higher means. These outcomes are consistent with the two methods applying different spectral treatments. Together, they demonstrate Hybrid's role as a complementary augmentation of Muon, with an architecture-dependent tradeoff relative to standalone Selective.

Dear Reviewer dbkQ,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_JZ1zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Muon-like orthogonalization approach to processing per-sample updates in order to exploit sample-space geometry. To incorporate these directions without disproportionately amplifying weak components, the authors introduce Selective Orthogonalization, a smooth transformation that normalizes strong singular directions while preserving the scale of vanishingly weak ones. The paper demonstrates the practical effectiveness of the proposed method through synthetic experiments and several image classification tasks.

Critical strengths

In the experiments presented in the paper, the proposed method consistently outperforms the baselines. If these performance gains hold up and can be extended to larger scales, this would constitute a critical strength of the work.

Critical weaknesses

My main concern is what role the Muon-style orthogonalization approach to per-sample updates actually plays, why it is necessary, and what intuitive benefits it provides. I did not find a convincing intuitive explanation for this approach in the paper. The paper provides a so-called convergence proof in Theorem 4.4. However, this proof relies on Assumption 4.3, which is too strong to hold (indeed, it is easy to construct many simple counterexamples where Assumption 4.3 is violated). As a result, Theorem 4.4 offers little meaningful theoretical insight.

Questions for authors
  1. Algorithm clarification: What is the complete Selective–Muon recurrence, including its relationship to the shared momentum state, the momentum update, and the alternation schedule?

  2. It appears that the eigendecomposition, the computation of per-sample updates, and the construction of their Gram matrix in Algorithm 1 could incur substantial computational and memory overhead. What are the wall-clock and peak-memory costs on the reported tasks, and do the accuracy improvements persist under comparable training-time budgets and a longer training schedule?

3 comments

Thank you for the thoughtful review and for recognizing the potential of the empirical improvements. Our central idea is to augment Muon's aggregated-matrix orthogonalization with the complementary co-update geometry retained across training examples. We clarify below what this adds, what the convergence result establishes, and how the hybrid is implemented and measured.

To W1 — Why is sample-space spectral shaping useful?

Aggregation discards directional information that the proposed transformation can use. For a per-sample update matrix S∈RB×pS\in\mathbb R^{B\times p} with mean update sˉ=B−1S⊤1\bar s=B^{-1}S^\top\mathbf1, Section 2.1 and Theorem 4.2 establish

$ C=S^\top S=B\widehat{\operatorname{Cov}}(s)+B\bar s\bar s^\top, \qquad u_{\mathrm{sel}}=h_\epsilon(C)\bar s. $

This identity yields three complementary reasons for the construction.

  1. Information. Two minibatches may have identical aggregated updates but different second moments CC, and thus different Selective directions. Muon orthogonalizes the aggregated m×nm\times n parameter matrix. Selective additionally uses the co-update structure present before aggregation.
  2. Spectral access. The sample Gram matrix K=SS⊤∈RB×BK=SS^\top\in\mathbb R^{B\times B} shares all nonzero eigenvalues with C∈Rp×pC\in\mathbb R^{p\times p}. When B≪pB\ll p, the relevant sample spectrum is available without constructing the large parameter-side second-moment matrix. The implemented thin SVD is an equivalent spectral route.
  3. Controlled shaping. Rather than setting every nonzero singular value to one, Selective uses gϵ(σ)=σϵ/(σ2+ϵ)g_\epsilon(\sigma)=\sigma^{\epsilon/(\sigma^2+\epsilon)}. For fixed ϵ>0\epsilon>0, Theorem 4.1 gives gϵ(σ)/σ→1g_\epsilon(\sigma)/\sigma\to1 as σ→0+\sigma\to0^+. Across five weak-spectrum parameter structures, Selective achieves lower validation loss than Hard (Table 1). The matched-RMS ablation further shows that retuning Hard's overall update scale does not recover these results over the tested range.

The contribution is therefore a second geometric view and a selective treatment of its spectrum, rather than another normalization of the aggregated Muon matrix.

To W2 — What does Theorem 4.4 guarantee?

Theorem 4.4 establishes conditional first-order stationarity, and Proposition B.5 supplies sufficient conditions for its alignment assumption. For standalone Selective, write ut=Htsˉtu_t=H_t\bar s_t with Ht≻0H_t\succ0. If

$ \kappa(H_t)\le\kappa_\star,\qquad |\bar s_t-\nabla f(\theta_t)|\le\delta|\nabla f(\theta_t)|,\qquad \delta<\kappa_\star^{-1}, $

then

$ \cos!\bigl(\nabla f(\theta_t),u_t\bigr) \ge\frac{\kappa_\star^{-1}-\delta}{1+\delta}>0. $

A uniform bound on ∥sˉt∥\|\bar s_t\| also controls the adaptive RMS target. The spectral condition is computable from the update spectrum. The relative discrepancy from the true gradient remains an additional condition, rather than an automatic property of minibatch sampling. This distinction separates the conditional stationarity guarantee of Theorem 4.4 from the deterministic spectral results in Theorems 4.1--4.2, which do not require Assumption 4.3.

To Q1 — How is Selective--Muon implemented?

Both branches operate on one shared Nesterov momentum state, as specified in Section 3(5) and Appendix E.2.1 (Algorithm 2):

$ M_t=0.6M_{t-1}+G_t,\qquad Z_t=G_t+0.6M_t, \qquad G_t\in\mathbb R^{B\times p}. $

Both branches extract per-sample gradients. With zero-based indexing, even steps apply Selective to ZtZ_t. Odd steps sum its rows, restore the m×nm\times n parameter-matrix shape, and apply Muon's Newton--Schulz transform. The shared state is never reset. Each branch retains its own scaling rule, and non-matrix parameters use auxiliary SGD. The evaluated 1:1 schedule is an implementation choice, not a restriction of the shared-state construction. Other fixed or adaptive branch-selection rules can use the same state. Determining the optimum schedule for a given training setting is a direction for future study.

To Q2 — What are the measured GPU costs and training budgets?

Appendix E.2.1 (Table 4) reports RTX 5090 step times (ms/step) and peak allocated memory (GiB).

OptimizerBatch 64Batch 128Batch 256
Muon5.360 / 1.5595.338 / 1.5635.630 / 1.846
Selective179.593 / 3.205285.768 / 4.835467.785 / 10.464
Selective--Muon67.866 / 1.593133.428 / 3.945357.404 / 6.290

Per-sample gradient extraction accounts for 79.2% of Selective's measured time at batch 256. The eight-epoch, drop-last step budget follows the convention used in Keller Jordan's publicly released Airbench94 Muon implementation. With 50,000 training images,

$ T(B)=8\left\lfloor\frac{50{,}000}{B}\right\rfloor, $

which gives 6,248 / 3,120 / 1,560 steps for B=64/128/256B=64/128/256. These data report implementation cost and fixed-epoch performance.

Dear Reviewer JZ1z,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Thank you for your response. However, it seems to me that you introduced a new assumption and claimed that Theorem 4.4 holds under this new assumption. But what exactly does this assumption mean, and why should we expect it to hold in practice? This new assumption still appears to be rather strong. I am inclined to maintain my current rating. Thanks.

Reviewer_fYj3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes using per-sample update geometry to augment matrix orthogonalization. It stacks flattened per-sample updates and applies a selective spectral transformation that moves strong singular values toward one while preserving the scale of sufficiently weak ones. It also introduces Selective–Muon, which alternates this method with Muon using shared momentum. The paper analyzes the spectral transformation and evaluates the methods on synthetic problems and CIFAR-10/100.

Critical strengths
  1. Incorporating sample geometry into orthogonalization is a natural and intuitive idea. It uses information beyond the aggregated update, and the experiments show improvements over the corresponding baselines.
  2. Theorem 4.2 provides an explicit connection to parameter-space preconditioning. This gives a useful interpretation of the proposed update.
Critical weaknesses
  1. Section 3(5) states that Selective and Muon alternate, but does not specify the switching criterion or schedule, or how the shared momentum enters each update. Algorithm 1 only describes Selective. Providing the hybrid's update equations and switching rule would make this part more reproducible.

  2. Theorem 4.1 verifies the intended behavior of the chosen spectral map for fixed epsilon, but does not establish why preserving weak directions' original scale improves optimization. Large relative amplification alone does not show that amplifying these directions is harmful. While the empirical improvements are encouraging, further evidence connecting them to weak-direction preservation and the use of sample geometry would strengthen the motivation.

  3. The experiments include Hard as a sample-geometry-only baseline, but do not include the same selective transformation applied directly to the aggregated momentum matrix. Adding this baseline would help separate the contributions of sample geometry and selective normalization.

  4. Assumption 4.3 requires the cosine similarity between the transformed update and the true gradient to remain bounded below by a positive constant at every nonstationary iteration. This is a strong, restrictive condition that seems not to be established specifically for the proposed algorithm. Verifiable conditions under which the update satisfies this assumption would better connect the convergence guarantee to the design.

  5. All real-data experiments are conducted on CIFAR-10/100, with no transformer language-model evaluation. This leaves the generality of the proposed algorithms unclear, as Muon was specifically designed for matrix-structured QKV weights in transformers. Evaluation on a modest transformer language model would provide useful evidence in comparison with Muon.

Minor comments: The following sentences read a bit unclear to the reviewer.

  • Lines 44–46: “Muon therefore uses spectral normalization to act on the row and column directions of matrix-shaped aggregated momentum updates.” It would help to state explicitly that Muon preserves the left and right singular vectors while approximately flattening the nonzero singular values.
  • Line 60: “Hard polar normalization, however, removes magnitude distinctions across the active spectrum”. Clarifying that this is a potential drawback of the same orthogonalization operation described in the preceding sentence would make the transition easier to follow.
Questions for authors
  1. What is the exact switching rule for Selective–Muon, and how is the shared momentum used to construct each branch's update?
  2. Could the authors add the selective-normalization-only baseline described above and provide further evidence for why preserving weak directions' scale is beneficial?
  3. Under what verifiable conditions does the proposed algorithm satisfy Assumption 4.3?
  4. Could the authors report runtime and memory overhead relative to Muon in the real-data experiments?
4 comments

Thank you for the constructive review and for recognizing the value of sample-space geometry and the parameter-side interpretation in Theorem 4.2. Our central contribution is to complement Muon's orthogonalization of aggregated parameter matrices with co-update geometry retained before aggregation.

To W1 / Q1 — Shared-momentum recurrence and alternation:

The hybrid algorithm is fully specified in Section 3(5) and Appendix E.2.1 (Algorithm 2). For each eligible parameter block, it maintains a single batch-row momentum state and Nesterov readout:

$ M_t=0.6M_{t-1}+G_t,\qquad Z_t=G_t+0.6M_t,\qquad G_t\in\mathbb R^{B\times p}. $

Both branches extract per-sample gradients and consume the same ZtZ_t. With zero-based indexing, even steps apply Selective to ZtZ_t. Odd steps sum its rows, reshape the aggregate into the original m×nm\times n parameter-matrix geometry, and apply Muon's Newton--Schulz transform. The momentum state is never reset at a branch switch, and the branches retain their respective scaling rules. Non-matrix parameters use auxiliary SGD. The evaluated 1:1 schedule is an implementation choice, not a restriction of the shared-state construction. Other fixed or adaptive branch-selection rules can use the same state. Determining the optimum schedule for a given training setting is a direction for future study.

To W2 / Q2 — Why preserve weak-direction scale?

Selective controls weak-direction amplification without discarding strong-direction normalization. Hard polar normalization maps every nonzero singular value to one, whereas, for fixed ϵ>0\epsilon>0, Theorem 4.1 establishes

$ \lim_{\sigma\to0^+}\frac{g_\epsilon(\sigma)}{\sigma}=1, \qquad \lim_{\sigma\to0^+}\frac{1}{\sigma}=\infty. $

Three observations support the optimization relevance. (1) Selective achieves lower validation loss than Hard across all five controlled weak-spectrum structures (Table 1). (2) Retuning Hard's global RMS does not recover Selective's results over the tested range (Section 5.1). (3) On ResNet-18, layer-4 weak-direction amplification is 251.3±36.7251.3\pm36.7 for Hard versus 5.15±1.765.15\pm1.76 for Selective (Section 5.2). The matched-RMS control is particularly important: the difference is not explained by global update scaling alone.

To W3 / Q2 — Why sample-space geometry instead of aggregated-matrix Selective?

The two spectral views retain different information. Sample space also makes its own active spectrum accessible at low dimension. For S∈RB×pS\in\mathbb R^{B\times p} with row mean sˉ\bar s, Section 2.1 and Theorem 4.2 yield

$ C=S^\top S=B\widehat{\operatorname{Cov}}(s)+B\bar s\bar s^\top, \qquad u_{\mathrm{sel}}=h_\epsilon(C)\bar s. $

  1. Information. Aggregation determines sˉ\bar s but not the sample-level second moment CC. Consequently, an operation on the aggregated m×nm\times n matrix cannot, in general, recover the co-update structure used by Selective.
  2. Spectral access. The B×BB\times B Gram matrix K=SS⊤K=SS^\top shares all nonzero eigenvalues with the p×pp\times p matrix CC. For B=128B=128 and p=106p=10^6, these matrices have 16,38416{,}384 and 101210^{12} entries, respectively. Thus the sample spectrum can be obtained without forming or eigendecomposing CC. The implementation can equivalently use a thin SVD of SS. This provides low-dimensional access to the sample spectrum. Per-sample gradient extraction remains a separate cost. Exact SVD of large aggregated matrices can also be expensive, motivating iterative methods such as Muon's Newton--Schulz iteration.
  3. Complementarity. Muon shapes the spectrum of the aggregated parameter matrix, while Selective shapes the spectrum before aggregation. Their combination therefore introduces a second geometric view into one optimization trajectory. Our Hard-versus-Selective comparison isolates spectral shaping within sample space. An aggregated-matrix Selective control would further separate the empirical contributions of geometry and spectral normalization.

To W4 / Q3 — Verifiable conditions for Assumption 4.3:

Theorem 4.4 establishes conditional first-order stationarity. Proposition B.5 makes its alignment premise explicit. For the standalone Selective update, ut=Htsˉtu_t=H_t\bar s_t with Ht≻0H_t\succ0. If

$ \kappa(H_t)\le\kappa_\star,\quad |\bar s_t-\nabla f(\theta_t)|\le\delta|\nabla f(\theta_t)|,\quad \delta<\kappa_\star^{-1}, $

then

$ \cos\bigl(\nabla f(\theta_t),u_t\bigr) \ge\frac{\kappa_\star^{-1}-\delta}{1+\delta}>0. $

A uniform bound on ∥sˉt∥\|\bar s_t\| also bounds the adaptive RMS target. The spectral condition is computable from the update spectrum, while control of the true-gradient discrepancy is an additional requirement. Theorem 4.4 concerns standalone Selective. In contrast, Theorems 4.1--4.2 do not require Assumption 4.3. This distinction separates the conditional stationarity guarantee from the spectral properties established without alignment assumptions.

To W5 — Transformer evaluation and scale:

We appreciate the suggestion to evaluate Transformer language models. The spectral distinction is independent of model size: hard polar normalization flattens active singular values, whereas Selective preserves the scale of vanishingly weak directions under the fixed-ϵ\epsilon limit. The update spectrum reflects data- and model-dependent geometry, which can be examined in our controlled and CIFAR experiments rather than requiring a large training scale.

To Q4 — GPU runtime, memory, and training budget:

Appendix E.2.1 (Table 4) reports measured RTX 5090 step times and peak allocated memory. At batch size 256, Muon requires 5.6305.630 ms/step and 1.8461.846 GiB, versus Selective at 467.785467.785 ms/step and 10.46410.464 GiB. Per-sample gradient extraction accounts for 79.2% of the measured Selective runtime. Selective--Muon records 67.866/133.428/357.40467.866/133.428/357.404 ms/step at batch sizes 64/128/25664/128/256, with corresponding reported memory peaks of 1.593/3.945/6.2901.593/3.945/6.290 GiB.

We follow the eight-epoch, drop-last step-budget convention used in Keller Jordan's publicly released Airbench94 Muon implementation. With 50,000 training images, T(B)=8⌊50,000/B⌋T(B)=8\lfloor50{,}000/B\rfloor, giving 6,248 / 3,120 / 1,560 optimizer steps for our batch sizes 64 / 128 / 256, respectively. Learning rates decay linearly over these budgets, and the weight-decay coefficient follows the same batch-size scaling, 2×10−6B2\times10^{-6}B.

To minor comments — Muon's spectral operation:

The Introduction specifies that Muon's finite-step Newton--Schulz transformation approximates polar orthogonalization, retaining active singular directions while approximately flattening their singular values. The weak-tail discussion concerns this same hard-polar operation.

Dear Reviewer fYj3,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Thanks for the response. It addresses some of my concerns, however some of my major concerns remain.

The matched-RMS comparison is useful, but I am still less convinced by the choice to preserve weak directions' scale. The aggregated-matrix Selective baseline is still missing, which would help separate the contributions of sample geometry and selective normalization. Also, the sufficient condition helps explain Assumption 4.3, but it remains unclear whether or how well the proposed algorithm satisfies it during training. In addition, the experiments remain limited to CIFAR-10/100. The spectral distinction alone does not establish improvement on transformer language models.

I would therefore keep my score.

Comments on the paper

1 comment

Dear Reviewers,

The authors have finished uploading their responses to reviewers, and the discussion phase is now underway. If you haven't already, please carefully review the authors' responses to understand their perspectives. Please engage in thoughtful and constructive discussions with the authors, and update your review or rating if necessary.

The discussion period lasts until November 18th AoE, which means you have less than five days left to interact with the authors. Please start engaging well before the period ends, so that the authors will have sufficient time to respond to any follow-up questions/comments.

Your active participation and contribution to the ongoing discussion are highly encouraged. Thank you very much for your contribution to ICLR.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.