- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a change to Muon-style optimizers. Instead of only normalizing the combined (averaged) update, it also looks at how the individual training examples in a batch push the parameters, and rebalances those per-example directions.
They also combine it with Muon in a hybrid optimizer that alternates between the two methods. Experiments on small synthetic tasks and short CIFAR-10/100 training runs show that Selective avoids the blow-up and does slightly better than Hard, and that the hybrid beats plain Muon
Critical strengths
Strength:
- Clear and verifiable math. All results seems to be correct, and the two equivalent views (T(S) = U g(Σ) Vᵀ = S h(C)) are cleanly presented. The B×B implementation is practical for small B.
- It works on any parameter shape. It applies to vectors, matrices and heterogeneous blocks, which Muon does not.
- Some real gains. On CIFAR-100 Airbench, both Hard and Selective strongly beat Muon and AdamW.
- Diagnostics match the motivation. Weak-direction amplification is measured during training, layer by layer and across five parameter structures, and Hard really does blow up the tail.
Critical weaknesses
Weakness: Providing more experimental results, under different hyper-parameters setting and on more relevant tasks is pertinent to proving the efficacy of the proposed method. The experiments are small (8-epoch CIFAR), have no error bars, give minimum tuning protocol for the real-data baselines, include no wall-clock comparison, and leave out the obvious damped, truncated, identity and SGD baselines.
- Statistics: multiple seeds with standard deviations, and the native Airbench94 setup (batch 1024, ~94%) as a reference point.
- Equal wall-clock comparisons and time-to-accuracy. (compared to standard Muon & SGD)
- Scale-up: transformer or LM training (e.g., NanoGPT speedrun-scale) and ImageNet-scale vision, where Muon is actually used.
- Ablations of the hybrid: alternation schedule and the shared-versus-separate momentum
Questions for authors
Questions:
- Do we have the wall-clock time per epoch and the peak memory for Selective versus Muon and SGD?
- What is the possible reason for the hybrid underperform Selective alone on all Airbench settings?
3 comments
To Q2 — Why does Hybrid trail standalone Selective on Airbench?
The hybrid is designed to augment Muon with sample-space geometry, not to reproduce an all-Selective trajectory. Its shared-state 1:1 schedule alternates between sample-space Selective and matrix-space Muon, with branch-specific scaling. Accordingly, the two algorithms follow different optimization trajectories.
The following comparisons use Table 2 four-seed mean TTA-2 accuracy. Differences are in percentage points (pp).
| Setting | Batch size | Hybrid Selective | Hybrid Muon |
|---|---|---|---|
| CIFAR-10 / Airbench94 | 64 | +0.63 | |
| 128 | +0.29 | ||
| 256 | +0.14 | ||
| CIFAR-10 / ResNet-18 | 64 | +0.08 | +0.25 |
| 128 | +0.50 | +0.10 | |
| 256 | +0.77 | +0.99 | |
| CIFAR-100 / Airbench-style | 64 | +3.94 | |
| 128 | +2.67 | ||
| 256 | +1.87 |
Hybrid improves over Muon in all nine settings and achieves the highest reported mean accuracy among all five methods at every CIFAR-10/ResNet-18 batch size. On the two Airbench-style architectures, standalone Selective instead achieves higher means. These outcomes are consistent with the two methods applying different spectral treatments. Together, they demonstrate Hybrid's role as a complementary augmentation of Muon, with an architecture-dependent tradeoff relative to standalone Selective.
Dear Reviewer dbkQ,
We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.
Thank you for recognizing the mathematical clarity, parameter-shape generality, and weak-spectrum diagnostics, and for requesting a more complete assessment of experimental reproducibility and computational cost. We distinguish the behavior of standalone Selective from the role of Selective--Muon, whose objective is to augment Muon's aggregated-matrix orthogonalization with sample-space geometry.
To W1 — Baselines, tuning, and controlled comparisons:
The experiments separate two comparisons with different purposes. (1) Hard and Selective act on the same per-sample update matrix, isolating the effect of the weak-direction spectral map. (2) Muon and Selective--Muon compare aggregated matrix orthogonalization with its shared-momentum, sample-space augmentation. Table 2 also includes AdamW as a conventional reference. In Section 5.1, retuning Hard's global RMS does not recover Selective's weak-spectrum validation losses across the tested range. Figure 2(c) reports sensitivity to Selective's transition parameter τ. These are the controls included in the study. Damped or truncated spectral transforms, identity updates, and a separately tuned SGD baseline are not part of the reported real-data comparison.
To W2 — Repeated runs, uncertainty, and the Airbench94 reference:
Table 2 presents TTA-2 accuracy as mean ± sample standard deviation over four seeds for all five optimizers across three settings and three physical batch sizes. The experiments use eight epochs and B∈{64,128,256}, with T(B)=8⌊50,000/B⌋ under the drop-last convention also used in Keller Jordan's publicly released Airbench94 Muon implementation. These settings examine batch-size sensitivity under a common training-horizon convention. They are distinct from the reviewer-requested batch-1024, approximately 94%-accuracy Airbench94 reference and should be read as the specified experimental configurations rather than a reproduction of that reference result.
To W3 / Q1 — Step time, peak memory, and computational cost:
Appendix E.2.1 (Table 4) provides measured RTX 5090 GPU step times and peak allocated memory. At batch 256, AdamW requires 6.896 ms/step and 3.692 GiB, Muon 5.630 ms/step and 1.846 GiB, Hard 466.214 ms/step and 10.464 GiB, and Selective 467.785 ms/step and 10.464 GiB. Hybrid records 67.866 / 133.428 / 357.404 ms/step for batches 64 / 128 / 256, with reported memory peaks of 1.593 / 3.945 / 6.290 GiB. These measurements characterize the implemented methods, with the respective timing protocols specified in the appendix. For Selective at batch 256, exact per-sample gradient extraction accounts for 79.2% of measured runtime. The small B×B Gram representation reduces the dimension of the spectral problem, while the extraction of per-sample updates remains the main implementation cost. The accuracy comparison is based on eight epochs, not a separately measured time-to-target or equal-wall-clock experiment. AdamW is not presented as a measured SGD runtime baseline.
To W4 — Broader training scales:
We appreciate the suggestion to examine Transformers and larger-scale vision tasks. The spectral mechanism itself is not tied to the CIFAR architecture or training scale. Hard polar normalization flattens nonzero singular values, while Selective has unit relative amplification in the vanishing-singular-value limit for fixed ϵ>0. The sample Gram and parameter-side second-moment matrices share the same nonzero spectrum for any parameter-block dimension. Our five controlled parameter structures and the CIFAR-10/100 experiments test these mathematical properties across different update geometries. They establish the reported spectral behavior and image-classification performance, without relying on a large-scale model to define the mechanism.
To W5 — Hybrid schedule and shared momentum:
Selective--Muon combines two distinct spectral operations within a single momentum trajectory, rather than maintaining independent optimizer states. Section 3(5) and Appendix E.2.1 (Algorithm 2) specify
$ M_t=0.6M_{t-1}+G_t,\qquad Z_t=G_t+0.6M_t. $
Even iterations apply sample-space Selective to Zt. Odd iterations sum the rows, restore the original parameter-matrix geometry, and apply Muon's Newton--Schulz transformation. Both branches extract per-sample gradients, and switching never resets the shared buffer. The evaluated 1:1 schedule is an implementation choice, not a restriction of the shared-state construction. Other fixed or adaptive branch-selection rules can use the same state. Determining the optimum schedule for a given training setting is a direction for future study.