acceptodds
Under review as a conference paper at ICLR 2027

Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization

Abstract

Low-rank gradient compression reduces communication in distributed training by representing updates with rank- factors. Dion is a recent method that approximates Muon, a spectral optimizer that orthogonalizes momentum, using one step of power iteration followed by column normalization (rescaling each column of the right factor to unit length). This makes it compatible with fully sharded data parallel training, but it converges more slowly than full-rank spectral methods. We show that this gap is geometric: column normalization does not yield the rank- polar factor that Muon implicitly targets, so the resulting direction violates the dual-norm constraint of the low-rank spectral geometry, and the rate picks up an extra factor of even though the low-rank approximation of the gradient itself is accurate. The same mismatch enters the smoothness term and the error-feedback recursion in the analysis, which has a knock-on effect on empirical performance. We propose Orth-Dion, which replaces column normalization with QR orthogonalization of the right factor. Under non-Euclidean smoothness, with the curvature constant along rank- directions, Orth-Dion attains rate , matching exact spectral methods at the same per-step communication cost as Dion. The proof removes the bounded-drift assumption common in prior error-feedback analyses via a self-consistent fixed-point argument, and uses a time-averaged contraction that only requires the error sequence to contract on average rather than at every step. Experiments on large-scale language model pre-training validate the predicted scaling and show that Orth-Dion closes the convergence gap to Muon at Dion's communication cost.

Primary area:
optimization
Keywords:
Optimization, Distributed Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4t8BOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies efficient low-rank optimization for distributed language model training. It proposes Orth-Dion, a variant of Dion that replaces column normalization with QR orthogonalization to better align updates with spectral optimization geometry. The paper also introduces an adaptive-rank extension to balance optimization progress and computational cost. It provides a theoretical analysis and language model pretraining experiments to evaluate the proposed methods.

Critical strengths

S1: Identifies a directly testable geometric quantity

Sections 3–4 focus on ν=∥Vˉ∥op\nu=\|\bar V\|_{\mathrm{op}} and measure its distribution across layers, training steps, and ranks in Figure 2. This provides a more informative mechanism diagnostic than final loss alone. The conclusion that QR enforces ν=1\nu=1 is valid and explains the change in the smoothness term of the descent inequality.

S2: Matched-rank experiments isolate the algorithmic change reasonably well

Table 1 reports paired differences across three seeds at r=96,192,384r=96,192,384, with loss improvements of approximately 0.007 and the same direction across all three seeds. Appendix L.1 documents baseline tuning and training settings. Although small, these improvements support the practical value of QR more directly than comparisons involving adaptive rank alone.

S3: System overhead is disclosed in useful detail

Appendix L.3 shows that QR overhead varies with the setting and reaches 9.6% at 1.5B. Appendix L.4 explicitly reports a wall-clock ratio of 1.02 when reaching Dion's loss. This distinction between step savings and actual speedups helps identify the communication regimes in which the method is useful.

Critical weaknesses

W1: QR guarantees a partial isometry, not polar alignment; the central defect bound has a counterexample

Algorithm 1, lines 4–5, sets W=M⊤UW=M^\top U, Vˉ=Q(W)\bar V=Q(W), and D^=UVˉ⊤\hat D=U\bar V^\top. Lemma 4.1 establishes only that the nonzero singular values of D^\hat D equal one. It does not establish that D^\hat D is the polar factor of the selected low-rank matrix. Appendix E nevertheless claims δ≤C∥M∥Fε\delta\le C\|M\|_F\varepsilon, where ε\varepsilon is the left-subspace projector error.

Take r=2r=2 and M=diag⁡(2,1,0)M=\operatorname{diag}(2,1,0), and let the columns of UU be an orthonormal basis obtained by rotating the first two coordinate directions by 45°. Then UU spans the exact left singular subspace of MM, so ε=0\varepsilon=0. In a thin QR decomposition W=M⊤U=QRW=M^\top U=QR with positive diagonal entries, the two diagonal entries of RR are 5/2\sqrt{5/2} and 2/5/22/\sqrt{5/2}. Hence

=\operatorname{tr}(R) =\sqrt{5/2}+\frac{2}{\sqrt{5/2}} \approx2.84605<3=\|M\|_{\mathrm{KF},2}.$$ Thus $\delta\approx0.15395>0$, directly contradicting a bound requiring $\delta=0$ when $\varepsilon=0$. This is not a QR sign-convention issue: the construction already uses a positive-diagonal convention. If $W=QH$, retaining the small matrix $H$ and inserting the polar rotation of $H^\top$ between $U$ and $Q$ would recover the correct polar pairing for the projected matrix. Such an algorithmic correction requires new analysis and experiments. The unsupported claim of recovering the polar rate is the main reason for rejection. ### W2: The proof of Theorem 5.1 does not establish the claimed leading constant Steps 2–3 of Appendix H first obtain an error sum of $O(T\eta^2)$. Dividing by $T\eta$ leaves $O(\eta)$, not the stated $O(1/T)$. With $\eta=c/\sqrt T$, this has the same order as the smoothness contribution. The stated proof therefore yields, at most, $$\min_t\|G_t\|_{\mathrm{KF},r} \le\frac{f_0-f_\infty}{T\eta} +\left(\frac{L_r}{2}+C\right)\eta +O\!\left(\frac{1}{T}\right).$$ rather than the same leading constant $\sqrt{2(f_0-f_\infty)L_r/T}$. Retaining the latter requires a stronger error bound or a specific cancellation argument. The theorem statement also omits $A^{\prime}$ and $B^{\prime\prime}$, which are used in the appendix. Moreover, $B^{\prime}$ requires a positive lower bound on $\sigma_r(G_t)$, restricting the applicable horizon to stages before approaching stationarity. The appendix acknowledges this restriction, but the convergence claims in the main text should be narrowed accordingly. ### W3: The residual fixed-point argument is not closed, and identical tracking is overstated Combining the inequality in Appendix F.4 with Weyl's inequality yields only a bound resembling $R_{t+1}\le\tau+R_t+\text{leakage}$, without a residual coefficient strictly below one. Appendix F.5 asserts an $O(\eta)$ Lipschitz constant without specifying a complete single-valued map, an invariant neighborhood, or explicit derivative bounds. A small-tail assumption does not automatically fill this gap. The equal-column-space observation in Lemma 4.3 applies only to the same input $W$. The two algorithms update parameters differently, so their subsequent $G$, $M$, and spectral gaps follow different trajectories. It does not imply identical gradient subspaces and errors throughout training, as asserted in Section 9. A conditional comparison of subspace-iteration structure is reasonable; an identical-trajectory conclusion is not. ### W4: Scale and adaptation evidence does not support the combined claim in the conclusion The 17.1B timing experiment in Appendix L.2 runs for only 50 steps and forces Ada-Orth-Dion's rank to 93% of Dion's, using a ratio obtained at 320M. It measures implementation overhead at a chosen rank, not genuine online rank selection at 17.1B. The 1.5B result in L.4 uses one seed and shows no wall-clock speedup. Section 8.3 and the conclusion should avoid combining separate experiments into a claim of equal speed and lower loss on a large model. A complete training comparison at an affordable intermediate scale, using actual adaptive rank and a fixed-rank control under the same budget, would be most useful. ### W5: The $\beta$ mechanism discussion lacks derivation and contains internal tensions Section 6 and Appendix K infer optimal $\beta$, random-walk behavior, and even divergence from the sign of adjacent residual correlations. This is insufficient: negatively correlated inputs under a fixed projection can cancel rather than make the buffer diverge. The formula $\beta^*=1-2\hat\varepsilon$ gives approximately 0.3 for $\hat\varepsilon\approx0.35$, which does not precisely match the optimum near 0.1 in Table 6. Appendix K.2 also states that ColNorm+EF outperforms Orth-Dion+EF; the relevant setting and its relationship to Table 1 need clarification. These claims should be presented as hypotheses for controlled settings, with the algorithm receiving $\beta$, explicit-momentum controls, and multiple-seed evidence specified.
Questions for authors
  1. How do you reconcile the ε=0\varepsilon=0, δ>0\delta>0 counterexample in W1? Does the implementation contain a small-matrix polar rotation omitted from the algorithm?
  2. How does the O(Tη2)O(T\eta^2) term in Appendix H become O(1/T)O(1/T) after division by TηT\eta? Please provide the final bound with all constants retained.
  3. What is the single-valued residual map in F.5? Which inequality establishes its contraction coefficient and invariant neighborhood?
7 comments
AuthorsOpenReview ↗

Regarding Weakness 1 and Question 1

QR guarantees a partial isometry and does not guarantee polar alignment. With r=2r=2, M=diag(2,1,0)M=\mathrm{diag}(2,1,0) and UU an exact basis rotated by 45°, ε=0\varepsilon=0 but δ≈0.154>0\delta\approx 0.154>0, contradicting δ≤C∥M∥Fε\delta\le C\lVert M\rVert_F\varepsilon in Appendix E. Does the implementation contain a small-matrix polar rotation omitted from the algorithm?

Thank you for taking the time to build this example. It concerns the Appendix E bound of the oracle defect δt\delta_t by the tracking error εt\varepsilon_t, which holds for the polar factor of WtW_t and is too strong for the QR factor. The rest of the analysis does not depend on that bound, however. Lemma 4.1 (νt=1\nu_t=1) and Lemma 4.3 (the shared column space) hold as stated, and the removal of the r\sqrt r factor in Theorem 5.1 holds for QR up to a multiplicative constant that we measure below. In your example UU spans the top-rr subspace exactly, giving ε=0\varepsilon=0, but its basis is rotated, so QR leaves a defect of δ≈0.154\delta\approx0.154 that the polar factor removes. Our implementation applies no rotation after QR, so in the revised paper we corrected Appendix E with the three results below and left the algorithm unchanged.

  1. Among the directions inside the tracked subspace with unit dual norm, the polar factor of WtW_t maximizes ⟨Mt,D⟩\langle M_t, D\rangle, and the maximum is ∥Ut⊤Mt∥∗\lVert U_t^\top M_t\rVert_\ast. This is the correction you suggested.
  2. For the polar factor, 0≤δt≤εt2 ∥Mt∥KF,r0\le\delta_t\le\varepsilon_t^2\,\lVert M_t\rVert_{\mathrm{KF},r} with no spectral gap assumed, so the defect vanishes when tracking is exact.
  3. For the QR factor QtQ_t, let Δt=∥Qt−Pt∥op\Delta_t=\lVert Q_t-P_t\rVert_{\mathrm{op}} be its distance to the polar factor PtP_t of WtW_t. Then ⟨Mt,UtQt⊤⟩≥(1−Δt2/2) ∥Ut⊤Mt∥∗\langle M_t, U_tQ_t^\top\rangle\ge\big(1-\Delta_t^2/2\big)\,\lVert U_t^\top M_t\rVert_\ast. The loss is second order because QtQ_t and PtP_t both have orthonormal columns, and on your example the inequality holds with equality.

In the revised Theorem 5.1 we carry this defect through the convergence proof as an explicit additive term ΔˉQR\bar\Delta_{\mathrm{QR}}, the mean in-subspace defect of the QR factor, which is zero for the polar factor. A corollary gives the multiplicative form. Whenever the QR direction attains at least a fraction q>0q>0 of the restricted optimum at every step, and by result 3 above q=1−Δ2/2q=1-\Delta^2/2 suffices when max⁡tΔt≤Δ<2\max_t\Delta_t\le\Delta<\sqrt2, the bound holds divided by q (1−εˉ2)q\,(1-\bar\varepsilon^2) with no additive term. The factor qq does not depend on the rank and (1−εˉ2)−1=1+O(1/T)(1-\bar\varepsilon^2)^{-1}=1+O(1/T) at the optimized step size, so the r\sqrt r separation from column normalization is unchanged.

We measured that fraction on saved states of the 320M runs, and we implemented the polar factor and trained with it.

Share of the restricted optimum that QR loses (630 matrix-states, 320M)
in aggregate, across 5 checkpoints0.2% to 1.9%
on the worst single matrix6.3%
Right factor, rank 384, val. lossSeed 0Seed 1Seed 2Mean
QR (Orth-Dion)3.57643.56503.56273.5680
Polar factor3.57933.56683.56493.5703

On the saved states, the descent term of the QR direction, ⟨Mt,UtQt⊤⟩\langle M_t, U_tQ_t^\top\rangle, is within 2% of the polar factor's ∥Ut⊤Mt∥∗\lVert U_t^\top M_t\rVert_\ast in aggregate and within 6.3% on the worst single matrix. In training, the polar factor did not lower the validation loss, since QR ended slightly lower on every seed. We therefore keep QR as the default and include the polar factor as an option in the released code. We trained these runs with a smaller batch than Table 1, so we compare the two rows only with each other. In the abstract, Section 4 and the conclusion we now state the claim as QR removing the ν\nu inflation, which is what Lemma 4.1 and Theorem 5.1 establish.

AuthorsOpenReview ↗

Regarding Weakness 2 and Question 2

In Appendix H the error sum is O(Tη2)O(T\eta^2); dividing by TηT\eta leaves O(η)O(\eta), the same order as the smoothness term, so the leading constant 2(f0−f∞)Lr/T\sqrt{2(f_0-f_\infty)L_r/T} is not established. The theorem statement omits A′A' and B′′B'', and B′B' restricts the horizon. Please give the final bound with all constants.

We appreciate your careful review. Step 3 of Appendix H dropped the error sum, which after dividing by TηT\eta is of the same order as the smoothness term, so the leading constant 2(f0−f∞)Lr/T\sqrt{2(f_0-f_\infty)L_r/T} was not established; the rate stays 1/T1/\sqrt T and the separation from column normalization stays, and the correction is to the constant. In the revised paper we restate Theorem 5.1 with every term kept. For a constant step η\eta and every TT on the horizon,

min⁡t<T∥Gt∥KF,r≤f0−f∞ηT+Lr2η+εˉ2(r GF+rRˉ)+2rRˉ+ΔˉQR,\min_{t<T}\lVert G_t\rVert_{\mathrm{KF},r}\le\frac{f_0-f_\infty}{\eta T}+\frac{L_r}{2}\eta+\bar\varepsilon^2(\sqrt r\,G_F+r\bar R)+2r\bar R+\bar\Delta_{\mathrm{QR}},

with εˉ=2A(LFr η+4τM)\bar\varepsilon=2A(L_F\sqrt r\,\eta+4\tau_M), Rˉ=2(τM+GFεˉ)\bar R=2(\tau_M+G_F\bar\varepsilon) and A=4γ~eff/((1−ρ′)Δgap)A=4\tilde\gamma_{\mathrm{eff}}/((1-\rho')\Delta_{\mathrm{gap}}). Here ΔˉQR≥0\bar\Delta_{\mathrm{QR}}\ge0 is the mean in-subspace defect of the QR factor, which is zero for the polar factor (see our reply to W1). When τM≤cτη\tau_M\le c_\tau\eta, the third and fourth terms are at most CˉEF η\bar C_{\mathrm{EF}}\,\eta, and optimizing over η\eta gives

min⁡t<T∥Gt∥KF,r≤2(f0−f∞)(Lr+2CˉEF)T+ΔˉQR.\min_{t<T}\lVert G_t\rVert_{\mathrm{KF},r}\le\sqrt{\frac{2(f_0-f_\infty)(L_r+2\bar C_{\mathrm{EF}})}{T}}+\bar\Delta_{\mathrm{QR}}.

The leading constant is therefore Lr+2CˉEFL_r+2\bar C_{\mathrm{EF}}, which replaces the LrL_r alone of the submission. The same proof applied to column normalization gives rLrrL_r in place of LrL_r and (1+r)(1+\sqrt r) instead of the factor 2 in front of the residual, so the rank-dependent separation holds with the corrected constants.

We also added A′A', the tail condition (which we now state on the buffer, see our reply to W3) and the horizon to the theorem statement. Since B′B' requires σr(Gt)\sigma_r(G_t) to stay bounded away from zero, the theorem covers the part of training in which the gradient keeps its rank-rr structure, and we state this after the theorem. We revised the abstract and contribution (ii) to match.

AuthorsOpenReview ↗

Regarding Weakness 3 and Question 3

The residual fixed-point argument is not closed: F.4 with Weyl gives no residual coefficient below one, and F.5 asserts an O(η)O(\eta) Lipschitz constant without a single-valued map, invariant neighborhood or derivative bounds. Lemma 4.3 holds for the same input WW and does not imply identical tracking throughout training (Section 9).

Thank you for raising these points. The residual is still controlled at order η\eta, and we now have proved it by induction under a tail condition stated on the buffer. Weyl's inequality returns ∥Rt∥op\lVert R_t\rVert_{\mathrm{op}} in the bound of F.4 with coefficient one, so the Banach argument of F.5 does not close, and we withdrew it. We replaced Proposition F.5 with an induction on the pair (εt,∥Rt∥op)(\varepsilon_t,\lVert R_t\rVert_{\mathrm{op}}) over the invariant set [0,εˉ]×[0,Rˉ][0,\bar\varepsilon]\times[0,\bar R], with εˉ\bar\varepsilon and Rˉ\bar R as in our reply to W2.

  1. For the residual, F.4 and Assumption C′C' give ∥Rt+1∥op≤σr+1(Mt)+εt σ1(Mt)≤τM+2GFεˉ≤Rˉ\lVert R_{t+1}\rVert_{\mathrm{op}}\le\sigma_{r+1}(M_t)+\varepsilon_t\,\sigma_1(M_t)\le\tau_M+2G_F\bar\varepsilon\le\bar R.
  2. For the tracking error, ∥Rt∥op≤Rˉ≤Δgap/4\lVert R_t\rVert_{\mathrm{op}}\le\bar R\le\Delta_{\mathrm{gap}}/4 keeps the buffer gap above Δgap/2\Delta_{\mathrm{gap}}/2, so Wedin's theorem bounds the drift DtD_t of the buffer's top-rr subspace, and the tracking recursion gives εt+1≤ρ′εt+γ~effDt≤εˉ\varepsilon_{t+1}\le\rho'\varepsilon_t+\tilde\gamma_{\mathrm{eff}}D_t\le\bar\varepsilon.
  3. The coefficient on εt\varepsilon_t is ρ′<1\rho'<1, and the coupling between the two components closes when 8AGF≤18AG_F\le1. In Theorem G.1 of the revised paper we extend this induction to a contraction that holds over windows of steps rather than at every step, which allows transient ρt>1\rho_t>1 and recovers the per-step condition as the case Cρ=1C_\rho=1.

For the induction we use the tail condition $\sigma_{r+1}(M_t)\le\tau_M$ on the buffer, with which we control the residual and the tracking error together. A tail condition on the gradient alone would leave $\lVert R_t\rVert_{\mathrm{op}}$ with coefficient one through Weyl's inequality, as you kindly point out. We now state in the abstract and contribution (iii) that this buffer condition replaces the bounded-drift assumption.

We also restated Lemma 4.3 as a comparison of one step of the recursion for the same input $W_t$, which is the comparison the analysis relies on, and we aligned Sections 4 and 9 with this statement.

Reviewer_vxUZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the normalization used in Dion, a low-rank spectral optimizer for distributed training. It argues that column normalization can inflate the dual norm of the update, introducing a geometric penalty beyond low-rank approximation error. Orth-Dion replaces column normalization with QR orthogonalization, while Ada-Orth-Dion additionally adjusts the rank to offset the extra computation. The paper presents convergence arguments based on coupled residual and subspace tracking, together with language-model pretraining experiments and distributed timing measurements.

Critical strengths
  1. The paper is easy to follow. The motivation for replacing column normalization with QR orthogonalization is clear, and the proposed modification is simple to implement.
  2. The paper provides direct measurements of the update norm during training. These results support the observation that column normalization introduces norm inflation and that QR removes it.
  3. Orth-Dion consistently improves validation loss across the tested ranks and random seeds, providing useful evidence of the effectiveness of the proposed modification.
Critical weaknesses
  1. The theoretical analysis appears to confuse orthogonality with optimal alignment. QR produces orthogonal factors, but it does not generally recover the polar update.
  2. The evidence for large-scale efficiency is somewhat limited. The 1.5B experiment does not improve time to the target loss, while the 17.1B experiment only measures short-run timing with a fixed rank transferred from the smaller model.
Questions for authors

Could the authors clarify why exact subspace tracking should imply zero oracle defect for the QR update?

2 comments
AuthorsOpenReview ↗

Regarding Weakness 1 and Question 1

The theoretical analysis appears to confuse orthogonality with optimal alignment. QR produces orthogonal factors but does not generally recover the polar update. Why should exact subspace tracking imply zero oracle defect for the QR update?

We appreciate you raising this question, we could've been more precise about this point. The exact tracking gives zero defect for the polar factor of Wt=Mt⊤UtW_t=M_t^\top U_t, and for the QR factor it also needs the columns of WtW_t to be orthogonal, so we revised the paper to state the implication for the polar factor only and more clearly. Lemma 4.1 (νt=1\nu_t=1) and the removal of the factor rr from the smoothness term don’t use it and hold as stated.

We separated the two in Appendix E of the revised paper. The polar factor is the best direction of unit dual norm inside the tracked subspace, and its defect is at most εt2∥Mt∥KF,r\varepsilon_t^2\lVert M_t\rVert_{\mathrm{KF},r} with no spectral gap assumed. The QR factor attains at least the fraction 1−Δt2/21-\Delta_t^2/2 of that, where Δt\Delta_t is its operator-norm distance to the polar factor. We carry the remaining defect through the convergence proof as an explicit additive term, so Theorem 5.1 keeps the O(1/T)O(1/\sqrt{T}) rate and a leading constant free of the r\sqrt r inflation, and a corollary divides the bound by that fraction when it is bounded below.

We measured that fraction on saved states of the 320M runs, and we implemented the polar factor and trained with it.

Share of the restricted optimum that QR loses (630 matrix-states, 320M)
in aggregate, across 5 checkpoints0.2% to 1.9%
on the worst single matrix6.3%
Right factor, rank 384, val. lossSeed 0Seed 1Seed 2Mean
QR (Orth-Dion)3.57643.56503.56273.5680
Polar factor3.57933.56683.56493.5703

The descent term of the QR direction, ⟨Mt,UtQt⊤⟩\langle M_t, U_tQ_t^\top\rangle, is within 2% of the polar factor's ∥Ut⊤Mt∥∗\lVert U_t^\top M_t\rVert_\ast in aggregate and within 6.3% on the worst single matrix. In training, the polar factor did not lower the validation loss, since QR ended slightly lower on every seed, so we keep QR as the default and include the polar factor as an option in the released code. We trained these runs with a smaller batch than Table 1, so we compare the two rows only with each other.

AuthorsOpenReview ↗

Regarding Weakness 2

Evidence for large-scale efficiency is limited: the 1.5B experiment does not improve time to target loss, and the 17.1B experiment only measures short-run timing with a rank transferred from the smaller model.

Thank you for raising this, since we can make the evidence for the large-scale claim much clearer. We establish that benefit through the cost of the optimizer step, which is where the low-rank update helps and is how Dion is evaluated at scale as well [1]. A full convergence run at a size where the low-rank step is much cheaper than Muon's is out of reach on an academic budget unfortunately. The 1.5B run therefore does not reduce time to a target loss, and the 17.1B run measures step time at a rank carried over from 320M. We revised the paper to say so, and we added three results.

  1. We trained Ada-Orth-Dion against the same code with the rank frozen, as complete runs at 320M with three seeds. At the same average rank the adaptive run ends below the frozen control by several times the spread across seeds, so online rank selection is what produces the gain and not the adaptive code path.
  2. We plotted validation loss against wall-clock time at 320M with Muon as the reference (Figure 4 in Appendix L.1 of the revised paper). Muon finishes sooner and at a lower loss at this size, and we revised the paper to say so. The matrices of a 320M model are small enough that a full orthogonalization is cheap and communication is a small part of the step, which is the regime that favors Muon.
  3. We measured one complete training step of a 4.3B model at rank fraction 1/4 on 4 A40, a size at which the low-rank step is the cheaper one.
Transport between GPUsMuonDionOrth-Dion
host shared memory6.34 s4.97 s5.22 s
sockets14.68 s13.00 s13.63 s

The Orth-Dion step is shorter than Muon's under both transports, since it exchanges a small factor between GPUs instead of the full matrix. The same pattern holds across matrix sizes in our update benchmark, where the low-rank methods overtake Muon once the matrix reaches 8192 and Orth-Dion reaches 9.5 times Muon's update speed at rank fraction 1/16.

We now say in the paper which evidence supports the efficiency claim, the step cost at large matrices and low rank fraction, and we state as a limitation that we cannot run a full convergence comparison at that size in our budget, but is a valuable next step this work opens.

Thank you again for your time and thoughtfulness in this review. Both of your points led to changes that made the paper stronger, and we hope we have addressed your concerns.

[1] Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, John Langford. Dion: Distributed Orthonormalized Updates. arXiv:2504.05295.

Reviewer_KnMUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a geometric mismatch in Dion's update rule from the perspective of steepest descent under the Ky Fan norm. Specifically, the authors show that column normalization in Dion does not guarantee an orthogonal update direction, introducing a rank-dependent penalty in the convergence bound. To address this issue, they propose Orth-Dion, which replaces column normalization with QR orthogonalization and achieves the same convergence rate as exact rank-r spectral optimization under the stated assumptions, while maintaining Dion's communication complexity. LLM pretraining experiments demonstrate that Orth-Dion consistently outperforms Dion at matched ranks. Furthermore, the authors introduce Ada-Orth-Dion, which adaptively adjusts the rank to compensate for the additional computational overhead of QR orthogonalization, further improving empirical performance and efficiency. Finally, the paper analyzes the effect of the error-feedback coefficient β\beta, showing that its optimal value depends on the correlation of residual gradients across steps and that tuning β\beta below 1 can substantially improve performance in stochastic training settings.

Critical strengths

The paper identifies a geometric mismatch in Dion's update rule from the perspective of steepest descent under the Ky Fan norm and proposes a simple correction using QR orthogonalization. This modification removes the rank-dependent penalty in the convergence bound, achieving the same convergence rate as exact rank-r spectral optimization under the stated assumptions. The theoretical improvement is supported by LLM pretraining experiments, where Orth-Dion outperforms Dion at matched ranks. Furthermore, the authors propose an adaptive rank selection algorithm to mitigate the additional computational overhead introduced by QR orthogonalization, achieving further empirical improvements while maintaining computational efficiency.

Critical weaknesses
  • The paper primarily frames its experimental results and discussions around comparisons with Dion, whereas I believe Muon should serve as the main baseline. The fundamental motivation behind Dion and other communication-efficient Muon variants is to reduce communication overhead in distributed training and ultimately achieve faster convergence than Muon in terms of wall-clock time. Indeed, the original Dion paper evaluates its effectiveness against Muon from this perspective. However, this paper primarily emphasizes improvements over Dion, reporting convergence in terms of training steps and measuring speedups relative to Dion's best validation loss. Although Muon is included in the results, it is not the central baseline for evaluating the proposed method's overall training efficiency. In particular, the main convergence plots use training steps rather than wall-clock time as the x-axis. It is unsurprising that a low-rank approximation method may converge more slowly than full-rank Muon in terms of training steps. The more important question is whether its reduced communication overhead translates into faster convergence in wall-clock time. Therefore, I believe the main experimental evaluation should focus on validation loss versus wall-clock time, with Muon as the primary baseline, rather than primarily comparing Orth-Dion against Dion. Furthermore, the paper lacks comparisons with more recent communication-efficient Muon variants, such as Dion2 [1] and MuonBP [2], which would be important for establishing the practical competitiveness of the proposed method.
  • Although the paper identifies a suboptimal update in Dion and demonstrates both theoretical and empirical improvements by correcting it, the proposed modification is relatively modest. The core difference between Dion and Orth-Dion is simply replacing column normalization with QR orthogonalization, while the remaining algorithmic components are essentially unchanged. Despite the additional theoretical analysis, the main algorithmic contribution appears incremental.

References

[1] K. Ahn, N. Amsel, and J. Langford. Dion2: A Simple Method to Shrink Matrix in Muon. arXiv:2512.16928, 2025.

[2] A. Khaled, K. Ozkara, T. Yu, M. Hong, and Y. Park. MuonBP: Faster Muon via Block-Periodic Orthogonalization. arXiv:2510.16981, 2025.

Questions for authors
  1. Could the authors provide convergence plots with wall-clock time, rather than training steps, on the x-axis, using Muon as the primary baseline?
  2. Could the authors include comparisons with other recent communication-efficient Muon variants, such as Dion2 and MuonBP?
  3. Could the authors confirm whether the error-feedback coefficient β\beta was fixed to 1 in the main Llama 3 320M experiments? Table 2 shows that tuning β\beta can substantially improve validation loss on GPT-2 45M. How would the main experimental results change if β\beta were tuned separately for Dion and Orth-Dion?
4 comments
AuthorsOpenReview ↗

Regarding Weakness 1 and Question 1

We appreciate these concrete suggestions, and we agree that Muon is an important reference for any spectral optimizer. We have added the plot you requested of validation loss against wall-clock time with Muon as the reference, and a measurement of one complete training step of a 4.3B model, where the Orth-Dion step is shorter than Muon's (details below).

We keep Dion as the main baseline because the scientific claim of the paper concerns the geometry of the update. With Orth-Dion, we intentionally changed only Dion's normalization step and kept its rank, communication pattern and schedule, so a matched-rank comparison against Dion isolates the effect of that change only. Ahn et al. evaluate Dion against Muon in wall-clock time at 3B parameters on 16 GPUs, and report that its reduced-rank variants roughly match Muon only at that size [1]. In our 320M setting Dion does not beat Muon either, so the gap to Muon is shared by the whole low-rank family and says little about the change we propose. Muon orthogonalizes and communicates the full matrix at every step, so as the full-rank reference it is expected to reach a lower loss, and the low-rank methods trade some of that loss for less communication per step.

The plot covers all methods on the 320M model, with Muon as the reference (Figure 4 in Appendix L.1 of the revised paper). At this size Muon reaches a lower loss in less time than every low-rank method, and we say so in Section 8.3. The weight matrices of a 320M model are small enough that a full orthogonalization is cheap and communication is a small part of each step, which is the regime that favors Muon.

This is consistent with what is known about communication compression more broadly. Across more than 200 data-parallel setups, Agarwal et al. find that compressing communication speeds up training over an optimized full-communication baseline in only a few cases, because on fast interconnects and at small scale the saved communication is a small part of each step [2].

The low-rank methods are designed for the regime where the full orthogonalization and its communication dominate the step, so we measured the cost of the optimizer update as the matrices grow (Figure 1(b) of the revised paper). Muon's update time grows faster with the matrix size than the low-rank methods' update time. At rank fraction 1/16 the low-rank methods overtake Muon once the matrix size reaches 8192, and at size 32768 Orth-Dion completes one update 9.5 times faster than Muon. Ahn et al. evaluate Dion against Muon at scale with the same kind of update benchmark [1].

We also measured one complete training step of a 4.3B model at rank fraction 1/4 on 4 A40 GPUs, a size at which the low-rank step is the cheaper one.

Transport between GPUsMuonDionOrth-Dion
host shared memory6.34 s4.97 s5.22 s
sockets14.68 s13.00 s13.63 s

The Orth-Dion step is shorter than Muon's under both transports, since it exchanges a small factor between GPUs in place of the full matrix. A complete convergence run at a size where the low-rank step is much cheaper than Muon's is beyond the compute available to us, and we state this as a limitation in the revised paper.

Also, among the communication-efficient methods we compare, Ada-Orth-Dion reaches the lowest validation loss at 320M (3.278, against 3.281 for MuonBP, 3.310 for Dion and 3.319 for Dion2), at a lower average rank than fixed-rank Dion (Table 1, and our reply to W1 and Q2).

[1] K. Ahn, B. Xu, N. Abreu, Y. Fan, G. Magakyan, P. Sharma, Z. Zhan, J. Langford. Dion: Distributed Orthonormalized Updates. arXiv:2504.05295, 2025.

[2] S. Agarwal, H. Wang, S. Venkataraman, D. Papailiopoulos. On the Utility of Gradient Compression in Distributed Training Systems. MLSys 2022.

AuthorsOpenReview ↗

Regarding Weakness 1 and Question 2

The paper lacks comparisons with recent communication-efficient Muon variants such as Dion2 (arXiv:2512.16928) and MuonBP (arXiv:2510.16981).

Thank you for pointing us to these two methods. We added both to Table 1 of the revised paper and trained them in the same 320M setting as the other methods, with three seeds each.

MethodVal. loss (mean ± 2 std, 3 seeds)
Muon3.255 ± 0.010
MuonBP (P=5P=5)3.281 ± 0.002
Dion2 (r=384r=384)3.319 ± 0.010
Dion (r=384r=384)3.310 ± 0.004
Orth-Dion (r=384r=384)3.303 ± 0.007
Ada-Orth-Dion (rmax⁡=384r_{\max}=384)3.278 ± 0.001

Ada-Orth-Dion reaches a lower loss than both methods. MuonBP reaches a lower loss than fixed-rank Dion and Orth-Dion, which fits how it saves communication. It keeps Muon's full-rank update and synchronizes the full matrix only every PP steps, while the Dion family communicates rank-rr factors at every step. Dion2 ends above every other method at a selected fraction of 0.5, the fraction that matches Dion's rank fraction at r=384r=384.

The two methods reduce communication in different ways, and neither uses the column normalization that Orth-Dion replaces. MuonBP orthogonalizes with Muon's Newton-Schulz iteration, and Dion2 applies Newton-Schulz only to a selected subset of rows or columns.

We used the authors' implementation of Dion2. MuonBP has no public code, so we reimplemented Algorithm 1 of the MuonBP paper on top of our Muon implementation, with blocks equal to the FSDP shards, period P=5P=5 and the paper's block-step learning-rate rule. We tuned both on one seed before running three seeds. For MuonBP we swept the learning rate over 0.004, 0.008, 0.012 and 0.016 and used 0.012, which tied with 0.016 for the lowest loss. For Dion2 we swept the learning rate over 0.004, 0.008, 0.012 and 0.016 and used 0.016, which had the lowest loss. One of its three seeds ran on 8 GH200 GPUs like the other methods, and two ran on 4 H100 GPUs with gradient accumulation to the same global batch. We describe both setups in the appendix of the revised paper.

AuthorsOpenReview ↗

Regarding Weakness 2

The proposed modification is modest: the only difference from Dion is replacing column normalization with QR, so the algorithmic contribution appears incremental.

We understand where this concern comes from, and thank you for raising it. We kept the change to one line so that Orth-Dion differs from Dion only in the geometry of its update. The architecture, schedule and communication pattern stay the same, so we can attribute any difference we measure to that geometry alone. The line we replace is the step Dion relies on for orthonormality. Dion's authors present column normalization as the step that makes the update orthonormal (their Section 3.1) [1]. We find that the normalized update is not orthonormal, and that its distance from orthonormal grows with the rank.

Around this change, we contribute three results: measurements of that distance and of the loss it costs Dion at matched rank, a convergence proof that covers both normalizations, and an adaptive-rank method that only works once the geometry is corrected.

  • The rank rr is Dion's central design parameter, since it sets the dimension of the low-rank subspace that each step orthonormalizes and communicates [1]. Under Dion, the update's distance from orthonormal grows with rr. On Llama 3 320M, the measured dual norm of Dion's update rises from 1.20 at r=96r=96 to 1.34 at r=384r=384, and it reaches 2.01 on the 1.5B model, while at matched rank Orth-Dion reaches a lower loss than Dion at every rank on every seed (Table 1 and Figure 2 of the revised paper). Raising rr therefore improves the low-rank approximation and worsens the geometry at the same time. More compute does not remove this error, as it would for the Newton-Schulz approximation in Muon, so closing it requires changing the algorithm. Replacing column normalization with QR makes the update orthonormal at every rank, so in Orth-Dion a larger rr no longer worsens the geometry.

  • We prove a convergence rate for Orth-Dion under non-Euclidean smoothness with error feedback (Theorem 5.1). We control the residual buffer and the tracked subspace together by induction, under a tail condition on the buffer that replaces the bounded-buffer-drift assumption of the original Dion analysis, and the tracking error needs to contract only over windows of steps rather than at every step (Theorem G.1). The same proof applied to column normalization gives the rate with rLrrL_r in place of LrL_r, so the analysis covers Dion as well and holds whether or not one adopts QR.

  • Once a larger rr no longer worsens the geometry (due to our modification), we can adapt the rank during training. Under Dion, raising the rank during training would also worsen the geometry. Ada-Orth-Dion selects rr per layer as training proceeds. It reaches the lowest validation loss among the low-rank methods, at a lower average rank than fixed-rank Dion, and it reaches Dion's best loss in 18.9% fewer steps. Ada-Orth-Dion with rank selection turned off ends at the fixed-rank result, so the improvement comes from choosing the rank during training.

[1] K. Ahn, B. Xu, N. Abreu, Y. Fan, G. Magakyan, P. Sharma, Z. Zhan, J. Langford. Dion: Distributed Orthonormalized Updates. arXiv:2504.05295, 2025.

AuthorsOpenReview ↗

Regarding Question 3

Was β\beta fixed to 1 in the main Llama 3 320M experiments? Table 2 shows tuning β\beta helps on GPT-2 45M. How would the main results change if β\beta were tuned separately for Dion and Orth-Dion?

We fixed β\beta to 1 for every low-rank method in the 320M runs. These runs use Dion's full recursion, Mt+1=Gt+1+β (I−UtUt⊤)Mt+μ UtUt⊤MtM_{t+1}=G_{t+1}+\beta\,(I-U_tU_t^\top)M_t+\mu\,U_tU_t^\top M_t with μ=0.95\mu=0.95, since they are the comparison against Dion. In Algorithm 1 we give the case μ=0\mu=0, which we analyse in Theorem 5.1, and we now write out the full recursion in Section 8.1 of the revised paper.

We then tuned β\beta separately for Dion and Orth-Dion on the 320M model.

β\betaOrth-Dion, val. lossDion, val. loss
1.04.01754.0230
0.74.06254.0713
0.44.06794.0752
0.14.06804.0781

Both methods reach their lowest loss at β=1\beta=1, the value we use in Table 1, so tuning β\beta separately selects the same setting and leaves the main results unchanged. Orth-Dion also reaches a lower loss than Dion at every value of β\beta, so the comparison between the two does not depend on this choice. The benefit of a smaller β\beta in Table 2 is specific to the GPT-2 45M setting, which uses the recursion of Algorithm 1 and so has no momentum inside the tracked subspace. Each run has one seed and 1,500 steps, which is what the rebuttal period allowed, and we now state in the revised paper that both sweeps use one seed.

Thank you for your engagement with both the theory and the experiments. Your questions led us to compare Orth-Dion against stronger baselines and under a wider range of settings, and the paper is better for it. We would be thankful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank very much all three reviewers for their time and constructive feedback. We uploaded the revised paper with every change in red, answered each weakness and question in the individual replies, and summarize the changes here for easier reading.

  1. QR and the polar update (4t8B W1, vxUZ W1). We extended Appendix E to bound the QR update directly, and we carried the resulting term through Theorem 5.1, where the separation from column normalization is unchanged. We also ran experiments showing that QR stays close to the polar update in practice.

  2. Convergence theorem (4t8B W2, W3). We restated Theorem 5.1 with all of its constants and assumptions, and we replaced the fixed-point argument for the residual with a proof by induction.

  3. Adaptive rank and evidence at scale (4t8B W4, vxUZ W2, KnMU W1 and Q1). We ran a frozen-rank control showing that the gain of Ada-Orth-Dion comes from selecting the rank online. We added a wall-clock comparison against Muon and a timing of one full training step on a 4.3B model, and we rewrote Section 8.3 and the conclusion so that each claim names the experiment behind it.

  4. Recent baselines (KnMU W1 and Q2). We added Dion2 and MuonBP to Table 1, trained in the same setting as the other methods with three seeds each.

  5. The role of β\beta (4t8B W5, KnMU Q3). We now present the β\beta regimes as hypotheses, added their derivation to Appendix K, and ran a β\beta sweep for Dion and Orth-Dion showing that the main results do not depend on this choice.

  6. Scope of the contribution (KnMU W2). In our reply we explain why correcting the geometry of Dion's update matters as the rank grows, and how the correction makes it possible to adapt the rank during training.

We hope these changes address the concerns raised, and we would be grateful if you would take them into account in your final evaluations and consider raising your scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.