- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies efficient low-rank optimization for distributed language model training. It proposes Orth-Dion, a variant of Dion that replaces column normalization with QR orthogonalization to better align updates with spectral optimization geometry. The paper also introduces an adaptive-rank extension to balance optimization progress and computational cost. It provides a theoretical analysis and language model pretraining experiments to evaluate the proposed methods.
Critical strengths
S1: Identifies a directly testable geometric quantity
Sections 3–4 focus on and measure its distribution across layers, training steps, and ranks in Figure 2. This provides a more informative mechanism diagnostic than final loss alone. The conclusion that QR enforces is valid and explains the change in the smoothness term of the descent inequality.
S2: Matched-rank experiments isolate the algorithmic change reasonably well
Table 1 reports paired differences across three seeds at , with loss improvements of approximately 0.007 and the same direction across all three seeds. Appendix L.1 documents baseline tuning and training settings. Although small, these improvements support the practical value of QR more directly than comparisons involving adaptive rank alone.
S3: System overhead is disclosed in useful detail
Appendix L.3 shows that QR overhead varies with the setting and reaches 9.6% at 1.5B. Appendix L.4 explicitly reports a wall-clock ratio of 1.02 when reaching Dion's loss. This distinction between step savings and actual speedups helps identify the communication regimes in which the method is useful.
Critical weaknesses
W1: QR guarantees a partial isometry, not polar alignment; the central defect bound has a counterexample
Algorithm 1, lines 4–5, sets , , and . Lemma 4.1 establishes only that the nonzero singular values of equal one. It does not establish that is the polar factor of the selected low-rank matrix. Appendix E nevertheless claims , where is the left-subspace projector error.
Take and , and let the columns of be an orthonormal basis obtained by rotating the first two coordinate directions by 45°. Then spans the exact left singular subspace of , so . In a thin QR decomposition with positive diagonal entries, the two diagonal entries of are and . Hence
=\operatorname{tr}(R) =\sqrt{5/2}+\frac{2}{\sqrt{5/2}} \approx2.84605<3=\|M\|_{\mathrm{KF},2}.$$ Thus $\delta\approx0.15395>0$, directly contradicting a bound requiring $\delta=0$ when $\varepsilon=0$. This is not a QR sign-convention issue: the construction already uses a positive-diagonal convention. If $W=QH$, retaining the small matrix $H$ and inserting the polar rotation of $H^\top$ between $U$ and $Q$ would recover the correct polar pairing for the projected matrix. Such an algorithmic correction requires new analysis and experiments. The unsupported claim of recovering the polar rate is the main reason for rejection. ### W2: The proof of Theorem 5.1 does not establish the claimed leading constant Steps 2–3 of Appendix H first obtain an error sum of $O(T\eta^2)$. Dividing by $T\eta$ leaves $O(\eta)$, not the stated $O(1/T)$. With $\eta=c/\sqrt T$, this has the same order as the smoothness contribution. The stated proof therefore yields, at most, $$\min_t\|G_t\|_{\mathrm{KF},r} \le\frac{f_0-f_\infty}{T\eta} +\left(\frac{L_r}{2}+C\right)\eta +O\!\left(\frac{1}{T}\right).$$ rather than the same leading constant $\sqrt{2(f_0-f_\infty)L_r/T}$. Retaining the latter requires a stronger error bound or a specific cancellation argument. The theorem statement also omits $A^{\prime}$ and $B^{\prime\prime}$, which are used in the appendix. Moreover, $B^{\prime}$ requires a positive lower bound on $\sigma_r(G_t)$, restricting the applicable horizon to stages before approaching stationarity. The appendix acknowledges this restriction, but the convergence claims in the main text should be narrowed accordingly. ### W3: The residual fixed-point argument is not closed, and identical tracking is overstated Combining the inequality in Appendix F.4 with Weyl's inequality yields only a bound resembling $R_{t+1}\le\tau+R_t+\text{leakage}$, without a residual coefficient strictly below one. Appendix F.5 asserts an $O(\eta)$ Lipschitz constant without specifying a complete single-valued map, an invariant neighborhood, or explicit derivative bounds. A small-tail assumption does not automatically fill this gap. The equal-column-space observation in Lemma 4.3 applies only to the same input $W$. The two algorithms update parameters differently, so their subsequent $G$, $M$, and spectral gaps follow different trajectories. It does not imply identical gradient subspaces and errors throughout training, as asserted in Section 9. A conditional comparison of subspace-iteration structure is reasonable; an identical-trajectory conclusion is not. ### W4: Scale and adaptation evidence does not support the combined claim in the conclusion The 17.1B timing experiment in Appendix L.2 runs for only 50 steps and forces Ada-Orth-Dion's rank to 93% of Dion's, using a ratio obtained at 320M. It measures implementation overhead at a chosen rank, not genuine online rank selection at 17.1B. The 1.5B result in L.4 uses one seed and shows no wall-clock speedup. Section 8.3 and the conclusion should avoid combining separate experiments into a claim of equal speed and lower loss on a large model. A complete training comparison at an affordable intermediate scale, using actual adaptive rank and a fixed-rank control under the same budget, would be most useful. ### W5: The $\beta$ mechanism discussion lacks derivation and contains internal tensions Section 6 and Appendix K infer optimal $\beta$, random-walk behavior, and even divergence from the sign of adjacent residual correlations. This is insufficient: negatively correlated inputs under a fixed projection can cancel rather than make the buffer diverge. The formula $\beta^*=1-2\hat\varepsilon$ gives approximately 0.3 for $\hat\varepsilon\approx0.35$, which does not precisely match the optimum near 0.1 in Table 6. Appendix K.2 also states that ColNorm+EF outperforms Orth-Dion+EF; the relevant setting and its relationship to Table 1 need clarification. These claims should be presented as hypotheses for controlled settings, with the algorithm receiving $\beta$, explicit-momentum controls, and multiple-seed evidence specified.Questions for authors
- How do you reconcile the , counterexample in W1? Does the implementation contain a small-matrix polar rotation omitted from the algorithm?
- How does the term in Appendix H become after division by ? Please provide the final bound with all constants retained.
- What is the single-valued residual map in F.5? Which inequality establishes its contraction coefficient and invariant neighborhood?
7 comments
Regarding Weakness 2 and Question 2
In Appendix H the error sum is ; dividing by leaves , the same order as the smoothness term, so the leading constant is not established. The theorem statement omits and , and restricts the horizon. Please give the final bound with all constants.
We appreciate your careful review. Step 3 of Appendix H dropped the error sum, which after dividing by is of the same order as the smoothness term, so the leading constant was not established; the rate stays and the separation from column normalization stays, and the correction is to the constant. In the revised paper we restate Theorem 5.1 with every term kept. For a constant step and every on the horizon,
with , and . Here is the mean in-subspace defect of the QR factor, which is zero for the polar factor (see our reply to W1). When , the third and fourth terms are at most , and optimizing over gives
The leading constant is therefore , which replaces the alone of the submission. The same proof applied to column normalization gives in place of and instead of the factor 2 in front of the residual, so the rank-dependent separation holds with the corrected constants.
We also added , the tail condition (which we now state on the buffer, see our reply to W3) and the horizon to the theorem statement. Since requires to stay bounded away from zero, the theorem covers the part of training in which the gradient keeps its rank- structure, and we state this after the theorem. We revised the abstract and contribution (ii) to match.
Regarding Weakness 3 and Question 3
The residual fixed-point argument is not closed: F.4 with Weyl gives no residual coefficient below one, and F.5 asserts an Lipschitz constant without a single-valued map, invariant neighborhood or derivative bounds. Lemma 4.3 holds for the same input and does not imply identical tracking throughout training (Section 9).
Thank you for raising these points. The residual is still controlled at order , and we now have proved it by induction under a tail condition stated on the buffer. Weyl's inequality returns in the bound of F.4 with coefficient one, so the Banach argument of F.5 does not close, and we withdrew it. We replaced Proposition F.5 with an induction on the pair over the invariant set , with and as in our reply to W2.
- For the residual, F.4 and Assumption give .
- For the tracking error, keeps the buffer gap above , so Wedin's theorem bounds the drift of the buffer's top- subspace, and the tracking recursion gives .
- The coefficient on is , and the coupling between the two components closes when . In Theorem G.1 of the revised paper we extend this induction to a contraction that holds over windows of steps rather than at every step, which allows transient and recovers the per-step condition as the case .
For the induction we use the tail condition $\sigma_{r+1}(M_t)\le\tau_M$ on the buffer, with which we control the residual and the tracking error together. A tail condition on the gradient alone would leave $\lVert R_t\rVert_{\mathrm{op}}$ with coefficient one through Weyl's inequality, as you kindly point out. We now state in the abstract and contribution (iii) that this buffer condition replaces the bounded-drift assumption.
We also restated Lemma 4.3 as a comparison of one step of the recursion for the same input $W_t$, which is the comparison the analysis relies on, and we aligned Sections 4 and 9 with this statement.
Regarding Weakness 1 and Question 1
Thank you for taking the time to build this example. It concerns the Appendix E bound of the oracle defect δt by the tracking error εt, which holds for the polar factor of Wt and is too strong for the QR factor. The rest of the analysis does not depend on that bound, however. Lemma 4.1 (νt=1) and Lemma 4.3 (the shared column space) hold as stated, and the removal of the r factor in Theorem 5.1 holds for QR up to a multiplicative constant that we measure below. In your example U spans the top-r subspace exactly, giving ε=0, but its basis is rotated, so QR leaves a defect of δ≈0.154 that the polar factor removes. Our implementation applies no rotation after QR, so in the revised paper we corrected Appendix E with the three results below and left the algorithm unchanged.
In the revised Theorem 5.1 we carry this defect through the convergence proof as an explicit additive term ΔˉQR, the mean in-subspace defect of the QR factor, which is zero for the polar factor. A corollary gives the multiplicative form. Whenever the QR direction attains at least a fraction q>0 of the restricted optimum at every step, and by result 3 above q=1−Δ2/2 suffices when maxtΔt≤Δ<2, the bound holds divided by q(1−εˉ2) with no additive term. The factor q does not depend on the rank and (1−εˉ2)−1=1+O(1/T) at the optimized step size, so the r separation from column normalization is unchanged.
We measured that fraction on saved states of the 320M runs, and we implemented the polar factor and trained with it.
On the saved states, the descent term of the QR direction, ⟨Mt,UtQt⊤⟩, is within 2% of the polar factor's ∥Ut⊤Mt∥∗ in aggregate and within 6.3% on the worst single matrix. In training, the polar factor did not lower the validation loss, since QR ended slightly lower on every seed. We therefore keep QR as the default and include the polar factor as an option in the released code. We trained these runs with a smaller batch than Table 1, so we compare the two rows only with each other. In the abstract, Section 4 and the conclusion we now state the claim as QR removing the ν inflation, which is what Lemma 4.1 and Theorem 5.1 establish.