- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces a truncated-SVD framework for low-rank LLM pretraining that maintains orthonormality of the factorized matrices via QR decomposition, and uses a "spectral energy" heuristic to assign a (per-matrix) rank to the low-rank factors. The authors report performance comparable to full-parameter training at a fraction of the compute cost, supported by large-scale experiments and ablations. The core components are a QR-based replacement for SVD, an amortization/caching mechanism, and the spectral-energy rank selection.
Critical strengths
- Clear, direct writing. The paper is not convoluted and avoids unnecessary jargon, which makes it easy to follow from the first paragraph onward.
- Important problem. Stable low-rank pretraining is a genuinely important and timely problem to tackle.
- Large-scale experiments and ablations. The inclusion of large-scale model experiments is valuable, and the ablations are a real strength.
Critical weaknesses
- Baseline experiments are questionable. Several results suggest the baselines are not tuned well:
- In Table 2, matching (or beating) full-parameter perplexity at roughly half the compute budget reads like a free lunch and points to an under-tuned baseline. Full-parameter vs. TSVD compute for Llama 1.3B works out to ~6 × 1339 × D vs. ~6 × 644 × D; getting better perplexity at half the budget is not credible without longer training.
- The low-rank comparison is unfair: the baseline uses rank 128 while the proposed method uses different (per-matrix) ranks. The low-rank baseline should be run with matched rank configurations.
- A brief look at the code suggests the full-rank models are untuned and not using μP. This needs to be confirmed — or clarified whether the supplementary code is not the actual baseline code.
- Motivation for orthonormality is not supported. The paper's logic is "low rank saves parameters but is unstable, so enforce orthonormality." But there are other ways to address instability (Spectron[1], CoLA, POET, etc.). The authors don't argue why orthonormality is the right or best solution.
- Rank selection requires a fully pretrained model. Using spectral energy to set rank is reasonable, but requiring a pretrained model to determine the compression rank defeats the purpose. If you already have a dense pretrained model, why not just compress it — and what is the point of low-rank training if the method depends on a dense model to begin with? The method should be shown to work from arbitrarily selected ranks, competitively with other low rank methods. It's also unclear what to do when the architecture is known but base pretrained weights are unavailable.
- The caching / amortization framing does not hold. Weights change only on update, not during the forward pass. The honest description is: after each optimizer step the weights change, so QR is applied to fix orthonormality. Framing it as a forward-pass issue that gradient accumulation "amortizes" is a reframing that doesn't actually describe an amortization.
- Forward-pass QR projection is unclear. It's not clear why U_k must be re-projected to orthonormal via QR in the forward pass when it is already orthonormal (at least at init), rather than simply doing it after the optimizer step.
- Poor scaling at 7B. The method appears to scale poorly at 7B, and the claim that further training would close the gap is unsupported — learning-rate decay effects would change the picture.
Minor / Presentation Notes
- Line 029: Wei et al. is cited as the workshop version, but it has a conference version[2] — please fix. Also missing: [1]
- Line 045: "reduced memory helps scale to larger architectures" is misleading — a larger model still needs more memory. Better framed as higher performance at the same hardware (or possibly wider models, not larger).
- Line 087: wrong quotation mark.
- Line 204: should be the squared sum of these values.
[1] Janson, Paul, Edouard Oyallon, and Eugene Belilovsky. "Stabilizing native low-rank LLM pretraining." ICML (2026).
[2] Wei, Xiuying, et al. "Building on efficient foundations: Effective training of llms with structured feedforward layers." NeurIPS (2024).
Questions for authors
-
QR variant Which QR decomposition is used (Householder vs. Cholesky)?
-
Forward-pass projection (line 163). Why is projected onto the set of orthonormal matrices via QR during the forward pass if is already orthonormal at initialization? Is this only necessary after the first optimizer step? If so, please state this clearly.
-
Complexity of forming (line/eq. 3). The reported cost appears to be for forming , whereas the goal is to form . The cost should roughly double, or alternatively, please state the complexity in big-O notation.
-
Rank-selection model. Is the pretrained model used for rank selection fully trained, or is it trained only for a few steps to determine the rank?
-
Memory measurement. Does the reported memory usage include the additional cache ?
-
Alternative to QR. What happens if the polar factor is computed using Newton–Schulz instead of QR for orthogonalization?
5 comments
Response by Authors (2)
Rank selection requires a fully pretrained model.
To answer, the rank selection strategy needs to be more clearly specified. In Section 4 (Figure 1), we analyzed the layer-wise compression ranks (derived via spectral energy decomposition), across a suite of full-parameter Llama-style models. Our empirical analysis revealed consistent structural relationships between the compression ranks of different functional components across parameter scales. For instance, the compression ranks for the Value and Output projection layers consistently measured ~18% higher than those of the Query and Key projection layers across all evaluated scales.
This invariance enables a key practical efficiency: the relative rank proportions identified on a small proxy model (e.g., a 60M parameter model) reliably project to much larger architectures (up to 7B parameters). Consequently, selecting non-uniform ranks for larger TSVD models requires pretraining only a single small full-parameter baseline to extract component-wise rank ratios, which are then scaled to guide rank allocation across the larger model family.
To verify the generality of this property, we extended our spectral analysis to open-weight model families, including Llama-2, Llama-3.1, and OLMo. As detailed in Appendix D, the component-wise rank relationships observed at small scales consistently hold for larger model variants when accounting for hyperparameter scaling. We will clarify the rank selection strategy in the revised paper.
The caching / amortization framing does not hold.
Indeed, orthonormality is strictly required at each optimization update step. The TSVD implementation specifically addresses scenarios involving gradient accumulation, where a naive execution would recompute QR decomposition across every micro-batch unnecessarily. The caching scheme ensures that QR decomposition is executed strictly once per optimization step (rather than per micro-batch), maintaining exact mathematical equivalence while avoiding redundant computations.
We appreciate your clarification regarding the terminology and will replace the term “amortization” with “caching” in the revised manuscript to accurately reflect this mechanism.
Forward-pass QR projection is unclear.
In TSVD, the matrices U_k and V_k are stored as unconstrained model parameters (i.e., they are not explicitly constrained to be orthonormal). Orthonormality is enforced during training as follows:
Forward Pass: We perform a QR decomposition on U_k and V_k, caching the resulting orthonormal factors (Q_u and Q_v). The layer's forward computation operates directly on these cached orthonormal matrices rather than the raw unconstrained weights.
Backward Pass: Gradients flow back to update the underlying unconstrained parameters U_k and V_k .
Subsequent Forward Pass: The updated parameters U_k and V_k undergo QR decomposition again, refreshing the cached orthonormal matrices used for execution.
This dynamic projection step ensures that while optimization occurs over unconstrained parameter space, the layer computation strictly maintains orthonormal properties at every execution step. We will clarify the orthonormality enforcement in the revised paper.
Poor scaling at 7B.
We have since updated our 7B model experiments to correct a rank allocation mismatch in the preliminary setup. With the corrected rank configuration, TSVD achieves a slightly improved perplexity of 6.53. Furthermore, as discussed in Section 5.2.1, the performance gap between the full-parameter 7B baseline and the TSVD 7B model steadily narrows as pretraining progresses, suggesting that TSVD will match full-parameter performance over continued training.
The main reason for smaller improvement at 7B is the computational cost of training. Compute-optimal scaling guidelines (e.g., Hoffmann et al., 2022) suggest training on at least 20 tokens per parameter, and modern production models typically train on even significantly higher ratios. Given available compute resources, the 7B model was trained on 19.6B tokens, i.e. roughly a 3:1 token-to-parameter ratio. Despite this undertrained regime, the trajectory demonstrates that TSVD scales effectively and maintains its relative advantages, offering a practical signal for resource-constrained pretraining. Thus, the 7B experiment should be seen as a proof-of-concept to evaluate TSVD's behavior at a larger scale under a fixed compute budget. We will incorporate this explanation into the revised paper.
Response by Authors (3)
Minor / Presentation Notes
Line 029: Wei et al. is cited as the workshop version, but it has a conference version[2] — please fix. Also missing: [1]
Thanks, we will add the conference version of the paper by Wei et al. and also the paper by Janson et al.
Line 045: "reduced memory helps scale to larger architectures" is misleading — a larger model still needs more memory. Better framed as higher performance at the same hardware (or possibly wider models, not larger).
Good point, "scaling to larger architectures" is ambiguous. To be precise, our claim is that parameter reduction enables training models with higher parameter counts under a fixed hardware memory budget. For example, as shown in Table 3, our TSVD framework reduces the parameter count of a 7B Llama model down to 4.574 billion (a 32% reduction). Consequently, a GPU node constrained by memory limits that cannot fit the full-parameter 7B baseline can successfully train this TSVD configuration. We will rephrase this statement in the revised manuscript to explicitly emphasize "improved parameter capacity under fixed hardware memory constraints" rather than scaling to larger architectures.
Line 087: wrong quotation mark.
Will fix it. Thank you.
Line 204: should be the squared sum of these values.
Will fix it. Thank you.
Questions For Authors:
QR variant Which QR decomposition is used (Householder vs. Cholesky)?
We use the qr() method in PyTorch’s linear algebra module, which uses Householder reflections.
Forward-pass projection (line 163). Why is (U_k) projected onto the set of orthonormal matrices via QR during the forward pass if (U_k) is already orthonormal at initialization? Is this only necessary after the first optimizer step? If so, please state this clearly.
Please refer to the response to “Forward-pass QR projection is unclear” above.
Complexity of forming (Q) (line/eq. 3). The reported cost appears to be for forming (R), whereas the goal is to form (Q). The cost should roughly double, or alternatively, please state the complexity in big-O notation.
Per “Algorithm 5.2.1” in the reference cited below, the big-O notation for an m by n matrix is .
Golub, Gene H., and Charles F. Van Loan. Matrix computations. Vol. 3. JHU press, 2013.
We will add this to the paper's bibliography.
Rank-selection model. Is the pretrained model used for rank selection fully trained, or is it trained only for a few steps to determine the rank?
Please refer to the response to “Rank selection requires a fully pretrained model.” above.
Memory measurement. Does the reported memory usage include the additional cache ?
Thanks for pointing out this necessary clarification. The initial memory usage figures reported in Table 2 included only parameters, gradients, and optimizer states, excluding the transient matrix cache. Because the cache temporarily stores the orthonormalized matrix factors alongside the unconstrained base parameters during forward execution, accounting for this cached state effectively adds an additional parameter-sized footprint during training. The updated memory measurements to fully account for this cache overhead across all model sizes are
TSVD Llama-60M: 0.40 GiB (increased from 0.32 GiB)
TSVD Llama-130M: 0.85 GiB (increased from 0.68 GiB)
TSVD Llama-350M: 1.65 GiB (increased from 1.32 GiB)
TSVD Llama-1.3B: 5.99 GiB (increased from 4.80 GiB)
Note that even with the transient cache included, TSVD maintains an overall lower memory footprint compared to the full-parameter baseline. We will update the memory usage results in the revised paper.
Alternative to QR. What happens if the polar factor is computed using Newton–Schulz instead of QR for orthogonalization?
Thanks again for an insightful suggestion. Comparing QR decomposition with Newton–Schulz (NS) iteration highlights an important trade-off between exactness and hardware throughput. While QR decomposition guarantees exact orthonormality up to machine precision in a single step, NS is an iterative polynomial approximation. However, because NS iteration consists entirely of dense matrix multiplications, it aligns exceptionally well with GPU Tensor Cores and can reduce hardware execution bottlenecks.
Thus, exploring NS iteration as a GPU-friendly proxy for exact orthogonalization is a promising direction. In the revised manuscript, we have added it as a key area for future work.
Response by Authors (1)
The hyperparameter selection methodology needs to be explained more thoroughly in the revision.
For TSVD, we utilized Maximal Update Parameterization (MuP) to perform hyperparameter tuning on a 60M parameter proxy model, transferring the resulting optimal hyperparameters directly to all larger TSVD scale experiments. For the three baseline models in Table 2 (full-parameter, Low-Rank, and CoLA), we used the exact hyperparameter configurations recommended in their respective original publications, under the standard protocol of utilizing author-tuned baselines.
Additionally, we highlight that TSVD preserves full-parameter rank for embedding and output head layers, applying low-rank decomposition exclusively to the transformer decoder blocks. Consequently, parameter savings scale more favorably in larger architectures, such as 350M and 1.3B models, where decoder blocks represent a significantly higher proportion of the total parameter budget compared to the embedding and head layers.
For this response, we conducted a new set of Low-Rank experiments using the exact same compression rank distribution as TSVD. The Low Rank perplexity results on the validation set are summarized below. For reference, the TSVD perplexity results on the validation set (from Table 2 in the paper ) are also provided.
These results demonstrate that TSVD consistently outperforms standard Low-Rank pretraining under identical rank distributions. This ablation directly isolates and confirms the positive impact of maintaining orthonormality, and will strengthen the paper. Thanks for the suggestion.
Please refer to the response to the first question above.
While alternative approaches provide valuable mechanisms for mitigating low-rank training instabilities, Truncated SVD (TSVD) offers a uniquely principled geometric and computational advantage through its explicit orthonormality. TSVD decomposes any linear transformation into its fundamental geometric components: W=UΣVT. Here, the orthogonal matrices U and VT perform pure rotations or reflections (preserving vector lengths and angular relationships), while all scaling and magnitude adjustments are completely isolated within the diagonal matrix Sigma. This clean separation of responsibility yields two distinct benefits compared to existing alternatives:
Wrt. POET: POET freezes the base weight matrix W and trains external orthonormal matrices on its left and right. While effective for certain adaptation tasks, this approach prevents the model from modifying the underlying singular values of W, if they are suboptimal. In contrast, TSVD keeps the singular values in Sigma fully trainable, preserving the network's expressive capacity.
Wrt. Spectron: Spectron successfully stabilizes training by using iterative subroutines (e.g., power iteration) to dynamically estimate the largest singular values at every step to bound gradient updates. In TSVD, the singular values are natively part of the parameter representation. Because they are explicitly maintained and updated during training, magnitude control and regularization are directly accessible with zero approximation or computational overhead.
We will clarify this motivation for TSVD’s clean separation between direction (orthonormal rotations) and magnitude (scaling) in the revised manuscript.