acceptodds
Under review as a conference paper at ICLR 2027

Efficient Pretraining of LLMs through Truncated SVD Representations

Abstract

LLM pretraining is extremely costly; therefore, parameter-efficient LLM architectures have recently emerged as a compelling research direction. One such promising approach is to represent the parameters as orthonormal low-rank weight matrices. However, maintaining orthonormality during training is computationally expensive, making it impractical. This paper presents the TSVD (Truncated Singular Value Decomposition) framework which efficiently maintains orthonormality through QR decomposition and caching. Furthermore, a spectral energy heuristic is introduced to select the rank of the resulting low-rank weight matrices. Empirical evaluations across model sizes show that TSVD matches or outperforms full-parameter baselines at a fraction of the compute cost. TSVD thus provides a scalable, computationally efficient foundation for LLM pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, pretraining, low-rank, TSVD

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cYS6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a truncated-SVD framework for low-rank LLM pretraining that maintains orthonormality of the factorized matrices via QR decomposition, and uses a "spectral energy" heuristic to assign a (per-matrix) rank to the low-rank factors. The authors report performance comparable to full-parameter training at a fraction of the compute cost, supported by large-scale experiments and ablations. The core components are a QR-based replacement for SVD, an amortization/caching mechanism, and the spectral-energy rank selection.

Critical strengths
  • Clear, direct writing. The paper is not convoluted and avoids unnecessary jargon, which makes it easy to follow from the first paragraph onward.
  • Important problem. Stable low-rank pretraining is a genuinely important and timely problem to tackle.
  • Large-scale experiments and ablations. The inclusion of large-scale model experiments is valuable, and the ablations are a real strength.
Critical weaknesses
  • Baseline experiments are questionable. Several results suggest the baselines are not tuned well:
    • In Table 2, matching (or beating) full-parameter perplexity at roughly half the compute budget reads like a free lunch and points to an under-tuned baseline. Full-parameter vs. TSVD compute for Llama 1.3B works out to ~6 × 1339 × D vs. ~6 × 644 × D; getting better perplexity at half the budget is not credible without longer training.
    • The low-rank comparison is unfair: the baseline uses rank 128 while the proposed method uses different (per-matrix) ranks. The low-rank baseline should be run with matched rank configurations.
    • A brief look at the code suggests the full-rank models are untuned and not using μP. This needs to be confirmed — or clarified whether the supplementary code is not the actual baseline code.
  • Motivation for orthonormality is not supported. The paper's logic is "low rank saves parameters but is unstable, so enforce orthonormality." But there are other ways to address instability (Spectron[1], CoLA, POET, etc.). The authors don't argue why orthonormality is the right or best solution.
  • Rank selection requires a fully pretrained model. Using spectral energy to set rank is reasonable, but requiring a pretrained model to determine the compression rank defeats the purpose. If you already have a dense pretrained model, why not just compress it — and what is the point of low-rank training if the method depends on a dense model to begin with? The method should be shown to work from arbitrarily selected ranks, competitively with other low rank methods. It's also unclear what to do when the architecture is known but base pretrained weights are unavailable.
  • The caching / amortization framing does not hold. Weights change only on update, not during the forward pass. The honest description is: after each optimizer step the weights change, so QR is applied to fix orthonormality. Framing it as a forward-pass issue that gradient accumulation "amortizes" is a reframing that doesn't actually describe an amortization.
  • Forward-pass QR projection is unclear. It's not clear why U_k must be re-projected to orthonormal via QR in the forward pass when it is already orthonormal (at least at init), rather than simply doing it after the optimizer step.
  • Poor scaling at 7B. The method appears to scale poorly at 7B, and the claim that further training would close the gap is unsupported — learning-rate decay effects would change the picture.

Minor / Presentation Notes

  • Line 029: Wei et al. is cited as the workshop version, but it has a conference version[2] — please fix. Also missing: [1]
  • Line 045: "reduced memory helps scale to larger architectures" is misleading — a larger model still needs more memory. Better framed as higher performance at the same hardware (or possibly wider models, not larger).
  • Line 087: wrong quotation mark.
  • Line 204: should be the squared sum of these values.

[1] Janson, Paul, Edouard Oyallon, and Eugene Belilovsky. "Stabilizing native low-rank LLM pretraining." ICML (2026).

[2] Wei, Xiuying, et al. "Building on efficient foundations: Effective training of llms with structured feedforward layers." NeurIPS (2024).

Questions for authors
  1. QR variant Which QR decomposition is used (Householder vs. Cholesky)?

  2. Forward-pass projection (line 163). Why is UkU_k projected onto the set of orthonormal matrices via QR during the forward pass if UkU_k is already orthonormal at initialization? Is this only necessary after the first optimizer step? If so, please state this clearly.

  3. Complexity of forming QQ (line/eq. 3). The reported cost appears to be for forming RR, whereas the goal is to form QQ. The cost should roughly double, or alternatively, please state the complexity in big-O notation.

  4. Rank-selection model. Is the pretrained model used for rank selection fully trained, or is it trained only for a few steps to determine the rank?

  5. Memory measurement. Does the reported memory usage include the additional cache ?

  6. Alternative to QR. What happens if the polar factor is computed using Newton–Schulz instead of QR for orthogonalization?

5 comments
AuthorsOpenReview ↗

Response by Authors (1)

Baseline experiments are questionable.

In Table 2, matching (or beating) full-parameter perplexity at roughly half the compute budget reads like a free lunch and points to an under-tuned baseline.

The hyperparameter selection methodology needs to be explained more thoroughly in the revision.

For TSVD, we utilized Maximal Update Parameterization (MuP) to perform hyperparameter tuning on a 60M parameter proxy model, transferring the resulting optimal hyperparameters directly to all larger TSVD scale experiments. For the three baseline models in Table 2 (full-parameter, Low-Rank, and CoLA), we used the exact hyperparameter configurations recommended in their respective original publications, under the standard protocol of utilizing author-tuned baselines.

Additionally, we highlight that TSVD preserves full-parameter rank for embedding and output head layers, applying low-rank decomposition exclusively to the transformer decoder blocks. Consequently, parameter savings scale more favorably in larger architectures, such as 350M and 1.3B models, where decoder blocks represent a significantly higher proportion of the total parameter budget compared to the embedding and head layers.

The low-rank comparison is unfair: the baseline uses rank 128 while the proposed method uses different (per-matrix) ranks. The low-rank baseline should be run with matched rank configurations

For this response, we conducted a new set of Low-Rank experiments using the exact same compression rank distribution as TSVD. The Low Rank perplexity results on the validation set are summarized below. For reference, the TSVD perplexity results on the validation set (from Table 2 in the paper ) are also provided.

ModelCompression rankLow Rank PerplexityTSVD Perplexity
Llama_60m96/112/16014.9714.40
Llama_130m168/192/28011.6911.36
Llama_350m168/192/2809.239.14
Llama_1.3b392/456/6407.657.56

These results demonstrate that TSVD consistently outperforms standard Low-Rank pretraining under identical rank distributions. This ablation directly isolates and confirms the positive impact of maintaining orthonormality, and will strengthen the paper. Thanks for the suggestion.

A brief look at the code suggests the full-rank models are untuned and not using μP. This needs to be confirmed — or clarified whether the supplementary code is not the actual baseline code.

Please refer to the response to the first question above.

Motivation for orthonormality is not supported.

While alternative approaches provide valuable mechanisms for mitigating low-rank training instabilities, Truncated SVD (TSVD) offers a uniquely principled geometric and computational advantage through its explicit orthonormality. TSVD decomposes any linear transformation into its fundamental geometric components: W=UΣVTW = U \Sigma V^T. Here, the orthogonal matrices UU and VTV^T perform pure rotations or reflections (preserving vector lengths and angular relationships), while all scaling and magnitude adjustments are completely isolated within the diagonal matrix Sigma. This clean separation of responsibility yields two distinct benefits compared to existing alternatives:

Wrt. POET: POET freezes the base weight matrix W and trains external orthonormal matrices on its left and right. While effective for certain adaptation tasks, this approach prevents the model from modifying the underlying singular values of W, if they are suboptimal. In contrast, TSVD keeps the singular values in Sigma fully trainable, preserving the network's expressive capacity.

Wrt. Spectron: Spectron successfully stabilizes training by using iterative subroutines (e.g., power iteration) to dynamically estimate the largest singular values at every step to bound gradient updates. In TSVD, the singular values are natively part of the parameter representation. Because they are explicitly maintained and updated during training, magnitude control and regularization are directly accessible with zero approximation or computational overhead.

We will clarify this motivation for TSVD’s clean separation between direction (orthonormal rotations) and magnitude (scaling) in the revised manuscript.

AuthorsOpenReview ↗

Response by Authors (2)

Rank selection requires a fully pretrained model.

To answer, the rank selection strategy needs to be more clearly specified. In Section 4 (Figure 1), we analyzed the layer-wise compression ranks (derived via spectral energy decomposition), across a suite of full-parameter Llama-style models. Our empirical analysis revealed consistent structural relationships between the compression ranks of different functional components across parameter scales. For instance, the compression ranks for the Value and Output projection layers consistently measured ~18% higher than those of the Query and Key projection layers across all evaluated scales.

This invariance enables a key practical efficiency: the relative rank proportions identified on a small proxy model (e.g., a 60M parameter model) reliably project to much larger architectures (up to 7B parameters). Consequently, selecting non-uniform ranks for larger TSVD models requires pretraining only a single small full-parameter baseline to extract component-wise rank ratios, which are then scaled to guide rank allocation across the larger model family.

To verify the generality of this property, we extended our spectral analysis to open-weight model families, including Llama-2, Llama-3.1, and OLMo. As detailed in Appendix D, the component-wise rank relationships observed at small scales consistently hold for larger model variants when accounting for hyperparameter scaling. We will clarify the rank selection strategy in the revised paper.

The caching / amortization framing does not hold.

Indeed, orthonormality is strictly required at each optimization update step. The TSVD implementation specifically addresses scenarios involving gradient accumulation, where a naive execution would recompute QR decomposition across every micro-batch unnecessarily. The caching scheme ensures that QR decomposition is executed strictly once per optimization step (rather than per micro-batch), maintaining exact mathematical equivalence while avoiding redundant computations.

We appreciate your clarification regarding the terminology and will replace the term “amortization” with “caching” in the revised manuscript to accurately reflect this mechanism.

Forward-pass QR projection is unclear.

In TSVD, the matrices U_k and V_k are stored as unconstrained model parameters (i.e., they are not explicitly constrained to be orthonormal). Orthonormality is enforced during training as follows:

Forward Pass: We perform a QR decomposition on U_k and V_k, caching the resulting orthonormal factors (Q_u and Q_v). The layer's forward computation operates directly on these cached orthonormal matrices rather than the raw unconstrained weights.

Backward Pass: Gradients flow back to update the underlying unconstrained parameters U_k and V_k .

Subsequent Forward Pass: The updated parameters U_k and V_k undergo QR decomposition again, refreshing the cached orthonormal matrices used for execution.

This dynamic projection step ensures that while optimization occurs over unconstrained parameter space, the layer computation strictly maintains orthonormal properties at every execution step. We will clarify the orthonormality enforcement in the revised paper.

Poor scaling at 7B.

We have since updated our 7B model experiments to correct a rank allocation mismatch in the preliminary setup. With the corrected rank configuration, TSVD achieves a slightly improved perplexity of 6.53. Furthermore, as discussed in Section 5.2.1, the performance gap between the full-parameter 7B baseline and the TSVD 7B model steadily narrows as pretraining progresses, suggesting that TSVD will match full-parameter performance over continued training.

The main reason for smaller improvement at 7B is the computational cost of training. Compute-optimal scaling guidelines (e.g., Hoffmann et al., 2022) suggest training on at least 20 tokens per parameter, and modern production models typically train on even significantly higher ratios. Given available compute resources, the 7B model was trained on 19.6B tokens, i.e. roughly a 3:1 token-to-parameter ratio. Despite this undertrained regime, the trajectory demonstrates that TSVD scales effectively and maintains its relative advantages, offering a practical signal for resource-constrained pretraining. Thus, the 7B experiment should be seen as a proof-of-concept to evaluate TSVD's behavior at a larger scale under a fixed compute budget. We will incorporate this explanation into the revised paper.

AuthorsOpenReview ↗

Response by Authors (3)

Minor / Presentation Notes

Line 029: Wei et al. is cited as the workshop version, but it has a conference version[2] — please fix. Also missing: [1]

Thanks, we will add the conference version of the paper by Wei et al. and also the paper by Janson et al.

Line 045: "reduced memory helps scale to larger architectures" is misleading — a larger model still needs more memory. Better framed as higher performance at the same hardware (or possibly wider models, not larger).

Good point, "scaling to larger architectures" is ambiguous. To be precise, our claim is that parameter reduction enables training models with higher parameter counts under a fixed hardware memory budget. For example, as shown in Table 3, our TSVD framework reduces the parameter count of a 7B Llama model down to 4.574 billion (a 32% reduction). Consequently, a GPU node constrained by memory limits that cannot fit the full-parameter 7B baseline can successfully train this TSVD configuration. We will rephrase this statement in the revised manuscript to explicitly emphasize "improved parameter capacity under fixed hardware memory constraints" rather than scaling to larger architectures.

Line 087: wrong quotation mark.

Will fix it. Thank you.

Line 204: should be the squared sum of these values.

Will fix it. Thank you.

Questions For Authors:

QR variant Which QR decomposition is used (Householder vs. Cholesky)?

We use the qr() method in PyTorch’s linear algebra module, which uses Householder reflections.

Forward-pass projection (line 163). Why is (U_k) projected onto the set of orthonormal matrices via QR during the forward pass if (U_k) is already orthonormal at initialization? Is this only necessary after the first optimizer step? If so, please state this clearly.

Please refer to the response to “Forward-pass QR projection is unclear” above.

Complexity of forming (Q) (line/eq. 3). The reported cost appears to be for forming (R), whereas the goal is to form (Q). The cost should roughly double, or alternatively, please state the complexity in big-O notation.

Per “Algorithm 5.2.1” in the reference cited below, the big-O notation for an m by n matrix is 2mn2−23n32mn^2 - \frac{2}{3}n^3.

Golub, Gene H., and Charles F. Van Loan. Matrix computations. Vol. 3. JHU press, 2013.

We will add this to the paper's bibliography.

Rank-selection model. Is the pretrained model used for rank selection fully trained, or is it trained only for a few steps to determine the rank?

Please refer to the response to “Rank selection requires a fully pretrained model.” above.

Memory measurement. Does the reported memory usage include the additional cache ?

Thanks for pointing out this necessary clarification. The initial memory usage figures reported in Table 2 included only parameters, gradients, and optimizer states, excluding the transient matrix cache. Because the cache temporarily stores the orthonormalized matrix factors alongside the unconstrained base parameters during forward execution, accounting for this cached state effectively adds an additional parameter-sized footprint during training. The updated memory measurements to fully account for this cache overhead across all model sizes are

TSVD Llama-60M: 0.40 GiB (increased from 0.32 GiB)
TSVD Llama-130M: 0.85 GiB (increased from 0.68 GiB)
TSVD Llama-350M: 1.65 GiB (increased from 1.32 GiB)
TSVD Llama-1.3B: 5.99 GiB (increased from 4.80 GiB)

Note that even with the transient cache included, TSVD maintains an overall lower memory footprint compared to the full-parameter baseline. We will update the memory usage results in the revised paper.

Alternative to QR. What happens if the polar factor is computed using Newton–Schulz instead of QR for orthogonalization?

Thanks again for an insightful suggestion. Comparing QR decomposition with Newton–Schulz (NS) iteration highlights an important trade-off between exactness and hardware throughput. While QR decomposition guarantees exact orthonormality up to machine precision in a single step, NS is an iterative polynomial approximation. However, because NS iteration consists entirely of dense matrix multiplications, it aligns exceptionally well with GPU Tensor Cores and can reduce hardware execution bottlenecks.

Thus, exploring NS iteration as a GPU-friendly proxy for exact orthogonalization is a promising direction. In the revised manuscript, we have added it as a key area for future work.

Reviewer_8NeuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose Truncated Singular Value Decomposition (TSVD), which is for reducing the computational and memory costs of LLMs pretraining. In prior works, it has been shown that imposing the orthonormality constraints on low rank factors can improve the optimization stability, but it may also introduce more computation.

Critical strengths

This paper studies an important and practical problem. It improves the computational and memory costs of LLM pretraining.

I think the techniques and algorithmic design of TSVD is relatively easy to follow. Replacing SVD with QR is helpful for ensuring orthonormality. It can be simply incorporated into existing LLM training process.

It has shown impressive experimental results on multiple model scales. For example, experiments have been conducted on models ranging from 60M to 7B parameters. More importantly, the perplexity result can be better than the full parametric model in many cases.

Critical weaknesses

However, I think my biggest concern is that the experiment does not show how much of the performance improvement comes from maintaining orthonormality. I think it is the main claim of the paper that imposing the orthonormality constraints may improve the optimization stability. I think it would be better if the authors may consider strengthening the ablation study of QR decomposition: at this moment, it only compares the computational costs of QR and SVD.

In addition, as in Appendix A, with W=1rUΣV⊤W=\frac{1}{\sqrt r}U\Sigma V^\top, the authors have shown ∣Wx∣2≤σmax⁡r∣x∣2\\|Wx\\|_2\leq \frac{\sigma_{\max}}{\sqrt r}\\|x\\|_2. This is a direct result follows from the definition of SCD, but the authors did not show a novel optimization guarantee. In addition, the singular values in Σ\Sigma may still be trainable, so maintaining the orthonormal UU and VV might not present exploding or vanishing signals.

In Figure 2, the authors show the training time for 60M and 130M models. It would be better if the authors may consider including the time comparisons for larger models such as 1.3B and 7B.

Questions for authors

Please see weaknesses.

1 comment
AuthorsOpenReview ↗

Response by Authors

Critical Weaknesses:

However, I think my biggest concern is that the experiment does not show how much of the performance improvement comes from maintaining orthonormality. I think it is the main claim of the paper that imposing the orthonormality constraints may improve the optimization stability. I think it would be better if the authors may consider strengthening the ablation study of QR decomposition: at this moment, it only compares the computational costs of QR and SVD.

Thank you for your feedback.

In Table 2 of the original manuscript, the baseline Low-Rank performance was reported using a uniform rank allocation, whereas the TSVD experiments utilized a compression (non-uniform) rank allocation across layers. To isolate and directly quantify the effect of orthonormality in the revision, we conducted a new set of Low-Rank experiments using the exact same compression rank distribution as TSVD. Under this controlled setup, the primary structural difference between the two approaches is TSVD’s explicit orthonormality constraint. The Low Rank perplexity results on the validation set are summarized below. For reference, the TSVD perplexity results on the validation set (from Table 2 in the paper ) are also provided.

ModelCompression rankLow Rank PerplexityTSVD Perplexity
Llama_60m96 / 112 / 16014.9714.40
Llama_130m168 / 192 / 28011.6911.36
Llama_350m168 / 192 / 2809.239.14
Llama_1.3b392 / 456 / 6407.657.56

These results demonstrate that TSVD consistently outperforms standard Low-Rank pretraining under identical rank distributions. This ablation directly isolates and confirms the positive impact of maintaining orthonormality. We will incorporate these results into the revised paper.

In addition, as in Appendix A, with W=1rUΣVTW = \frac{1}{\sqrt{r}} U \Sigma V^T, the authors have shown ∥Wx∥2≤σmaxr∥x∥2\lVert Wx \rVert_2 \leq \frac{\sigma_{max} }{\sqrt{r}} \lVert x \rVert_2. This is a direct result follows from the definition of SCD, but the authors did not show a novel optimization guarantee. In addition, the singular values in Σ\Sigma may still be trainable, so maintaining the orthonormal UU and VV might not present exploding or vanishing signals.

Good point. While output magnitudes are bounded by the top singular value, the current implementation does not explicitly constrain the singular value spectrum during training. Incorporating spectral norm regularization (such as bounding the maximum singular value), to enhance generalization is an immediate priority for future work, and will be outlined in the revised paper.

In Figure 2, the authors show the training time for 60M and 130M models. It would be better if the authors may consider including the time comparisons for larger models such as 1.3B and 7B.

For this revision, we conducted profiling experiments on the Llama-350M and Llama-1.3B models to assess how training throughput scales with gradient accumulation steps. The measured execution times (seconds per step) were

Model12481632
Llama_350m_TSVD1579971615349
Llama_350m_full_param6557525250.850.3
Model12481632
Llama_1.3b_TSVD416275205171154145
Llama_1.3b_full_param310281266261258256

The results highlight a clear efficiency crossover point: for the 350M model, TSVD outperforms full-parameter training in step latency between 16 and 32 gradient accumulation steps. For the larger 1.3B model, TSVD becomes faster starting at a gradient accumulation setting of 4. These findings will strengthen the paper and confirm that the relative computational efficiency of TSVD improves with both model capacity and effective batch size. Thank you for your suggestion.

Reviewer_ih5jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TSVD, a framework for parameter-efficient LLM pretraining that represents weight matrices using trainable low-rank factors with orthonormal constraints. The method uses QR-based reparameterization to maintain orthonormality of the left and right factors, and caches the resulting factors across gradient accumulation steps to amortize the computational overhead. It also introduces a spectral-energy-based heuristic that uses pretrained models to guide rank allocation across projection types and transfer relative rank configurations across model scales. Experiments on C4 cover Llama-style architectures ranging from 60M to 7B. TSVD achieves lower perplexity than dense baselines at scales up to 1.3B with fewer parameters, while retaining a modest perplexity gap at the 7B scale. Additional experiments evaluate downstream performance and examine the effects of caching and nonuniform rank allocation.

Critical strengths

1.The paper provides an empirical demonstration of training LLMs from scratch with orthonormal low-rank factors, contributing useful evidence on the applicability of this parameterization to pretraining. The combination of QR-based reparameterization and caching across gradient accumulation steps is an interesting practical design. It maintains orthonormal factors at every optimization step while amortizing the cost of orthonormalization.

2.The experiments extend to a 7B-scale architecture and demonstrate meaningful reductions in parameter count and reported memory footprint. At this scale, TSVD reduces the parameter count from approximately 6.74B to 4.57B while retaining reasonably competitive perplexity, suggesting a useful parameter–quality trade-off.

Critical weaknesses

The empirical comparison with LORO, a closely related method addressing the same task of low-rank LLM pretraining from scratch, is missing. Although the paper discusses LORO, the reported experiments do not establish whether TSVD offers better model quality or efficiency than this relevant baseline. Evaluating LORO within the same implementation and data-processing pipeline, with matched parameter or compute budgets and comparable hyperparameter tuning, would help establish the practical benefit of the proposed approach.

Questions for authors

1.TSVD achieves lower validation perplexity than the dense baselines at scales up to 1.3B, but this advantage reverses at the 7B scale, where its perplexity is 6.57 compared with 6.38 for the dense model. Could the authors provide stronger evidence or analysis?

2.How much additional benefit comes from maintaining strict orthonormality at every optimization step? Would enforcing orthogonality every K steps lead to a meaningful degradation in model quality, or could it achieve comparable performance with lower overhead?

If the authors convincingly address these concerns and provide the necessary analyses, I would consider raising my score to 3.

3 comments
AuthorsOpenReview ↗

Response by Authors

Critical Weaknesses:

The empirical comparison with LORO, a closely related method addressing the same task of low-rank LLM pretraining from scratch, is missing. Although the paper discusses LORO, the reported experiments do not establish whether TSVD offers better model quality or efficiency than this relevant baseline. Evaluating LORO within the same implementation and data-processing pipeline, with matched parameter or compute budgets and comparable hyperparameter tuning, would help establish the practical benefit of the proposed approach.

Thank you for your feedback. We appreciate the suggestion to compare our work with LORO.

We have implemented LORO within our codebase following its standard specifications. We conducted pretraining experiments across four model scales (60M, 130M, 350M, and 1.3B parameters) under two allocation settings:

  1. Uniform Rank: Equal rank distribution across all layers, as specified in the LORO paper
  2. Compressed (Non-Uniform) Rank: Differentiated rank allocation for Key, Query/ Value, Output Projection/MLP layers.

The LORO perplexity results on the validation, for uniform and compression ranks, set are summarized below.

ModelUniform rankLORO Perplexity
Llama_60m12814.89
Llama_130m25611.51
Llama_350m2569.28
Llama_1.3b5127.67
ModelCompression rankLORO Perplexity
Llama_60m96 / 112 / 16014.9
Llama_130m168 / 192 / 28011.54
Llama_350m168 / 192 / 2809.28
Llama_1.3b392 / 456 / 6407.59

Comparing these results with Table 2 in the paper shows that TSVD yields lower perplexity across both uniform and compressed rank allocations at all tested scales. We will incorporate these results into the revised paper.

Questions For Authors:

.1.TSVD achieves lower validation perplexity than the dense baselines at scales up to 1.3B, but this advantage reverses at the 7B scale, where its perplexity is 6.57 compared with 6.38 for the dense model. Could the authors provide stronger evidence or analysis?

We have since updated our 7B model experiments to correct a rank allocation mismatch in the preliminary setup. With the corrected rank configuration, TSVD achieves a slightly improved perplexity of 6.53. Furthermore, as discussed in Section 5.2.1, the performance gap between the full-parameter 7B baseline and the TSVD 7B model steadily narrows as pretraining progresses, suggesting that TSVD will match full-parameter performance over continued training.

The main reason for smaller improvement at 7B is the computational cost of training. Compute-optimal scaling guidelines (e.g., Hoffmann et al., 2022) suggest training on at least 20 tokens per parameter, and modern production models typically train on even significantly higher ratios. Given available compute resources, the 7B model was trained on 19.6B tokens, i.e. roughly a 3:1 token-to-parameter ratio. Despite this undertrained regime, the trajectory demonstrates that TSVD scales effectively and maintains its relative advantages, offering a practical signal for resource-constrained pretraining. Thus, the 7B experiment should be seen as a proof-of-concept to evaluate TSVD's behavior at a larger scale under a fixed compute budget. We will incorporate this explanation into the revised paper.

Questions For Authors:

2.How much additional benefit comes from maintaining strict orthonormality at every optimization step? Would enforcing orthogonality every K steps lead to a meaningful degradation in model quality, or could it achieve comparable performance with lower overhead?

To clarify the architectural difference in enforcing orthonormality: in TSVD, each layer parameterizes U, Sigma, and V as unconstrained matrices (i.e., they are not explicitly maintained as orthonormal). To ensure orthonormality during inference and training, we apply a QR decomposition to U and V in every forward pass to generate the orthonormal matrices used for the layer's output. During the backward pass, gradients update the underlying unconstrained U and V matrices directly. Because these raw parameters are unconstrained before and after the update step, applying QR decomposition dynamically in the next forward pass re-establishes orthonormality. Consequently, unlike LORO, periodic orthonormality updates (e.g., once every K steps) are structurally inapplicable to TSVD method, as continuous projection via QR is required on every forward pass. In other words, the approach was designed specifically to take advantage of orthonormality at every step without extra overhead. This is an important point and we’ll make it clear in the revision—thanks for pointing it out.

Thank you for your response; most of my concerns have been addressed. However, I still have concerns regarding Q1. I agree that the models were not trained with the token count required by scaling laws. But even after updating 6.57 to 6.53, it still does not appear to show a reversal trend in the later stages. In contrast, there is a clear improvement on smaller models (up to 1.3B). As the model scales up, I think it is reasonable for the performance gap to narrow, but a reversal is indeed strange. I suspect that the baselines for the smaller models were not properly tuned. I hope to receive a stronger explanation.

Thank you for taking the time to review our rebuttal and share your final perspective. We appreciate your thoughtful feedback throughout this process, which will be very helpful as we continue refining the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.