acceptodds
Under review as a conference paper at ICLR 2027

DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training

Abstract

Large language models have driven recent progress in language and multimodal AI, yet pre-training them at scale is prohibitively expensive. Low-rank pre-training, which factorizes each weight matrix into a rank-r product to reduce both parameters and FLOPs, is a promising response but typically lags behind full-rank training in quality. We propose Duplicated Latent Residual (DLR), a training-only, parameter-free, foldable plug-in for low-rank pre-training. DLR augments the standard low-rank output with a fixed structured residual that expands the latent representation by replicating each latent coordinate across the output dimension. The replication factor is determined by the ratio between the output dimension and the latent rank. With a fixed residual scale, DLR adds zero learnable parameters per layer. After training, the residual is absorbed into the up-projection in closed form, so the deployment parameter count, FLOPs, and memory match the underlying low-rank backbone exactly. Across LLaMA models from 60M to 7B parameters, DLR consistently strengthens low-rank pre-training on C4 validation perplexity; folded checkpoints transfer cleanly to supervised fine-tuning on standard benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
efficient pre-training, Large language model, model folding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SUPoOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces DLR, a method that adds a replicated latent into the output of a low-rank parameterization to stabilize training. The replication acts as a simulated residual connection that improves gradient flow during low-rank pretraining. Because the replication can be folded back into the weights via a replication matrix at inference time, the method applies to any low-rank approach as a plugin with no added inference cost. The authors show it works for both non-linear and linear low-rank factorizations, and report consistent gains across several baselines.

Critical strengths
  • Strong, consistent results: the method improves on multiple low-rank baselines, which is the main driver of my recommendation.
  • Simple, general plugin: it attaches to most existing low-rank methods and, by design, folds away at inference so the low-rank speedup is preserved.
  • Supports linear low-rank factorization via the replication matrix, which keeps it compatible with online-decoding pre-merging (Wei et al. 2024).

Wei, Xiuying, et al. "Building on efficient foundations: Effective training of llms with structured feedforward layers." Advances in neural information processing systems 37 (2024)

Critical weaknesses
  • Missing baselines: no comparison to training-only low-rank methods that are also usable as plugins, specifically self-guided training (Wei et al. 2024) and Spectron (Janson et al. 2026). This is my main concern.
  • Undefined notation in abstract and intro: Several symbols are used before being defined, which makes the core idea hard to follow on a first read. For example, the abstract uses z, B (without stating it is the AB product), α, R, and Expand_K() without introduction, and the intro writes Bz at line 048 before defining B or z. Line 044 also refers to an “existing low-rank inference stack” without a citation or example.
  • Online-decoding discussion missing: when a non-linearity sits between the factors (the chosen non-linear baseline), the dense model cannot be pre-merged for online decoding. This is useful when the model is being infered with batch size 1 and seq len 1 where the model is parallelism bound. This limitation should be stated (e.g., around line 150, re: CoLA).

Wei, Xiuying, et al. "Building on efficient foundations: Effective training of llms with structured feedforward layers." Advances in neural information processing systems 37 (2024)

Janson, Paul, Edouard Oyallon, and Eugene Belilovsky. "Stabilizing native low-rank LLM pretraining." ICML (2026).

Questions for authors
  • For the non-linear low-rank setting, do you agree the dense model cannot be pre-merged for online decoding, and can you discuss this trade-off?
  • Minor - In the codebase, I don't see the folding code when the model is in eval mode ? Is this missed during shipping the code ? This does not affect my score , just a clarification.
Reviewer_EPPTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a novel method called Duplicated Latent Residual (DLR), a plugin style method for improving low-rank pre-training methods. This method is parameter free and does not change anything to the model at the end of the training or during inference time. Specifically, in DLR, the output of the low-rank block is augmented with a fixed structured residual which includes scaled expansion of the first low-rank matrix output. After training, DLR is merged into the model making no changes to the base low-rank model. This plug in style proposed method is shown to work with various low -rank methods across multiple model scales on pre-training as well as on SFT task.

Critical strengths

The paper is very well written, enough details on mathematical formulas, background, with easy to follow and well connected sections. The idea that the plug-in technique do not change the base low-rank model on the inference side is very interesting. I also like that this method is compatible with various low-rank style pre-training methods and cheaper to adapt as no new trainable parameters are introdued, makes DLR very useful to broader community. The proposed method is evaluated against baseline across many models ranging from 60M to 7B models and also beyond on SFT task showing broader set to compare methods.

Critical weaknesses

My main concern is on how much gain is the true gains DLR gives on top of the current low-rank methods. As the perplexity numbers are close for various methods, any changes to the experiment setup including hardware can bring in some noise. If authors can clarify which baselines in Table1 were reproduced and which ones are taken from prior work would give more clarity on how to compare results. If low-rank baseline numbers were from prior work, I suggest training a subset of experiments e.g. 130m CoLA and LOST baselines on the same experimental setup as CoLA+DLR and LOST +DLR respectively to see if the gains remain consistent. This will increase my confidence on the evaluation.

The low-rank baselines (without DLR) are missing from other results, making it harder to evaluate DLR performance. For example, either including LOST+DLR or CoLA in Table 2 is a better way to compare DLR true gain.

Questions for authors

None

Reviewer_GUtvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DLR adds a fixed, structured duplication of the latent vector to the learned up-projection of a low-rank layer. A shape-dependent scale controls the duplicated signal's energy. Since both terms are linear in the same latent vector, the fixed branch can be absorbed exactly into the up-projection, preserving the baseline deployment graph and parameter count. The paper evaluates this intervention on several low-rank LLaMA-style backbones trained on C4, with reported improvements at most scales from 130M upward, a single-seed 7B result, system measurements, and downstream evaluation of folded checkpoints. It also reports gradient-component diagnostics and an initialization-offset ablation. The empirical improvements are promising, but their attribution to an active residual pathway, rather than structured initialization and changed regularization, is not adequately isolated.

Critical strengths
  1. The deployment property is exact and easy to use. Equation (12) follows directly by collecting the two linear terms in the latent vector. Appendix A supplies an explicit folding implementation and checks a trained checkpoint, reporting a small BF16 perplexity difference without changing the parameter count. Thus the claimed absence of an additional inference branch is well supported and does not depend on distillation or approximate post-training compression.

  2. The reported quality improvements have useful breadth. Table 1 pairs several backbones with their DLR variants and reports multiple seeds for the DLR rows. For example, CoLA improves from 15.76 to 14.26 PPL at 1B. Tables 3–4 and Appendix B further examine system cost, folded-checkpoint transfer, normalization, and rank sensitivity. The paper also acknowledges the single-seed nature of 7B and the limited scope of downstream improvements.

Critical weaknesses
  1. The “active gradient pathway” interpretation is confounded with a change in regularization. Write C = (α/√K)Rᵀ and D = B + C. For fixed C, the forward function and data-loss gradients are exactly those of an ordinary decoder D, including the gradient Dᵀg to the latent representation. Without weight decay, shifted initialization and explicit DLR have equivalent optimizer trajectories under matched initialization and optimizer state, up to numerical differences. With AdamW, which uses weight decay 0.1 here (Appendix A/Table 6), decaying B corresponds to decaying D toward C; the ordinary init-offset baseline instead decays D toward zero. Consequently, Table 8 does not isolate a benefit from keeping a residual branch active, and the gradient decomposition cannot distinguish these explanations. This changes the interpretation of the central contribution and suggests that the branch could potentially be eliminated during training as well.

  2. The matched-token basis of the main comparisons is not reconstructible from the supplied settings. Tables 5–6 specify 140,000 steps and a token batch of 131,072 for the 1B run, implying approximately 18.35B tokens, whereas Table 1 and the surrounding text state 13.1B. Other scales also do not match exactly. Since Table 1 combines reproduced and literature baselines without a row-level provenance/budget breakdown, it is unclear whether this is a reporting error or a difference in actual training exposure. Equal-token quality improvements are a central claim, so this discrepancy needs resolution before the empirical advantage can be judged reliably.

Questions for authors
  1. [Evidence request] Can you repeat the initialization-offset comparison with the same effective-decoder regularization: decay D toward C, or remove decoder weight decay in both variants, with matched initial tensors and optimizer states? If DLR still differs, which implementation operation breaks the reparameterization equivalence? This directly tests whether the proposed active-path explanation is necessary.

  2. [Clarification] What are the actual optimizer-step counts and non-padding training-token totals for each Table 1 row? Please reconcile the 1B 140K-step/131,072-token-batch configuration with 13.1B tokens and identify which baseline rows were rerun under exactly that budget. A corrected accounting could resolve my main empirical-validity concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.