- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces DLR, a method that adds a replicated latent into the output of a low-rank parameterization to stabilize training. The replication acts as a simulated residual connection that improves gradient flow during low-rank pretraining. Because the replication can be folded back into the weights via a replication matrix at inference time, the method applies to any low-rank approach as a plugin with no added inference cost. The authors show it works for both non-linear and linear low-rank factorizations, and report consistent gains across several baselines.
Critical strengths
- Strong, consistent results: the method improves on multiple low-rank baselines, which is the main driver of my recommendation.
- Simple, general plugin: it attaches to most existing low-rank methods and, by design, folds away at inference so the low-rank speedup is preserved.
- Supports linear low-rank factorization via the replication matrix, which keeps it compatible with online-decoding pre-merging (Wei et al. 2024).
Wei, Xiuying, et al. "Building on efficient foundations: Effective training of llms with structured feedforward layers." Advances in neural information processing systems 37 (2024)
Critical weaknesses
- Missing baselines: no comparison to training-only low-rank methods that are also usable as plugins, specifically self-guided training (Wei et al. 2024) and Spectron (Janson et al. 2026). This is my main concern.
- Undefined notation in abstract and intro: Several symbols are used before being defined, which makes the core idea hard to follow on a first read. For example, the abstract uses z, B (without stating it is the AB product), α, R, and Expand_K() without introduction, and the intro writes Bz at line 048 before defining B or z. Line 044 also refers to an “existing low-rank inference stack” without a citation or example.
- Online-decoding discussion missing: when a non-linearity sits between the factors (the chosen non-linear baseline), the dense model cannot be pre-merged for online decoding. This is useful when the model is being infered with batch size 1 and seq len 1 where the model is parallelism bound. This limitation should be stated (e.g., around line 150, re: CoLA).
Wei, Xiuying, et al. "Building on efficient foundations: Effective training of llms with structured feedforward layers." Advances in neural information processing systems 37 (2024)
Janson, Paul, Edouard Oyallon, and Eugene Belilovsky. "Stabilizing native low-rank LLM pretraining." ICML (2026).
Questions for authors
- For the non-linear low-rank setting, do you agree the dense model cannot be pre-merged for online decoding, and can you discuss this trade-off?
- Minor - In the codebase, I don't see the folding code when the model is in eval mode ? Is this missed during shipping the code ? This does not affect my score , just a clarification.