- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether low-rank pre-training methods for LLMs reach solutions comparable to full-rank training, beyond matching validation perplexity. The authors propose BASIN, a diagnostic framework with 16 metrics across four dimensions: loss landscape, checkpoint interpolation, weight spectra, and activation similarity to full-rank training. They use it to compare five low-rank methods with full-rank training at 60M, 130M, and 350M parameters, using three seeds each. They also look at behavior after fine-tuning and continual pre-training. The results show that low-rank methods are equivalent neither to full-rank training nor to one another, even when their perplexities are close. Each method converges to a distinct basin with its own sharpness and spectral properties, and its activations drift further from full-rank training in later layers and as training goes on. The authors link these differences to post-training behavior, such as forgetting. In two case studies on Fira and SLTrain, they show that simple fixes guided by BASIN bring each method closer to full-rank training.
Critical strengths
-
The paper proposes BASIN, a unified diagnostic framework that goes beyond perplexity. It combines loss barriers, weight spectra, internal activations, and the loss landscape to systematically characterize the solutions found by low-rank pretraining.
-
The authors show that models with similar perplexity can still have clearly different geometric and representational properties. They connect these differences to fine-tuning adaptability and forgetting.
-
The paper analyzes the specific ways Fira and SLTrain deviate from full-rank training. It then shows that simple corrections bring their training behavior closer to full-rank training, which demonstrates that the diagnostics can guide method improvements.
Critical weaknesses
-
The largest model tested has only 350M parameters. It is still unclear whether the conclusions hold for models with 1B parameters or more.
-
The paper does not study training behavior under larger token budgets. It remains to be seen whether the conclusions apply when models are trained to convergence.
-
The experiments do not include other optimizers such as Muon as comparisons. This makes it hard to tell whether the observed phenomena depend on Adam-family optimization.
Questions for authors
Questions for the Authors:
-
Do the main conclusions still hold for models with 1B parameters or more?
-
Would the observed differences between low-rank and full-rank training remain under a larger token budget?
-
Are the observed phenomena specific to Adam-family optimizers, or would they also appear with other optimizers such as Muon?
3 comments
R4.3 The experiments do not include other optimizers such as Muon as comparisons. This makes it hard to tell whether the observed phenomena depend on Adam-family optimization.
We agree this is an important question. Low-rank pre-training methods have so far been developed and evaluated almost exclusively with Adam-family optimizers: GaLore and FiRA are defined as modifications of Adam's update, and CoLA, SLTrain and ReLoRA all report results with AdamW. How these methods behave under other optimizers is therefore largely unexplored. To test whether our findings depend on Adam, we retrained all six methods at 60M with Muon in place of AdamW (rank 128, 10k steps on C4 and the optimizer is the only difference). Muon is applied to the 2D hidden weights, embeddings, the output head and 1D parameters use AdamW, for GaLore and FiRA the orthogonalization is applied inside the low-rank subspace. All metrics were computed with the revised pipeline.
Method orderings at the final checkpoint, Adam vs Muon, single seed:
- Normalized sharpness (flattest → sharpest)
- Adam: Full-rank < ReLoRA < SLTrain < FiRA < GaLore < CoLA
- Muon: ReLoRA < SLTrain < Full-rank < FiRA < GaLore < CoLA
The order is identical for ReLora, SLtrain, Fira, Galore and CoLA,
- Weight stable rank (high → low)
- Adam: Full-rank > GaLore > FiRA > ReLoRA > SLTrain
- Muon: ReLoRA > Full-rank > GaLore > SLTrain > FiRA
Full-rank>GaLore> Sltrain is common is both optimizers
- Activation CKA to full-rank (closest → farthest)
- Adam: ReLoRA > GaLore > FiRA > CoLA > SLTrain
- Muon: GaLore > FiRA > CoLA > ReLoRA > SLTrain
GaLore > FiRA > CoLA > SLTrain is common in both
- Barrier to full-rank after weight matching (lowest → highest)
- Adam: FiRA < ReLoRA < GaLore < SLTrain
- Muon: ReLoRA < SLTrain ≈ FiRA < GaLore
Fira < Galore and ReLoRA < GaLore are common to both optimizers. Among the shared-init methods, GaLore has the highest barrier under both.
Thus, several method-level patterns persist under Muon, suggesting they stem from the low-rank parameterization rather than only from Adam. Note that this analysis is based on a single-seed run at 60M. We leave a multi-seed study at larger model scales to future work.
Questions
R4.Q1 Do the main conclusions still hold for models with 1B parameters or more?
See R4.1
R4.Q2 Would the observed differences under a larger token budget?
See R4.2
R4.Q3 Are the observed phenomena specific to Adam-family optimizers, or would they also appear with other optimizers such as Muon?
See R4.3
We hope the additional results, new sections, and clarifications address your concerns. If our responses have sufficiently addressed your feedback, we kindly ask you to consider revisiting your score.
Gentle reminder
Dear Reviewer qSzZ,
As the discussion period ends today (AoE), we wanted to kindly follow up on our earlier responses. Based on your valuable feedback, we have added new experiments and analyses that we believe substantially strengthen the paper. We would greatly appreciate it if you could let us know whether these updates resolve your remaining concerns.
Thank you again for your time and effort throughout the reviewing process.
R4 response
R4.1 The largest model tested has only 350M parameters. It is still unclear whether the conclusions hold for models with 1B parameters or more.
We agree this is an important question. Our study covers 60M, 130M and 350M models with 54 pre-training runs (6 methods × 3 seeds × 3 scales) and 324 SFT runs and 54 CPT runs. Within this range, our diagnostics separate methods more clearly at larger scales than at the smaller scale 60M:
This suggests the differences we report, and the ability of our diagnostics to detect them, do not weaken as models grow. Validating this at 1B+ parameters, which would need multi-seed pre-training and post-training of all six methods, was beyond our compute budget. We state this in the Limitations section, and we release our code so the community can apply these diagnostics to larger models.
R4.2 The paper does not study training behavior under larger token budgets. It remains to be seen whether the conclusions apply when models are trained to convergence.
To test whether these geometric differences persist under larger token budgets, we trained 60M models with full-rank training and with GaLore (rank 128). Each method was trained at two budgets: 10k steps (1x) (existing runs) and 40k steps (4x training tokens) and other experimental setup was kept constant. Each configuration uses two seeds, and we report the mean. At both budgets, we computed one main metric per lens, each with the same pipeline as in the paper:
The differences do not diminish with 4× more training, and most of them grow.
Taken together, these results suggest the geometric and spectral differences between low-rank and full-rank training are not an artifact of short training budgets and as seen from our 60M experiments, they persist with 4× more tokens and, in most cases, grow, even as both methods reach lower validation loss.