acceptodds
Under review as a conference paper at ICLR 2027

Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training

Abstract

Pre-training large language models is dominated by the memory cost of storing full-rank weights, gradients, and optimizer states. Low-rank pre-training has emerged to address this, and the space of methods has grown rapidly. A central question remains open: do low-rank methods produce models that generalize comparably to full-rank training, or does the rank constraint fundamentally alter the solutions reached? Existing comparisons rely almost entirely on validation perplexity from single-seed runs, often carried forward from prior literature. Yet perplexity is a poor proxy for solution quality — two methods can match on perplexity while converging to very different loss landscape regions and internal representations. To look beyond perplexity, we introduce BASIN (Barrier, Activations, Spectra, and Intrinsic geometry of Networks), a framework that characterizes the solutions found by five low-rank pre-training methods — GaLore and Fira (memory-efficient optimizers), CoLA and SLTrain (architecture reparameterizations), and ReLoRA (adapter-style updates with periodic resets) — against full-rank training at three model scales (60M, 130M, 350M) under three seeds. We evaluate each along 16 metrics across four dimensions: 1-D loss landscape along random/top-K PCA directions, 1-D interpolation between checkpoints, spectral structure of the weights and learned updates, and activation similarity to full-rank training. We also examine how each pre-trained model adapts after task and instruction fine-tuning, and continual pre-training with and without replay. We show that low-rank methods are not equivalent to full-rank training, nor to one another, even when their validation perplexity is close. Each method converges to a distinct basin with different sharpness and spectrum. Low-rank method activations drift further from full-rank in later layers and as training progresses. These differences shape post-training behavior. For example, the path from the pre-trained to the fine-tuned model crosses a barrier for ReLoRA and climbs a steep wall for SLTrain, matching their high forgetting. Finally, in two case studies, BASIN traces how Fira and SLTrain deviate from full-rank, and a simple fix guided by this diagnosis brings each method closer to full-rank training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
low-rank pre-training, geometric metrics, spectral metrics, loss landscape

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_qSzZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether low-rank pre-training methods for LLMs reach solutions comparable to full-rank training, beyond matching validation perplexity. The authors propose BASIN, a diagnostic framework with 16 metrics across four dimensions: loss landscape, checkpoint interpolation, weight spectra, and activation similarity to full-rank training. They use it to compare five low-rank methods with full-rank training at 60M, 130M, and 350M parameters, using three seeds each. They also look at behavior after fine-tuning and continual pre-training. The results show that low-rank methods are equivalent neither to full-rank training nor to one another, even when their perplexities are close. Each method converges to a distinct basin with its own sharpness and spectral properties, and its activations drift further from full-rank training in later layers and as training goes on. The authors link these differences to post-training behavior, such as forgetting. In two case studies on Fira and SLTrain, they show that simple fixes guided by BASIN bring each method closer to full-rank training.

Critical strengths
  1. The paper proposes BASIN, a unified diagnostic framework that goes beyond perplexity. It combines loss barriers, weight spectra, internal activations, and the loss landscape to systematically characterize the solutions found by low-rank pretraining.

  2. The authors show that models with similar perplexity can still have clearly different geometric and representational properties. They connect these differences to fine-tuning adaptability and forgetting.

  3. The paper analyzes the specific ways Fira and SLTrain deviate from full-rank training. It then shows that simple corrections bring their training behavior closer to full-rank training, which demonstrates that the diagnostics can guide method improvements.

Critical weaknesses
  1. The largest model tested has only 350M parameters. It is still unclear whether the conclusions hold for models with 1B parameters or more.

  2. The paper does not study training behavior under larger token budgets. It remains to be seen whether the conclusions apply when models are trained to convergence.

  3. The experiments do not include other optimizers such as Muon as comparisons. This makes it hard to tell whether the observed phenomena depend on Adam-family optimization.

Questions for authors

Questions for the Authors:

  1. Do the main conclusions still hold for models with 1B parameters or more?

  2. Would the observed differences between low-rank and full-rank training remain under a larger token budget?

  3. Are the observed phenomena specific to Adam-family optimizers, or would they also appear with other optimizers such as Muon?

3 comments
AuthorsOpenReview ↗

R4 response

R4.1 The largest model tested has only 350M parameters. It is still unclear whether the conclusions hold for models with 1B parameters or more.

We agree this is an important question. Our study covers 60M, 130M and 350M models with 54 pre-training runs (6 methods × 3 seeds × 3 scales) and 324 SFT runs and 54 CPT runs. Within this range, our diagnostics separate methods more clearly at larger scales than at the smaller scale 60M:

  • Barrier: the full-rank seed-to-seed barrier becomes far less noisy with scale. As a result, the weight-matched barrier separates only 1 of 4 low-rank methods from full-rank at 60M, but 3 of 4 at 130M and 350M (independent init, p < 0.05).
  • Sharpness: the normalized sharpness gap to full-rank grows with model size for GaLore (2.06× → 2.18× → 2.23×) and FiRA (1.86× → 2.32× → 2.78×).
  • CKA: the gap between full-rank seed-to-seed CKA and low-rank-to-full-rank CKA widens from 0.19 at 60M to 0.31 at 130M and 0.27 at 350M.
  • Post-training: method differences in forgetting grow relative to pre-training perplexity differences (15.6×, 23× and 25× at 60M, 130M and 350M).

This suggests the differences we report, and the ability of our diagnostics to detect them, do not weaken as models grow. Validating this at 1B+ parameters, which would need multi-seed pre-training and post-training of all six methods, was beyond our compute budget. We state this in the Limitations section, and we release our code so the community can apply these diagnostics to larger models.

R4.2 The paper does not study training behavior under larger token budgets. It remains to be seen whether the conclusions apply when models are trained to convergence.

To test whether these geometric differences persist under larger token budgets, we trained 60M models with full-rank training and with GaLore (rank 128). Each method was trained at two budgets: 10k steps (1x) (existing runs) and 40k steps (4x training tokens) and other experimental setup was kept constant. Each configuration uses two seeds, and we report the mean. At both budgets, we computed one main metric per lens, each with the same pipeline as in the paper:

LensMetricBudgetFull-rankGaLoreGaLore vs FR
PerformanceVal loss1x3.5273.551+0.024
4x3.3723.425+0.053
LandscapeNormalized sharpness (ES_wn)1x4.389.122.08x
4x5.8714.492.47x
SpectralStable rank (W)1x61.9852.280.84x
4x61.7750.070.81x
ActivationCKA to FR (FR column = FR seed-to-seed floor)1x0.780.61-0.17
4x0.810.62-0.19
BarrierWeight-matched barrier, independent init (FR col = FR-FR)1x4.225.401.28x
4x5.397.351.36x

The differences do not diminish with 4× more training, and most of them grow.

  • Val loss: GaLore's gap to full-rank widens from 0.024 to 0.053.
  • Sharpness: the GaLore/full-rank ratio rises from 2.08× to 2.47×.
  • Stable rank: GaLore's stable rank decreased a little (52.3 → 50.1), while full-rank's stays flat (62.0 → 61.8).
  • CKA: GaLore stays about 0.2 below the full-rank seed-to-seed floor at both budgets.
  • Barrier: with independent inits, the GaLore–full-rank barrier stays above the full-rank reference (1.28× → 1.36×).

Taken together, these results suggest the geometric and spectral differences between low-rank and full-rank training are not an artifact of short training budgets and as seen from our 60M experiments, they persist with 4× more tokens and, in most cases, grow, even as both methods reach lower validation loss.

R4.3 The experiments do not include other optimizers such as Muon as comparisons. This makes it hard to tell whether the observed phenomena depend on Adam-family optimization.

We agree this is an important question. Low-rank pre-training methods have so far been developed and evaluated almost exclusively with Adam-family optimizers: GaLore and FiRA are defined as modifications of Adam's update, and CoLA, SLTrain and ReLoRA all report results with AdamW. How these methods behave under other optimizers is therefore largely unexplored. To test whether our findings depend on Adam, we retrained all six methods at 60M with Muon in place of AdamW (rank 128, 10k steps on C4 and the optimizer is the only difference). Muon is applied to the 2D hidden weights, embeddings, the output head and 1D parameters use AdamW, for GaLore and FiRA the orthogonalization is applied inside the low-rank subspace. All metrics were computed with the revised pipeline.

Method orderings at the final checkpoint, Adam vs Muon, single seed:

  • Normalized sharpness (flattest → sharpest)
    • Adam: Full-rank < ReLoRA < SLTrain < FiRA < GaLore < CoLA
    • Muon: ReLoRA < SLTrain < Full-rank < FiRA < GaLore < CoLA

The order is identical for ReLora, SLtrain, Fira, Galore and CoLA,

  • Weight stable rank (high → low)
    • Adam: Full-rank > GaLore > FiRA > ReLoRA > SLTrain
    • Muon: ReLoRA > Full-rank > GaLore > SLTrain > FiRA

Full-rank>GaLore> Sltrain is common is both optimizers

  • Activation CKA to full-rank (closest → farthest)
    • Adam: ReLoRA > GaLore > FiRA > CoLA > SLTrain
    • Muon: GaLore > FiRA > CoLA > ReLoRA > SLTrain

GaLore > FiRA > CoLA > SLTrain is common in both

  • Barrier to full-rank after weight matching (lowest → highest)
    • Adam: FiRA < ReLoRA < GaLore < SLTrain
    • Muon: ReLoRA < SLTrain ≈ FiRA < GaLore

Fira < Galore and ReLoRA < GaLore are common to both optimizers. Among the shared-init methods, GaLore has the highest barrier under both.

Thus, several method-level patterns persist under Muon, suggesting they stem from the low-rank parameterization rather than only from Adam. Note that this analysis is based on a single-seed run at 60M. We leave a multi-seed study at larger model scales to future work.

Questions

R4.Q1 Do the main conclusions still hold for models with 1B parameters or more?

See R4.1

R4.Q2 Would the observed differences under a larger token budget?

See R4.2

R4.Q3 Are the observed phenomena specific to Adam-family optimizers, or would they also appear with other optimizers such as Muon?

See R4.3

We hope the additional results, new sections, and clarifications address your concerns. If our responses have sufficiently addressed your feedback, we kindly ask you to consider revisiting your score.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer qSzZ,

As the discussion period ends today (AoE), we wanted to kindly follow up on our earlier responses. Based on your valuable feedback, we have added new experiments and analyses that we believe substantially strengthen the paper. We would greatly appreciate it if you could let us know whether these updates resolve your remaining concerns.

Thank you again for your time and effort throughout the reviewing process.

Reviewer_yqT2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces BASIN, a 16-metric framework (interpolation barriers, activation similarity, spectral/rank structure, 1-D loss-landscape sharpness) to compare five low-rank pre-training methods against full-rank training at three scales, claiming that (i) low-rank methods converge to geometrically separated basins, (ii) basin geometry explains post-training forgetting, and (iii) BASIN diagnostics can guide method design.

Critical strengths
  1. Important motivating question; substantial empirical coverage (6 methods × 3 scales × 3 seeds × 4 post-training protocols).
  2. Some genuinely interesting observations (perplexity rankings fail to transfer to post-training; activation drift grows with depth).
Critical weaknesses
  1. The core claim about methods converging to "distinct basins" is not supported by the metric used. To measure inter-method barriers (IMBH), the paper simply does linear interpolation in weight space between models trained with different optimizers. However, it is well known in the mode connectivity literature (Entezari et al., 2022; Ainsworth et al., 2023, both uncited here) that naive interpolation produces high barriers simply due to neuron permutation mismatch. Even two runs of the same method from different seeds will show high barriers without permutation alignment. Thus, observing large barriers here is just an artifact of the metric and says almost nothing about whether the methods truly sit in separate basins. The activation and spectral metrics show the models are different, but that does not mean they are separated by loss barriers.

  2. Cross-method comparisons of sharpness and rank are confounded by weight scale. The paper measures sharpness along unnormalized random directions. As shown by Li et al. (2018) (cited in the paper), unnormalized perturbations make networks with larger weights artificially appear sharper. We can see a clear symptom of this in the results: ReLoRA's sharpness is 10 to 100 times higher than everything else, which strongly suggests a scale issue rather than a true geometric property. The fixed threshold (τ=0.1) used for effective rank has the same scale-sensitivity problem. The Limitations section briefly notes that normalization is an open question, but the main text still relies heavily on these unnormalized numbers to explain forgetting in Section 5.3.

  3. The explanation of post-training forgetting is post-hoc and lacks predictive testing. The "task direction" in Section 5.3 is computed from the final fine-tuned weights, meaning the diagnostic direction is defined by the very outcome it is trying to explain. Moreover, both the barrier and the forgetting metric are just the C4 loss. The analysis only looks at two qualitative examples at one scale, without computing statistical correlations across all six methods or testing out-of-sample prediction (like predicting 350M behavior from 130M metrics). While the paper successfully shows that perplexity is a poor predictor of downstream performance, it never actually proves that BASIN is a good predictor. Without that, we are just left with the observation that different methods have different training dynamics.

  4. The case studies in the "diagnosis to design" section are too weak to support the claim. Both experiments only use one seed at the smallest scale (130M). In the SLTrain case, two changes (QR init and DCT basis) are applied at the same time with no ablation, so we do not know which one fixed the issue. In the Fira case, the 0.5 scaling factor is just a single hyperparameter choice. Neither fix is validated at 350M/1B or checked to see if it hurts the memory efficiency of low-rank training. Also, framing the gains as "71.8% closer to full-rank" overstates the result, since relative percentages depend heavily on how small the denominator is.

Questions for authors
  1. Do inter-method barriers survive permutation alignment (e.g., Git Re-Basin-style weight matching)? If they vanish, as prior work strongly suggests, the central finding of Section 5.2 would need to be retracted.
  2. Do the sharpness and threshold-rank orderings survive filter-wise normalized perturbations and a relative singular-value cutoff? How much of ReLoRA's 10–100× sharpness is explained by weight norm alone?
  3. Can you provide quantitative, out-of-sample evidence that BASIN metrics predict forgetting (e.g., correlation with CIs across all six methods, or cross-scale prediction), rather than post-hoc qualitative matches for two methods at one scale?
  4. Which component of the SLTrain fix drives the improvement, does it replicate across seeds/scales, and does it preserve the method's memory advantage?
2 comments
AuthorsOpenReview ↗

R3 response

Thank you for your valuable feedback.

R3.1 The core claim about methods converging to "distinct basins" is not supported by the metric used.

See Common Response 2.

We added mode connectivity literature (Entezari et al., 2022; Ainsworth et al., 2023) in the related work section in the revised work.

R3.2 Cross-method comparisons of sharpness and rank are confounded by weight scale

We agree, and have addressed this in "Common response section 1. Normalized perturbations". With matrix-normalized directions and ReLoRA/SLTrain perturbed in their materialized weights, ReLoRA's sharpness falls to 0.93–1.18× full-rank's, suggesting that its excess was a weight-scale effect. Our main takeaways are now based on normalized perturbations, not only on raw ones. The fixed threshold τ=0.1\tau = 0.1 applies to threshold rank. We now compute threshold rank with a scale-invariant relative cutoff, σi>0.1σ˙max⁡\sigma_i > 0.1\.\sigma_{\max}, on materialized weights. We discuss more on this while answering R3.Q2.

R3.3 The explanation of post-training forgetting is post-hoc and lacks predictive testing.

See Common Response 3. We added a quantitative study that links BASIN metrics to post-training behaviour across methods and scales, and that predicts held-out scales (including 350M from 60M and 130M).

R3.4 The case studies in the "diagnosis to design" section are too weak to support the claim.

See common response 4

Questions

R3. Q1 Do inter-method barriers survive permutation alignment (e.g., Git Re-Basin-style weight matching)? If they vanish, as prior work strongly suggests, the central finding of Section 5.2 would need to be retracted.

See Common Response 2 for detailed results. Barriers after alignment survive, as alignment lowers the full-rank barrier by 14–18%, and we withdraw the “distinct basins” wording and explain the barrier result with linear mode connectivity.

R3.Q2 Do the sharpness and threshold-rank orderings survive filter-wise normalized perturbations and a relative singular-value cutoff? How much of ReLoRA's 10–100× sharpness is explained by weight norm alone?

See Common Response 1 for normalized permutation analysis.

  • Sharpness: Under normalized perturbations, the raw ordering largely reverses: full-rank becomes among the flattest, FiRA is among the sharpest, and SLTrain sharper than full-rank.

  • Threshold rank: partly. With a relative cutoff, SLTrain remains the lowest-rank method, but FiRA no longer ranks highest. ReLoRA: almost all of its excess sharpness is weight norm: it is 7–15× full-rank's under raw perturbations but about the same as full-rank's (0.93–1.18×) once normalized.

R3.Q3 Can you provide quantitative, out-of-sample evidence that BASIN metrics predict forgetting (e.g., correlation with CIs across all six methods, or cross-scale prediction), rather than post-hoc qualitative matches for two methods at one scale?

See common response 3

R3.Q4 Which component of the SLTrain fix drives the improvement, does it replicate across seeds/scales, and does it preserve the method's memory advantage?

See common response 4

We hope the additional results, new sections, and clarifications address your concerns. If our responses have sufficiently addressed your feedback, we kindly ask you to consider revisiting your score.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer yqT2,

As the discussion period ends today (AoE), we wanted to kindly follow up on our earlier responses. Based on your valuable feedback, we have added new experiments and analyses that we believe substantially strengthen the paper. We would greatly appreciate it if you could let us know whether these updates resolve your remaining concerns.

Thank you again for your time and effort throughout the reviewing process.

Reviewer_sWcjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that validation perplexity is not enough to compare low-rank pre-training methods, and introduces BASIN, a diagnostic framework of 16 metrics grouped into four lenses:

  • loss barriers along linear interpolations, both between consecutive checkpoints and between different methods (the proposed inter-method barrier, IMBH);
  • activation similarity to the full-rank model (L2 distance, cosine similarity, linear CKA);
  • spectral and rank statistics of the weights and their updates;
  • sharpness from 1-D loss landscapes along random and top-1 PCA directions.

The authors pretrain LLaMA-style models at 60M, 130M and 350M on C4 with Chinchilla-optimal budgets and three seeds, comparing full-rank training with GaLore, Fira, CoLA, SLTrain and ReLoRA. They also evaluate the final checkpoints after task SFT on five benchmarks, instruction SFT on LaMini, and continued pre-training on German C4 with and without replay (130M only). Perplexity ranks Fira best and ReLoRA worst. Zero-shot accuracy separates only those two extremes, while fine-tuning and continued pre-training separate the remaining methods, with forgetting patterns that differ by protocol. BASIN shows that every pair of methods is separated by a sizable loss barrier that grows during training, that activations drift further from full-rank with depth and training time, and that the methods differ in spectra and sharpness (e.g. SLTrain has the lowest rank and largest spectral gap; ReLoRA is the sharpest along random directions). Interpolating from pre-trained to fine-tuned checkpoints, ReLoRA crosses a barrier and SLTrain climbs a steep wall in C4 loss, which the authors link to their higher forgetting. In two single-seed case studies at 130M, scaling Fira's residual term by 0.5 and changing SLTrain's initialization and sparse basis bring both methods closer to full-rank on the BASIN metrics and on post-training forgetting. A unified codebase is promised upon acceptance.

Critical strengths

1. A broader and more careful evaluation than is standard for low-rank pre-training. The paper compares five methods from all three design families (optimizer-based, reparameterization-based and adapter-based) against full-rank training at three scales, with three pre-training seeds per configuration. Zero-shot differences are checked with a permutation ANOVA. More importantly, every checkpoint is also evaluated after task SFT, instruction SFT and continued pre-training with and without replay. This shows that methods that look alike under perplexity and zero-shot accuracy separate once they are fine-tuned, and that the forgetting pattern can even flip between SFT and continued pre-training (Fira forgets less than full-rank in SFT but more in CPT). Most prior work reports single-seed perplexity, often copied from earlier papers. A concrete demonstration that this hides real differences in post-training behavior is useful to the field, and it is the main point in the paper's favor.

2. A clear spectral picture of how the methods update the weights. Because the spectral metrics are computed on materialized weights, all methods except CoLA are compared in the same weight space. This makes the update-spectrum results easier to trust than the landscape metrics. They show a consistent split across scales: GaLore and Fira make near full-rank updates throughout training, while SLTrain and ReLoRA make much lower-rank updates concentrated at small singular values, and SLTrain ends with the lowest weight rank and the largest spectral gap. This gives a mechanistic handle on why methods with similar perplexity differ, and it is the kind of diagnostic that future method papers could adopt directly, especially if the promised codebase is released.

Critical weaknesses

1. The headline geometric claims are not calibrated against a baseline. The abstract states that each method converges to a distinct basin, based on large inter-method barriers (BH≈2\mathrm{BH} \approx 2–88). However, linear interpolation between independently trained networks almost always shows a high barrier, regardless of optimizer, unless the networks are aligned for permutation symmetry or branched from a shared checkpoint late in training. The paper reports no full-rank vs. full-rank barrier across seeds, and no alignment step. Without that reference, the barrier does not show anything specific to low-rank training. The same problem applies to activation L2 distance and cosine similarity, which are not invariant to rotations of the hidden space. The sharpness comparison has a related issue. Random directions are drawn as δ∼N(0,I)\delta \sim \mathcal{N}(0, I) without filter or norm scaling, so methods with different weight scales or parameterizations are perturbed by different relative amounts. CoLA is even perturbed in its own factorized parameter space. Since BASIN is the main contribution and these metrics carry its main findings, the lack of calibration is the central reason I lean toward rejection.

2. The "diagnosis to design" story rests on confounded and single-seed evidence. The link between basin shape and forgetting in Section 5.3 is drawn from interpolation plots for six methods, with no quantitative correlation across methods or scales. The post-training protocols also differ in ways that matter. In continued pre-training, each method keeps its own low-rank optimizer and its own tuned learning rate (5×10−35\times10^{-3} for GaLore and Fira vs. 10−310^{-3} for full-rank). Fira's larger forgetting in CPT may therefore simply reflect a 5× larger step size rather than the shape of its basin. The case studies are run on a single seed and report "percent closer to full-rank" without noise estimates. For Fira, moving closer to full-rank means more forgetting, since Fira forgets less than full-rank in SFT, so the "fix" makes the model worse on that axis. More broadly, full-rank is used as the reference point throughout, but the full-rank baseline reuses the GaLore codebase settings and is clearly beaten by Fira (17.26 vs. 19.08 perplexity at 350M), and ReLoRA is run without its warm start. This suggests the reference itself may be under-tuned. These problems weaken the claim that BASIN explains post-training behavior and can guide method design.

Questions for authors

Clarifications and additional results that could change my assessment

  1. Noise floor for barriers and activations. What is the inter-model barrier between two full-rank runs with different seeds, at the same steps and scales as Fig. 3? The same question applies to activation L2 distance, cosine similarity and CKA between two full-rank seeds. Do methods with the same architecture (full-rank, GaLore, Fira) share initialization when they share a seed? If the full-rank seed-to-seed barrier is of similar size (BH≈2\mathrm{BH} \approx 2–88), the claim that each method converges to a distinct basin would need to be restated. Results after permutation alignment (e.g. weight matching) would also help separate method effects from symmetry effects.

  2. Normalization of perturbations. How are SLTrain, ReLoRA and CoLA perturbed in the random-direction analysis: in materialized weights or in their native parameters? Do the sharpness rankings (ReLoRA sharpest, Fira and SLTrain flatter than full-rank) still hold with filter- or layer-normalized directions, i.e. with δ\delta scaled to the norm of each weight matrix?

  3. Post-training protocol. In CPT, each method uses its own optimizer and a per-method learning rate (5×10−35\times10^{-3} for GaLore and Fira vs. 10−310^{-3} for full-rank). Could you rerun CPT from the materialized checkpoints with one shared recipe, so that only the pre-trained checkpoint differs? Does Fira still forget more than full-rank in that setting? For LaMini SFT, please clarify whether each checkpoint is fine-tuned with its own low-rank method or as a dense model with AdamW.

  4. Quantitative link to forgetting. Can you report a correlation, across methods, scales and seeds, between a BASIN quantity (e.g. the C4 loss slope along the pre-trained-to-fine-tuned path, or IMBH to full-rank) and measured forgetting or gain? At present the explanation in Section 5.3 rests on interpolation plots for six methods.

  5. Reference point and case studies.

    • Was the full-rank learning rate tuned at each scale, or taken from the GaLore codebase? Does Fira still beat a tuned full-rank baseline?
    • Why is ReLoRA run without the warm start that its original paper requires?
    • For case study 1, Fira forgets less than full-rank in SFT. Why is moving its forgetting 37.6% closer to full-rank, i.e. increasing it, counted as an improvement?
    • Could both case studies be repeated with three seeds?
5 comments
AuthorsOpenReview ↗

R2 response

Thank you for your valuable feedback.

R2.1. The headline geometric claims are not calibrated against a baseline.

We agree and added the missing calibrations for barriers, activations and sharpness.

  • Barriers: see Common Response 2
  • Sharpness see Common Response 1.
  • Activations see below

We agree that L2 distance and cosine similarity are not invariant to rotations of the hidden space. We now use CKA, which is rotation-invariant, as the primary activation metric, calibrated against full-rank seed pairs.

The table shows that CKA to full-rank at the final checkpoint (mean ± s.d. over 3 pairs), higher the better:

FR ↔ FR (diff. seeds)ReLoRAGaLoreFiRASLTrainCoLA
60M0.59 ± 0.170.47 ± 0.180.52 ± 0.160.47 ± 0.150.24 ± 0.030.31 ± 0.07
130M0.74 ± 0.040.61 ± 0.030.53 ± 0.110.45 ± 0.070.30 ± 0.140.28 ± 0.05
350M0.71 ± 0.060.57 ± 0.050.51 ± 0.080.37 ± 0.040.36 ± 0.150.39 ± 0.08
  • Every low-rank method stays below the full-rank seed reference at 130M and 350M. At 60M, the reference is noisy (0.59 ± 0.17), so only SLTrain and CoLA separate clearly from it.
  • The broad pattern is consistent across scales: ReLoRA and GaLore are among the closest to full-rank, and SLTrain and CoLA are among the furthest. FiRA lies in between, and drops to the furthest group at 350M.
  • L2 and cosine do not survive this calibration. We therefore keep CKA as the primary activation metric. We have revised the results on activation in Figure 4 and Section 5.2.1 in the uploaded revised version.

R2.2. The "diagnosis to design" story rests on confounded and single-seed evidence.

We have addressed each point in different parts:

  • No quantitative link between BASIN and forgetting: see Common Response 3.
  • CPT learning-rate confound: see R2.Q3, Common Response 3.2
  • Single-seed case studies without noise estimates: see Common Response 4 (three seeds, mean ± sd, and an ablation of SLTrain's two components).
  • FiRA's "fix" increases forgetting: see R2.Q5. We no longer call it an improvement.
  • Under-tuned full-rank reference and ReLoRA without warm start: see R2.Q5.

Questions

R2.Q1 Noise floor for barriers and activations.

  • Barriers, shared initialization and alignment: see Common Response 2. For barrier across training trajectory, we have revised Figure 3 in the revised paper uploaded.
  • Activations: see R2.1.

R2.Q2 Normalization of perturbations.

See "Common Response 1. Normalized perturbations".

In the submission version, low-rank methods were perturbed in their native parameters. In the revised draft, ReLoRA and SLTrain are also perturbed in their materialized dense weights and CoLA has no dense equivalent and remains in its native parameters. Under matrix-normalized directions, the raw ranking largely reverses i.e., ReLoRA is close to full-rank, FiRA becomes the sharpest from 130M on, SLTrain is sharper than full-rank, and full-rank is among the flattest. We have revised Figure 6 and Section 5.2.3 to report sharpness and ranking under all three protocols (raw, normalized and top-1 PCA). Full results are in Common Response 1.

R2.Q3 Post-training protocol.

See common response 3.2 for the matched CPT recipe and result. LaMini SFT. All low-rank methods were fine-tuned as dense models with AdamW. ReLoRA and SLTrain were materialized to dense weights first. CoLA is the exception: it has no dense equivalent, so it is fine-tuned in its own architecture and with AdamW.

R2.Q4 Quantitative link to forgetting.

See common response 3

R2.Q5 Reference point and case studies.

1. Was the full-rank learning rate tuned at each scale, or taken from the GaLore codebase? Does Fira still beat a tuned full-rank baseline?

Our full-rank (3 seeds)GaLore paper full-rankOur FiRA
60M34.04 ± 0.0834.0631.04
130M25.07 ± 0.0325.0823.18
350M19.08 ± 0.3518.8017.26

(continuing R2Q5 point 1) Our full-rank baseline is the standard one in the low-rank pre-training literature. GaLore (Zhao et al., 2024) tuned the learning rate of every method, including full-rank, over {0.01, 0.005, 0.001, 0.0005, 0.0001} by validation perplexity. FiRA (Chen et al., 2024) and SLTrain (Han et al., 2024) both follow GaLore's setup and take their full-rank results directly from it ("Results of all baselines are taken from Zhao et al. (2024)"). Every method we study was therefore originally compared against this same tuned baseline, so we use the same in our study. We use the same recipe, and our runs reproduce the published full-rank perplexity closely: 34.04 / 25.07 / 19.08 (3 seeds) against 34.06 / 25.08 / 18.80 reported by GaLore at 60M / 130M / 350M. FiRA's lower perplexity (17.26 at 350M) therefore holds against a tuned full-rank baseline and matches the FiRA paper's own report that it outperforms full-rank under this setup. We do not conduct additional full-rank tuning.

2. Why is ReLoRA run without the warm start that its original paper requires?

We follow GaLore (Zhao et al., 2024), which states: "We evaluate ReLoRA without a full-rank training warmup for a fair comparison." SLTrain and FiRA take their ReLoRA results from that setup, so it is the standard ReLoRA baseline in this literature. It also matters for our study specifically: we measure metrics along the whole training trajectory, and with a full-rank warm start ReLoRA's early checkpoints would be full-rank training, so its early-trajectory metrics would measure full-rank rather than low-rank training. We therefore keep the no-warm-start setting and state it as a deviation from the original ReLoRA recipe in the setup section.

3. For case study 1, Fira forgets less than full-rank in SFT. Why is moving its forgetting 37.6% closer to full-rank, i.e. increasing it, counted as an improvement?

We agree that "improvement" was the incorrect description for the fira case study. The goal of the case studies is not to reduce forgetting. It is to find a mechanistic handle, a component or set of components, identified by our diagnostics, that moves a low-rank model closer to full-rank training. Full-rank is the reference because a low-rank method is meant to be a cheaper way to train the same model. Its properties and post-training behaviour should not change just because it was trained with low rank, in either direction. FiRA's lower forgetting is such a change, just as SLTrain's higher forgetting is in the second case study. The two cases show that the diagnosed component moves both the metrics and the post-training behaviour toward full-rank, whichever side the deviation lies on. We will describe the results as "closer to full-rank" rather than "improved".

4. Could both case studies be repeated with three seeds?

See common response 4

We hope the additional results, new sections, and clarifications address your concerns. If our responses have sufficiently addressed your feedback, we kindly ask you to consider revisiting your score.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer sWcj,

As the discussion period ends today (AoE), we wanted to kindly follow up on our earlier responses. Based on your valuable feedback, we have added new experiments and analyses that we believe substantially strengthen the paper. We would greatly appreciate it if you could let us know whether these updates resolve your remaining concerns.

Thank you again for your time and effort throughout the reviewing process.

Reviewer_ygMgOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether low-rank pre-training methods produce comparable solutions and adaptation behavior beyond validation perplexity. It introduces BASIN, a diagnostic framework combining checkpoint interpolation, activation similarity, weight/update spectra, and directional loss-landscape measurements. The study compares GaLore, Fira, CoLA, SLTrain, and ReLoRA against full-rank training on C4 at 60M, 130M, and 350M parameters, with three seeds. It additionally evaluates zero-shot performance, task and instruction fine-tuning, and German continual pre-training with and without replay. The reported results reveal method-dependent representations, spectra, and adaptation/forgetting trade-offs. Two exploratory interventions at 130M illustrate how these diagnostics can guide modifications to Fira and SLTrain.

Critical strengths
  1. A valuable comparison across low-rank method families. Studying five methods at three scales with three seeds provides substantially more useful evidence than isolated perplexity comparisons. Combining activation, spectral, and geometric measurements reveals that similarity to full-rank training depends on which property is measured.

  2. Evaluation follows the pre-trained models into adaptation. Task SFT, LaMini SFT, and German CPT examine the consequences of low-rank pre-training rather than merely reusing low-rank methods as fine-tuning algorithms. The reported reversal in Fira’s retention behavior between SFT and CPT is particularly interesting.

Critical weaknesses
  1. The geometric conclusions are stronger than the measurements establish. A high loss along one straight interpolation path establishes a barrier on that path, but does not establish that two solutions occupy disconnected basins. Parameter alignment and initialization also matter

  2. Git Re-Basin illustrates why unaligned interpolation can be misleading. Moreover, Section 3.1 subtracts the mean endpoint loss, whereas Appendix G describes excess above the endpoint chord. These quantities differ for unequal endpoint losses, and the former can be positive even for a monotonic linear loss profile.

  3. Cross-method sharpness comparisons also need a clearer account of perturbation normalization, particularly across different parameterizations. These issues affect the central interpretation of BASIN, although they need not invalidate its descriptive measurements.

  4. The post-training evidence needs clarification before it supports the proposed explanation.

Questions for authors
  1. Task-accuracy consistency: How is the task accuracy in Figure 1 aggregated?
  2. Why does the 350M full-rank curve start around 13–14%, while Table 7 gives approximately 39.1% when averaging the five task-SFT benchmarks?
  3. Are the step-zero checkpoints, evaluation splits, scoring metrics, and harness settings identical, as Appendix B.1 indicates?
  4. Which barrier formula was actually used for CCBH, IMBH, and Figure 7: subtraction of the mean endpoint loss or subtraction of the interpolation chord?
  5. Do the main rankings survive a consistent treatment of unequal endpoint losses?
  6. Were cross-method checkpoints initialized from matched materialized weights and compared against within-method, cross-seed barriers?
  7. In which parameter space are random perturbations applied for each method, how are their magnitudes normalized, and how is CoLA handled given its nonlinearity?
  8. Is there evidence that the principal sharpness rankings are robust to a reasonable relative or layerwise normalization?
3 comments
AuthorsOpenReview ↗

R1 response

Thank you for your valuable feedback.

R 1.1 The geometric conclusions are stronger than the measurements establish.

We agree that the linear barrier shows only that two solutions are not linearly mode connected. We removed "each method converges to a distinct basin" from the abstract, introduction, Section 5.2, and other sections of the paper, and now state results in terms of linear mode connectivity (LMC) (See common response 2.4). We also addressed both factors the reviewer mentions:

  • Alignment: all inter-method barriers are now computed after weight-matching alignment (addressed in Common response 2).
  • Initialization: we report shared-init and independent-init pairs separately, each against a matched full-rank reference (addressed in Common response 2).

R 1.2 Git Re-Basin illustrates why unaligned interpolation can be misleading.

See Common Response 2 (Barriers: permutation alignment, formula, and calibration): all barriers are now computed after weight-matching alignment (Common Response section 2.1) and with the chord formula throughout (Common Response section 2.2).

R1.3 Cross-method sharpness comparisons also need a clearer account of perturbation normalization…

See “Common response section 1. Normalized perturbations”.

R 1.4 The post-training evidence needs clarification before it supports the proposed explanation.

We clarified the post-training evidence in two ways:

  • Link BASIN to post-training behavior: we added a quantitative correlation and prediction study, and kept the interpolation analysis as a qualitative illustration (see Common Response section 3).
  • Figure 1: we explain how its task accuracy is aggregated (see Q1.1–Q1.3 below).

Questions

Q 1.1 Task-accuracy consistency: How is the task accuracy in Figure 1 aggregated?

Figure 1 plots chance-normalized accuracy, (acc − chance)/(1 − chance), averaged over the five task-SFT benchmarks (ARC-Easy, HellaSwag, LogiQA2, PIQA, SWAG) and then over three SFT seeds. Normalization keeps PIQA (chance 0.5) from dominating the tasks with chance 0.25. We will add this to the experimental setup and to the Figure 1 description.

Q 1.2 Why does the 350M full-rank curve start around 13–14%, while Table 7 gives approximately 39.1% when averaging the five task-SFT benchmarks?

Chance normalization is applied per task, since PIQA is 2-way (chance 50%) and the other four tasks are 4-way (chance 25%): ARC-Easy (31.62 − 25)/75 = 8.83%, HellaSwag (32.47 − 25)/75 = 9.96%, LogiQA2 (25.93 − 25)/75 = 1.24%, PIQA (63.15 − 50)/50 = 26.3%, and SWAG (42.37 − 25)/75 = 23.16%, which average to 13.89%, against a raw average of 39.1%.

Q1.3 Are the step-zero checkpoints, evaluation splits, scoring metrics, and harness settings identical, as Appendix B.1 indicates?

Yes. Step zero in Figure 1 is the zero-shot evaluation of the same pre-trained checkpoint, with the same lm-evaluation-harness version, evaluation split, metric, and 0-shot setting as Table 7. The difference is the chance normalization.

Q1.4 Which barrier formula was actually used for CCBH, IMBH, and Figure 7: subtraction of the mean endpoint loss or subtraction of the interpolation chord?

The submitted CCBH and IMBH used the mean-endpoint formula, while Figure 7 used the chord, since its endpoints (pre- and post-training) have very different losses. In the revised draft uploaded, all barriers use the chord formula (Common Response 2.2).

Q 1.5 Do the main rankings survive a consistent treatment of unequal endpoint losses?

  • IMBH: yes. The two formulas differ by at most 0.15 nats on any pair, against barriers of 2–8 nats at the final checkpoint, rankings change only for two near-ties (gaps ≤ 0.06).
  • CCBH: no. Under the chord formula, consecutive checkpoints are LMC for all methods except ReLoRA and one CoLA run, so we withdrew the submitted CCBH ranking (Common Response 2.3).

Q 1.6 Were cross-method checkpoints initialized from matched materialized weights and compared against within-method, cross-seed barriers?

Yes, for materialized weights: ReLoRA and SLTrain are aligned and interpolated in their materialized dense weights, and CoLA has no dense equivalent and is excluded. In the revision (see Common Response 2), we compare each low-rank method only against full-rank, not low-rank methods with each other, so the cross-seed baseline is full-rank's own i.e. two full-rank runs from different seeds for independent-init pairs, and full-rank retrained from the same initialization for shared-init pairs. We report the two kinds of pairs separately, each against its matched reference.

Q 1.7 In which parameter space are random perturbations applied for each method, how are their magnitudes normalized, and how is CoLA handled given its nonlinearity?

We have answered the question in “Common response section 1. Normalized perturbations”.

Q 1.8 Is there evidence that the principal sharpness rankings are robust to a reasonable relative or layerwise normalization?

Partly (see Common Response 1 on Normalized perturbations). Under per-matrix normalization, ReLoRA's extreme raw sharpness disappears (0.93–1.18× full-rank), full-rank is among the flattest methods at every scale, and FiRA moves from flattest to among sharpest. All cross-method sharpness claims now use the normalized perturbation.

We hope the additional results, new sections, and clarifications address your concerns. If our responses have sufficiently addressed your feedback, we kindly ask you to consider revisiting your score.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer ygMg,

As the discussion period ends today (AoE), we wanted to kindly follow up on our earlier responses. Based on your valuable feedback, we have added new experiments and analyses that we believe substantially strengthen the paper. We would greatly appreciate it if you could let us know whether these updates resolve your remaining concerns.

Thank you again for your time and effort throughout the reviewing process.

Comments on the paper

11 comments
AuthorsOpenReview ↗

Common response

The reviews share four main concerns. We ran new experiments for each:

  1. Sharpness was measured with unnormalized perturbations (ygMg, sWcj, yqT2).
  2. Barriers were not aligned for permutation symmetry, used an inconsistent formula, and were read as evidence of distinct basins (ygMg, sWcj, yqT2).
  3. Post-training link: there was no quantitative link between BASIN metrics and post-training behavior (ygMg, sWcj, yqT2).
  4. Case studies used a single seed (sWcj, yqT2).

The revised paper already includes the new sharpness and barrier results (Sections 1 and 2) in the main text and figures. Additional tables from these sections, along with Sections 3 and 4, will be added in the camera-ready version.

AuthorsOpenReview ↗

Common Response 1: Normalized perturbations

1. Normalized perturbations (ygMg, sWcj, yqT2)

In the initial submission, we applied raw random directions in each method's native parameter space i.e. the low-rank factors for ReLoRA, SLTrain and CoLA, and the dense weights for full-rank, GaLore and FiRA. We chose this to measure sharpness in the space the low-rank methods actually update, and it gave one protocol for all six methods, including CoLA, whose factors are separated by a nonlinearity and cannot be merged into a single dense W. As the reviewers point out, however, sharpness measured at different weight scales and different parameterizations is not directly comparable across methods. We therefore conduct additional loss-landscape analysis under weight-normalized perturbations and on materialized weights.

1a. Normalized perturbations

We repeated the loss-landscape experiment with each direction rescaled per matrix to that matrix's norm, δ^W=δW∥δW∥F∥˙W∥F\hat{\delta}_W = \frac{\delta_W}{\|\delta_W\|_F}\.\|W\|_F, so the step size is relative to each matrix's scale and the weight-norm confound is removed.

The table below shows normalized sharpness at the final checkpoint (mean ± s.d. over 3 seeds; lower = flatter; ReLoRA and SLTrain are materialized to dense matrices):

ModelFull-rankReLoRASLTrainGaLoreFiRACoLA (native)
60M4.38 ± 0.034.90 ± 0.085.09 ± 0.049.04 ± 0.198.15 ± 0.0910.52 ± 0.79
130M4.94 ± 0.095.85 ± 0.1610.75 ± 0.5610.76 ± 0.3111.47 ± 0.338.72 ± 0.29
350M5.84 ± 0.295.42 ± 0.098.50 ± 0.2213.00 ± 0.0616.22 ± 0.1711.09 ± 0.65

Result. Under normalized perturbations, full-rank is consistently among the flattest methods (flattest at 60M and 130M, close second to ReLoRA at 350M). FiRA and GaLore are markedly sharper at every scale, and the gap grows with scale. FiRA goes from 1.9× full-rank's sharpness at 60M to 2.8× at 350M, becoming the sharpest method from 130M on (CoLA is sharpest at 60M).

1b. Materialized weights.

For all three perturbation protocols (raw, normalized and top-1 PCA), we also perturb ReLoRA and SLTrain in their materialized dense weights (W+sBA and sBA+S), so every method except CoLA is perturbed in the same space. CoLA remains in its native parameterization and is marked as such in the results.

Effect of materialization (final checkpoint, mean over 3 seeds; native → materialized):

Full-rank (reference)ReLoRASLTrain
Normalized60M4.384.90 → 4.905.85 → 5.09
130M4.945.83 → 5.8510.87 → 10.75
350M5.845.42 → 5.429.13 → 8.50
Raw60M0.01170.0781 → 0.07710.0069 → 0.0802
130M0.00910.0799 → 0.07990.0037 → 0.0894
350M0.00890.136 → 0.1360.0062 → 0.241
Top-1 PCA60M0.00300.0048 → 0.00480.0030 → 0.0023
130M0.00170.0058 → 0.00580.0018 → 0.0028
350M0.00220.0099 → 0.00980.0024 → 0.0018

Result. Under normalized and top-1 PCA perturbations, materialization changes sharpness only slightly: ReLoRA by at most 2% under every protocol, and SLTrain by 1–13% under normalized perturbations and by 0.0006–0.0010 in absolute terms along top-1 PCA. No ordering changes beyond seed noise, except that SLTrain moves just below full-rank along top-1 PCA at 350M. At every scale, full-rank and SLTrain remain among the flattest methods along top-1 PCA. Under raw perturbations, ReLoRA is again unchanged at every scale, but SLTrain's sharpness rises 12×, 24× and 39× at 60M, 130M and 350M once materialized (from 0.004–0.007 to 0.080–0.241), making ReLoRA and SLTrain the two sharpest methods at every scale. We therefore withdraw the claim that SLTrain is flatter than full-rank, and every cross-method claim in the revision uses normalized, materialized sharpness.

Figures. The revised Figure 6 in the uploaded draft shows sharpness over the training trajectory at 350M under all three protocols (normalized, raw and top-1 PCA), with ReLoRA and SLTrain shown both natively and materialized. We added the corresponding figures for 60M and 130M in the appendix.

Revision. We have revised the results in Section 5.2.3 accordingly. Our main takeaway, that low-rank methods converge to solutions whose flatness differs from that of full-rank methods, still holds under all three protocols. Specifically, under normalized perturbations, full-rank reaches flatter solutions than most low-rank methods.

AuthorsOpenReview ↗

Common response 2: Barriers

2. Barriers: permutation alignment, formula, and calibration (ygMg, sWcj, yqT2)

The reviewers raised four related points on Barrier lens: the barriers were not aligned for permutation symmetry (ygMg, sWcj, yqT2), the formula was inconsistent (ygMg), there was no full-rank seed-to-seed reference (sWcj, yqT2), and initialization sharing was not reported (ygMg, sWcj). We address each below.

2.1 Permutation alignment

  • Method. Before interpolating, we align the two models under comparison with weight matching (Ainsworth et al., 2023), using the codebase of Theus et al. (2025), which extends weight matching to transformers. We use the data-free weight matching method and adapted it to the LLaMA architecture used in our study. It permutes the residual stream, MLP units, and attention heads. Every alignment is checked to leave the model's loss unchanged (tolerance 5e-3).

  • Scope. All full-rank ↔ low-rank pairs at 60M, 130M and 350M, ReLoRA and SLTrain are aligned and interpolated in their materialized dense weights. CoLA has no dense equivalent and is excluded.

2.2 Barrier formula

We redid all barrier experiments with weight-matching alignment (2.1 Permutation alignment) and used a single formula consistently throughout the draft. In the submission, the CCBH and IMBH barriers subtracted the mean endpoint loss, while Figure 7 and Appendix G defined the barrier with the chord, an inconsistency ygMg pointed out. All barriers now use the chord formula of Entezari et al. (2022):

BH(θ1,θ2)=max⁡β∈(0,1)[L(βθ1+(1−β)θ2)−(βL(θ1)+(1−β)L(θ2))].\mathrm{BH}(\theta_1,\theta_2) = \max_{\beta\in(0,1)}\Big[\mathcal{L}\big(\beta\theta_1+(1-\beta)\theta_2\big) - \big(\beta\mathcal{L}(\theta_1)+(1-\beta)\mathcal{L}(\theta_2)\big)\Big].

2.3 Calibration and initialization: matched full-rank references

We compute two types of full-rank references, one for each kind of low-rank ↔ full-rank initialization:

  • Independent initialization (60M, 130M, 350M): the barrier between two full-rank runs trained from different seeds, i.e. different initializations (3 pairs per size). This is the reference for low-rank ↔ full-rank pairs from different seeds (6 pairs per method). We report the mean over pairs.

  • Shared initialization (130M, one seed): this is the reference for low-rank ↔ full-rank pairs at the same seed, which start from identical weights (FiRA, GaLore, ReLoRA). SLTrain never shares the full-rank initialization, because it is initialized in its own sparse + low-rank form. We retrained full-rank 130M twice from the original run's initialization:

    • Same initialization, same data order: the barrier between the two full-rank runs is 0.
    • Same initialization, different data-order seed: we use this barrier as the data-order reference for shared-init pairs.
    • 350M: we did not rerun full-rank at this scale. Since the code is deterministic, we assume from the 130M rerun that the same-initialization, same-data-order barrier is also ~0. We do not have a different-data-order reference at 350M.

Results

The inter-method barrier height (IMBH) at the final checkpoint is given in the table below.

SettingFR ↔ FR referenceGaLoreFiRAReLoRASLTrain
130M, indep. init4.80 ± 0.016.26 ± 0.235.53 ± 0.165.53 ± 0.325.95 ± 0.35
130M, shared init≈0 (same data) / 1.21 (different data order)*4.36 ± 0.433.20 ± 0.264.57 ± 0.12n/a
350M, indep. init5.46 ± 0.067.21 ± 0.476.43 ± 0.185.27 ± 0.196.80 ± 0.22
350M, shared init≈0 (same data)†6.32 ± 0.405.30 ± 0.665.06 ± 0.17n/a

*One seed. †Assumed from the 130M rerun (same deterministic code).

The revised Figure 3 in the uploaded draft shows IMBH across the training trajectory.

  • Alignment does not remove the barriers. It lowers the full-rank seed-to-seed barrier by 14–18% at the final checkpoint. Two full-rank seeds are not linearly mode connected (LMC) even after alignment, and neither is any low-rank/full-rank pair.

  • Independent init. GaLore, FiRA and SLTrain stay above the full-rank reference at every checkpoint and scale (1.10–1.53×). At the final checkpoint, GaLore is highest (1.31× / 1.32× at 130M / 350M), then SLTrain (1.24× / 1.25×) and FiRA (1.15× / 1.18×). ReLoRA stays near the reference (0.89–1.15×).

  • Shared init.

    • 130M: the same-data rerun gives ~0 and the data-order reference gives 1.21. The low-rank pairs are 2.6–3.8× above the data-order reference at the final checkpoint. ReLoRA is highest (4.57, 3.8×), then GaLore (4.36, 3.6×) and FiRA (3.20, 2.6×).
    • 350M: we assume a same-data rerun also gives ~0, since training uses the same deterministic code. GaLore (6.32) exceeds the barrier between two independent full-rank seeds (5.46).

Takeaway. Neither full-rank/full-rank nor full-rank/low-rank pairs are linearly mode connected, and at 350M GaLore is further from full-rank than another full-rank seed is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.