- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies intruder dimensions in LoRA fine-tuning, these are new singular directions that are almost orthogonal to those of the pretrained weights and have been linked to forgetting.
The central contribution is a prediction of the LoRA update strength at which these intruders start to appear in each layer, based on the pretrained weight spectrum and the LoRA update. The authors evaluate this prediction across several model architectures and show that it estimates the empirical threshold reasonably well in 82% of layers.
They also show that going beyond this threshold is related to forgetting. Based on this, they propose a simple threshold-based method to reduce forgetting, which shows improvements in some of their experiments.
Critical strengths
-
The main idea is interesting. It seems useful and important for forgetting to predict when intruder dimensions appear in each layer, and the connection to random matrix theory gives a reasonable theoretical explanation for this behavior.
-
The experimental part is good. The authors tested the prediction on many adapters and different model architectures, including Transformers, state-space models, and MoEs. which is exactly what I would want to see to back up their arguments.
-
The norm-matched ablations are interesting and can show that this has practical consequences.
Critical weaknesses
-
The improvement over simpler spectral predictors is marginal. In Appendix C, the authors show that a fitted multiple of the bulk spectral edge performs almost identically to the proposed prediction. This makes me question how much the D-transform actually contributes.
-
The relation between the threshold and forgetting needs better evidence. For example, the large difference in the norm matched experiment (270x) is from comparing 221 supercritical layers against 3 subcritical ones. In the case of a more balanced experiment, the gap shrinks to around 1.5x.
-
The practical benefit of the proposed method is limited. Table 2 shows that simply reducing the learning rate can give better results.
Questions for authors
-
Given that a simple scaled spectral-edge baseline performs almost as well as the proposed method, what is the main advantage of using the D-transform? Can the authors better isolate its contribution?
-
Can the authors provide a more balanced norm-matched intervention with a similar number of supercritical and subcritical layers? I think this would give stronger evidence for the connection to forgetting.
-
Can the authors compare the spike-budget rule against uniform scaling and learning-rate tuning at matched task performance? It is not clear whether the layer-wise allocation provides a meaningful advantage.
2 comments
(part 2 of 2)
At equal count and norm the predicted-supercritical set forgets more than every random set in 49 of 50 draws over the ten balanced splits. Against the subcritical set the differential grows with s on every Mistral adapter; against the random sets it rises only on the default adapter and falls on the cap-rule seeds, because at higher s the random sets themselves contain more matrices past threshold. On the original Qwen adapter the matched asymmetry is 0.99× at s = 2 (45/151) and 1.56× at s = 4 (143/53); the submission reported only the second, and Appendix J now reports both. The two norm controls bracket the result: the largest-norm set, scaled down by matching to 0.3–0.8 of s, forgets less than the random mean at every strength; the smallest-norm set, scaled up 1.4–4× past most of its thresholds, forgets most of all matched conditions (+0.5 to +508 PPL). At the 221/3 split the random sets forget as much as the supercritical set, which is why that ratio is uninformative.
Limits, stated in the appendix. The crossed count is coarse: at s = 1 the smallest-norm set crosses fewer thresholds than the supercritical set and forgets more, because matching pushes it much further past them (the margin effect the reviewer notes). Below threshold an update still perturbs the model: at s = 1 the cap-rule adapter's 151–157 subcritical matrices carry 3.2–3.4× the norm of the supercritical ones (2.2× per matrix), cross no threshold, and add +0.97 to +1.11 PPL against +0.25. And on Qwen, which deploys below half of its thresholds, the equal-norm asymmetry is small (its matched subcritical set at s = 3 sits at a median 0.79 of threshold and forgets +1.11 against +1.16 for the supercritical set at 1.46×). The law's claim, and the revised sentence in §4, is the asymmetry at equal norm: on the balanced splits whose matched subcritical set stays below threshold, 1.5–5.5× on Mistral and Llama and 1.0–1.1× on Qwen, rather than 270× (at s = 1.5 matching pushes most of the cap-rule seeds' subcritical sets past threshold, the 6.8× included); the abstract now reports these ranges.
Reply to W3 and Q3 (the rule against uniform scaling and learning-rate tuning at matched task performance). Short answer: the reviewer's reading of Table 2 of the submission (Table 3 of the revision) is right and holds generally; the rule is not the best knob. Figure 4 plots every Mistral knob run of the paper (six learning rates, six ranks, the α patterns, DoRA, rsLoRA, post-hoc downscaling) as forgetting against task perplexity at deployment. At equal task perplexity (≈ 1.33): learning rate ÷4 +0.55 PPL, rank 64 +0.79, learning rate ÷2 +1.12, cap rule +1.59, uncapped pattern +1.77, matched uniform α +2.11. Per-layer allocation adds 16% over a uniform scale of the same mean, which GSM8K (n = 500) and HellaSwag (n = 2,000) do not resolve. What the thresholds add is not a better knob but (i) the diagnosis: Mistral deploys at 1.5× its median threshold and needs shrinking; Qwen deploys below half of its thresholds and gains at most 0.2 PPL and no resolvable accuracy from halving the learning rate (Table 3: +0.35 → +0.14 PPL, GSM8K 83.8 → 83.0); (ii) the dose: 0.5–0.7 of threshold, which halving the learning rate achieves (the law puts 55% of layers past threshold before and 13% after; measured intruder fractions 91% and 27%); (iii) the layer attribution tested in Q2. Figure 4b shows forgetting rising with the deployed spike over threshold along every knob series, with family offsets that follow the number of directions near the edge. The paragraph after "Spike-budgeted LoRA" is rewritten in these terms, with the learning rate as the cheaper implementation of the same dose.
We would be grateful if the reviewer would weigh the revised §4 and Appendix K in their assessment; we will answer any further question during the discussion period.
(part 1 of 2)
We thanks for your careful reviewing Summary. W1/Q1: the contribution of the transform is isolated with paired-bootstrap intervals (Table 2, p. 5; Appendix C). W2/Q2: the balanced norm-matched experiment with random same-count controls is in Appendix K (Table 11, Figure 7, pp. 21–23): 1.7–6.8× on Mistral and 1.05–1.14× on Qwen against the subcritical set, 1.4–3.2× and 1.1–1.4× against random sets of the same count and norm, rather than 270×; §4 now says so next to the 221/3 split. W3/Q3: every Mistral knob run is compared at matched task perplexity (Figure 4, p. 9; Table 10), and §4 is rewritten: the rule is not the best knob. Revised passages are in blue in the uploaded PDF.
Reply to W1 and Q1 (what the D-transform adds over a scaled spectral edge). Short answer: a small, consistent gain at the bulk edge, none at the full edge, and the scale without fitting from the geometric mean of the two edges. Each law variant is compared with the fitted multiple of its own edge by paired bootstrap over adapters (95% intervals; full tables in our response to Reviewer 2YnX, Table 2 and Appendix C):
ρ and AUC do not depend on the calibration constant. The bulk-edge transform also removes the error by module type that the fitted bulk-edge baseline shows (spread of per-type median errors 0.22 against 0.52, variance share 0.06 against 0.16; Table 9). The full edge predicts late where the top singular value is an isolated outlier, exactly where the spike escapes the bulk early. The scale: the geometric mean of the two edges needs no constant (98–100% of layers within ×2 at κ = 1, where the toy condition σ_max needs 0.65), and the bulk-edge transform has the most family-stable one (16% over 25 model groups, against 20–36% for the edge baselines); its value, 1.5, reflects the bulk edge lying below the effective edge, which the two pre-specified edges bracket. The revised §3 keeps the full edge as the pre-specified falsification test and recommends the bulk edge as the instrument.
Reply to W2 and Q2 (a balanced norm-matched intervention). Agreed: the 270× comes from a 221/3 split in the collapse regime, and the revision says so where it reports the number (§4, §7). We ran the experiment the question asks for, with the control it needs (Appendix K, Table 11, Figure 7; script
src/causal_probe2.py).Design. At a test strength s the law splits the adapted matrices into predicted supercritical (s* ≤ s) and subcritical. Conditions at matched injected Frobenius norm: the supercritical set at s; the subcritical set scaled to the same norm; five random sets with the supercritical count (and the same sets at s, unmatched); the same-count sets with the largest and with the smallest update norms. Each condition records how many of its matrices the applied scale pushes past their own predicted threshold ("crossed"), the quantity the law says forgetting should follow, random sets included. Adapters: the two cap-rule seeds of Appendix N, and Mistral r256 and Qwen r256 retrained at seed 0 because the originals were not kept (they reproduce Table 3 to within 0.3 PPL, 1.6 GSM8K points and 0.005 HellaSwag).
Results (ΔPPL on WikiText-2; ratios of the supercritical set to each control):