acceptodds
Under review as a conference paper at ICLR 2027

The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning

Abstract

LoRA fine-tuning creates intruder dimensions, new leading singular vectors of the updated weight matrix that are nearly orthogonal to every pretrained singular vector and that drive catastrophic forgetting, and no theory has predicted, layer by layer on measured spectra, when they appear. We derive a per-layer critical update strength from the measured spectrum of alone, through the rectangular spiked-deformation transform together with an exact secular-equation characterization of the updated spectrum, with no fitted parameters. In a pre-specified study of 18 adapters across seven base models, including a state-space model, a mixture of experts, and an encoder-decoder (9,840 layer scans), the law localizes the empirical threshold within a factor of two on 82% of layers, and a parameter-free combination of its two pre-specified edge evaluations reaches 98 %, confirmed out of sample at on six third-party adapters. The law separates intruder-bearing from intruder-free layers at deployment with a mean AUC of and places the WikiText-2 forgetting knee at Spearman over 16 adapters. Full fine-tuning disperses its update below the threshold of every layer, by a median of –, which accounts for the asymmetry between LoRA and full fine-tuning. Norm-matched interventions show that threshold-crossing layers carry to the forgetting of subcritical layers at equal injected norm, the larger figure where nearly every layer is supercritical, and a spike-budget rule derived from the thresholds, set from one SVD and one calibration run in place of a validation sweep, reduces forgetting by 66% on the most fragile model at no task cost and, applied stage by stage in a three-task continual-learning benchmark, removes over 70% of the accumulated WikiText-2 forgetting at higher task accuracy.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LoRA, parameter-efficient fine-tuning, intruder dimensions, random matrix theory, spiked matrix models, SVD, LLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Fod7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies intruder dimensions in LoRA fine-tuning, these are new singular directions that are almost orthogonal to those of the pretrained weights and have been linked to forgetting.

The central contribution is a prediction of the LoRA update strength at which these intruders start to appear in each layer, based on the pretrained weight spectrum and the LoRA update. The authors evaluate this prediction across several model architectures and show that it estimates the empirical threshold reasonably well in 82% of layers.

They also show that going beyond this threshold is related to forgetting. Based on this, they propose a simple threshold-based method to reduce forgetting, which shows improvements in some of their experiments.

Critical strengths
  1. The main idea is interesting. It seems useful and important for forgetting to predict when intruder dimensions appear in each layer, and the connection to random matrix theory gives a reasonable theoretical explanation for this behavior.

  2. The experimental part is good. The authors tested the prediction on many adapters and different model architectures, including Transformers, state-space models, and MoEs. which is exactly what I would want to see to back up their arguments.

  3. The norm-matched ablations are interesting and can show that this has practical consequences.

Critical weaknesses
  1. The improvement over simpler spectral predictors is marginal. In Appendix C, the authors show that a fitted multiple of the bulk spectral edge performs almost identically to the proposed prediction. This makes me question how much the D-transform actually contributes.

  2. The relation between the threshold and forgetting needs better evidence. For example, the large difference in the norm matched experiment (270x) is from comparing 221 supercritical layers against 3 subcritical ones. In the case of a more balanced experiment, the gap shrinks to around 1.5x.

  3. The practical benefit of the proposed method is limited. Table 2 shows that simply reducing the learning rate can give better results.

Questions for authors
  1. Given that a simple scaled spectral-edge baseline performs almost as well as the proposed method, what is the main advantage of using the D-transform? Can the authors better isolate its contribution?

  2. Can the authors provide a more balanced norm-matched intervention with a similar number of supercritical and subcritical layers? I think this would give stronger evidence for the connection to forgetting.

  3. Can the authors compare the spike-budget rule against uniform scaling and learning-rate tuning at matched task performance? It is not clear whether the layer-wise allocation provides a meaningful advantage.

2 comments
AuthorsOpenReview ↗

(part 1 of 2)

We thanks for your careful reviewing Summary. W1/Q1: the contribution of the transform is isolated with paired-bootstrap intervals (Table 2, p. 5; Appendix C). W2/Q2: the balanced norm-matched experiment with random same-count controls is in Appendix K (Table 11, Figure 7, pp. 21–23): 1.7–6.8× on Mistral and 1.05–1.14× on Qwen against the subcritical set, 1.4–3.2× and 1.1–1.4× against random sets of the same count and norm, rather than 270×; §4 now says so next to the 221/3 split. W3/Q3: every Mistral knob run is compared at matched task perplexity (Figure 4, p. 9; Table 10), and §4 is rewritten: the rule is not the best knob. Revised passages are in blue in the uploaded PDF.

Reply to W1 and Q1 (what the D-transform adds over a scaled spectral edge). Short answer: a small, consistent gain at the bulk edge, none at the full edge, and the scale without fitting from the geometric mean of the two edges. Each law variant is compared with the fitted multiple of its own edge by paired bootstrap over adapters (95% intervals; full tables in our response to Reviewer 2YnX, Table 2 and Appendix C):

law variant − fitted multiple of its edge53 out-of-sample adapters: ρAUC6 external adapters: ρAUC
bulk edge − κσ_bulk+0.041 [+0.031, +0.051] (better on 87%; family-clustered [+0.021, +0.060])+0.012 [+0.007, +0.018]+0.047 [+0.024, +0.072] (6/6)+0.026 [+0.004, +0.051] (4/5)
geometric mean − κ√(σ_max σ_bulk)−0.006 [−0.012, −0.000]−0.000 [−0.005, +0.005]−0.011 [−0.019, +0.000]+0.007 [−0.009, +0.027]
full edge (pre-specified primary) − κσ_max−0.063 [−0.079, −0.049]−0.020 [−0.027, −0.013]−0.084 [−0.121, −0.048]−0.034 [−0.063, −0.005]

ρ and AUC do not depend on the calibration constant. The bulk-edge transform also removes the error by module type that the fitted bulk-edge baseline shows (spread of per-type median errors 0.22 against 0.52, variance share 0.06 against 0.16; Table 9). The full edge predicts late where the top singular value is an isolated outlier, exactly where the spike escapes the bulk early. The scale: the geometric mean of the two edges needs no constant (98–100% of layers within ×2 at κ = 1, where the toy condition σ_max needs 0.65), and the bulk-edge transform has the most family-stable one (16% over 25 model groups, against 20–36% for the edge baselines); its value, 1.5, reflects the bulk edge lying below the effective edge, which the two pre-specified edges bracket. The revised §3 keeps the full edge as the pre-specified falsification test and recommends the bulk edge as the instrument.

Reply to W2 and Q2 (a balanced norm-matched intervention). Agreed: the 270× comes from a 221/3 split in the collapse regime, and the revision says so where it reports the number (§4, §7). We ran the experiment the question asks for, with the control it needs (Appendix K, Table 11, Figure 7; script src/causal_probe2.py).

Design. At a test strength s the law splits the adapted matrices into predicted supercritical (s* ≤ s) and subcritical. Conditions at matched injected Frobenius norm: the supercritical set at s; the subcritical set scaled to the same norm; five random sets with the supercritical count (and the same sets at s, unmatched); the same-count sets with the largest and with the smallest update norms. Each condition records how many of its matrices the applied scale pushes past their own predicted threshold ("crossed"), the quantity the law says forgetting should follow, random sets included. Adapters: the two cap-rule seeds of Appendix N, and Mistral r256 and Qwen r256 retrained at seed 0 because the originals were not kept (they reproduce Table 3 to within 0.3 PPL, 1.6 GSM8K points and 0.005 HellaSwag).

Results (ΔPPL on WikiText-2; ratios of the supercritical set to each control):

adapterssuper / subvs random, same count and norm (mean of 5; draws won)vs subcritical, norm-matchedSpearman(crossed, ΔPPL), 9 matched sets
Mistral cap rule, seed 11 / 1.25 / 1.567/157, 121/103, 161/633.2×, 3.1×, 2.8× (15/15)3.1×, 5.5×, 6.8×0.48, 0.87, 0.90
Mistral cap rule, seed 21 / 1.25 / 1.573/151, 122/102, 159/652.6×, 2.0×, 1.6× (15/15)2.2×, 2.9×, 2.9×0.57, 0.97, 0.92
Mistral r256, retrained0.75 / 192/132, 124/1001.4×, 2.3× (10/10)1.7×, 3.8×0.93, 0.93
Qwen r256, retrained2 / 346/150, 95/1011.4×, 1.1× (9/10)1.14×, 1.05×0.51, 0.46
Mistral r256, the 221/3 split2221/31.0× (2/5): random sets of 221 contain 218 supercritical matrices770× (3 matrices scaled 21×)0.37
AuthorsOpenReview ↗

(part 2 of 2)

At equal count and norm the predicted-supercritical set forgets more than every random set in 49 of 50 draws over the ten balanced splits. Against the subcritical set the differential grows with s on every Mistral adapter; against the random sets it rises only on the default adapter and falls on the cap-rule seeds, because at higher s the random sets themselves contain more matrices past threshold. On the original Qwen adapter the matched asymmetry is 0.99× at s = 2 (45/151) and 1.56× at s = 4 (143/53); the submission reported only the second, and Appendix J now reports both. The two norm controls bracket the result: the largest-norm set, scaled down by matching to 0.3–0.8 of s, forgets less than the random mean at every strength; the smallest-norm set, scaled up 1.4–4× past most of its thresholds, forgets most of all matched conditions (+0.5 to +508 PPL). At the 221/3 split the random sets forget as much as the supercritical set, which is why that ratio is uninformative.

Limits, stated in the appendix. The crossed count is coarse: at s = 1 the smallest-norm set crosses fewer thresholds than the supercritical set and forgets more, because matching pushes it much further past them (the margin effect the reviewer notes). Below threshold an update still perturbs the model: at s = 1 the cap-rule adapter's 151–157 subcritical matrices carry 3.2–3.4× the norm of the supercritical ones (2.2× per matrix), cross no threshold, and add +0.97 to +1.11 PPL against +0.25. And on Qwen, which deploys below half of its thresholds, the equal-norm asymmetry is small (its matched subcritical set at s = 3 sits at a median 0.79 of threshold and forgets +1.11 against +1.16 for the supercritical set at 1.46×). The law's claim, and the revised sentence in §4, is the asymmetry at equal norm: on the balanced splits whose matched subcritical set stays below threshold, 1.5–5.5× on Mistral and Llama and 1.0–1.1× on Qwen, rather than 270× (at s = 1.5 matching pushes most of the cap-rule seeds' subcritical sets past threshold, the 6.8× included); the abstract now reports these ranges.

Reply to W3 and Q3 (the rule against uniform scaling and learning-rate tuning at matched task performance). Short answer: the reviewer's reading of Table 2 of the submission (Table 3 of the revision) is right and holds generally; the rule is not the best knob. Figure 4 plots every Mistral knob run of the paper (six learning rates, six ranks, the α patterns, DoRA, rsLoRA, post-hoc downscaling) as forgetting against task perplexity at deployment. At equal task perplexity (≈ 1.33): learning rate ÷4 +0.55 PPL, rank 64 +0.79, learning rate ÷2 +1.12, cap rule +1.59, uncapped pattern +1.77, matched uniform α +2.11. Per-layer allocation adds 16% over a uniform scale of the same mean, which GSM8K (n = 500) and HellaSwag (n = 2,000) do not resolve. What the thresholds add is not a better knob but (i) the diagnosis: Mistral deploys at 1.5× its median threshold and needs shrinking; Qwen deploys below half of its thresholds and gains at most 0.2 PPL and no resolvable accuracy from halving the learning rate (Table 3: +0.35 → +0.14 PPL, GSM8K 83.8 → 83.0); (ii) the dose: 0.5–0.7 of threshold, which halving the learning rate achieves (the law puts 55% of layers past threshold before and 13% after; measured intruder fractions 91% and 27%); (iii) the layer attribution tested in Q2. Figure 4b shows forgetting rising with the deployed spike over threshold along every knob series, with family offsets that follow the number of directions near the edge. The paragraph after "Spike-budgeted LoRA" is rewritten in these terms, with the learning rate as the cheaper implementation of the same dose.

We would be grateful if the reviewer would weigh the revised §4 and Appendix K in their assessment; we will answer any further question during the discussion period.

Reviewer_GyLHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LoRA fine-tuning can introduce intruder dimensions, as shown by prior work. This paper studies at what LoRA update strength these intruder dimensions emerge. The authors derive a layer-specific threshold using the pretrained weight matrix's singular-value spectrum and the rectangular spiked-deformation transform.

Critical strengths

The paper addresses an interesting and meaningful problem by proposing a theoretically motivated approach to predicting the emergence of intruder dimensions during LoRA fine-tuning. The extensive experimental validation across different model architectures and LoRA configurations is a major strength. Studying such per-layer intruder thresholds is a useful contribution.

  1. The paper provides a principled, layer-specific approach to predicting when intruder dimensions emerge, connecting pretrained weight spectra to LoRA update strength using established spectral theory.

  2. The authors evaluate their predictions across diverse models, architectures, LoRA ranks, aka perform dense evaluations.

  3. Connecting intruders with forgetting is an interesting results, optimal lora strength reduces forgetting maintaining downstream performance.

Critical weaknesses
  1. Compared to simpler spectral baselines of uniform lora scaling, performance is not much different with the proposed per layer beta-cap rule.

  2. Secondly, for mistral and qwen, both approaches naively reducing the lora learinng rate reduced forgetting and achieves best accuracy.

  3. Limited practical advantage over direct SVD-based approaches: A simpler alternative to identifying safe LoRA update strengths is to directly measure the singular-vector alignment between pretrained and LoRA-tuned weights at different update scales. Although such repeated SVD scans require additional computation. Given that the proposed theory does not yield substantial improvements over simpler scaling baselines, the practical benefit of deriving a more complex theoretical framework for automatically determining per-layer LoRA scaling remains unclear.

Questions for authors

My main concern is the limited practical advantage of the proposed threshold-based scaling method over simpler alternatives, such as uniform LoRA scaling or reducing the learning rate. Can the authors comment on this?

Nevertheless, the theoretical motivation and potential implications for understanding forgetting make me lean toward a weak accept.

3 comments

We thanks for your careful reviewing Summary. W1: §4 (p. 7) now states that per-layer allocation adds 16% over a uniform scale of the same mean and that the accuracy metrics do not resolve it. W2 and Q: every Mistral knob run is compared at matched task perplexity (Figure 4, p. 9; Table 10); §4 and §7 are rewritten: the rule is not the best knob, the law gives the diagnosis and the dose. W3: the direct alignment measurement is the s-scan of §3, whose outcome the law predicts (Table 2, p. 5). Revised passages are in blue in the uploaded PDF.

Reply to W1 (uniform scaling performs about as well as the per-layer cap). Agreed. The uniform control matched to the mean strength of the pattern gives +2.11 PPL against +1.77 for the uncapped pattern and +1.59 for the rule, a 16% difference that GSM8K (n = 500) and HellaSwag (n = 2,000) do not resolve. The gain of the rule is the global scale set from the spectra, not the allocation across layers; the revised §4 says this where the rule is introduced, not in an appendix.

Reply to W2 and Q (a lower learning rate reduces forgetting and gives the best accuracy). Also agreed, and the revision shows it is general. Figure 4 plots every Mistral knob run of the paper (six learning rates, six ranks, the α patterns, DoRA, rsLoRA, post-hoc downscaling) as forgetting against task perplexity at deployment; at equal task perplexity (≈ 1.33): learning rate ÷4 +0.55 PPL, rank 64 +0.79, learning rate ÷2 +1.12, cap rule +1.59, uniform control +2.11. On Qwen, halving the learning rate changes little (+0.35 → +0.14 PPL; GSM8K 83.8 → 83.0; HellaSwag 0.598 → 0.596), as the law says of a model that deploys below half of its median threshold. The practical content of the law is therefore (i) which models need shrinking at all, from one SVD per layer and the spike scale of the recipe (Mistral at 1.5× its median threshold: yes; Qwen, OLMo-2: no); (ii) how much: to 0.5–0.7 of threshold, which halving the learning rate achieves (the law puts 55% of Mistral's layers past threshold before and 13% after; measured intruder fractions 91% and 27%); (iii) a common yardstick: along every knob series forgetting rises with the deployed spike over the per-layer threshold (Figure 4b), with family offsets that follow the number of directions near the edge. A learning-rate sweep finds the operating point; the law says where it is and whether it is needed. §4 and the limitations are rewritten in these terms.

Reply to W3 (directly measuring singular-vector alignment at different update scales). That measurement is the s-scan of §3: for every adapted matrix we rebuild W + sγBA on a grid of 26 strengths and read off the alignment of the leading singular vectors with the pretrained basis (9,840 scans in the first batch, 29,170 in the second). The law predicts its outcome from one number per layer, before the scan: it separates the layers that carry an intruder at deployment from those that do not at a mean AUC of 0.88 (full edge) to 0.97 (bulk edge) on the 53 held-out adapters, and with the adapter's spike scale its predicted crossing fraction ranks the deployment forgetting of 77 non-collapsed probed adapters at Spearman 0.74, the same as the measured intruder fraction from the scans (0.73), where the update magnitude alone gives 0.11. The scan remains the measurement; the law replaces it where one wants the answer first, and it makes the layer-wise attribution of Appendix K possible without a scan per candidate split.

On the transform against the simpler scaling baselines (also raised in W1). The new paired-bootstrap comparison (Table 2; Appendix C): at the bulk edge the transform ranks layers better than a fitted multiple of the edge by +0.041 [+0.031, +0.051] in Spearman ρ on 53 out-of-sample adapters (better on 87%), improves the deployment AUC by +0.012 [+0.007, +0.018] and removes the error by module type that the fitted baseline shows; at the full edge it does not. Its scale needs no constant for the geometric mean of the two edges (98–100% of layers within ×2 at κ = 1), and at the bulk edge its constant is the most family-stable (16% against 20–36% for the edge baselines). The revised text claims exactly this and no more.

We would be grateful if the reviewer would consider the revised §4 and Figure 4 in their assessment; we will answer any further question during the discussion period.

Thanks for your answering my questions. I will keep my positive score.

Thank you for the follow-up and for keeping your positive assessment. Your questions about uniform scaling and the learning rate led to the matched-task-fit comparison of Figure 4 and to the rewritten §4.

Reviewer_2YnXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper develops an exact low-rank spectral computation and a D-transform-based threshold intended to predict when LoRA updates create new dominant singular directions, then evaluates the predictor across adapters and architectures and uses it to motivate update-strength controls. The supplement adds reproducibility material and additional checks but does not resolve the main concerns below.

Critical strengths
  1. The exact low-rank SVD/spectral computation is a useful technical contribution because it makes large layer-wise scans tractable while preserving the singular-value and overlap information needed for the paper's empirical analysis.
  2. The study design is careful. The authors fixed their pass/fail thresholds in code before measuring any layer, evaluated on a held-out second batch of 53 adapters and six third-party adapters, released per-layer data from which Table 3 can be reproduced, and reported in Appendix C the baseline comparison that weakens their own headline claim.
Critical weaknesses
  1. The paper does not show that its primary full-spectrum D-transform predicts intruding better than simpler baselines. The authors acknowledge in Appendix C (p. 14, Table 3) that “the shared factor carries much of the per-adapter ρ\rho” and that “the transform contributes little spectral-shape information beyond the location of the edge.” Fitted σmax\sigma_{max} and σbulk\sigma_{bulk} rules are as good as or better then the primary transform on every reported aggregate metric across the three batches. Independent recomputation via a coding agent using the supplied Table 3 CSV inputs in the supplementary material found that the bulk-edge D-transform is stronger than fitted σbulk\sigma_{bulk} externally in mean rank correlation and AUC (0.939/0.986 versus 0.891/0.960) but weaker in hit@×2\text{hit}@\times2 (0.917 versus 0.988), so there are settings under which a D-transform approach can be superior, but the evaluation and analysis are insufficient.
  2. The abstract and main text compress exact computation, model assumptions, empirical proxies, and interventions into a single “law,” while qualifications that materially bound the headlines are largely deferred to appendices or compressed into the Section 7 limitations paragraph. This makes the argument substantially harder to follow and assess than its technical content requires.
  3. The writing in this paper is dense, jargon-laden, and sloppy. Terms are introduced without definition (e.g. "bulk" on p. 3, which is then used 33 more times). A paper is a scholarly artifact intended to motivate, educate, persuade, report, and synthesize. There's a lot of reporting here and a deficit of the other four qualities.
Questions for authors

With respect to Appendix C: on the frozen second batch and the six external adapters, do the D-transform variants produce statistically supported gains over σmax and σbulk after each method receives one multiplicative calibration factor? For each evaluation set, please report hit@×2\text{hit}@\times2, mean per-adapter ρ\rho, and mean AUC, along with a 95% paired bootstrap confidence interval over adapters for each D-transform-versus-baseline difference. For the six external adapters, where a confidence interval based on six observations is weak evidence, please also report every adapter-level difference and the positive and negative class counts used to calculate AUC. Please also include appendix plots equivalent to Figure 1 for the calibrated σmax\sigma_{max} and σbulk\sigma_{bulk} baselines. Use the same held-out layers, axes, identity line, factor-of-two bands, and frozen calibration factors as the D-transform plots, and show the methods side by side. This would reveal whether the D-transform reduces systematic prediction errors that the aggregate metrics conceal.

5 comments
AuthorsOpenReview ↗

(part 1 of 2)

Thanks for your careful reviewing. Summary. W1 and the Question: the requested statistics (paired bootstrap under both calibration regimes, adapter-level values and class counts, Figure-1-style plots, systematic errors) are below and in Table 2 (p. 5) and Appendix C (Tables 5–9, Figure 5); the main text now states what the transform adds and what it does not. W2: the qualifications stand next to the claims in §3, §4 and §7, and the abstract is shorter. W3: a "Terms" paragraph opens §2 and the introduction says in what sense we use "law". Revised passages are in blue in the uploaded PDF.

Reply to W1 and the Question (the transform against a fitted multiple of the edge). Short answer: for the pre-specified primary, the full edge, the reviewer is right, a calibrated σ_max does as well or better; at the bulk edge the transform keeps a small, consistent advantage in ranking and in deployment AUC, with intervals clear of zero on both held-out sets; in every variant its main content is the scale, which needs no fitting. We thank the reviewer for recomputing Table 3 of the submission (Table 4 of the revision, p. 15) from the released data: the three numbers reported (0.939/0.986 against 0.891/0.960 in mean ρ and AUC, 0.917 against 0.988 in hit@×2) are our six external adapters to the third decimal, and the statistics below are built on that table (scripts/ext/rebuttal_baselines_ci.py in the updated supplementary code).

Setup. Every predictor is s* = κ X_ℓ / (γ σ₁(BA)). Regime (i), as in the submission: law variants at κ = 1, baselines with κ fitted once on the pooled first-batch layers. Regime (ii): every method with one κ fitted the same way (law: full edge 1.00, bulk edge 1.57, geometric mean 1.18; σ_max 0.65, σ_bulk 0.97, √(σ_max σ_bulk) 0.78). ρ and AUC are invariant to κ, so the regimes differ only in hit@×2. Entries are law variant minus baseline, mean over adapters, with 95% paired-bootstrap intervals over adapters (20,000 resamples); "better on" is the fraction of adapters where the law variant is strictly better.

Frozen second batch, 53 adapters (out of sample for every κ):

comparisonhit@×2, κ=1 vs fittedhit@×2, one κ eachρ (better on)AUC, 45 adapters with both classes (better on)
bulk-edge transform vs κσ_bulk0.92 vs 0.990.995 vs 0.988, +0.007 [+0.005, +0.010]0.907 vs 0.867, +0.041 [+0.031, +0.051] (87%)0.968 vs 0.956, +0.012 [+0.007, +0.018] (76%)
geometric mean vs κ√(σ_max σ_bulk)1.00 vs 1.000.992 vs 0.995, −0.003 [−0.006, −0.001]0.869 vs 0.875, −0.006 [−0.012, −0.000]0.950 vs 0.950, −0.000 [−0.005, +0.005]
full-edge transform (primary) vs κσ_max0.84 vs 0.960.840 vs 0.962, −0.122 [−0.137, −0.107]0.693 vs 0.756, −0.063 [−0.079, −0.049]0.876 vs 0.896, −0.020 [−0.027, −0.013]

Resampling the 17 base-model families instead of adapters leaves the bulk-edge intervals clear of zero (ρ [+0.021, +0.060], AUC [+0.005, +0.018]).

Six external adapters:

comparisonhit@×2, κ=1hit@×2, one κ eachρAUC, 5 adapters with both classes
bulk-edge transform vs κσ_bulk0.917 vs 0.9880.997 vs 0.988, +0.009 [+0.004, +0.014], 6/60.939 vs 0.891, +0.047 [+0.024, +0.072], 6/60.986 vs 0.960, +0.026 [+0.004, +0.051], 4/5
geometric mean vs κ√(σ_max σ_bulk)1.00 vs 1.00−0.004 [−0.009, +0.001]−0.011 [−0.019, +0.000]+0.007 [−0.009, +0.027]
full-edge transform vs κσ_max0.81 vs 0.96−0.143 [−0.152, −0.134]−0.084 [−0.121, −0.048]−0.034 [−0.063, −0.005]

Adapter-level values (hit@×2 with one κ per method / ρ / AUC) and class counts (layers with / without an intruder at s = 1):

adapterlayerspos / negbulk-edge transformκσ_bulkdifference (hit / ρ / AUC)
DIA-MVP Qwen2.5-7B1961 / 1951.000 / 0.928 / 1.0000.980 / 0.833 / 0.954+0.020 / +0.095 / +0.046
medical QA Mistral-7B (QLoRA)22475 / 1490.991 / 0.897 / 0.9370.987 / 0.814 / 0.868+0.004 / +0.083 / +0.068
Bengali Qwen2.5-7B (QLoRA)1963 / 1930.995 / 0.945 / 1.0000.980 / 0.902 / 1.000+0.015 / +0.043 / 0.000
byzhsm Mistral-7B2241 / 2230.996 / 0.942 / 1.0000.991 / 0.912 / 0.996+0.004 / +0.029 / +0.004
chloeli Llama-3.1-8B baseline2240 / 2241.000 / 0.960 / —0.996 / 0.938 / —+0.004 / +0.022 / —
chloeli Llama-3.1-8B spec.2244 / 2201.000 / 0.961 / 0.9920.996 / 0.948 / 0.982+0.004 / +0.012 / +0.010

Four of the five AUCs rest on one to four positive layers; the medical-QA adapter (75 / 149) is the only external AUC we would weight. The full-edge and σ_max rows are in Table 7 of the revision; the geometric-mean values are in baselines_ci.md of the supplementary code.

AuthorsOpenReview ↗

(part 2 of 2)

Figure-1-style plots. Figure 5 (p. 17) draws the eight panels of Figure 1 three times with the same layers, axes, identity line and factor-of-two bands: the law at the full edge, κσ_max (κ = 0.65) and κσ_bulk (κ = 0.97), both κ frozen from the first batch, hit rates in the titles.

Systematic errors the aggregates conceal. Table 9 (p. 19) splits the per-layer log₂ error by module type on the 53 held-out adapters. The full-edge law carries a type-dependent bias (per-type medians −0.55 to +0.28, η² 0.14): where the top singular value is an isolated outlier, the full edge predicts late on exactly the layers where the spike escapes the bulk early. At the bulk edge the transform removes it (spread of per-type medians 0.22, η² 0.058), while the edge itself carries a type bias of its own: κσ_bulk has the largest η² of any predictor (0.164, spread 0.52; κσ_max 0.065, 0.29; external adapters 0.019 and 0.09 against 0.330 and 0.63).

What we conclude, and now say in §3. (i) The transform is close to a fixed multiple of its edge (median within-adapter Spearman(θ̄, edge) 0.997 at the full edge and 0.982 at the bulk edge; median coefficient of variation of θ̄/edge 0.11–0.12). The reviewer's reading holds for the full edge, which we keep as the pre-specified falsification test; we recommend the bulk edge as the instrument. (ii) At the bulk edge the gains in ρ and AUC are small, but their intervals exclude zero on both held-out sets, also under family resampling, and the module-type bias is the smallest of any predictor. (iii) The geometric mean adds no ranking over √(σ_max σ_bulk); its advantage is the scale. (iv) A fitted baseline can match the hit@×2 ceiling but not exceed it; the geometric-mean law reaches it at κ = 1 (98–100% of layers within ×2 on all three sets), where the toy condition σ_max needs κ = 0.65. What the transform contributes beyond "a multiple of the edge" is a family-stable multiple: fitted per family over the 25 groups of Table 8, σ_max needs 0.52–0.71 (max/min 1.36), σ_bulk 0.92–1.19 (1.29), √(σ_max σ_bulk) 0.75–0.90 (1.20), the bulk-edge transform 1.45–1.68 (1.16); its value, 1.5, reflects the bulk edge lying below the effective edge, which the two pre-specified edges bracket. The abstract, the contribution statement and §3 are rewritten in these terms.

Reply to W2 (qualifications deferred to appendices). The 10th page is used for this: the comparison above enters §3 as Table 2 and a paragraph; the 221/3 split behind the 270×, the 16% gain of per-layer allocation over a matched uniform scale and the calibration run of the cap rule now stand next to the claims in §4, and the abstract reports the balanced splits instead of the 270×; §7 points to the balanced interventions (Appendix K). The abstract is shortened to one localization figure, one AUC range, one intervention ratio and the baseline result.

Reply to W3 (terms without definition). A "Terms" paragraph opens §2 (p. 2): spike (sγσ₁(BA); deployment is s = 1), edge (the top of the pretrained spectrum), full edge (σ₁), bulk edge (the largest singular value left after the gap test removes isolated outliers; the test is stated in Appendix A), supercritical (predicted s* ≤ the strength in question), evaluable layer (intruder inside the scan grid), hit@×2 and ρ. The introduction states the sense of "law": a per-layer prediction with no fitted constant, tested for falsification, not a claim of exactness, with §3 measuring how much of it the edge already carries.

On Informativeness. Over the toy condition λ > σ_max (Shuttleworth et al.) the paper adds: a per-layer critical strength whose scale needs no fitting when its two edges are combined (the fitted toy condition needs κ = 0.65, varying by 36% across families); the same formula holding on state-space, mixture-of-experts and encoder-decoder models (Table 8); a quantitative account of why full fine-tuning creates almost no intruders (its largest orthogonal-complement spike sits 100–200× below the same threshold); the anti-alignment bias and its dissection; and interventions on the predicted layers, now with random matched-count controls (Appendix K).

We would be grateful if the reviewer would reassess Soundness and Presentation against the revised text and the statistics above; we will answer any further question during the discussion period.

Dear Reviewer,

The author–reviewer discussion closes soon, so we would like to ask once more whether our response addresses your concerns. Every statistic you asked for is in our response above and in the revised PDF (Table 2, Appendix C and Figure 5; changed text in blue), and the revision states the comparison as the statistics show it: a small, consistent gain of the transform at the bulk edge and none at the full edge. On your point about qualifications, the abstract now reports the balanced-split ratios instead of the 270× figure. If anything is still unclear, we will answer it today; if your concerns are resolved, we would be grateful if you would consider updating your assessment. Thank you again for the careful review and for recomputing our results from the released data.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response to all reviewers

We thank the reviewers. The reviews converge on two questions: (A) what the spectral transform adds beyond the location of the spectral edge, and (B) what the spike-budget rule adds beyond simpler ways of shrinking the update; plus one request for a balanced causal design (C) and comments on presentation (D). Each item is answered from the released data or one new experiment; the revised PDF (main text still 10 pages) states the answers, and the individual responses give the full tables.

concernwhat we didwhere
A1. Transform vs. fitted σ_max / σ_bulk, with statistics (2YnX W1, Q; Fod7 W1, Q1; GyLH W1)paired bootstrap over 53 out-of-sample and 6 external adapters, both calibration regimes, family-clustered intervalsTable 2, §3 (p. 5); Appendix C, Tables 5–7
A2. Figure-1-style plots for the calibrated baselines (2YnX Q)same layers, axes and ×2 bands, frozen κ, beside the lawFigure 5 (p. 17)
A3. Systematic errors that aggregates conceal (2YnX Q)per-layer error by module typeTable 9 (p. 19)
A4. What "no fitted constant" buysper-family calibration factor: 16% spread vs 20–36%Table 8 (p. 19)
B. Cap rule vs. uniform scaling and lower learning rate at matched task fit (GyLH W1–W3, Q; Fod7 W3, Q3)every Mistral knob run on one forgetting-vs-task-fit plot; §4 rewrittenFigure 4 (p. 9), Table 10 (p. 22), §4, §7
C. Balanced norm-matched intervention (Fod7 W2, Q2)new experiment: ten balanced splits of four adapters with random same-count, same-norm controlsAppendix K, Table 11, Figure 7 (pp. 21–23)
D. Terms, qualifications in appendices, abstract (2YnX W2, W3)"Terms" paragraph (§2); qualifications moved next to the claims (§3, §4, §7); abstract shortened; the sense of "law" statedpp. 1–2, 5, 7–8, 10

A. The transform is close to a fixed multiple of its edge (median within-adapter Spearman 0.98–0.997), so each law variant is compared with the fitted multiple of its own edge. At the bulk edge the transform ranks layers better than κσ_bulk on 53 out-of-sample adapters (Spearman ρ +0.041 [+0.031, +0.051], better on 87%; family-clustered [+0.021, +0.060]), classifies deployment better (AUC +0.012 [+0.007, +0.018]), wins in ρ on all six external ones, and removes the module-type error that the fitted baseline shows. The geometric mean adds nothing in ranking over √(σ_max σ_bulk), and the full edge, our pre-specified primary, ranks worse than κσ_max. A fitted baseline matches the localization ceiling, which the geometric-mean law reaches at κ = 1 (the toy condition σ_max needs 0.65), and at the bulk edge the transform's constant is the most family-stable (16% over 25 model groups, against 20–36%). The revised §3 says this and recommends the bulk edge as the instrument.

B. Figure 4 places every Mistral knob run on one plot. At equal task perplexity (≈ 1.33) a four-times lower learning rate forgets +0.55 PPL, rank 64 +0.79, half learning rate +1.12, the cap rule +1.59, matched uniform α +2.11. The revised §4 states that the rule is not the best knob; its per-layer allocation adds 16% over a matched uniform scale, which the accuracy metrics do not resolve. What the thresholds add is the diagnosis and the dose: Mistral deploys at 1.5× its median threshold and needs shrinking to 0.5–0.7 of it, which halving the learning rate achieves; Qwen deploys below half of its thresholds and gains at most 0.2 PPL from either knob.

C. On ten balanced splits of four adapters (two cap-rule seeds; Mistral and Qwen r256 retrained), the predicted-supercritical set forgets more than every one of five random sets of the same count and injected norm in 49 of 50 draws: 1.4–3.2× their mean on Mistral, 1.1–1.4× on Qwen; against the norm-matched subcritical set 1.7–6.8× on Mistral and 1.05–1.14× on Qwen. The appendix states the limits: pooled across strengths the injected norm correlates with forgetting as well as the crossed count; below threshold an update still perturbs the model; on Qwen the equal-norm asymmetry is small. The abstract now reports the splits whose matched set stays subcritical (1.5–5.5× on Mistral and Llama, 1.0–1.1× on Qwen) in place of the 270×.

D. A "Terms" paragraph opens §2; qualifications formerly in Appendices C, H and J now accompany the claims; the abstract is shorter; the introduction defines our use of "law".

Changes in the revised PDF (changed text in blue): abstract and introduction (pp. 1–2); §2 "Terms" (p. 2); §3, Table 2 (p. 5); §4 around Figure 4 (pp. 7–9; dose panels now Figure 9); §7 (p. 10); Appendix C, Tables 5–9, Figure 5 (pp. 15–19); Appendices G, J, K, Table 10 (pp. 18–23); and corrections from a consistency check of the whole text (§2, §6, Appendices A, F, H, L–P), including a Qwen run at s = 2 (0.99×) that the submission did not report.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.