acceptodds
Under review as a conference paper at ICLR 2027

PRISM: A Geometric Risk Bound for Decomposing Drift into Scale, Shape, and Head

Abstract

A single base LLM now comes with dozens of post-training variants, quantized, LoRA-adapted, or distilled, and each has to be checked before release. Existing evaluations provide only a partial picture: benchmark scores and likelihood screens say that a variant has degraded, similarity scores such as CKA and SVCCA say how its features moved, and nothing connects the two. We connect them with one structural fact and one design choice: the prediction head is linear, so feature geometry reaches the loss, and we compare the two feature sets through an orthogonal map, which leaves the geometry being measured unchanged. From these we derive PRISM, a closed-form upper bound on the cross-entropy risk gap between a target model and a proxy variant, and prove that it splits exactly into three measurable axes: scale, shape, and head. Each axis names a failure mode and where to intervene: low-bit quantization distorts shape and, at the lowest bit-widths, also inflates activation scale; quantizing the output embedding inflates the head term, which dominates the bound at high bit-widths. Because the shape term is differentiable, the same geometry becomes a regularizer that curbs catastrophic forgetting more than experience replay. Across two model families and five benchmarks, PRISM ranks quantized and fine-tuned variants from a single forward pass at mean Spearman above , and eight reference sequences already recover the ranking that the risk gap itself needs over a hundred to reach.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation geometry, representational similarity, Procrustes analysis, generalization and risk bounds, large language models, post-training quantization, parameter-efficient fine-tuning, model diagnostics, interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_H946OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work tries to connect the performance of post-training variants and the feature geometry. A bound named PRISM is proposed, where the cross-entropy risk gap is decomposed into three parts: scale, shape, and head. A shape regularization term is further proposed to improve the post-training. Extensive empirical results somehow validate the effectiveness of PRISM.

Critical strengths
  1. The addressed topic is interesting. Leveraging analysis on features to help diagnose the post-training performance is important.
  2. The decomposition of PRISM is reasonable, and the experiment results somehow validate this point.
  3. Experiments cover a variety of models, protocols, tasks, and benchmarks.
Critical weaknesses
  1. PRISM derives its bound entirely from the feature immediately preceding the linear head. However, this feature is a highly abstracted representation that has passed through dozens of Transformer layers, and it may act as an information bottleneck and may miss rich information in intermediate representations. PRISM may under-report the true risk of a variant whose degradation is concentrated in intermediate layers, and it cannot provide the layer-wise or module-wise attribution that practitioners often need when debugging aggressive quantization. The authors should discuss this limitation explicitly and clarify whether the diagnostic can be extended to, or complemented by, richer intermediate representations (e.g., Q/K/V statistics or per-layer feature drift).

  2. The paper argues that each of the three axes, i.e., shape, scale, and head, “suggests a different fix,” but only the shape regularizer is implemented and empirically validated (Sec. 4.5). For the scale axis, the paper suggests per-channel scaling or outlier-channel preservation (line 417), and for the head axis, no corresponding intervention is tested. As a result, the paper does not verify that acting on the scale or head diagnosis actually reduces the bound or the risk gap, nor that the axis attribution correctly identifies the dominant failure source. The authors should either add intervention experiments for the scale and head axes, or temper the actionability claim and present the three-axis decomposition as descriptive rather than prescriptive.

Questions for authors
  1. Could similar analysis be potentially applied to early-layer features such as the Q, K, V modules in attention blocks?

  2. Regarding the 2nd weakness above, although the authors acknowledge this limitation in Section 5, it is suggested to supplement some preliminary intervention experiments to validate the claimed actionability of the scale and head axes.

4 comments
AuthorsOpenReview ↗

Response to Reviewer H946 (1/2): W1/Q1, intermediate layers; layer-wise and Q/K/V profile added

Thank you for pressing on exactly the two points where the paper needed more evidence. We ran the layer-wise profile and the scale and head interventions you asked for.

W1 / Q1: intermediate layers, and layer-wise attribution

Can the bound under-report risk when degradation sits in intermediate layers? The interior drift you describe does occur (table below), but it cannot make the bound under-report, for a structural reason. The logits are exactly Z·H, where Z is the final hidden state and H the output head, so the loss depends on the model only through (Z, H). Any upstream change that affects the loss must pass through Z, and PRISM reads Z. On the sequences PRISM is computed from, reading only Z therefore misses nothing; this is also what gives Theorem 1 its closed form.

Can the analysis be applied to Q, K, V and intermediate layers? Yes, as a descriptive profile, and we have run it (table below). At an intermediate layer the bound no longer covers the model's own loss, because it needs a linear map from features to logits, which only the final layer has. At layer l, a linear probe can supply that map, and Theorem 1 then bounds the probe's cross-entropy gap in the same way. If both models share one probe, the head term is zero; if each has its own, the head term measures how far the two probes differ. Either way the guarantee is on the probe's loss, not the model's, so the paper keeps the bound at the deployed head and uses the profile below for localization.

We ran four GGUF (llama.cpp) quantized variants on 128 MMLU sequences (about 14k tokens). In every block we read four points: the residual stream after the block and the outputs of its q, k and v projections. At each point we computed the shape drift 1−Ω between the original model (the target) and the variant. Here we use a per-token version of Ω: the cosine similarity between the target's and the variant's feature for the same token, averaged over tokens, so 1−Ω = 0 means no drift. We normalize each token's feature ourselves, as the RMSNorm in front of every block does, because in the residual stream a few tokens carry very large activations and would otherwise dominate. The profile tracks the shape term only: inside the network there is no head, and this normalization sets scale aside, so the scale term is read at the output (W2). Entries are the peak drift over depth, with the block where it occurs (blocks numbered from 1):

variant (entries: shape drift 1−Ω, 0 = no drift)residual stream, peak (block)at output (final hidden state)q_proj peak (block)k_proj peak (block)v_proj peak (block)
Qwen3-8B-Base Q6_K0.002 (33)0.0010.001 (34)0.001 (34)0.004 (36)
Qwen3-8B-Base Q4_K_M0.017 (33)0.0070.009 (30)0.010 (34)0.035 (23)
Qwen3-8B-Base Q2_K0.173 (22)0.0490.090 (22)0.122 (17)0.334 (17)
Llama-3.1-8B Q6_K0.003 (output)0.0030.001 (14)0.001 (14)0.004 (13)

What the profile adds to the output-level reading:

  • Where in depth. At Q2_K drift builds to a peak of 0.173 at block 22, and the last blocks and final norm bring it back to 0.049 at the output. So the intermediate layers can drift more than the output does, which is the situation you describe; the loss itself follows the output.
  • Which module. Value projections drift most (about 4× the queries at Q2_K, where q, k and v all peak mid-network), so for aggressive quantization the profile points to the value projections first.
  • Consistency. For the three Qwen3-8B-Base variants, the bit-width ordering Q2_K > Q4_K_M > Q6_K holds at every block and in every module, so ranking them at the output gives the same answer as ranking them at any intermediate point.

Together, the two levels give a workflow. The bound at the output ranks the variants and shows which term moved; the profile then shows where in depth and in which module the drift is largest, so a developer knows where to look first (here, the value projections of the middle blocks at Q2_K). The revision adds the profile and states this division of labor: the bound is the output-level guarantee, and the profile localizes drift by layer.

AuthorsOpenReview ↗

Response to Reviewer H946 (2/2): W2/Q2, head and scale interventions run

W2 / Q2: interventions on the head and scale axes

You offered two routes, interventions or a tempered claim; we took the first, on both axes. The bound holds in every cell below. We use a 0.02-nat noise floor; we only count differences above it.

Each axis has its own intervention:

axisinterventionevidence here
headkeep the output head at higher precisionthe head term ranks how much restoring the head lowers the gap (193 / 194 pairs)
scalekeep outlier channels in FP16scale term 6.1× to 95.7× lower; it is large on the model where this remedy changes the gap and small on the other
shape (1−Ω)shape regularizer during fine-tuningless forgetting than no regularization, anchor-CE and generic replay in all four cells and every seed (reply to Reviewer 5VCw)

Head axis

Experiment 1: does the gap split into a head part and a backbone part? From a GGUF variant we build two hybrids: its quantized backbone (everything before the output head) with the original head, and the original backbone with its quantized head (Qwen3-8B-Base at Q6_K, Q4_K_M and Q2_K; Llama-3.1-8B at Q6_K; five benchmarks; 20 cells). The two hybrids' gaps add up to the variant's gap (median ratio 0.999 over the ten cells whose gap exceeds the noise floor, all on Qwen3-8B-Base). In the hybrid with the original head, the feature term alone still ranks the three Qwen3-8B-Base bit-widths correctly on every benchmark.

Experiment 2: vary the head's precision. The head's precision can be set separately from the backbone's. For each of three backbones (original, GGUF Q4_K_M, GGUF Q2_K) we swap in seven heads: BF16, Q8_0, Q6_K, Q5_0, Q4_0, and 3- and 2-bit as a stress test. Two models, five benchmarks, 210 cells.

what the head term should rankpairs ranked correctlyQ8_0 to Q4_0 only
the gap caused by the head alone (original backbone)92 / 9218 / 18
how much restoring the original head lowers the gap (quantized backbones)193 / 19433 / 33

Head and backbone errors again add up (interaction: a median 2% of the gap). The head term also separates the two models. It is larger on Qwen3-8B-Base at every head format, and Qwen3-8B-Base's head is indeed far more fragile (gap from the head alone, median over five benchmarks):

headQ8_0Q6_KQ5_0Q4_02-bit
Qwen3-8B-Base0.00160.00300.02160.07591.44
Llama-3.1-8B0.00020.00030.00190.00050.13

Every Llama-3.1-8B value up to Q4_0 is within noise, so all pairs in the last column of the pairs table come from Qwen3-8B-Base.

Takeaway. Before any change is made, the head term ranks head settings by how much restoring the original head would lower the gap. On Qwen3-8B-Base, restoring a Q4_0 head lowers the gap by up to 0.15 nats (beyond noise in 9 of 10 cells), while restoring a Q8_0 or Q6_K head changes it by at most 0.018, within noise.

On attribution. Head and backbone gaps add up, so each source's part of the gap can be measured directly, and the dominant source is read from these parts. Each term ranks within its own axis: across head settings and across models.

Scale axis

8-bit quantization in bitsandbytes keeps outlier channels in FP16 by default, which is the scale remedy the paper names. Raising its llm_int8_threshold from 6 to 10⁹ switches this remedy off: every channel is quantized and nothing else changes. On Qwen3-8B-Base (n = 512 per benchmark; outliers-kept gaps match the paper's Table 10 to within 0.01 nats):

benchmarkgap (nats), outliers keptgap, all quantizedscale term, all quantized ÷ keptshape term, all quantized ÷ kept
SQuAD0.00360.064386.3×4.3×
TriviaQA0.02390.030712.3×3.4×
MMLU0.01200.017395.7×3.4×
ARC0.06810.02406.1×3.1×
GSM8K0.00060.031755.6×2.7×

Beyond the noise floor, keeping the outliers lowers the gap on SQuAD and GSM8K and raises it on ARC; on TriviaQA and MMLU the change is within noise. The head term stays at zero.

In absolute terms the shape term is still the larger of the two; the scale term is the one that responds most to the switch (6× to 96×, against 3× to 4×). With outliers kept, the bound is lower on all five benchmarks.

With outliers kept, the scale term already tells the two models apart: it is large on Qwen3-8B-Base (0.5 to 12.5) and small on Llama-3.1-8B (at most 0.115). The switch then moves the gap by up to 0.061 nats on Qwen3-8B-Base and by at most 0.011 on Llama-3.1-8B, within noise.

Sec. 5 will report the interventions on all three axes, so each axis's remedy is tested rather than only proposed.

We would be grateful if you would take the new evidence into account in your assessment. Thank you again: the layer profile and both interventions will be in the paper because you asked for them.

I appreciate the authors’ careful responses. My concerns are adequately addressed, and thereby I maintain my initial positive rating.

AuthorsOpenReview ↗

Thank you; W1 and W2 are now in the revised paper

Thank you for confirming that your concerns are addressed, and for the time you have given to our responses. Since your comment, the revised PDF is up, so what you found addressed can now be checked in the paper itself: Sec. 5 summarizes both points, and App. K gives the details.

  • W1 and Q1 (App. K.3, Table 33): the per-layer and q/k/v drift profile you suggested. On Qwen3-8B-Base at Q2_K, residual-stream drift peaks at block 22 (0.173) and falls to 0.049 at the output, and the value projections drift the most. K.3 states the scope you asked us to discuss: the bound is output-level, and the profile localizes the drift.
  • W2, head (App. K.1, Table 30): a variant's gap splits into a head part and a backbone part, which add up to it within 0.007 nats in all 20 cells; of the 194 pairs of head settings whose benefit from restoring the original head differs beyond noise, the head term orders 193 correctly.
  • W2, scale (App. K.2): quantizing the outlier channels that 8-bit bitsandbytes keeps in FP16 raises the scale term 6.1× to 95.7× on Qwen3-8B-Base, against 2.7× to 4.3× for the shape term. With outliers kept, the scale term is large on Qwen3-8B-Base, where the switch moves the gap by up to 0.061 nats, and small on Llama-3.1-8B, where it moves it within noise.
  • W2, shape (Sec. 4.5): the regularizer forgets less than no regularization in all four (model, task) cells and every seed.

Your review called "leveraging analysis on features to help diagnose the post-training performance" important. The paper now follows that diagnosis through: ranking the variants, seeing which axis moved and where in the network, and testing a remedy on each axis. Both additions address points from your review, and we are grateful for them.

Reviewer_5VCwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes PRISM, an upper bound on the cross-entropy risk gap between a base LLM (target) and a post-training variant (proxy) computed from one teacher-forced forward pass. Theorem 1 (eq. 4) bounds ∣RT−RP∣\lvert \mathcal{R}_T - \mathcal{R}_P\rvert by a feature term δ=Kfeat(ρT−ρP)2+2ρTρP(1−ΩW)\delta = K_{\mathrm{feat}}\sqrt{(\rho_T - \rho_P)^2 + 2\rho_T\rho_P(1 - \Omega_W)} plus a head term γ=Kpred∥ΣP1/2(WHT−HP)∥F\gamma = K_{\mathrm{pred}}\left\lVert\Sigma_P^{1/2}(W H_T - H_P)\right\rVert_F (using Kpred=2K_{\mathrm{pred}} = \sqrt{2}), for any orthogonal WW. Proposition 1 (eq. 2) states that the alignment residual splits exactly into a scale term and a shape term; KfeatK_{\mathrm{feat}} is the maximum pairwise distance between head columns (Appendix A.3). The shape term 1−Ω1 - \Omega at W=IW = I is differentiable and is used as a regularizer against forgetting during frozen-head LoRA (eq. 6).

Validation: on Llama-3.1-8B and Qwen3-8B over five benchmarks, the bound ranks 11-12 quantized variants per cell at mean Spearman 0.875 (Fig. 2) and LoRA checkpoints at 0.831 (Fig. 3); on the ten PTQ cells 1−CKA1-\mathrm{CKA} and 1−SVCCA1-\mathrm{SVCCA} reach 0.899 against PRISM's 0.875-0.906, a difference the paper reports as unresolved after multiplicity correction (Sec. 4.2, Table 16; no such baseline comparison is run for the LoRA ranking). With 8 reference sequences the bound recovers the full-slice ordering (rsr_s 0.93) where the directly measured gap reaches 0.795 and needs 128 sequences (Appendix F.2, three seeds). The shape regularizer reduces mean downstream forgetting from 0.843 to 0.681 versus 0.764 for replay on Llama/TruthfulQA (Table 2; three-seed replication in Table 22), on Llama/BBQ it is ahead of replay (0.195 vs 0.241) but 9% above no regularization (0.179), and on both Qwen3-8B tasks all three arms sit within a 0.02 noise floor (Appendix J, lines 1989-1995, 2017-2019). The theory is correct for the empirical quantities it is tested on, with a presentation mismatch in Theorem 1 noted below; the contribution is weakened because the paper's own comparison shows no ranking advantage over CKA/SVCCA, the "exact decomposition" is the law of cosines, and a substantive reduction of forgetting by the regularizer is confined to one cell while the abstract states the advantage without qualification.

Critical strengths
  • Significance: The small-sample result is a genuinely useful finding for practitioners. Appendix F.2 (12 variants, three seeds) shows the Procrustes-form bound BN\mathcal{B}_N holds rs=0.932±0.016r_s = 0.932 \pm 0.016 against the full-512 risk-gap ordering with only 8 reference sequences, while the risk gap measured on those same 8 sequences reaches 0.795±0.0590.795 \pm 0.059 and needs 128 sequences to match; Qwen3-8B replicates (0.827 vs 0.755). If a cheap screen is the goal, this is the evidence that matters, and it is seeded.

  • Soundness: The theory is correct, self-contained and honest. Kfeat=max⁡j,k∥hT,j−hT,k∥2K_{\mathrm{feat}} = \max_{j,k}\lVert h_{T,j} - h_{T,k}\rVert_2 via the convex-combination argument (Appendix A.3) is tighter than the spectral constant and vocabulary-independent; the head term is correctly covariance-weighted so that only directions with data support contribute; and the empirical form of Theorem 1 holds verbatim (lines 203-206). The code implements exactly these quantities (metrics.py lines 98-135, bounds.py line 38).

Critical weaknesses
  • Significance: The paper's own experiment shows no demonstrated ranking advantage on full-reference PTQ. Section 4.2 (lines 360-367, Table 16) reports that on the ten PTQ cells 1−CKA1-\mathrm{CKA} and 1−SVCCA1-\mathrm{SVCCA} reach mean rsr_s 0.899 and PRISM quantities 0.875-0.906 (BN\mathcal{B}_N 0.906), with per-metric SEM 0.02-0.05 and "no pairwise difference survives multiplicity correction"; the LoRA ranking has no baseline comparison at all. The bound is also 10210^2–10410^4 times the measured gap (Table 1: FP16 B=6.09\mathcal{B} = 6.09 vs ∣ΔR∣=0.0017\lvert \Delta\mathcal{R}\rvert = 0.0017; Q2_K 267.7 vs 0.366), so it is not usable as an estimate. What is left of contribution 1 is Proposition 1, which is the law of cosines for ∥ZT−ZPW∥F2\lVert Z_T - Z_P W\rVert_F^2 (expand the square and use ∥ZPW∥F=∥ZP∥F\lVert Z_P W\rVert_F = \lVert Z_P\rVert_F): it holds for any two matrices and any orthogonal WW and encodes nothing about the models beyond ρT\rho_T, ρP\rho_P and one inner product. Presenting it as an "exact Procrustes residual decomposition" and a theoretical contribution overstates a one-line identity. The remaining novelty is the small-sample stability (strength above) and the head-term readout.
  • Soundness: The regularizer claim is stated without the qualification the paper's own results require, and the baseline it is measured against is not experience replay. The abstract says the shape term "curbs catastrophic forgetting more than experience replay" and contribution 3 says it "outperforms experience replay in aggregate." But the "replay" arm rehearses cross-entropy on the same 32-sequence reference set as the regularizer, drawn "from the held-out split of the fine-tuning task" and "disjoint from every benchmark on which forgetting is measured" (lines 237-239, 424-426); the shipped trainer loads args.task with split="test" for both controls (train_forgetting.py lines 1191-1206, 350-386). It therefore replays new-task examples and contains no old-domain data; it is a held-out-CE control, not experience replay in the continual-learning sense, and beating it says nothing about replay of old-domain data. On point estimates the regularizer is ahead of replay in three of four (model, task) cells, but it reduces forgetting substantively in only one: Llama/TruthfulQA (0.681 vs replay 0.764 vs no-reg 0.843, Table 2; three seeds in Table 22: 0.680±0.0160.680 \pm 0.016 vs 0.771±0.0270.771 \pm 0.027). On Llama/BBQ the regularizer is 9% above no regularization (0.195 vs 0.179, replay 0.241; Appendix J lines 1992-1994), and on both Qwen3-8B tasks the three arms differ by under 0.02, which the authors call the evaluation-noise floor (lines 2017-2019). "Curbs forgetting more than replay" therefore means "replay hurts more than the regularizer" in two cells and "no difference" in one. The main-text Table 2 is seed 42 alone (lines 1969-1970), and the criterion by which "the best λ\lambda in its sweep" (line 427) was chosen is not stated, so whether both arms were selected on the reported downstream metric cannot be judged; Appendix I's target-loss-matched protocol (lines 1935-1943) is the clean one and belongs in the main text.
Questions for authors

Questions that could change assessment

  1. Please rename the replay-CE arm as what it is (cross-entropy on held-out fine-tuning examples) and add a genuine experience-replay baseline that rehearses old-domain data (e.g. 32 sequences from the pre-training-like corpora or from the downstream benchmarks' training splits) at matched plasticity, on all four (model, task) cells with three seeds. Score-change criterion: if the shape regularizer still forgets less than old-domain replay on at least three of four cells, Soundness rises to 3 and Rating to 3; if old-domain replay matches or beats it, the abstract's claim must be withdrawn and the Rating stands.
  2. Theorem 1 as printed bounds the population gap of lines 148-149 by Ddiag\mathcal{D}_{\mathrm{diag}} statistics; please restate it for empirical risk (eq. 35) and leave the population claim to Corollary 1 / eq. 36, and add a CKA/SVCCA baseline for the LoRA ranking cells. Score-change criterion: presentation only for the theorem; if PRISM's LoRA-checkpoint ranking beats 1−CKA1-\mathrm{CKA} on the ten LoRA cells with seeds, Informativeness rises to 3.
  3. Given Table 16, what is the concrete decision a practitioner makes differently with PRISM than with 1−CKA1-\mathrm{CKA} on the same forward pass? If the answer is "the small-sample stability of Appendix F.2 and the head term," please move F.2 into the main text, report it on all five benchmarks for both families with the same three seeds, and reframe contribution 3 around it. Score-change criterion: a main-text, seeded demonstration that PRISM's ordering is stable at n=8n = 8 across all ten cells while CKA and the risk gap are not would raise Informativeness to 3 and Rating to 3.
  4. What criterion selected "the best λ\lambda in its sweep" for the shape regularizer and for replay in Table 2 and Fig. 4 (line 427): the downstream forgetting metric being reported, target-task loss as in Appendix I, or something else? Score-change criterion: if selection was on target-task loss or a held-out split, no change; if both arms were selected on the reported downstream ∣ΔR∣\lvert \Delta\mathcal{R}\rvert, the main-text comparison must be replaced by the Appendix I protocol for Soundness to stay at 2 rather than fall to 1.

Clarifications and minor points (do not affect my assessment)

  1. Proposition 1 should be stated as the standard cosine-law identity with a one-line proof in the main text, and the word "exact" used once rather than as a selling point.
  2. The abstract's "curbs catastrophic forgetting more than experience replay" should read "is ahead of a held-out-CE control on three of four cells and reduces forgetting relative to no regularization on one"; Appendix J already gives these numbers.
  3. "the head term ... dominates the bound at high bit-widths" (abstract) is true for Qwen3 (γ\gamma 77.4 vs δ\delta 26.8 at Q6_K/SQuAD) and false for Llama (18.8 vs 37.9 at Q6_K/MMLU, Table 1); qualify by family.
  4. Each per-cell Spearman is over 11-12 points; please print nn next to every rsr_s.
  5. Williams et al. 2021 already define rotation- and scale-invariant residuals in neural shape space; the related-work paragraph should credit that lineage for the scale/shape split.
  6. The regularizer is a feature-distillation penalty on a reference set; LwF-style functional regularisation and representation KD are the natural baselines alongside EWC/L2-SP in Appendix I.
  7. Reproducibility: quantization.py loads every model with trust_remote_code=True; pin revisions or make it opt-in. Only seed 42 is in the shipped configs; the three-seed rounds of Appendices F.2 and I have no launch script.
  8. Future direction: scale and head remediations are proposed (lines 416-421) but untested; a single experiment keeping lm_head in FP16 for GGUF Q6_K on Qwen3 would close the loop on the head diagnosis.
7 comments
AuthorsOpenReview ↗

Response to Reviewer 5VCw (1/4): your four criteria at a glance; Q1, genuine old-domain replay

Thank you for reading the code line by line and attaching a criterion to each question. We ran every requested experiment; the table reports each against your criterion.

your questionyour criterionresult
Q1: genuine old-domain replay, four cells, three seeds, matched plasticityshape regularizer forgets less on ≥ 3 of 4 cellsmet: less than generic replay in all four (model, task) cells and in every seed, with every method's target-task loss at most 0.01 nats above the unregularized run's
Q2: 1−CKA for the LoRA ranking, with seedsPRISM beats 1−CKA on the ten LoRA cellshigher on the ten-cell mean, and in six of the ten cells: Spearman r_s 0.825 against 0.790 for 1−CKA and 0.719 for 1−SVCCA; the bound holds in all 360 LoRA (checkpoint, cell, seed) cases
Q3: n = 8 stability on all ten cells, in the main textPRISM stable while 1−CKA and the measured gap are notmet: with eight sequences, PRISM's worst reference set still ranks at 0.564, against 0.382 for 1−CKA and −0.064 for the gap, and PRISM ranks variants better than both in all ten cells (p = 0.002 against each); the screen moves to the main text
Q4: how λ was selectedno change if λ was selected on target-task lossmet: the regularizer's λ was the lowest target-task-loss point of its sweep; the revision tunes every method by Q1's rule, which never consults forgetting

W2 and Q1: genuine replay at matched plasticity

Short answer: met (table below).

Thank you for this correction: the 32 sequences come from the fine-tuning task, so the method that rehearses them is not experience replay. The revision makes the data and the names explicit:

  • Anchor set: 32 sequences from the fine-tuning task, on which the shape regularizer compares the fine-tuned model's features with the base model's. The revision makes them a split of their own, excluded from the fine-tuning data and from every target-task loss in the new results. They come from the fine-tuning task because a practitioner always has that data; the regularizer needs no old-domain data.
  • anchor-CE: the held-out-CE control you named, cross-entropy on the anchor set. We keep it as a same-data control: it sees exactly the sequences the regularizer sees, so the two differ only in the penalty.
  • generic replay (new, old-domain): 32 sequences of a pre-training-like corpus (WikiText), rehearsed with the same cross-entropy.

How λ is chosen. Every method (the regularizer and each baseline) is tuned by one rule at matched plasticity, and forgetting is never consulted:

steprule
referenceno regularization, trained in the same loop as every other method
budgettarget-task loss at most 0.01 nats above the reference (1% in perplexity), the same for every method
selectionthe largest λ within budget, searched upward on the grid λ ∈ {1, 5} × 10^k up to 10⁴, the same grid for every method; three seeds at the selected λ

We take the largest λ within budget, so each method is compared at the strongest setting the budget allows, much as the one-standard-error rule takes the most regularized model within tolerance (Breiman et al., 1984). Raw λ values are not comparable across methods, since each penalty has its own scale; Appendix I of the revision will list every selected λ with its weighted penalty λ·P.

Entries are downstream forgetting ↓ (mean |ΔR| over the five benchmarks; mean ± sd over three seeds), with target-task loss in parentheses.

modeltaskno regularizationanchor-CEgeneric replayshape regularizer
Llama-3.1-8BTruthfulQA0.771 ± 0.032 (0.862)0.837 ± 0.059 (0.871)0.632 ± 0.033 (0.833)0.180 ± 0.005 (0.872)
Llama-3.1-8BBBQ0.258 ± 0.002 (0.114)0.270 ± 0.015 (0.122)0.369 ± 0.008 (0.088)0.189 ± 0.010 (0.057)
Qwen3-8B-BaseTruthfulQA0.316 ± 0.024 (0.835)0.334 ± 0.015 (0.828)0.308 ± 0.018 (0.837)0.165 ± 0.015 (0.843)
Qwen3-8B-BaseBBQ0.108 ± 0.013 (0.191)0.135 ± 0.032 (0.162)0.355 ± 0.010 (0.196)0.068 ± 0.003 (0.127)
  • Not a matter of the baselines' λ: even if we give each baseline its best λ (the one at which it forgets least within budget), generic replay and anchor-CE still forget more than the regularizer in all four cells; the closest case is 0.104 against 0.068, on Qwen3-8B-Base/BBQ.
  • Robustness: a budget of 0.005 or 0.02 nats changes none of these comparisons.

What changed from the submission. All methods now share one training loop. The submission trained the unregularized run with the library's default trainer, so that run's numbers differ from the submission's, most visibly on Llama-3.1-8B/BBQ. The rule can also select a λ above 1, where the submission's sweep stopped; at λ = 1 the new runs reproduce the submission's forgetting.

The protocol and table will move to the main text.

AuthorsOpenReview ↗

Response to Reviewer 5VCw (2/4): Q2, LoRA ranking against 1−CKA; W1/Q3, the eight-sequence screen

Q2: Theorem 1 on empirical risk, and 1−CKA for the LoRA cells

Short answer: the theorem is restated as you ask. On the LoRA ranking, PRISM's mean r_s over the ten cells is higher than 1−CKA's (0.825 against 0.790): PRISM is higher in six cells, and in the other four the mean ± sd ranges overlap. Unlike 1−CKA, PRISM is a bound that holds in every case.

The revision restates Theorem 1 on the empirical risk gap (eq. 35) and moves the population statement to Corollary 1.

We re-ran the ten Llama-3.1-8B LoRA cells as checkpoint series (12 checkpoints, one every 25 steps, three seeds), computed 1−CKA on the same features as the bound, and ranked the checkpoints against |ΔR| within each cell. Entries are Spearman r_s, mean ± sd over three seeds (the submission's 0.831 is a single run).

taskbenchmarkPRISM bound, r_s ↑1−CKA, r_s ↑
TruthfulQAARC0.846 ± 0.1700.471 ± 0.458
TruthfulQAGSM8K0.998 ± 0.0040.986 ± 0.000
TruthfulQAMMLU0.904 ± 0.0980.876 ± 0.085
TruthfulQASQuAD0.965 ± 0.0120.984 ± 0.016
TruthfulQATriviaQA0.895 ± 0.0240.888 ± 0.036
BBQARC0.967 ± 0.0250.972 ± 0.007
BBQGSM8K0.979 ± 0.0070.958 ± 0.014
BBQMMLU0.937 ± 0.0570.958 ± 0.043
BBQSQuAD−0.114 ± 0.189−0.056 ± 0.244
BBQTriviaQA0.874 ± 0.1230.867 ± 0.124
mean over the ten cells0.8250.790

1−SVCCA, on the same features, is lower than both (0.719). The bound holds in all 360 LoRA (checkpoint, cell, seed) cases.

W1 and Q3: what PRISM adds to 1−CKA, and F.2 in the main text

Short answer: PRISM's aim is not to out-rank 1−CKA but to guide two decisions: which variants are worth testing on the full benchmarks and, when a variant looks degraded, which axis to act on. Both come from one forward pass. This is the answer you anticipated, the small-sample stability of Appendix F.2 and the head term; the screen is now run on all ten cells. On the same forward pass PRISM adds what a similarity score leaves out, a bound and which axis moved; at eight sequences it also ranks better (below).

1−CKA is unchanged when the features are rescaled globally, and it does not see the output head. On the same forward pass, PRISM adds four things:

  1. a guarantee: for any number of sequences, the bound cannot under-report the risk gap on the sequences it is computed from (Theorem 1);
  2. which axis moved, and so where to look. Example: 8-bit quantization in bitsandbytes keeps outlier channels in FP16 by default. Quantizing them too on Qwen3-8B-Base (llm_int8_threshold raised from 6 to 10⁹) makes the scale term 6.1× to 95.7× larger, while the shape term moves by 2.7× to 4.3× and the head term stays at zero (reply to Reviewer H946). Likewise, the head term ranks how much restoring the output head lowers the gap (minor 8);
  3. a remedy for the shape axis: the shape term doubles as a regularizer that reduces forgetting (Q1);
  4. a better screen at eight sequences: it ranks variants better than 1−CKA (below).

With only eight sequences, a score can change with which eight are drawn, so we score many sets rather than one. For each benchmark we drew three pools of 512 sequences, each with its own seed; the risk gap measured on all 512 of a pool gives the ordering to be recovered. From each pool we score 100 sets of eight: the first is the set Appendix F.2 used, the other 99 are drawn uniformly without replacement by a rule fixed before any of them was scored. All 300 sets per cell are reported. On each set, a score's ordering of the variants is compared with its pool's ordering by Spearman r_s. B_N is the bound under the Procrustes alignment; "gap@8" is the risk gap measured directly on the same eight sequences.

at n = 8, ten cells × 300 reference setsgap@81−CKA@8PRISM B_N@8
mean r_s0.7650.8510.877
worst single reference set−0.0640.3820.564
cells where B_N is higher than this column's score (exact paired Wilcoxon)10 / 10 (p = 0.002)10 / 10 (p = 0.002)
AuthorsOpenReview ↗

Response to Reviewer 5VCw (3/4): Q3 per-cell results; W1, the bound's magnitude; Q4, λ selection

W1 and Q3 (continued): per-cell results at eight sequences

Compared on the same reference sets, B_N is higher than 1−CKA on average within every one of the ten cells (p < 10⁻⁴ in each, over its 300 sets, after Holm correction). We test within each cell because the test across cells (2/4) has only ten samples, one per (model, benchmark) cell. All tests are paired, and a paired t-test gives the same conclusions. Per cell (Llama = Llama-3.1-8B, Qwen = Qwen3-8B-Base; mean ± sd over the 300 sets; the sd is how much r_s varies across the 300 sets of eight, that is, how stable each score's ordering is, not an error bar on the mean):

modelbenchmarkvariantsgap@81−CKA@8PRISM B_N@8
LlamaARC120.700 ± 0.1780.907 ± 0.0460.942 ± 0.016
LlamaGSM8K120.912 ± 0.0510.907 ± 0.0160.914 ± 0.010
LlamaMMLU120.708 ± 0.1750.879 ± 0.0670.938 ± 0.022
LlamaSQuAD120.730 ± 0.1570.856 ± 0.0560.869 ± 0.040
LlamaTriviaQA120.787 ± 0.1470.963 ± 0.0160.973 ± 0.006
QwenARC110.7420 ± 0.1570.691 ± 0.0960.7423 ± 0.055
QwenGSM8K110.896 ± 0.0640.934 ± 0.0240.936 ± 0.024
QwenMMLU110.641 ± 0.2130.701 ± 0.1370.759 ± 0.114
QwenSQuAD110.815 ± 0.1220.834 ± 0.0530.853 ± 0.052
QwenTriviaQA110.724 ± 0.1620.838 ± 0.0400.847 ± 0.036

On the sequences it is computed from, the bound is guaranteed to lie above the gap (Theorem 1). Empirically, a bound computed from eight sequences also lies above the risk gap measured on all 512: this holds for every variant in every one of the 300 sampled sets, in all ten cells; at n = 32, 128 and 512 it holds for every variant and seed.

On the bound's magnitude. As you note, the bound sits far above the measured gap; it is a worst-case bound, and the paper does not use it as a point estimate. The four additions listed in (2/4) need validity, ordering and attribution, not tightness: the bound holds on all 360 quantization (variant, benchmark) pairs of the paper and in every case above; it orders variants (table above); and it shows which axis moved (item 2). The theoretical contribution is Theorem 1; Proposition 1 will be stated as the cosine-law identity (minor 1).

F.2 will move to the main text as you ask, and contribution 3 will lead with the eight-sequence screen.

Q4: what selected "the best λ"

Short answer: target-task loss for the regularizer. The revised main text will use the protocol of Q1.

In the submission, the shape regularizer's λ = 1 was chosen by target-task loss (the cross-entropy on the target task's held-out answers): it is the lowest-loss point of its sweep, which went only up to λ = 1.

shape regularizer, λ0.010.050.10.51
target loss0.89460.89610.89280.87750.8670
downstream |ΔR|0.80040.77570.76030.73580.6813

For held-out-CE (now anchor-CE), Table 2 uses λ = 0.01, the point of its sweep at which it forgets least. So the baseline was shown at its best setting, and the regularizer was still ahead by 0.083; both methods ran in the same training loop. Appendix I matched each baseline to the regularizer's own target-task loss; Q1 instead caps every method's target-task loss at 0.01 nats above the unregularized run's, so no method sets the budget for the others.

Reviewer_sfB7OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PRISM (Proxy Risk Inference via Structural Mapping), a geometric framework for diagnosing performance degradation in post-training variants of large language models. The central contribution is a closed-form upper bound on the cross-entropy risk gap between a target model and a proxy variant, decomposing the bound into three interpretable components: scale mismatch, shape distortion, and prediction-head discrepancy. The theoretical analysis exploits the linear prediction head of language models and an orthogonal feature alignment to derive an exact scale–shape decomposition of the feature alignment residual.

Critical strengths
  1. The paper establishes a closed-form upper bound on the cross-entropy risk gap and decomposes feature mismatch into scale and shape components, supplemented by a covariance-weighted head discrepancy.
  2. PRISM achieves mean Spearman correlations of 0.875 for post-training quantization and 0.831 for LoRA checkpoints across the reported model family and benchmark configurations. The experiments cover different quantization schemes and fine-tuning tasks, suggesting that the proposed diagnostic captures useful information about relative degradation.
Critical weaknesses
  1. The ranking advantage over established similarity metrics is not clearly demonstrated. The authors report that PRISM performs comparably to CKA and SVCCA on the full reference sets, with no statistically significant pairwise differences after multiplicity correction.
  2. The main experiments focus on two model families, Llama-3.1-8B and Qwen3-8B, with additional replications in the appendix. The quantization and LoRA experiments cover useful but specific configurations, and the mitigation experiments are concentrated on a small set of fine-tuning tasks.
Questions for authors
  1. The reported full-reference-set ranking performance is comparable to CKA and SVCCA, with no statistically significant pairwise differences after multiplicity correction. Could the authors clarify under which settings PRISM provides a statistically robust advantage over these baselines?
  2. The shape regularizer reduces downstream risk degradation in the main Llama-3.1-8B TruthfulQA experiment, but its effects are smaller in several low-drift settings. Does the regularizer consistently improve retention across different fine-tuning tasks and model families without substantially compromising adaptation performance on the target task?
3 comments
AuthorsOpenReview ↗

Response to Reviewer sfB7 (1/2): summary and W1/Q1, where PRISM has a robust advantage

Thank you for two questions that go to the heart of what the paper should claim. We ran a new experiment for each. In brief:

your pointwhat we didresult
W1 / Q1: a statistically robust advantage over similarity scoreseight-sequence screen on all ten cells, three pools × 100 reference sets eachranks better than 1−CKA and the directly measured gap in every cell (p = 0.002); within each cell, better than 1−CKA (p < 10⁻⁴)
Q2: consistency, and cost to adaptationfour (model, task) cells, three seeds, every method's target-task loss at most 0.01 nats above the unregularized run'sless forgetting in all four cells, in every seed
W2: breadthseven models, three toolchains, two sources of drift; the regularizer on four cells × five benchmarks × three seedsthe bound is never violated (360 quantization pairs, 360 LoRA cases), and its rank correlation with the measured gap is high (mean r_s 0.825 to 0.875)

W1 and Q1: under which settings does PRISM provide a statistically robust advantage?

Short answer: when only eight reference sequences are used. There, PRISM ranks variants better than both 1−CKA and the directly measured gap in all ten cells (p = 0.002). Beyond ranking, it shows which axis moved (scale, shape or the output head), which 1−CKA cannot report by construction.

Eight reference sequences. Screening every candidate variant is cheap only when the reference set is small. We computed each score from eight reference sequences and asked how well it ranks the quantized variants by their risk gap measured on a pool of 512. Setup: ten cells (two model families × five benchmarks, 11 to 12 variants each). For each benchmark we drew three such pools, each with its own seed; the risk gap measured on a whole pool gives the ordering to be recovered, and 100 sets of eight drawn from that pool are scored against it. That makes 300 sets per cell, so the result does not hinge on one choice of the eight.

score at n = 8 (ten cells, 300 reference sets each)mean Spearman r_s (higher is better)worst of all 3,000 sets
risk gap measured directly on the same 8 sequences0.765−0.064
1−CKA0.8510.382
PRISM bound (B_N, Procrustes-aligned)0.8770.564

Across cells, PRISM's mean r_s is higher than both 1−CKA's and the directly measured gap's in all ten cells (p = 0.002 for each). This test has only one value per cell, and p = 0.002 is the smallest value ten samples can give. Within each cell, we test over the 300 sets of eight: both scores are computed on every set, so the comparison is paired, and PRISM's r_s is higher than 1−CKA's on average in every cell, with p < 10⁻⁴ (Llama = Llama-3.1-8B, Qwen = Qwen3-8B-Base):

modelbenchmarkPRISM − (1−CKA), mean over 300 setsp (paired, Holm-corrected)
LlamaARC+0.0355 × 10⁻³⁰
LlamaGSM8K+0.0074 × 10⁻⁹
LlamaMMLU+0.0593 × 10⁻³⁶
LlamaSQuAD+0.0131 × 10⁻⁷
LlamaTriviaQA+0.0096 × 10⁻²⁰
QwenARC+0.0517 × 10⁻²⁰
QwenGSM8K+0.0021 × 10⁻⁵
QwenMMLU+0.0581 × 10⁻²⁹
QwenSQuAD+0.0197 × 10⁻²⁰
QwenTriviaQA+0.0092 × 10⁻⁶

All tests are paired Wilcoxon tests; every p-value survives Holm correction, and a paired t-test gives the same conclusions. The comparison with the directly measured gap is the practical one: without PRISM, a practitioner would measure the gap on those same eight sequences. (1−SVCCA is degenerate at this size on most benchmarks: eight sequences give fewer token rows than feature dimensions.)

The bound computed from eight sequences also stayed above the gap throughout. On those eight this is guaranteed (Theorem 1); on all 512 it is what we observed, for every variant, in every one of the 300 sets and all ten cells. At n = 32, 128 and 512 it holds for every variant and seed.

Beyond ranking. PRISM's aim is not to replace similarity scores, but to show a developer where the current bottleneck is. From the same features 1−CKA uses, plus the head weights already in the model, it separates the drift into scale, shape and the output head. 1−CKA cannot make this split: it is unchanged when the features are rescaled globally, and it does not see the output head. For example, 8-bit quantization in bitsandbytes keeps outlier channels in FP16 by default. On Qwen3-8B-Base we raised its llm_int8_threshold from 6 to 10⁹, so that every channel is quantized. The scale term then rises 6.1× to 95.7×, against 2.7× to 4.3× for the shape term, while the head term stays at zero: the term that responds most is the one this setting targets (reply to Reviewer H946).

AuthorsOpenReview ↗

Response to Reviewer sfB7 (2/2): Q2 consistency and cost, and W2 breadth

Q2: is the improvement consistent, and at what cost to adaptation?

Short answer: yes, on both fine-tuning tasks and both model families, without substantially compromising adaptation. The shape regularizer forgets less than the unregularized run in all four (model, task) cells and in every seed. Adaptation is protected by design: every method must keep its target-task loss at most 0.01 nats (1% in perplexity) above the unregularized run's.

We made "without substantially compromising adaptation" a fixed rule rather than a judgment. For the regularizer, as for every baseline, the strength λ is the largest λ at which the target-task loss stays within that budget; forgetting never enters the choice. Generic replay, our old-domain replay baseline, rehearses 32 WikiText sequences with cross-entropy (the full protocol is in our reply to Reviewer 5VCw). Entries are downstream forgetting (the risk gap |ΔR| on five other benchmarks, averaged; lower is better; mean ± sd over three seeds), with target-task loss in parentheses (lower means better adaptation):

modeltaskno regularizationgeneric replayshape regularizerforgetting vs. no regularization
Llama-3.1-8BTruthfulQA0.771 ± 0.032 (0.862)0.632 ± 0.033 (0.833)0.180 ± 0.005 (0.872)−77%
Llama-3.1-8BBBQ0.258 ± 0.002 (0.114)0.369 ± 0.008 (0.088)0.189 ± 0.010 (0.057)−27%
Qwen3-8B-BaseTruthfulQA0.316 ± 0.024 (0.835)0.308 ± 0.018 (0.837)0.165 ± 0.015 (0.843)−48%
Qwen3-8B-BaseBBQ0.108 ± 0.013 (0.191)0.355 ± 0.010 (0.196)0.068 ± 0.003 (0.127)−37%
  • Consistency. Less forgetting than the unregularized run in every seed of every cell, from 27% less on Llama-3.1-8B/BBQ to 77% less on Llama-3.1-8B/TruthfulQA.
  • Cost. The regularizer's target-task loss is below the unregularized run's on both BBQ cells, and within the budget on both TruthfulQA cells (0.010 and 0.008 nats above).
  • Against old-domain replay, under the same budget, the regularizer forgets less in all four cells, in every seed. This does not depend on replay's λ: even at the λ where replay forgets least within budget, replay still forgets more than the regularizer in all four cells (closest case: 0.104 against 0.068, on Qwen3-8B-Base/BBQ; reply to Reviewer 5VCw).

Two things changed from the submission. All methods, the unregularized run included, now share one training loop, which is why the unregularized numbers differ from the submission's, most visibly on Llama-3.1-8B/BBQ. And the rule can select a λ above 1, where the submission's sweep stopped; at λ = 1 the new runs reproduce the submission's forgetting. The revised abstract will state the benefit per cell, as in this table.

W2: breadth

The evidence spans each factor that changes what the bound must capture:

  • Models. Seven models from four developers, covering base, instruction-tuned and reasoning models (Llama-3.1-8B and its Instruct version, DeepSeek-R1-Distill-Llama-8B, Qwen3-8B Base and Instruct, Ministral-3-8B Base and Instruct), with no violation of the bound in 360 quantization (variant, benchmark) pairs.
  • Compression. Three quantization toolchains (GGUF, bitsandbytes, GPTQ), from 2 to 8 bits.
  • Source of drift. Post-training quantization and LoRA fine-tuning, two different ways of changing the weights, under the same bound.
  • The regularizer. The four (model, task) cells, each now scored on all five downstream benchmarks with three seeds, span unregularized forgetting from 0.108 to 0.771, so Q2 covers both high- and low-forgetting regimes. The comparison now also includes old-domain replay and an anchor-CE control (cross-entropy on the regularizer's own 32 reference sequences), and, on Llama-3.1-8B/TruthfulQA, LwF and representation KD (reply to Reviewer 5VCw).

In short: PRISM names which of three axes moved, with a bound that was never violated; from eight sequences it ranks variants more reliably than 1−CKA or the gap measured directly; and its shape axis yields a regularizer that removes up to three quarters of the forgetting. We would be grateful if you would take these results into account in your assessment. Thank you for the two questions: the eight-sequence screen and the three-seed protocol are central to the revision because you asked them.

AuthorsOpenReview ↗

Your Q1 and Q2 in the revised PDF (Table 2 and Table 3)

Thank you again for your review. In one line: with eight reference sequences, PRISM ranks quantized variants better than 1−CKA in all ten (model, benchmark) cells, and within a shared target-task-loss budget the regularizer forgets less than each of its three references in all four (model, task) cells. Both results are now in the main text of the revised PDF.

Q1 (under which settings PRISM has a statistically robust advantage over CKA and SVCCA): with eight reference sequences.

  • There, PRISM's mean Spearman r_s is higher than that of 1−CKA and of the risk gap on the same eight sequences, in all ten cells (p = 0.002 against each, the smallest value ten samples can give).
  • Over the 300 reference sets per cell, its worst single set still ranks at r_s = 0.564, against 0.382 for 1−CKA (Sec. 4.3, Table 2).
  • 1−SVCCA is degenerate at this size on most benchmarks: eight sequences give fewer token rows than feature dimensions (App. F.2).

Q2 (consistent retention without substantially compromising adaptation): yes.

  • Every regularized method's target-task loss stays at most 0.01 nats (1% in perplexity) above the unregularized run's.
  • Within that budget the regularizer forgets less than each of the three references in Table 3, on both tasks and both model families, in every seed: 27–77% less than no regularization.
  • The other two references are anchor-CE (cross-entropy on the same anchor set, so only the penalty differs) and a 32-sequence old-domain replay.
  • On the cell with the least forgetting to begin with (Qwen3-8B, BBQ), it forgets 37% less (Sec. 4.5, Table 3).

Breadth (your second weakness)

  • The bound held in all 360 quantization (variant, benchmark) rows across seven 8B models (Sec. 4.1; App. A.6, D.3).
  • The regularizer results cover both model families, both fine-tuning tasks and three seeds (Sec. 4.5).

We would be glad to hear whether these answer your two questions. If anything in the paper still seems unclear or incomplete to you, we would be grateful to know.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary for all reviewers and the AC: every requested experiment run, three main results

We thank the three reviewers. We ran every experiment you asked for. You found the theory "correct, self-contained and honest" and the eight-sequence screen "a genuinely useful finding for practitioners" (5VCw), read the experiments as "suggesting that the proposed diagnostic captures useful information about relative degradation" (sfB7), and noted that "leveraging analysis on features to help diagnose the post-training performance is important" (H946).

What PRISM is for

LLMs are now released, quantized and fine-tuned faster than each variant can be evaluated in full. PRISM is a diagnostic for this pace. Its aim is not to replace similarity scores such as 1−CKA, but to fill what they leave out: from one forward pass on a few reference sequences, it ranks variants, bounds their risk gap and shows where to act. 1−CKA cannot do the last two, since it is not a bound, ignores global rescaling and does not see the output head. PRISM adds:

  • A guarantee. On the sequences it is computed from, the bound cannot under-report the risk gap (Theorem 1); it held on all 360 quantization (variant, benchmark) pairs across seven models.
  • Where to act. The bound splits into scale, shape and output-head terms that separate in practice; results 1 and 3 show what acting on them achieves.
  • A cheap screen. From eight sequences it ranks variants better than 1−CKA and than the gap measured directly (result 2).

Three results that answer the central concerns

  1. Less forgetting than genuine old-domain replay (5VCw Q1, sfB7 Q2). In all four (model, task) cells and all three seeds, the shape regularizer forgets less than old-domain replay and the unregularized run. Reviewer 5VCw's Q1 criterion asked for at least three cells.
  2. A statistically robust advantage at small reference sets (sfB7 Q1, 5VCw Q3). With eight sequences, PRISM ranks quantized variants better than both 1−CKA and the gap measured on those sequences in all ten cells (300 reference sets per cell; p = 0.002 against each), and better than 1−CKA within every cell (p < 10⁻⁴).
  3. The axes separate, and the head and scale terms tell where an intervention matters (H946 W2, 5VCw minor 8). Varying the output head's precision with the backbone fixed, the head term correctly ranks how much restoring the head lowers the gap (193 of 194 comparisons), and head and backbone errors add up. Switching off the outlier preservation of 8-bit quantization raises the scale term 6–96×, the shape term 3–4×, and the scale term, read before the switch, separates the model on which the switch changes the gap from the one on which it does not.

New experiments

experimentasked byresult
Old-domain replay and an anchor-CE control, 4 cells × 3 seeds, one target-loss budget for every method5VCw Q1, sfB7 Q2result 1
1−CKA and 1−SVCCA on the ten LoRA cells × 3 seeds5VCw Q2mean r_s 0.825 (PRISM) against 0.790 (1−CKA) and 0.719 (1−SVCCA); bound valid in all 360 LoRA (checkpoint, cell, seed) cases
Eight-sequence screen on all ten quantization cells5VCw Q3, sfB7 Q1result 2
Head axis: restore the original output head (20 cells), and vary its precision with the backbone fixed (210 cells)H946 W2, 5VCw minor 8result 3
Scale axis: toggle outlier-channel preservationH946 W2result 3
Layer-wise and module-wise drift profile (residual stream, q/k/v)H946 W1 / Q1at Q2_K drift peaks mid-network, most in the value projections

Changes in the revised paper (in preparation)

changeprompted by
The "replay" control is renamed anchor-CE; old-domain replay is added; every method is tuned by one target-loss budget, never consulting forgetting, with three seeds5VCw, sfB7
LwF and representation KD will be added to Appendix I; launch scripts for every three-seed round will be released5VCw
The eight-sequence screen (Appendix F.2) moves to the main text on all ten cells, and contribution 3 is reframed around it; the abstract and contributions state the regularizer's benefit per cell5VCw, sfB7
Proposition 1 and Theorem 1 are restated as requested5VCw
The head-term claim is stated per model family, with the head-precision experiment showing why; n is printed next to every r_s5VCw
The layer-wise profile is added with the output-level scope stated; Sec. 5 reports the interventions on all three axes, so each remedy is tested rather than only proposedH946

All new results are in our replies and can be assessed now. The revised PDF and supplementary material are still being prepared; we apologize for the delay and will upload both during the discussion period.

Taken together, PRISM complements similarity scores rather than replacing them: a bound that held in every case we tested, axes that separate in practice, and a screen that works from eight sequences. Your reviews made this a sharper paper; thank you.

Dear Reviewers,

Thank you for your reviews. The authors have submitted their rebuttal. Could you please take a look and check whether it addresses your questions and concerns? Please leave a comment for the authors, and update your review if your assessment has changed.

Thank you again for your time and effort!

Best regards,

Area Chair

AuthorsOpenReview ↗

Revised PDF and code uploaded: where each of your questions is answered

The revised PDF and the supplementary code are now uploaded. Every change listed under "Changes in the revised paper" in our summary above is in the PDF, on a light-blue background, and the code README maps each three-seed round to its launch script, with model versions and dependencies pinned. Table numbers in our earlier replies refer to the submitted PDF; its regularization table (Table 2), for example, is replaced by Table 3.

Your questions, and where the revised PDF answers them

revieweryour questionanswer
sfB7 Q1Under which settings does PRISM provide a statistically robust advantage over CKA and SVCCA?Small reference sets (Sec. 4.3, Table 2). With eight sequences, PRISM's mean r_s is higher than 1−CKA's and than that of the risk gap on the same sequences in all ten cells (p = 0.002 against each, the smallest value ten cells allow). Even its worst single set, over 300 per cell, ranks at 0.564, against 0.382 for 1−CKA and −0.064 for the risk gap. 1−SVCCA is degenerate at this size on most benchmarks, since eight sequences give fewer token rows than feature dimensions (App. F.2).
sfB7 Q2Does the regularizer improve retention across tasks and families without compromising adaptation?Yes, with adaptation held fixed: every method's target-task loss stays within 0.01 nats (1% in perplexity) of the unregularized run's. Within that budget it forgets less than no regularization, a cross-entropy control on the same anchor sequences and a 32-sequence old-domain replay, in all four (model, task) cells and all three seeds, 27–77% less than no regularization (Sec. 4.5, Table 3).
H946 W1, W2Intermediate layers and Q/K/V; interventions on the scale and head axes.Confirmed as addressed in your thread; now in the paper, where Sec. 5 tests an intervention on each axis rather than only proposing one. Head: the head term orders 193 of the 194 pairs of head settings whose benefit from restoring the original head differs beyond noise (App. K.1). Scale: on Qwen3-8B, quantizing the outlier channels raises the scale term 6.1–95.7×, against 2.7–4.3× for the shape term (App. K.2). Shape: the regularizer (Sec. 4.5). Layers and q/k/v: App. K.3 (Table 33), with the output-level scope stated.
5VCwThe revised PDF matching the replies; representation KD on all four cells.Both are in this upload (Sec. 4.5, App. I, Table 29); point by point in the thread with Reviewer 5VCw.

What holds across the paper. On the sequences it is computed from, the bound held in every check we ran: all 360 quantization (variant, benchmark) rows over seven 8B models, all 360 LoRA (checkpoint, cell, seed) cases, and every cell of the head and scale interventions; a bound from eight sequences also stayed above the gap on all 512, for every variant in every set. PRISM ranks variants on par with CKA and SVCCA on full reference sets and adds what they do not give: a bound on the risk gap at the head the model is deployed with, a split into scale, shape and head with a tested intervention on each axis, and a better ordering from eight sequences.

Your questions shaped this revision: the eight-sequence screen now leads contribution 3 in the main text (5VCw Q3, sfB7 Q1), every regularized method shares one target-task-loss budget (5VCw Q1, sfB7 Q2), and each axis has a tested intervention (H946 W2). Thank you. We hope the final discussion can rest on the revised version; if anything in it is still open, we will answer it here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.