- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work tries to connect the performance of post-training variants and the feature geometry. A bound named PRISM is proposed, where the cross-entropy risk gap is decomposed into three parts: scale, shape, and head. A shape regularization term is further proposed to improve the post-training. Extensive empirical results somehow validate the effectiveness of PRISM.
Critical strengths
- The addressed topic is interesting. Leveraging analysis on features to help diagnose the post-training performance is important.
- The decomposition of PRISM is reasonable, and the experiment results somehow validate this point.
- Experiments cover a variety of models, protocols, tasks, and benchmarks.
Critical weaknesses
-
PRISM derives its bound entirely from the feature immediately preceding the linear head. However, this feature is a highly abstracted representation that has passed through dozens of Transformer layers, and it may act as an information bottleneck and may miss rich information in intermediate representations. PRISM may under-report the true risk of a variant whose degradation is concentrated in intermediate layers, and it cannot provide the layer-wise or module-wise attribution that practitioners often need when debugging aggressive quantization. The authors should discuss this limitation explicitly and clarify whether the diagnostic can be extended to, or complemented by, richer intermediate representations (e.g., Q/K/V statistics or per-layer feature drift).
-
The paper argues that each of the three axes, i.e., shape, scale, and head, “suggests a different fix,” but only the shape regularizer is implemented and empirically validated (Sec. 4.5). For the scale axis, the paper suggests per-channel scaling or outlier-channel preservation (line 417), and for the head axis, no corresponding intervention is tested. As a result, the paper does not verify that acting on the scale or head diagnosis actually reduces the bound or the risk gap, nor that the axis attribution correctly identifies the dominant failure source. The authors should either add intervention experiments for the scale and head axes, or temper the actionability claim and present the three-axis decomposition as descriptive rather than prescriptive.
Questions for authors
-
Could similar analysis be potentially applied to early-layer features such as the Q, K, V modules in attention blocks?
-
Regarding the 2nd weakness above, although the authors acknowledge this limitation in Section 5, it is suggested to supplement some preliminary intervention experiments to validate the claimed actionability of the scale and head axes.
4 comments
Response to Reviewer H946 (2/2): W2/Q2, head and scale interventions run
W2 / Q2: interventions on the head and scale axes
You offered two routes, interventions or a tempered claim; we took the first, on both axes. The bound holds in every cell below. We use a 0.02-nat noise floor; we only count differences above it.
Each axis has its own intervention:
| axis | intervention | evidence here |
|---|---|---|
| head | keep the output head at higher precision | the head term ranks how much restoring the head lowers the gap (193 / 194 pairs) |
| scale | keep outlier channels in FP16 | scale term 6.1× to 95.7× lower; it is large on the model where this remedy changes the gap and small on the other |
| shape (1−Ω) | shape regularizer during fine-tuning | less forgetting than no regularization, anchor-CE and generic replay in all four cells and every seed (reply to Reviewer 5VCw) |
Head axis
Experiment 1: does the gap split into a head part and a backbone part? From a GGUF variant we build two hybrids: its quantized backbone (everything before the output head) with the original head, and the original backbone with its quantized head (Qwen3-8B-Base at Q6_K, Q4_K_M and Q2_K; Llama-3.1-8B at Q6_K; five benchmarks; 20 cells). The two hybrids' gaps add up to the variant's gap (median ratio 0.999 over the ten cells whose gap exceeds the noise floor, all on Qwen3-8B-Base). In the hybrid with the original head, the feature term alone still ranks the three Qwen3-8B-Base bit-widths correctly on every benchmark.
Experiment 2: vary the head's precision. The head's precision can be set separately from the backbone's. For each of three backbones (original, GGUF Q4_K_M, GGUF Q2_K) we swap in seven heads: BF16, Q8_0, Q6_K, Q5_0, Q4_0, and 3- and 2-bit as a stress test. Two models, five benchmarks, 210 cells.
| what the head term should rank | pairs ranked correctly | Q8_0 to Q4_0 only |
|---|---|---|
| the gap caused by the head alone (original backbone) | 92 / 92 | 18 / 18 |
| how much restoring the original head lowers the gap (quantized backbones) | 193 / 194 | 33 / 33 |
Head and backbone errors again add up (interaction: a median 2% of the gap). The head term also separates the two models. It is larger on Qwen3-8B-Base at every head format, and Qwen3-8B-Base's head is indeed far more fragile (gap from the head alone, median over five benchmarks):
| head | Q8_0 | Q6_K | Q5_0 | Q4_0 | 2-bit |
|---|---|---|---|---|---|
| Qwen3-8B-Base | 0.0016 | 0.0030 | 0.0216 | 0.0759 | 1.44 |
| Llama-3.1-8B | 0.0002 | 0.0003 | 0.0019 | 0.0005 | 0.13 |
Every Llama-3.1-8B value up to Q4_0 is within noise, so all pairs in the last column of the pairs table come from Qwen3-8B-Base.
Takeaway. Before any change is made, the head term ranks head settings by how much restoring the original head would lower the gap. On Qwen3-8B-Base, restoring a Q4_0 head lowers the gap by up to 0.15 nats (beyond noise in 9 of 10 cells), while restoring a Q8_0 or Q6_K head changes it by at most 0.018, within noise.
On attribution. Head and backbone gaps add up, so each source's part of the gap can be measured directly, and the dominant source is read from these parts. Each term ranks within its own axis: across head settings and across models.
Scale axis
8-bit quantization in bitsandbytes keeps outlier channels in FP16 by default, which is
the scale remedy the paper names. Raising its llm_int8_threshold from 6 to 10⁹
switches this remedy off: every channel is quantized and nothing else changes. On
Qwen3-8B-Base (n = 512 per benchmark; outliers-kept gaps match the paper's Table 10 to
within 0.01 nats):
| benchmark | gap (nats), outliers kept | gap, all quantized | scale term, all quantized ÷ kept | shape term, all quantized ÷ kept |
|---|---|---|---|---|
| SQuAD | 0.0036 | 0.0643 | 86.3× | 4.3× |
| TriviaQA | 0.0239 | 0.0307 | 12.3× | 3.4× |
| MMLU | 0.0120 | 0.0173 | 95.7× | 3.4× |
| ARC | 0.0681 | 0.0240 | 6.1× | 3.1× |
| GSM8K | 0.0006 | 0.0317 | 55.6× | 2.7× |
Beyond the noise floor, keeping the outliers lowers the gap on SQuAD and GSM8K and raises it on ARC; on TriviaQA and MMLU the change is within noise. The head term stays at zero.
In absolute terms the shape term is still the larger of the two; the scale term is the one that responds most to the switch (6× to 96×, against 3× to 4×). With outliers kept, the bound is lower on all five benchmarks.
With outliers kept, the scale term already tells the two models apart: it is large on Qwen3-8B-Base (0.5 to 12.5) and small on Llama-3.1-8B (at most 0.115). The switch then moves the gap by up to 0.061 nats on Qwen3-8B-Base and by at most 0.011 on Llama-3.1-8B, within noise.
Sec. 5 will report the interventions on all three axes, so each axis's remedy is tested rather than only proposed.
We would be grateful if you would take the new evidence into account in your assessment. Thank you again: the layer profile and both interventions will be in the paper because you asked for them.
I appreciate the authors’ careful responses. My concerns are adequately addressed, and thereby I maintain my initial positive rating.
Thank you; W1 and W2 are now in the revised paper
Thank you for confirming that your concerns are addressed, and for the time you have given to our responses. Since your comment, the revised PDF is up, so what you found addressed can now be checked in the paper itself: Sec. 5 summarizes both points, and App. K gives the details.
- W1 and Q1 (App. K.3, Table 33): the per-layer and q/k/v drift profile you suggested. On Qwen3-8B-Base at Q2_K, residual-stream drift peaks at block 22 (0.173) and falls to 0.049 at the output, and the value projections drift the most. K.3 states the scope you asked us to discuss: the bound is output-level, and the profile localizes the drift.
- W2, head (App. K.1, Table 30): a variant's gap splits into a head part and a backbone part, which add up to it within 0.007 nats in all 20 cells; of the 194 pairs of head settings whose benefit from restoring the original head differs beyond noise, the head term orders 193 correctly.
- W2, scale (App. K.2): quantizing the outlier channels that 8-bit bitsandbytes keeps in FP16 raises the scale term 6.1× to 95.7× on Qwen3-8B-Base, against 2.7× to 4.3× for the shape term. With outliers kept, the scale term is large on Qwen3-8B-Base, where the switch moves the gap by up to 0.061 nats, and small on Llama-3.1-8B, where it moves it within noise.
- W2, shape (Sec. 4.5): the regularizer forgets less than no regularization in all four (model, task) cells and every seed.
Your review called "leveraging analysis on features to help diagnose the post-training performance" important. The paper now follows that diagnosis through: ranking the variants, seeing which axis moved and where in the network, and testing a remedy on each axis. Both additions address points from your review, and we are grateful for them.
Response to Reviewer H946 (1/2): W1/Q1, intermediate layers; layer-wise and Q/K/V profile added
Thank you for pressing on exactly the two points where the paper needed more evidence. We ran the layer-wise profile and the scale and head interventions you asked for.
W1 / Q1: intermediate layers, and layer-wise attribution
Can the bound under-report risk when degradation sits in intermediate layers? The interior drift you describe does occur (table below), but it cannot make the bound under-report, for a structural reason. The logits are exactly Z·H, where Z is the final hidden state and H the output head, so the loss depends on the model only through (Z, H). Any upstream change that affects the loss must pass through Z, and PRISM reads Z. On the sequences PRISM is computed from, reading only Z therefore misses nothing; this is also what gives Theorem 1 its closed form.
Can the analysis be applied to Q, K, V and intermediate layers? Yes, as a descriptive profile, and we have run it (table below). At an intermediate layer the bound no longer covers the model's own loss, because it needs a linear map from features to logits, which only the final layer has. At layer l, a linear probe can supply that map, and Theorem 1 then bounds the probe's cross-entropy gap in the same way. If both models share one probe, the head term is zero; if each has its own, the head term measures how far the two probes differ. Either way the guarantee is on the probe's loss, not the model's, so the paper keeps the bound at the deployed head and uses the profile below for localization.
We ran four GGUF (llama.cpp) quantized variants on 128 MMLU sequences (about 14k tokens). In every block we read four points: the residual stream after the block and the outputs of its q, k and v projections. At each point we computed the shape drift 1−Ω between the original model (the target) and the variant. Here we use a per-token version of Ω: the cosine similarity between the target's and the variant's feature for the same token, averaged over tokens, so 1−Ω = 0 means no drift. We normalize each token's feature ourselves, as the RMSNorm in front of every block does, because in the residual stream a few tokens carry very large activations and would otherwise dominate. The profile tracks the shape term only: inside the network there is no head, and this normalization sets scale aside, so the scale term is read at the output (W2). Entries are the peak drift over depth, with the block where it occurs (blocks numbered from 1):
What the profile adds to the output-level reading:
Together, the two levels give a workflow. The bound at the output ranks the variants and shows which term moved; the profile then shows where in depth and in which module the drift is largest, so a developer knows where to look first (here, the value projections of the middle blocks at Q2_K). The revision adds the profile and states this division of labor: the bound is the output-level guarantee, and the profile localizes drift by layer.