acceptodds
Under review as a conference paper at ICLR 2027

Compression-Induced Representational Dissonance: Anticipating and Suppressing Prediction Flips

Abstract

A compressed model can preserve benchmark accuracy while altering individual predictions, leading to potentially severe reliability failures. Across accuracy- preserving quantization, pruning, and low-rank compression of 58 vision models on 9 datasets, we find that models lose only 0.42 percentage points of accuracy on average while changing 9.4% of their predictions. We trace these changes to the interaction between compression error and the constraints it imposes, and to how they affect features of different energies. Compression disproportionately distorts low-variance feature directions, drowning them in noise. Whether these changes alter a prediction depends on the base model’s decision margin, linking differential representational change to decision instability. This relationship generalizes across compression families and model classes, allowing prediction flips to be anticipated before evaluating the compressed model on the target inputs. Across 453 configurations of vision and language models spanning quantization, sparsification, and low-rank, we predict the flip rates with a median relative error of 5.7%. And as we focus more on preserving representation during compression, we substantially restore decision fidelity and reduce flips at matched accuracy by 50 to 93%.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Model compression, efficient deep learning, representation learning, model reliability, quantization, knowledge distillation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FcHeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how model compression can preserve overall accuracy while changing individual predictions. It analyzes compression error in the principal directions of the original representation and finds that low-variance directions can have poor signal-to-noise ratios and recover less under a persistent compression constraint. The paper also estimates changes in the original winner–runner-up margin using base-model sensitivities and candidate compression perturbations, without evaluating the compressed model on the target inputs. Finally, it combines spectral energy matching with feature distillation to reduce disagreement with the original model. Experiments cover quantization, pruning, and low-rank compression across vision models, with additional language-model evaluations. The reported median relative error for estimating winner–runner-up reversal rates is 5.7% across 453 configurations.

Critical strengths
  1. The paper provides a useful empirical account of what aggregate accuracy misses after compression. It connects representation changes with decision changes across several compression families and many model configurations. The evaluation before applying the task-retention screen in Table 8 is also valuable: it shows that the forecasting result is not simply a consequence of retaining favorable compression settings. This breadth makes the findings more useful than an isolated example of compression-induced disagreement.
  2. The analysis leads to practical improvements in preserving decisions. The signed margin estimate improves over the magnitude-based and model-transfer baselines in Table 3. The suppression experiments also show gains beyond feature distillation alone. For example, Table 13 reduces disagreement from 12.61% after post-training quantization to 4.24% with feature distillation and 1.86% with SEM plus distillation, while maintaining similar task accuracy. These are meaningful results even though several underlying compression components build on existing methods.
Critical weaknesses
  1. The headline forecasting result is narrower than “prediction flips” suggests. Equations (6)–(7) predict whether the original runner-up overtakes the original winner, rather than whether any class changes the final prediction. Appendix F reports that this event covers a median of 87% of full prediction disagreement, with coverage as low as 57%. The reported errors for estimating full disagreement are 10.0% after a coverage correction and 13% using the top five competitors. Table 7 also shows substantial false-positive and false-negative counts for rounding that largely cancel in the estimated rate. This does not invalidate aggregate rate estimation, but it means that a small rate error should not be interpreted as accurate identification of individual changed predictions. These distinctions are acknowledged in the appendix and should qualify the main claims more clearly.
  2. The recovery experiments do not fully isolate the effect of maintaining the compression constraint. Appendix C explicitly acknowledges that the Gaussian controls match aggregate initial damage rather than its directional structure. In its three-bit weight-only comparison, the control starts with 219 directions below unit SNR, compared with 355 under compression. Table 2 also shows substantially different initial accuracy losses in some matched pairs. The observed recovery gap is informative, but these different starting conditions make it difficult to attribute that gap specifically to the persistence of the constraint. The evidence supports the empirical finding more strongly than the causal interpretation of why the gap occurs.
Questions for authors
  1. Forecast target and input-level reliability: Can you provide a compact breakdown by compression family of the errors for winner–runner-up reversals and full-argmax prediction changes, together with input-level precision and recall? This would clarify whether the strongest result is accurate rate estimation, reliable identification of fragile inputs, or both.
  2. Recovery mechanism: For weight pruning, can the constrained and unconstrained recovery runs start from the same pruned weight tensors, giving identical initial predictions and directional errors, with only one run retaining the mask? Alternatively, what existing evidence separates the effect of persistence from the different initial perturbation structures acknowledged in Appendix C?
1 comment
AuthorsOpenReview ↗

Author's rebuttal to reviewer FcHe

We thank Reviewer FcHe for their review.

W1 / Q1. Forecast target and input-level reliability.

"The headline forecasting result is narrower than "prediction flips" suggests." "Can you provide a compact breakdown by compression family of the errors for winner–runner-up reversals and full-argmax prediction changes, together with input-level precision and recall?"

The reviewer is correct that the 5.7% describes one kind of flip. Our estimate works from the margin between the winner and the runner-up, so the flip it predicts is the runner-up overtaking the winner (Eq. 7). This is by far the most common way a prediction changes: it accounts for a median 87% of all flips. The remaining flips go to a third class, and because the estimate does not see them, its error rises to about 10% when every flip is counted. Even then, it orders the 420 configurations by their total flip rate almost perfectly (rank correlation 0.98), which is what choosing between compressions requires. We now state both numbers together in the abstract.

Here is the breakdown asked for, by family (medians over 420 configurations):

familyrate error, runner-uprate error, any class (raw / corrected)precision / recall, runner-upprecision / recall, any classavg. precision (margin alone)
rounding8.0%20.5 / 11.5%58 / 52%62 / 48%0.58 (0.44)
pruning4.2%12.3 / 10.2%80 / 80%81 / 70%0.82 (0.44)
low rank2.6%15.7 / 6.5%88 / 88%90 / 76%0.90 (0.45)

The rate is well estimated in all three compression families. The same score also identifies which inputs flip.

W2 / Q2. Recovery mechanism.

"The recovery experiments do not fully isolate the effect of maintaining the compression constraint." "can the constrained and unconstrained recovery runs start from the same pruned weight tensors, giving identical initial predictions and directional errors, with only one run retaining the mask?"

We agree that our Gaussian control matched only the total amount of damage, so it might not separate the effect of the constraint from the effect of a different starting point. To separate the two, we ran the experiment proposed: two runs start from the same compressed weights, and only one of them keeps the constraint while it retrains. Any difference between them can then only come from the constraint.

3 seeds, for ResNet-18 on CIFAR-100 and a ViT trained from scratch on Tiny-ImageNet. Recovery is the decodability (0–1) of the directions below SNR 1 at the start.

modelsettingrecovery of weak directions, constrains kept vs releasedpredictions changed (%), constrains kept vs released
ResNet-182-bit weights0.738 ± 0.003 vs 0.816 ± 0.00317.7 ± 0.2 vs 14.0 ± 0.3
ResNet-18W3A30.619 ± 0.003 vs 0.768 ± 0.00519.2 ± 0.1 vs 11.2 ± 0.2
ResNet-18rank 1/40.764 ± 0.000 vs 0.808 ± 0.00416.1 ± 0.2 vs 12.8 ± 0.1
ResNet-1870% structured pruning0.679 ± 0.003 vs 0.702 ± 0.00023.4 ± 0.2 vs 21.3 ± 0.2
ViTW3A30.470 ± 0.012 vs 0.667 ± 0.00249.3 ± 2.1 vs 38.1 ± 0.7
ViTrank 1/40.481 ± 0.003 vs 0.511 ± 0.00141.4 ± 0.3 vs 39.2 ± 0.3

The two runs start from the same weights and train under the exact same configuration, differing only in whether the constraint remains. With the constraint kept, the weak directions recover less in every setting and in every run.

We thank the reviewer again and remain available for any further questions or concerns. The submitted PDF has been updated with the revision incorporating the flips remark and additional ablation (in green).

Reviewer_U7Z7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an issue often overlooked in previous research on model compression: even when a compressed model maintains its overall task accuracy, it may still produce different decisions from the original model for a significant number of specific inputs. The authors term this phenomenon "compression-induced representational dissonance" and attempt to explain its origins through three lenses: representation, decision boundaries, and compression error.

Critical strengths
  1. The experiments are extensive, and the details are clearly presented.
  2. The authors identify that low-energy feature directions are more susceptible to falling into low-SNR regions; they further link representation error to prediction flips by analyzing classifier sensitivity and decision margins.
  3. Beyond merely considering the magnitude of compression perturbations, the authors incorporate directional information relative to the decision margin. They use the base-model margin, weight sensitivity, and activation sensitivity to predict whether a margin crossing will occur post-compression.
Critical weaknesses
  1. It remains unclear whether the phenomenon is truly driven by "compression-induced representational dissonance." If the root cause of prediction flips is the geometry of the decision margin, why does the proposed method not optimize this geometry directly?
  2. There is a potential logical inconsistency: the paper initially argues that low-energy directions are critical yet easily obscured by compression noise; however, the subsequent methodology imposes larger penalties for compression errors in higher-energy input dimensions. This creates confusion regarding the underlying rationale.
  3. In many models, activation dimensions often exhibit significant correlations. Relying solely on the RMS of individual dimensions may overlook important joint directions. It is necessary to investigate the extent of information loss resulting from the diagonal approximation.
  4. Table 15 indicates that the application of techniques like QuaRot or GPTQ leads to a substantial reduction in token disagreement. This suggests that the reported "28%–81% change in answers" for LLMs may largely stem from the use of less robust compression pipelines.
Questions for authors

Please refer to the "Weaknesses" for further details. In my view, this paper leans more toward analysis and measurement; consequently, my current assessment is a borderline reject. I look forward to the authors' clarifications and response.

1 comment
AuthorsOpenReview ↗

Author's rebuttal to reviewer U7Z7

We thank Reviewer U7Z7 for their review.

W1. Is it dissonance, and why not optimise the margin?

"It remains unclear whether the phenomenon is truly driven by 'compression-induced representational dissonance.' If the root cause of prediction flips is the geometry of the decision margin, why does the proposed method not optimize this geometry directly?"

The margin shifts because the features shift (but not in the same way): compression introduces error into the representation, and the classifier reads it. The term "Dissonance" was introduced because, even after retraining, high-energy features adapt to that error while low-energy ones recover less.

  • We also trained on the margin directly. It helps, but less than matching the features, and applying SEM first helps more than any choice of loss. Predictions changed (%), ViT-B/16 W8A8, mean ± SD over 3 seeds:
training losswithout Spectral Energy Matchingwith Spectral Energy Matching
none7.032.88
margin5.57 ± 0.152.35 ± 0.06
logit distillation4.75 ± 0.271.91 ± 0.06
feature distillation4.57 ± 0.041.73 ± 0.03

W2. Low-energy directions vs energy weighting.

"There is a potential logical inconsistency: the paper initially argues that low-energy directions are critical yet easily obscured by compression noise; however, the subsequent methodology imposes larger penalties for compression errors in higher-energy input dimensions."

We thank the reviewer for this point, which the paper should have made clearer. Compression methods generally minimize an error that is dominated by the high-energy components, so the low-energy ones get little attention. we are not arguing that the low-energy directions are the most important. We argue that they still affect decisions and are the first to be drowned by compression noise. The high-energy components carry the general features and most of the signal, so errors there cost the most, which is why SEM also weights by input energy.

This weighting also helps the weak directions, since lowering the output error lowers the noise they sit in. Their SNR rises from 12.5 to 117 with SEM and distillation, against 28.7 with distillation alone.

W3. Diagonal approximation.

"It is necessary to investigate the extent of information loss resulting from the diagonal approximation."

We measured it as recommended, and the diagonal loses almost nothing in prediction change.

  • Rounding and pruning: it predicts the true layer-output error within 2% on all 48 layers of ViT-B, and a full-covariance method (SparseGPT) gives the same result (10.36% vs 10.57% changed).
  • Low rank: errors are correlated, so the full second moment lowers layer error further. Predictions changed stay the same: 19.41% with the diagonal, 19.20% with the full form (SVD-LLM).

W4. The "28–81%" for LLMs.

"This suggests that the reported "28%–81% change in answers" for LLMs may largely stem from the use of less robust compression pipelines."

We reran Llama-2-7B with strong pipelines. Answer changes go down, but they do not go away.

  • W8A8, QuaRot+GPTQ or SEM+GPTQ: exact match is unchanged or higher, and 7–12.5% of answers still change.
  • W4A8, SEM with distillation: exact match stays within 1–4 points of the base, and 37–38% of answers change.

Analysis vs method.

"This paper leans more toward analysis and measurement"

The method comes from that analysis and measurement. Flips are a recent notion (Dutta et al., NeurIPS 2024), so we devoted a large part of the paper to controlled experiments that explain their origin. Those experiments are what motivate the method and its results.

  • The analysis predicts, before compression is ran, which transforms reduce flips (predicted 1.9–2.1% vs 4.2–5.1%; measured 2.3–2.7% vs 6.1–6.2%), which can be reduced with our second contribution.

We thank the reviewer again for their feedback and remain available for any further questions or concerns. The submitted PDF has been updated with the revision incorporating your recommendations (in orange).

Reviewer_T3KEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies prediction changes that remain hidden when compressed models preserve aggregate accuracy.

It analyzes compression error through representation-space SNR, estimates flip rates using signed margin perturbations, and proposes spectral energy matching followed by feature distillation to reduce disagreement.

The problem is important and the evaluation is broad, but the proposed mechanism and reliability interpretation are not fully supported by the current controls.

Critical strengths
  1. Broad characterization of an important deployment problem.

Across many models, datasets, and compression families, the paper convincingly shows that preserving accuracy does not guarantee agreement with the base model.

Evaluating decision fidelity in addition to aggregate performance is practically valuable.

  1. Unified diagnosis, anticipation, and mitigation framework.

The signed estimator improves aggregate flip-rate prediction over margin- and variance-based alternatives, while SEM with feature distillation substantially reduces disagreement at similar task performance.

The paper also carefully distinguishes models, configurations, and seeds as statistical units.

Critical weaknesses
  1. The proposed mechanism and anticipation claims are overstated.

Low-SNR directions are not specific to flipped inputs, and some models exhibit substantial disagreement with few or no such directions.

Moreover, the recovery control matches only aggregate feature error, not its directional profile, so it does not isolate the effect of a persistent compression constraint.

The reported 5.7%5.7\% error also concerns aggregate flip rates: for quantization, high false-positive and false-negative rates largely cancel, limiting the method’s ability to identify individual fragile inputs.

  1. Experimental selection and method comparisons are insufficiently controlled.

It is unclear whether configurations are selected using the same evaluation accuracy later used to report disagreement, which could bias the headline accuracy-preserving population.

The suppression experiments also compare mainly against plain compression, feature distillation, and an in-house QuaRot implementation.

Comparisons with stronger modern compression methods under matched calibration data and optimization budgets are missing, and several pruning, low-rank, and language results are based on single runs.

Questions for authors
  • Configuration selection.

Were the compression configurations selected on a split independent of the one used to report accuracy and disagreement?

If not, do the headline results remain on a fully held-out test set?

  • Causal recovery control.

Does the recovery gap remain when the one-time perturbation matches the compression error spectrum in each principal direction, rather than only its total feature-error energy?

  • Practical meaning of prediction changes.

How do the results decompose into correct-to-incorrect, incorrect-to-correct, and incorrect-to-incorrect changes?

Please also report input-level precision and recall for identifying flips, rather than only aggregate rate error.

  • Stronger matched-budget baselines.

How does SEM with feature distillation compare with strong (or more recent) calibration-, rotation-, or distillation-based compression methods under the same data, optimization, bit-width, and budgets?

1 comment
AuthorsOpenReview ↗

Author's rebuttal to reviewer T3KE

We thank Reviewer T3KE for their review.

W1. Low-SNR directions.

"Low-SNR directions are not specific to flipped inputs, and some models exhibit substantial disagreement with few or no such directions."

We argue that compression damages weak directions more than strong ones, relative to their signal. We do not claim that they mark which inputs flip, or that the SNR needs to fall below 1 to take effect: Section 4 says the spectrum shows that the most damage is in the tail, and the margin determines whether a prediction changes.

  • SNR < 1 is only a reporting & plotting threshold, and signals when noise outweighs information; weak directions have a lower SNR that is also further reduced relative to stronger directions.

W2 / Q2. Causal recovery control.

"Does the recovery gap remain when the one-time perturbation matches the compression error spectrum in each principal direction […]?"

Yes, with a stricter control: both runs start from the same compressed weights, so their errors match in every direction, and only one keeps the constraint.

  • Weak directions recover less with the constraint kept, in every identical-start setting and every seed. Paired intervals exclude zero in all but rank 1/2, the mildest constraint.
  • The deficit falls most on weak directions: it shrinks as direction variance grows in every setting and seed.
  • Capacity sufficient for accuracy is not sufficient for recovery: at 70% sparsity the constrained model matches base accuracy and still recovers less.
  • Decisions follow under quantization and low rank (2-bit: 17.7% vs 14.0%). At 70% sparsity both runs sit within 1.5 points of the churn of continued training, so little is left to compare.
  • A ViT trained from scratch repeats the same result.

W3 / Q3. Meaning of prediction changes.

"high false-positive and false-negative rates largely cancel, limiting the method's ability to identify individual fragile inputs." "How do the results decompose into correct-to-incorrect, incorrect-to-correct, and incorrect-to-incorrect changes?"

A flip records what happened to one input after compression. Aggregate metrics such as accuracy average over inputs, while false-positive and false-negative rates can record diffrent samples, so individual failures cancel and stay hidden; the flip rate follows these samples and counts them. That rate is what the paper anticipates.

  • The same score also identifies the inputs that flip: 80–88% precision and recall for pruning and low rank, and 58 / 52% for rounding, eight times the 7.4% base rate. It beats the margin alone in 385 of 420 configurations.
  • The decomposition you asked for, on ImageNet-Sketch: 1.00% correct-to-incorrect, 0.93% incorrect-to-correct, 5.13% incorrect-to-incorrect. SEM with distillation cuts them to 0.26 / 0.26 / 1.21%.
  • On accurate models, most flips are harmful: on the six pairs above 70% base accuracy, 34–71% of changes are correct-to-incorrect.

W4 / Q1. Configuration selection.

"Were the compression configurations selected on a split independent of the one used to report accuracy and disagreement?"

No: in the submission, they were selected and scored on the full val sets. This is by design: flips matter where accuracy looks matched, because breakage stays hidden there; a visibly lossy compression is already reported by accuracy. To show this does not inflate the result, we now select on one half of the inputs and report on the other (50 splits, 44 models).

  • Held out, the hidden disagreement is slightly larger: 13.31% changed (12.65% in-sample) for 0.51 points lost (0.42).
  • The anticipation error is 6.6% with or without selection on the scored half.

W5 / Q4. Stronger matched-budget baselines.

"Comparisons with stronger modern compression methods under matched calibration data and optimization budgets are missing"

We added two recent methods, each run under the same conditions as ours (same data, budget and quantizer).

Ours is SEM with the same distillation on vision, and SEM + GPTQ without training on language. Flips are the share of inputs (vision) or tokens (language) whose prediction changes; mean ± SD over 3 seeds.

SettingStrong recent methodFlips, theirs (%)Flips, ours (%)Fewer flips (%)
ViT-B/16, W8A8RepQ-ViT + distillation4.11 ± 0.141.73 ± 0.0358
DeiT-B, W8A8RepQ-ViT + distillation4.28 ± 0.212.42 ± 0.1243
Llama-2-7B, W8A8QuaRot (Official) + GPTQ1.51 ± 0.131.02 ± 0.2132
Qwen2.5-0.5B, W8A8QuaRot (Official) + GPTQ3.02 ± 0.092.30 ± 0.0324

W6. Single runs.

"Several pruning, low-rank, and language results are based on single runs"

All experiments now use at least three seeds.

The submitted PDF has been updated with the revision incorporating your recommendations (in blue). We thank the reviewer again and remain fully available to address any further questions or concerns.

Comments on the paper

1 comment

Dear Reviewers, please take a moment to read the author responses and engage in the discussion. In particular, please comment on whether your questions and concerns have been adequately addressed and follow up on any points that remain unclear. Thank you! AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.