- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how model compression can preserve overall accuracy while changing individual predictions. It analyzes compression error in the principal directions of the original representation and finds that low-variance directions can have poor signal-to-noise ratios and recover less under a persistent compression constraint. The paper also estimates changes in the original winner–runner-up margin using base-model sensitivities and candidate compression perturbations, without evaluating the compressed model on the target inputs. Finally, it combines spectral energy matching with feature distillation to reduce disagreement with the original model. Experiments cover quantization, pruning, and low-rank compression across vision models, with additional language-model evaluations. The reported median relative error for estimating winner–runner-up reversal rates is 5.7% across 453 configurations.
Critical strengths
- The paper provides a useful empirical account of what aggregate accuracy misses after compression. It connects representation changes with decision changes across several compression families and many model configurations. The evaluation before applying the task-retention screen in Table 8 is also valuable: it shows that the forecasting result is not simply a consequence of retaining favorable compression settings. This breadth makes the findings more useful than an isolated example of compression-induced disagreement.
- The analysis leads to practical improvements in preserving decisions. The signed margin estimate improves over the magnitude-based and model-transfer baselines in Table 3. The suppression experiments also show gains beyond feature distillation alone. For example, Table 13 reduces disagreement from 12.61% after post-training quantization to 4.24% with feature distillation and 1.86% with SEM plus distillation, while maintaining similar task accuracy. These are meaningful results even though several underlying compression components build on existing methods.
Critical weaknesses
- The headline forecasting result is narrower than “prediction flips” suggests. Equations (6)–(7) predict whether the original runner-up overtakes the original winner, rather than whether any class changes the final prediction. Appendix F reports that this event covers a median of 87% of full prediction disagreement, with coverage as low as 57%. The reported errors for estimating full disagreement are 10.0% after a coverage correction and 13% using the top five competitors. Table 7 also shows substantial false-positive and false-negative counts for rounding that largely cancel in the estimated rate. This does not invalidate aggregate rate estimation, but it means that a small rate error should not be interpreted as accurate identification of individual changed predictions. These distinctions are acknowledged in the appendix and should qualify the main claims more clearly.
- The recovery experiments do not fully isolate the effect of maintaining the compression constraint. Appendix C explicitly acknowledges that the Gaussian controls match aggregate initial damage rather than its directional structure. In its three-bit weight-only comparison, the control starts with 219 directions below unit SNR, compared with 355 under compression. Table 2 also shows substantially different initial accuracy losses in some matched pairs. The observed recovery gap is informative, but these different starting conditions make it difficult to attribute that gap specifically to the persistence of the constraint. The evidence supports the empirical finding more strongly than the causal interpretation of why the gap occurs.
Questions for authors
- Forecast target and input-level reliability: Can you provide a compact breakdown by compression family of the errors for winner–runner-up reversals and full-argmax prediction changes, together with input-level precision and recall? This would clarify whether the strongest result is accurate rate estimation, reliable identification of fragile inputs, or both.
- Recovery mechanism: For weight pruning, can the constrained and unconstrained recovery runs start from the same pruned weight tensors, giving identical initial predictions and directional errors, with only one run retaining the mask? Alternatively, what existing evidence separates the effect of persistence from the different initial perturbation structures acknowledged in Appendix C?
Author's rebuttal to reviewer FcHe
We thank Reviewer FcHe for their review.
W1 / Q1. Forecast target and input-level reliability.
The reviewer is correct that the 5.7% describes one kind of flip. Our estimate works from the margin between the winner and the runner-up, so the flip it predicts is the runner-up overtaking the winner (Eq. 7). This is by far the most common way a prediction changes: it accounts for a median 87% of all flips. The remaining flips go to a third class, and because the estimate does not see them, its error rises to about 10% when every flip is counted. Even then, it orders the 420 configurations by their total flip rate almost perfectly (rank correlation 0.98), which is what choosing between compressions requires. We now state both numbers together in the abstract.
Here is the breakdown asked for, by family (medians over 420 configurations):
The rate is well estimated in all three compression families. The same score also identifies which inputs flip.
W2 / Q2. Recovery mechanism.
We agree that our Gaussian control matched only the total amount of damage, so it might not separate the effect of the constraint from the effect of a different starting point. To separate the two, we ran the experiment proposed: two runs start from the same compressed weights, and only one of them keeps the constraint while it retrains. Any difference between them can then only come from the constraint.
3 seeds, for ResNet-18 on CIFAR-100 and a ViT trained from scratch on Tiny-ImageNet. Recovery is the decodability (0–1) of the directions below SNR 1 at the start.
The two runs start from the same weights and train under the exact same configuration, differing only in whether the constraint remains. With the constraint kept, the weak directions recover less in every setting and in every run.
We thank the reviewer again and remain available for any further questions or concerns. The submitted PDF has been updated with the revision incorporating the flips remark and additional ablation (in green).