acceptodds
Under review as a conference paper at ICLR 2027

ResponseScope: Measuring Finite-Response Distortion in Compressed Models

Abstract

Model compression is commonly evaluated using clean-task performance and static teacher–student agreement, but these criteria do not characterize responses to finite, task-valid input changes. We study three related questions: how to measure this behavioral drift, predict compression vulnerability before full recovery and final utility evaluation, and preserve response behavior at a fixed model footprint. ResponseScope (RS) is a teacher-normalized finite-response distortion measure. The evaluation spans six datasets, nine teacher–student architecture pairs, four compression families, three storage budgets, 540 recipe means, and 2,700 independently stored seed checkpoints. Combining teacher layer fragility with operator-local distortion and applicable initialization compatibility yields task-equal leave-onedataset-out R2 = .687 and MAE 1.800 percentage points for predicting utility damage prospectively. After compression, adding measured RS to a scale- and endpoint-matched baseline raises R2 from .587 to .778 and reduces selection regret from 1.260 to .688 points. At footprint ratio .25, IF-Reg reduces mean held-out RS from .440 to .257 and improves the task-specific utility composite by 3.017 points relative to augmented logit distillation. A baseline matched for endpoints, teacher scale, and margin explains part of this gain. These results connect the measurement, prediction, and preservation of interventional fidelity

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Model Compression, Finite-Response Distortion, Behavioral Evaluation, Structured Input Transformations, Model Selection, Utility-Damage Prediction, Knowledge Distillation, Structured Pruning, Post-Training Quantization, Distribution Shift

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qhDZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces ResponseScope (RS), a teacher-normalized measure of finite-response distortion under task-valid transformations. It investigates whether RS can predict compressed-model utility, improve configuration selection, and guide behavior-preserving compression. The paper reports results over six datasets, four compression families, three budgets, and a large set of teacher-student configurations, claiming that RS adds predictive and selection value beyond endpoint and scale descriptors. It also proposes IF-Reg, which combines paired-response matching with endpoint and margin anchors to improve response fidelity and task utility at a fixed footprint.

Critical strengths
  1. The paper addresses a meaningful question beyond clean accuracy: whether a compressed model preserves the teacher's behavior under valid local interventions. The teacher-normalized RS construction has a reasonable motivation.

  2. The evaluation framework attempts to control endpoint performance, realized footprint, model size, compression operator, teacher access, and optimization budget, and employs nested leave-one-task-out validation and matched comparisons.

  3. The paper is unusually explicit about record reconstruction, auditing procedures, operating conditions, and evidence gaps. Although these disclosures reveal substantial limitations, the transparency helps reviewers identify the true scope of the claims.

Critical weaknesses
  1. RS depends critically on transformations being semantically valid, yet the validity audit uses no human annotators and partly relies on generated channels with known labels. The paper acknowledges that these checks cannot validate real transformation families or establish the provenance of the Waterbirds foreground masks.

  2. The work combines a new metric, predictive models, selection rules, and IF-Reg into a highly complex evaluation, while several central comparisons remain point estimates or have limited panel-level intervals. The end-to-end cost of RS relative to direct downstream validation is also insufficiently established.

Questions for authors
  1. Without human semantic annotation, what existing evidence establishes that the transformations used in the main experiments preserve task semantics rather than merely preserving generator-defined labels? Did the Waterbirds records with unresolved mask provenance enter the main conclusions, and do the conclusions change if they are excluded?

  2. Using the existing logs, can the authors compare the teacher-query count, wall-clock time, and complete selection cost of RS with directly evaluating the candidate compressed models? If RS is not cheaper than direct evaluation, its main practical motivation as a selection proxy would be weakened.

Reviewer_RUxVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how model compression distorts model responses under finite, task-valid input transformations. The authors argue that clean accuracy and static teacher–student agreement fail to capture behavioral drift. Therefore, four mechanisms are introduced

  • ResponseScope is a teacher normalized metric evaluating the finite-response difference between a compressed student fSf_S and a teacher fTf_T on canonical zero-sum logits.
  • Perspective Vulnerability Prediction uses teacher layer interventional fragility and operator-local distortion to predict performance drops and target layers for protection before full recovery.
  • Post-compression candidate selection uses RS as an additional feature in a meta regressor to select the best compression candidate under fixed storage budgets.
  • IF-Reg is a finetuning objective combining augmented supervision, endpoint matching, normalized response matching and margin matching.
Critical strengths
  • This paper studies an important problem formulation. Evaluating compressed models beyond in-distribution clean accuracy is a vital topic, especially under functional shifts and task-preserving transformations.
  • The benchmark includes diverse modalities (vision, NLP, time series), multiple compression modalities, and an extensive list of comparative feature descriptors (e.g., CKA, Jacobians, ECE, calibration shifts, Hessian-based metrics).
  • The authors provide extensive appendix sections detailing failure modes, ablations, seed-level variations, and audit checks.
Critical weaknesses

On potential circularity of synthetic and numerical experiments.

  • In several controlled numerical experiments designed to validate ResponseScope, the ground-truth utility metric is explicitly constructed as a function of the response difference, e.g., Appendix J.2, K.2 and M.1

On selection regret and outlier dominance

  • The paper reports substantial selection regret reduction when adding RS M0+→M1+M_0^+ \rightarrow M_1^+. A closer look at the actual decision records reveals sensitivity to single outlier cells. For example, in Appendix K.5, out of 54 decisions only 9 changed. A single cell (Task D04, Group 3, budget 0.125) accounts for an improvement of 10.9 pp. If this cell is omitted, the average regret reduction becomes negative.

On Scale Invariance

  • ResponseScope is not invariant to independent rescaling of student logits (line 158). As shown in Figure 7a, simply scaling student logits by a factor α∈[0.5,2.0]\alpha \in [0.5, 2.0] drastically alters ResponseScope (by over 60%), even though class argmax, rankings, and underlying classifications are completely unchanged.
  • While post-hoc temperature scaling on validation data can mitigate this (Figure 7a), temperature scaling is known to perform unpredictably under out-of-distribution shifts. If a metric intended to capture out-of-distribution behavioral drift is highly sensitive to arbitrary logit scaling, it risks evaluating calibration scale rather than genuine interventional fidelity.

On several discrepancies I am able to identify

  • Lines 254–256: “All numerical findings are drawn from author-supplied records of executed experiments. Manuscript preparation does not independently rerun those experiments or recompute their results.”
  • Lines 1615–1616: “Experiments were executed on a separate platform; manuscript preparation uses exported records without rerunning experiments.”
  • Appendix I.1 notes that an earlier benchmark export had corrupted alias assignments (c01--c05 sorted by utility, falsely showing c01 as a universal winner across budgets).
Questions for authors

refer to the weakness points above

Reviewer_9y7oOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper seeks to understand the behavioral changes induced by model compression, outside of clean task settings where they may be evaluate. They introduce ResponseScope, which seeks to measure the distortion induced by model compression. Using this approach, they show that they can predict performance differences on held-out datasets. They also show that the damage can be localized via their approach allowing for preservations of fragile modules.

Critical strengths
  • Interesting Problem Statement: Broadly, the problem of understanding how to measure compression-based performance degradations, particularly in perturbed tasks seems impactful and interesting to me. I think that the general finding that clean-accuracy can hide performance differences induced by model-compression techniques is a very important finding, particularly if it holds in more modern/applicable settings (such as the modern foundation models setting).
Critical weaknesses
  • Readability Concerns: In general, I found that the writing in this paper was hard to follow. Although the motivation was relatively easy to follow, the results/findings were difficult to decipher from the abstract/introduction. Similarly, I am not sure what is “IF-Reg” from the abstract/intro and what is the motivation of developing it. Similarly I am unsure what “LODO” refers to in this setting. Similarly, I am unsure what “endpoint” and “margin” anchors refer to in this context. I think that having such unexplained writing presents a bar to my full understanding of the paper and hence should be addressed before peer-review, let alone publication.

  • Tasks Considered: I think that the tasks considered in this work CIFAR-100, CivilComments, Amazon Reviews, etc are relatively out-dated and in a different regime than the settings where model compression is typically used in current settings. I think the applicability of this paper would definitely be significantly improved if the authors were to consider settings based off of the modern foundation-models setting. While this is not on its a reason for rejection, I think in conjunction with the poor presentation, it detracts from the impact of the paper.

Questions for authors

Please respond to the weakness points that I mentioned.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.