- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces ResponseScope (RS), a teacher-normalized measure of finite-response distortion under task-valid transformations. It investigates whether RS can predict compressed-model utility, improve configuration selection, and guide behavior-preserving compression. The paper reports results over six datasets, four compression families, three budgets, and a large set of teacher-student configurations, claiming that RS adds predictive and selection value beyond endpoint and scale descriptors. It also proposes IF-Reg, which combines paired-response matching with endpoint and margin anchors to improve response fidelity and task utility at a fixed footprint.
Critical strengths
-
The paper addresses a meaningful question beyond clean accuracy: whether a compressed model preserves the teacher's behavior under valid local interventions. The teacher-normalized RS construction has a reasonable motivation.
-
The evaluation framework attempts to control endpoint performance, realized footprint, model size, compression operator, teacher access, and optimization budget, and employs nested leave-one-task-out validation and matched comparisons.
-
The paper is unusually explicit about record reconstruction, auditing procedures, operating conditions, and evidence gaps. Although these disclosures reveal substantial limitations, the transparency helps reviewers identify the true scope of the claims.
Critical weaknesses
-
RS depends critically on transformations being semantically valid, yet the validity audit uses no human annotators and partly relies on generated channels with known labels. The paper acknowledges that these checks cannot validate real transformation families or establish the provenance of the Waterbirds foreground masks.
-
The work combines a new metric, predictive models, selection rules, and IF-Reg into a highly complex evaluation, while several central comparisons remain point estimates or have limited panel-level intervals. The end-to-end cost of RS relative to direct downstream validation is also insufficiently established.
Questions for authors
-
Without human semantic annotation, what existing evidence establishes that the transformations used in the main experiments preserve task semantics rather than merely preserving generator-defined labels? Did the Waterbirds records with unresolved mask provenance enter the main conclusions, and do the conclusions change if they are excluded?
-
Using the existing logs, can the authors compare the teacher-query count, wall-clock time, and complete selection cost of RS with directly evaluating the candidate compressed models? If RS is not cheaper than direct evaluation, its main practical motivation as a selection proxy would be weakened.