acceptodds
Under review as a conference paper at ICLR 2027

Screening Is Not Ranking: Resolution-Aware Model Selection Under Compression

Abstract

Fidelity metrics such as reconstruction error, perplexity, and teacher agreement are effective at detecting severely damaged compressed models, but they need not identify the best model among candidates that all retain high fidelity. We therefore formulate compressed-model selection as a decision problem and evaluate metrics through selection regret: the capability gap between the selected model and the best available candidate. We identify four structural limits on whether a proxy can achieve low regret: a proxy limit, when fidelity bounds are too loose to order viable candidates; a pool limit, when compression rate is confounded with design quality; a target limit, when the evaluation budget is insufficient to resolve capability differences; and a target-definition limit, when no single scalar capability target is appropriate across deployments. We derive a formal result for each limit and test them on pre-registered pools of lexical-interface compressed language models. Fidelity is highly reliable for screening broken models (AUROC 0.97–1.00), yet in a viable 3B-model pool, bytes and nine fidelity metrics recover essentially the same ranking because the pool collapses to a compression-rate ladder rather than distinguishing design quality. Evaluation budget sharply determines what can be resolved: across 11,734 pair-by-budget tests, our regret bound is never violated, and pairs classified as resolvable reverse ordering only 0.005% of the time, compared with 22.8% for unresolved pairs. Screening before capability evaluation reduces selection regret by up to an order of magnitude. More importantly, we observe direct proxy reversals at fixed compression rate: restoring row norms improves capability by +0.0089 across eight paired seeds while every measured fidelity proxy ranks the modified model worse; a pre-registered SmolLM2 replication yields a +0.0050 capability gain in 11/12 seeds while 4/5 fidelity proxies again prefer the baseline. These results support a simple evaluation protocol: use fidelity for screening, control for compression rate and model family, evaluate capability among the survivors, and report “unresolved at this budget” when the available evidence cannot support a ranking.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Compression, Model Evaluation, Model Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6fRDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies compressed-model selection through selection regret. It identifies four limitations involving fidelity bounds, compression-rate confounding, finite evaluation resolution, and deployment-dependent targets. Experiments show that fidelity reliably screens damaged candidates, while local fixed-budget design changes can improve capability despite worsening fidelity. The paper recommends screening, controlling for rate and family, evaluating survivors, and reporting unresolved comparisons.

Critical strengths

The controlled local experiment provides concrete evidence that fidelity and capability can favor opposite design choices. The norm-restoration intervention holds served bytes and the representation family fixed, improving capability while worsening all five principal proxies in the selected Qwen experiment.

Critical weaknesses
  • The incremental scientific contribution is limited relative to the breadth of the framing. The four formal limits largely combine standard softmax geometry, the law of total covariance, concentration-based best-arm selection, and Pareto incomparability. Section 6 explicitly describes the best-arm results as imported. The distinctive empirical contribution is therefore especially important, but the demonstrated local reversal remains narrow: SmolLM2 confirms it, fresh Qwen fails the directional test, and Llama is unresolved. Meanwhile, teacher KL correctly orders 92.5-100% of resolvable comparisons in the reported pools. These results establish a useful counterexample to universal fidelity ordering, but provide limited guidance about when fidelity-based selection will actually fail across practical compression settings. A broader characterization of those conditions would make the contribution more consequential.

  • The proposed resolution-aware protocol is not validated as a procedure operating under a genuinely limited evaluation budget. Appendix A.7 computes calibration gaps and variances from the complete draw-half scores, then tests their predictions through subsampling. This is a valid retrospective diagnostic, but it does not demonstrate how a practitioner estimates resolution before spending the evaluation budget needed to obtain those scores. Likewise, the screening experiments examine several survivor counts without a validated rule for selecting that count or deciding when to stop evaluation. The comparisons also exclude teacher-KL screening forwards from the candidate-document budget; the reported inexpensive matrix-MSE timing does not account for the KL screen actually used. The paper acknowledges these restrictions, but they leave the central practical recommendation without an end-to-end demonstration of its decision quality and total cost.

Questions for authors
  • Can the authors demonstrate a prospective procedure that estimates resolution from an initial evaluation budget, chooses a survivor count, and returns either a candidate or an unresolved decision without accessing complete candidate scores?

  • At matched total inference cost, including teacher and candidate forwards for screening, how does the protocol compare with proxy-only selection, uniform capability evaluation, and an established racing or elimination procedure?

  • Can the fixed-byte reversal be tested prospectively across additional whole-model compression families and generative tasks? Which measurable properties predict when fidelity ranking will fail?

  • What contribution beyond the established theoretical ingredients is intended to carry the paper: a new selection procedure, a broadly applicable empirical characterization, or an evaluation protocol? What additional evidence would substantiate that contribution?

Reviewer_a6QQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper separates fidelity-based screening from ranking compressed models, emphasizing compression-rate confounding and evaluation uncertainty. It supports this framework with candidate-pool experiments and a norm-restoration intervention.

Critical strengths
  1. Selection regret and explicit “unresolved” comparisons are more informative than pooled proxy correlations.
  2. The replication results are reported candidly, including failures. SmolLM2 supports the local reversal claim without implying that fidelity generally fails. I find this useful enough for weak acceptance.
Critical weaknesses
  1. The resolution analysis is largely retrospective: Appendix A.7 estimates gaps and variances using the complete H1 pool. It remains unclear how reliably a small pilot can provide these quantities; empirical variance also does not automatically satisfy Equation 4’s sub-Gaussian assumption.
  2. Practical deployment is less established. The survivor count m lacks a prospective selection rule, and Table 28 times matrix MSE rather than the teacher-KL screen used in the experiment.
Questions for authors
  1. Can a limited pilot predict resolution on independent examples, with estimation uncertainty included?
  2. How would you choose m before observing capability scores, and does screening remain worthwhile after including KL computation?
Reviewer_Mq4DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that fidelity metrics can effectively screen out severely degraded compressed models but do not necessarily identify the most capable candidate among viable alternatives. Through theoretical analysis and experiments, it distinguishes proxy misalignment, compression-rate confounding, and insufficient evaluation resolution, motivating a screen-then-evaluate protocol that permits an “unresolved” outcome.

Critical strengths
  • Relevant problem formulation. Focusing on selection regret rather than correlation, while distinguishing screening, cross-rate ranking, and selection at a fixed budget, better reflects practical decisions.
  • Careful methodology. Controlling compression rate, using paired seeds and independent scoring data, and separating evaluation noise from training variability help avoid attributing measurement instability to proxy failure.
Critical weaknesses
  • Limited theoretical novelty. Most formal results build on standard statistical and multi-objective principles; the contribution primarily lies in their integration. The theory also provides limited explanation for reversals between teacher KL and downstream capability.
  • Limited generalizability of local findings. The reversal does not replicate on fresh Qwen seeds, while KL generally performs well on resolvable comparisons. The main experiments focus on output-head and lexical-interface compression.
Questions for authors
  • Among candidates with matched compression rates, the same method family, and resolvable capability differences, how frequently do proxies misrank candidates, and how much selection regret results?
  • Why does norm restoration worsen KL while improving capability, and can this mechanism explain the differing replication outcomes across models?
  • Under a limited budget, how should the procedure account for uncertainty in estimated gaps, determine the screening size, and establish an advantage over adaptive elimination strategies?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Kindly Requesting Your Feedback on Our Rebuttal

Dear Reviewers,

Thank you again for your thoughtful and constructive reviews of our submission. We truly appreciate the time and effort you invested in evaluating our work.

We have carefully addressed each of the points raised in your reviews and have posted our rebuttal along with a revised manuscript. We believe the revisions substantively strengthen the paper and directly respond to your concerns. We would be very grateful if you could find the time to review our rebuttal and updated manuscript at your earliest convenience. Your feedback during this discussion phase is invaluable to us, and we welcome any further questions or suggestions you may have.

Thank you for your continued engagement with our work. We look forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.