- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies compressed-model selection through selection regret. It identifies four limitations involving fidelity bounds, compression-rate confounding, finite evaluation resolution, and deployment-dependent targets. Experiments show that fidelity reliably screens damaged candidates, while local fixed-budget design changes can improve capability despite worsening fidelity. The paper recommends screening, controlling for rate and family, evaluating survivors, and reporting unresolved comparisons.
Critical strengths
The controlled local experiment provides concrete evidence that fidelity and capability can favor opposite design choices. The norm-restoration intervention holds served bytes and the representation family fixed, improving capability while worsening all five principal proxies in the selected Qwen experiment.
Critical weaknesses
-
The incremental scientific contribution is limited relative to the breadth of the framing. The four formal limits largely combine standard softmax geometry, the law of total covariance, concentration-based best-arm selection, and Pareto incomparability. Section 6 explicitly describes the best-arm results as imported. The distinctive empirical contribution is therefore especially important, but the demonstrated local reversal remains narrow: SmolLM2 confirms it, fresh Qwen fails the directional test, and Llama is unresolved. Meanwhile, teacher KL correctly orders 92.5-100% of resolvable comparisons in the reported pools. These results establish a useful counterexample to universal fidelity ordering, but provide limited guidance about when fidelity-based selection will actually fail across practical compression settings. A broader characterization of those conditions would make the contribution more consequential.
-
The proposed resolution-aware protocol is not validated as a procedure operating under a genuinely limited evaluation budget. Appendix A.7 computes calibration gaps and variances from the complete draw-half scores, then tests their predictions through subsampling. This is a valid retrospective diagnostic, but it does not demonstrate how a practitioner estimates resolution before spending the evaluation budget needed to obtain those scores. Likewise, the screening experiments examine several survivor counts without a validated rule for selecting that count or deciding when to stop evaluation. The comparisons also exclude teacher-KL screening forwards from the candidate-document budget; the reported inexpensive matrix-MSE timing does not account for the KL screen actually used. The paper acknowledges these restrictions, but they leave the central practical recommendation without an end-to-end demonstration of its decision quality and total cost.
Questions for authors
-
Can the authors demonstrate a prospective procedure that estimates resolution from an initial evaluation budget, chooses a survivor count, and returns either a candidate or an unresolved decision without accessing complete candidate scores?
-
At matched total inference cost, including teacher and candidate forwards for screening, how does the protocol compare with proxy-only selection, uniform capability evaluation, and an established racing or elimination procedure?
-
Can the fixed-byte reversal be tested prospectively across additional whole-model compression families and generative tasks? Which measurable properties predict when fidelity ranking will fail?
-
What contribution beyond the established theoretical ingredients is intended to carry the paper: a new selection procedure, a broadly applicable empirical characterization, or an evaluation protocol? What additional evidence would substantiate that contribution?
Kindly Requesting Your Feedback on Our Rebuttal
Dear Reviewers,
Thank you again for your thoughtful and constructive reviews of our submission. We truly appreciate the time and effort you invested in evaluating our work.
We have carefully addressed each of the points raised in your reviews and have posted our rebuttal along with a revised manuscript. We believe the revisions substantively strengthen the paper and directly respond to your concerns. We would be very grateful if you could find the time to review our rebuttal and updated manuscript at your earliest convenience. Your feedback during this discussion phase is invaluable to us, and we welcome any further questions or suggestions you may have.
Thank you for your continued engagement with our work. We look forward to hearing from you.