- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents the first systematic compression benchmark for protein language models (PLMs), covering 13 models across three families (ESM2, ESM-C, ProGen2) spanning 8M–15B parameters, 19 compression configurations (quantization and pruning), and three levels of evaluation (reconstruction fidelity, representation geometry, downstream biological tasks). The authors report two main deployment findings: (1) calibration-aware 4-bit quantization is near-lossless across all scales, verified via TOST equivalence testing with a pre-registered margin (δ=0.02) and Holm correction; (2) 3-bit is "free" only at the top of each model family, while 2-bit collapses zero-shot variant effect prediction except for the largest models. Beyond measurement, the paper identifies a mechanistic phenomenon they call "readout dissociation": the same compressed representation can be restored by different label-free alignment maps depending on the readout — SAEs recover with per-channel rescaling, probes need basis alignment, while the zero-shot LM head cannot be fully restored by any tested label-free correction. The authors release the full benchmark harness and per-cell records.
Critical strengths
-
Statistical rigor rare in ML compression literature. The paper uses TOST with a pre-registered equivalence margin rather than relying on non-significance of standard tests to claim "4-bit is near-lossless." This is combined with Holm correction across methods within each model and paired bootstrap for confidence intervals. This methodology is more rigorous than typical in the field and converts "we didn't observe a difference" into "we have positive evidence of equivalence." This elevates the paper from a routine benchmark to a reliable deployment reference.
-
The readout dissociation finding is a non-trivial mechanistic insight. The same compressed trunk can be restored by different label-free maps depending on which readout reads. It is not clearly established in the LLM compression literature. The input-correction ladder combined with the identity-initialized nonlinear adapter control (Appendix A) provides a careful falsification argument that this is not an optimizer or capacity artifact. This finding has direct implications for which PLM deployment scenarios can tolerate aggressive quantization.
Critical weaknesses
-
Overlap with existing LLM compression findings. The authors themselves acknowledge that several core top-line results: (a) 4-bit calibration-aware quantization is near-lossless, (b) 2-bit collapses, (c) scale improves compression robustness, (d) calibration data matters more for pruning than quantization, (e) long-tail capabilities degrade first. They are protein analogs of findings already established in the LLM compression literature (Jaiswal et al. 2024; von Rad et al. 2026; Williams & Aletras 2024; Hooker et al. 2019/2020). Given that quantization is a fundamentally information-theoretic operation that is largely architecture-agnostic, it is unclear which of these reproductions would have been predicted incorrectly based on LLM intuition. The genuinely novel contributions occupy a relatively smaller portion of the paper. This positions the work closer to a domain-specific validation than a conceptual advance, which may be better suited to a Datasets & Benchmarks venue than ICLR's main track.
-
The label-free alignment analysis is a mechanistic diagnostic, not a deployable method. The alignment maps that recover SAE and probe readouts require full-precision activations to fit, which defeats the purpose of compression at deployment. The paper is upfront about this (Section 7 limitations), but it means the practical "decision rule" ultimately rests on the equivalence-tested 4-bit result, which itself is a reproduction of LLM-level knowledge. The novel insight (readout dissociation) therefore informs understanding without directly enabling better deployment beyond the already-known 4-bit recommendation.
Questions for authors
Q1: Can the authors identify a specific finding in this paper that would have been predicted incorrectly based on LLM compression literature? Concretely: which empirical claim in Sections 5.1, 5.2, or 5.4 could not have been anticipated from Jaiswal et al. (2024), von Rad et al. (2026), or Williams & Aletras (2024)? Clarifying this is central to assessing the paper's novelty beyond domain-specific reproduction.
Q2 : The readout dissociation claim rests partly on the zero-shot LM head not being recoverable by any tested label-free map. Appendix A notes that even an identity-initialized nonlinear adapter fails on compressed activations but succeeds on a learnable-distortion control. However, the adapter's capacity may still be insufficient relative to the per-token margin signal the head reads. Have the authors tested larger adapters, or multi-step adapters that could in principle learn arbitrary continuous maps? If so, do they also fail?
Q3 : For the ProGen2-xlarge RTN-int4 collapse localized to the tied output embedding, is this finding specific to tied-embedding architectures, or does it also appear in untied variants? This would help readers understand whether the "spare the embedding" heuristic generalizes.
Q4: The matched-bytes argument (Fig. 5 and Table 2) favors "smallest model past the knee at 4-bit" over a compressed larger model, but relies on fake-quantized memory accounting for low-bit points. For int3/int2 where fast kernels do not yet exist, could the authors clarify whether their deployment recommendation would change if real-kernel latency were considered?
Follow-up on our rebuttal and revision
Dear Reviewer uuyu, thank you for your feedback on our submission. We would be grateful if you could read our rebuttal and revised paper before the discussion period ends. We are happy to answer any further questions.