acceptodds
Under review as a conference paper at ICLR 2027

Compressing Protein Language Models: A Unified Benchmark

Abstract

Protein language models are becoming central tools across a growing range of tasks, and their scale and adoption are increasing in tandem, making efficiency a main concern, with compression as a popular way to achieve it. Yet what it actually costs remains unmeasured. We present the first compression bench- mark spanning model families, training objectives, and compression paradigms at a common protocol, evaluated at reconstruction, representation geometry, and downstream biology. For deployment, calibration-aware 4-bit is near-lossless at every scale, equivalence-tested with pre-declared margins; 3-bit is free only at the top of a family; and 2-bit collapses the zero-shot channel except at the very top of a family. How much compression costs depends on the readout: a probe retrained on the compressed features survives to 2-bit at scale, whereas a reused head or SAE does not. Underlying this is a structural dissociation: compression largely preserves the representation and shifts its coordinates, so a label-free realignment restores coarse readouts like the probe and SAE, each with a different correction, while the fine-grained LM head reads exactly the part no such map recovers.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
PLMs, Compression, Pruning, Benchmark, Fine Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_HW46OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents the first systematic compression benchmark for protein language models (PLMs), covering 13 models across three families (ESM2, ESM-C, ProGen2) spanning 8M–15B parameters, 19 compression configurations (quantization and pruning), and three levels of evaluation (reconstruction fidelity, representation geometry, downstream biological tasks). The authors report two main deployment findings: (1) calibration-aware 4-bit quantization is near-lossless across all scales, verified via TOST equivalence testing with a pre-registered margin (δ=0.02) and Holm correction; (2) 3-bit is "free" only at the top of each model family, while 2-bit collapses zero-shot variant effect prediction except for the largest models. Beyond measurement, the paper identifies a mechanistic phenomenon they call "readout dissociation": the same compressed representation can be restored by different label-free alignment maps depending on the readout — SAEs recover with per-channel rescaling, probes need basis alignment, while the zero-shot LM head cannot be fully restored by any tested label-free correction. The authors release the full benchmark harness and per-cell records.

Critical strengths
  1. Statistical rigor rare in ML compression literature. The paper uses TOST with a pre-registered equivalence margin rather than relying on non-significance of standard tests to claim "4-bit is near-lossless." This is combined with Holm correction across methods within each model and paired bootstrap for confidence intervals. This methodology is more rigorous than typical in the field and converts "we didn't observe a difference" into "we have positive evidence of equivalence." This elevates the paper from a routine benchmark to a reliable deployment reference.

  2. The readout dissociation finding is a non-trivial mechanistic insight. The same compressed trunk can be restored by different label-free maps depending on which readout reads. It is not clearly established in the LLM compression literature. The input-correction ladder combined with the identity-initialized nonlinear adapter control (Appendix A) provides a careful falsification argument that this is not an optimizer or capacity artifact. This finding has direct implications for which PLM deployment scenarios can tolerate aggressive quantization.

Critical weaknesses
  1. Overlap with existing LLM compression findings. The authors themselves acknowledge that several core top-line results: (a) 4-bit calibration-aware quantization is near-lossless, (b) 2-bit collapses, (c) scale improves compression robustness, (d) calibration data matters more for pruning than quantization, (e) long-tail capabilities degrade first. They are protein analogs of findings already established in the LLM compression literature (Jaiswal et al. 2024; von Rad et al. 2026; Williams & Aletras 2024; Hooker et al. 2019/2020). Given that quantization is a fundamentally information-theoretic operation that is largely architecture-agnostic, it is unclear which of these reproductions would have been predicted incorrectly based on LLM intuition. The genuinely novel contributions occupy a relatively smaller portion of the paper. This positions the work closer to a domain-specific validation than a conceptual advance, which may be better suited to a Datasets & Benchmarks venue than ICLR's main track.

  2. The label-free alignment analysis is a mechanistic diagnostic, not a deployable method. The alignment maps that recover SAE and probe readouts require full-precision activations to fit, which defeats the purpose of compression at deployment. The paper is upfront about this (Section 7 limitations), but it means the practical "decision rule" ultimately rests on the equivalence-tested 4-bit result, which itself is a reproduction of LLM-level knowledge. The novel insight (readout dissociation) therefore informs understanding without directly enabling better deployment beyond the already-known 4-bit recommendation.

Questions for authors

Q1: Can the authors identify a specific finding in this paper that would have been predicted incorrectly based on LLM compression literature? Concretely: which empirical claim in Sections 5.1, 5.2, or 5.4 could not have been anticipated from Jaiswal et al. (2024), von Rad et al. (2026), or Williams & Aletras (2024)? Clarifying this is central to assessing the paper's novelty beyond domain-specific reproduction.

Q2 : The readout dissociation claim rests partly on the zero-shot LM head not being recoverable by any tested label-free map. Appendix A notes that even an identity-initialized nonlinear adapter fails on compressed activations but succeeds on a learnable-distortion control. However, the adapter's capacity may still be insufficient relative to the per-token margin signal the head reads. Have the authors tested larger adapters, or multi-step adapters that could in principle learn arbitrary continuous maps? If so, do they also fail?

Q3 : For the ProGen2-xlarge RTN-int4 collapse localized to the tied output embedding, is this finding specific to tied-embedding architectures, or does it also appear in untied variants? This would help readers understand whether the "spare the embedding" heuristic generalizes.

Q4: The matched-bytes argument (Fig. 5 and Table 2) favors "smallest model past the knee at 4-bit" over a compressed larger model, but relies on fake-quantized memory accounting for low-bit points. For int3/int2 where fast kernels do not yet exist, could the authors clarify whether their deployment recommendation would change if real-kernel latency were considered?

Reviewer_uuyuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates 13 protein language models (ESM2, ESM-C, and ProGen2, from 8M to 15B parameters) under 19 quantization and pruning settings. It looks at the effects on logits, representations, and downstream tasks.

There is a lot of experimental work here, but I do not think the paper makes the case for why compression is needed in the settings it studies. The results often seem to favor simply choosing a smaller model. The novelty is limited, and the writing makes it harder than it should be to judge what the work adds.

Critical strengths

The experimental grid is large. I also appreciate the paired equivalence tests instead of treating a non-significant difference as evidence that performance is preserved.

Critical weaknesses

Overall, the paper feels like AI writing without much editing. The same sentences recur almost word for word. “A probe retrained on the compressed features survives to 2-bit at scale, whereas a reused head or SAE does not” appears in both the abstract and the intro. The 4-bit/3-bit/2-bit rule is restated in the abstract, intro, Sec. 5.2, Fig. 5, and the discussion. I would rather see that space used to bring the key evidence out of the appendix

  1. My main concern is that the paper does not establish the practical problem it is trying to solve. The introduction argues that compression is necessary because large PLMs are expensive for academic labs, but that cost is never really measured. The paper’s own results make the motivation harder to follow. Compression does not improve speed either: 4-bit is slower than full precision at every size tested. Given these results, why would a lab compress a large model rather than just use the smaller one?

  2. The paper’s recommendation seems to reinforce this concern. The authors recommend using “the smallest model on the plateau.” That is a model-selection result, and it leaves me wondering what compression adds here.

  3. The novelty is limited. These are existing LLM compression methods applied as-is to PLMs. The authors also frame several headline results as protein versions of known LLM findings, including those in UniComp and Williams & Aletras. There is value in checking whether those findings carry over, but the paper needs to be clearer about what new insight comes out of that exercise.

  4. The writing makes the paper unnecessarily hard to review. SAE appears in the abstract without being defined, and the reader does not learn which SAEs are used or what they represent until Appendix A. Terms such as “channel,” “readout,” “ladder,” “rung,” “floor,” and “Level-3” are also used without clear definitions.

Questions for authors

Did you read the paper a couple times, front to back? Can you show a concrete use case where compressing a large PLM beats simply running a smaller one at full precision? What is the main contribution intended to be: the benchmark, the deployment rule, or mechanism?

1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal and revision

Dear Reviewer uuyu, thank you for your feedback on our submission. We would be grateful if you could read our rebuttal and revised paper before the discussion period ends. We are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal and revision

Dear Reviewer oGSZ, thank you for your feedback on our submission. We would be grateful if you could read our rebuttal and revised paper before the discussion period ends. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.