- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ENCORE, a two-stage MLLM compression method that reduces the number of visual tokens at the projector level and supplements the compressed representations with additional information within the LLM. At the projector stage, instruction-conditioned queries first compress the encoder features into a small set of tokens. At a middle layer in the LLM, a single cross-attention retrieves information from the pre-compression features and adds it to the text hidden states through a stabilized residual update. Across vision, protein, and molecule tasks with LLM backbones from 0.5B to 8B, ENCORE reports higher Norm-Avg scores than MLP and Q-Former and substantially lower inference FLOPs than MLP.
Critical strengths
-
This work substantially reduces the visual token count across the evaluated vision backbones while maintaining aggregate performance comparable to the MLP baseline. On Qwen-2.5-7B, ENCORE also improves efficiency, lowering prefill latency from 34.13 to 20.91 ms and TTFT from 47.22 to 35.24 ms relative to MLP.
-
This work combines token compression at the input with a single recovery step inside the LLM that reads modality features during prefill to supplement the text hidden states. The recovery step adds little inference overhead, taking about 0.5 ms in the reported profiling setup.
Critical weaknesses
-
The accuracy and efficiency comparisons do not yet support the claimed performance–efficiency advantage of ENCORE.
- Norm-Avg makes performance retention under compression difficult to assess because it weights metrics using manually chosen anchors and counts correlated metrics from the same dataset separately. For example, five of the eight protein metrics are BLEU/ROUGE variants on Prot2Text. Reporting one primary metric per benchmark relative to the dense MLP baseline (100%) will make average performance retention easier to interpret. On vision, ENCORE remains close to the MLP baseline, indicating near-lossless compression rather than a clear accuracy improvement.
- The molecule setting provides limited evidence of practical compression benefits. With only about 28 nodes per molecule, the inputs are much shorter than the long scientific sequences motivating this work, and compression to 8 tokens does not improve throughput over MLP.
- Comparisons with recent projector-level compression methods are missing in both accuracy and efficiency. The main performance tables include only MLP and Q-Former. The projector-level baselines in the appendix date from 2022–2024 (the latest, TokenPacker, is over two years old) and are evaluated mainly on vision. Besides, FLOPs reductions relative to MLP alone do not establish an efficiency advantage over other compression methods. For example, ENCORE and TokenPacker have nearly identical prefill latency on the 7B backbone (Table 35).
-
The evidence that the recovery route restores information lost by compression is limited.
- This paper itself notes that compression weakens fine-grained signals. Since the recovery route accesses pre-compression features, one would expect it to mitigate a meaningful portion of the resulting performance loss. However, the observed gains on fine-grained perception are limited. In Table 19, adding recovery to Q-Former improves TextVQA and GQA by only 0.32 and 0.14 points, respectively, leaving most of the gap to MLP unchanged. Its main gains are on reasoning benchmarks, for example, MME-C alone accounts for more than half of the reported Norm-Avg gain from recovery. The effectiveness of the recovery function needs to be tested on fine-grained tasks, including OCR (e.g., OCR Bench) and grounding (e.g., RefCOCO) benchmarks.
- The claim that recovery helps more under a tighter bottleneck is not supported. This paper attributes the larger recovery gain on molecules than on vision (+13.56 vs. +1.78) to a tighter information bottleneck, but the two settings differ in modality, backbone, and task. On molecules, Q-Former already exceeds MLP in Norm-Avg at ENCORE's 8-token budget, whereas ENCORE without recovery falls below both. Nearly half of ENCORE's recovery gain closes its 6.40-point gap to Q-Former (Tables 16 and 23), suggesting that the large gain may partly reflect a weaker compression-only baseline.
Questions for authors
-
What does the recovery route read on protein and molecules? Section 2 describes recovery as reading all encoder features, but App. B.2.6 identifies query-token representations as the recovery memory for protein and molecules. The recovery costs in Table 38 also appear consistent with the query budgets rather than the full encoder lengths. If recovery only reads compressor outputs, it cannot restore information discarded by compression, as App. C.3.5 notes. This would substantially affect my assessment.
-
Does the recovery gain come from information read at inference? Table 30 reports hidden-state changes of only about 0.57%, with no prediction changes on the evaluated subset when the recovery scale varies by 20% in either direction. In Table 19, recovery improves Q-Former by 1.22 points on SQA but only 0.02 on SQA-I, suggesting that the gains may not primarily depend on visual input. Please evaluate a trained ENCORE model with recovery disabled only at inference. A small performance drop would suggest that much of the gain comes from training effects rather than information recovered at inference.