acceptodds
Under review as a conference paper at ICLR 2027

Compress at Input, Recover at Depth: Improving the Efficiency–Performance Trade-off for MLLMs

Abstract

In common MLLM pipelines, full-token connectors (e.g., Linear or MLP projectors) forward all modality tokens into the decoder, making attention cost grow quadratically with input length. Token compression alleviates this cost but can weaken multimodal conditioning, creating a core efficiency–performance trade-off. This issue is particularly pronounced for long or information-sparse scientific modalities, where aggressive compression can discard task-relevant input information. We propose ENCORE, a dual-stage connector built on a simple principle: compress at input, recover at depth. ENCORE constructs a compact set of instruction-conditioned query tokens to reduce the number of input modality tokens, and introduces a lightweight one-shot recovery route that uses intermediate text states to retrieve complementary information from pre-compression modality memory. A stabilized residual update integrates this information during prefill, with no repeated memory reads during cached decoding. Across vision, protein, and molecule tasks, ENCORE achieves the best overall Norm-Avg against the shared MLP and Q-Former references while improving the performance–efficiency trade-off. Relative to MLP, it reduces inference FLOPs by 16.3–62.6% and lowers training time in four of five settings. Extended comparisons further show that ENCORE remains competitive with stronger compression and multi-layer reinjection baselines while retaining a compact decoder interface. Ablations show that compression provides a strong primary representation, while one-shot recovery contributes complementary gains, especially under aggressive compression. Overall, ENCORE provides a practical, modality-agnostic connector that consistently improves the performance–efficiency trade-off across diverse settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Language and multimodal language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aUdyOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ENCORE, a two-stage MLLM compression method that reduces the number of visual tokens at the projector level and supplements the compressed representations with additional information within the LLM. At the projector stage, instruction-conditioned queries first compress the encoder features into a small set of tokens. At a middle layer in the LLM, a single cross-attention retrieves information from the pre-compression features and adds it to the text hidden states through a stabilized residual update. Across vision, protein, and molecule tasks with LLM backbones from 0.5B to 8B, ENCORE reports higher Norm-Avg scores than MLP and Q-Former and substantially lower inference FLOPs than MLP.

Critical strengths
  1. This work substantially reduces the visual token count across the evaluated vision backbones while maintaining aggregate performance comparable to the MLP baseline. On Qwen-2.5-7B, ENCORE also improves efficiency, lowering prefill latency from 34.13 to 20.91 ms and TTFT from 47.22 to 35.24 ms relative to MLP.

  2. This work combines token compression at the input with a single recovery step inside the LLM that reads modality features during prefill to supplement the text hidden states. The recovery step adds little inference overhead, taking about 0.5 ms in the reported profiling setup.

Critical weaknesses
  1. The accuracy and efficiency comparisons do not yet support the claimed performance–efficiency advantage of ENCORE.

    • Norm-Avg makes performance retention under compression difficult to assess because it weights metrics using manually chosen anchors and counts correlated metrics from the same dataset separately. For example, five of the eight protein metrics are BLEU/ROUGE variants on Prot2Text. Reporting one primary metric per benchmark relative to the dense MLP baseline (100%) will make average performance retention easier to interpret. On vision, ENCORE remains close to the MLP baseline, indicating near-lossless compression rather than a clear accuracy improvement.
    • The molecule setting provides limited evidence of practical compression benefits. With only about 28 nodes per molecule, the inputs are much shorter than the long scientific sequences motivating this work, and compression to 8 tokens does not improve throughput over MLP.
    • Comparisons with recent projector-level compression methods are missing in both accuracy and efficiency. The main performance tables include only MLP and Q-Former. The projector-level baselines in the appendix date from 2022–2024 (the latest, TokenPacker, is over two years old) and are evaluated mainly on vision. Besides, FLOPs reductions relative to MLP alone do not establish an efficiency advantage over other compression methods. For example, ENCORE and TokenPacker have nearly identical prefill latency on the 7B backbone (Table 35).
  2. The evidence that the recovery route restores information lost by compression is limited.

    • This paper itself notes that compression weakens fine-grained signals. Since the recovery route accesses pre-compression features, one would expect it to mitigate a meaningful portion of the resulting performance loss. However, the observed gains on fine-grained perception are limited. In Table 19, adding recovery to Q-Former improves TextVQA and GQA by only 0.32 and 0.14 points, respectively, leaving most of the gap to MLP unchanged. Its main gains are on reasoning benchmarks, for example, MME-C alone accounts for more than half of the reported Norm-Avg gain from recovery. The effectiveness of the recovery function needs to be tested on fine-grained tasks, including OCR (e.g., OCR Bench) and grounding (e.g., RefCOCO) benchmarks.
    • The claim that recovery helps more under a tighter bottleneck is not supported. This paper attributes the larger recovery gain on molecules than on vision (+13.56 vs. +1.78) to a tighter information bottleneck, but the two settings differ in modality, backbone, and task. On molecules, Q-Former already exceeds MLP in Norm-Avg at ENCORE's 8-token budget, whereas ENCORE without recovery falls below both. Nearly half of ENCORE's recovery gain closes its 6.40-point gap to Q-Former (Tables 16 and 23), suggesting that the large gain may partly reflect a weaker compression-only baseline.
Questions for authors
  1. What does the recovery route read on protein and molecules? Section 2 describes recovery as reading all encoder features, but App. B.2.6 identifies query-token representations as the recovery memory for protein and molecules. The recovery costs in Table 38 also appear consistent with the query budgets rather than the full encoder lengths. If recovery only reads compressor outputs, it cannot restore information discarded by compression, as App. C.3.5 notes. This would substantially affect my assessment.

  2. Does the recovery gain come from information read at inference? Table 30 reports hidden-state changes of only about 0.57%, with no prediction changes on the evaluated subset when the recovery scale varies by 20% in either direction. In Table 19, recovery improves Q-Former by 1.22 points on SQA but only 0.02 on SQA-I, suggesting that the gains may not primarily depend on visual input. Please evaluate a trained ENCORE model with recovery disabled only at inference. A small performance drop would suggest that much of the gain comes from training effects rather than information recovered at inference.

Reviewer_Mu78OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ENCORE, a dual-stage connector built on a simple principle: compress at input, recover at depth. ENCORE constructs a compact set of instruction-conditioned query tokens to reduce the number of input modality tokens, and introduces a lightweight one-shot recovery route that uses intermediate text states to retrieve complementary information from pre-compression modality memory.

Critical strengths

The paper evaluates ENCORE across three distinct modality families and multiple LLM backbones.

Critical weaknesses

The technical novelty appears limited, as the main components are closely related to existing methods. Instruction-conditioned query compression has been explored in InstructBLIP and Q-Former-based architectures, while decoder-side cross-attention and gated multimodal fusion are established in Flamingo and ViCA. ENCORE primarily combines these existing techniques with a single intermediate-layer recovery operation. The paper does not sufficiently establish the unique technical contribution of this combination beyond an architectural efficiency trade-off.

Questions for authors
  1. The motivation for performing recovery at an intermediate decoder layer remains insufficiently established. Although the depth ablations show that intermediate layers generally perform well, the default midpoint is not always optimal, and five recovery operations slightly outperform the single-recovery design. It remains unclear under which task conditions early, intermediate, or repeated recovery is preferable.

  2. The ablation study does not fully isolate the contribution of instruction-conditioned compression. In Table 22, compression-only ENCORE improves Norm-Avg from 74.29 to 80.74 over Q-Former (+6.45), whereas adding recovery provides a further gain of 1.78. However, the compression-only ENCORE differs from Q-Former in both instruction conditioning and extractor architecture/depth. Therefore, the 6.45-point improvement cannot be attributed to instruction conditioning alone.

  3. The recovery mechanism is applied to all text hidden states in Eq. (9), while Eqs. (10)–(11) compute global statistics across text positions. It is unclear whether teacher-forced answer tokens are included during training, although recovery is performed only during prefill at inference. If future answer tokens affect the global scaling factors, this may introduce information leakage and create a training–inference mismatch. The authors should clarify the training implementation and explain how causal consistency is maintained.

  4. Potentially Unfair Comparison with ViCA. The evaluated ViCA-based baseline differs substantially from the original ViCA architecture. In particular, it retains 729 visual tokens in the decoder while adding cross-attention at eight layers, whereas the original ViCA allows visual tokens to bypass standard decoder self-attention and feed-forward layers. This difference may inflate the computational cost of the evaluated baseline. A comparison with a faithful implementation of ViCA is needed to substantiate the claimed efficiency advantage.

Reviewer_gjjNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ENCORE, a dual-stage connector for multimodal large language models. The method compresses modality features into instruction-conditioned query tokens before entering the LLM, then uses intermediate text states to retrieve additional information from pre-compression modality memory. Recovery is performed once during prefill. Experiments on vision, protein, and molecule tasks suggest that ENCORE can improve the performance–efficiency trade-off compared with MLP and Q-Former connectors.

Critical strengths
  • The proposed framework is conceptually intuitive. Combining input compression with a lightweight recovery operation provides a practical way to reduce computation while preserving access to modality information.
  • The experiments cover multiple modalities and LLM backbones, providing useful evidence for the generality of the approach.
  • The paper includes extensive ablations on token budgets, recovery depth and frequency, and stabilization components, which help explain the design choices.
  • The efficiency evaluation includes measured latency, memory consumption, and batched decoding results, in addition to FLOPs.
Critical weaknesses
  • The discussion of closely related work is incomplete. Although FastV and FastV-FT are included, representative methods such as PyramidDrop (CVPR 2025), SparseVLM (ICML 2025), and ZipVL (ICCV 2025) are neither cited nor compared against. These methods manage visual information across decoder depth through progressive pruning, token recycling, or dynamic sparsity. The authors should clarify their relationship to ENCORE and provide accuracy–efficiency comparisons with representative methods using the same backbone and timing protocol.

  • The choice of recovery memory could be better justified. ENCORE uses intermediate LLM text states to query linearly projected modality-encoder features. A controlled comparison with memory constructed from visual hidden states processed by early LLM layers, while keeping the input compressor and recovery mechanism fixed, would help assess whether additional LLM processing improves cross-modal integration. Reporting both accuracy and computational overhead would clarify the trade-off underlying this design choice.

  • The technical distinction from ViCA needs further clarification. Both methods allow LLM text states to access projected modality features through decoder-side cross-attention. ENCORE combines this mechanism with instruction-conditioned input compression and a single stabilized recovery operation. However, the evaluated ViCA-based control retains a 729-token visual prefix, unlike the published design that bypasses visual self-attention and FFN updates. This modification may overstate ENCORE’s efficiency advantage. A faithful ViCA implementation, together with controlled comparisons of the input interface and memory-read frequency, would help establish ENCORE’s technical contribution and practical benefits.

Questions for authors
  • What are the main technical advantages of ENCORE over ViCA? Why does the evaluated ViCA-based control retain a 729-token visual prefix? Could the authors provide a faithful ViCA comparison and isolate the benefits of the compressed input interface and one-shot recovery?

  • How does recovery from linearly projected encoder features compare with recovery from visual hidden states processed by early LLM layers? Could the authors compare both accuracy and computational overhead while keeping the compressor and recovery mechanism fixed?

  • How does ENCORE compare with representative decoder-side token reduction methods, such as PyramidDrop, SparseVLM, and ZipVL, using the same backbone and timing protocol? Please also clarify differences in training requirements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.