acceptodds
Under review as a conference paper at ICLR 2027

PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding

Abstract

Large Vision-Language Models (LVLMs) map visual inputs into dense token sequences, increasing inference and memory costs. Elastic compression addresses this by training a single model that can run at multiple visual token budgets. However, existing approaches struggle under aggressive compression. Nested spatial pooling behaves as an imperfect low-pass filter and induces spectral aliasing that obscures fine-grained detail, while nested query resampling replaces explicit grid-aligned tokens with non-local summaries and substantially degrades spatial grounding. To resolve this representational conflict, we introduce PARCEL (**P**ool-**A**nchored **R**esampling with **C**onditioned **EL**astic Queries), a visual tokenization architecture that dynamically partitions the labor of feature extraction. PARCEL establishes spatial pool tokens as low-frequency layout anchors and conditions elastic query tokens on these anchors through Pool-Conditioned Query Resampling. This coupling encourages queries to capture complementary visual details while strengthening the pooled pathway itself, including at budgets where no query tokens are leveraged at inference. Extensive evaluations across 30+ benchmarks and two LVLMs show that PARCEL improves the performance-efficiency Pareto frontier, consistently outperforming existing matryoshka baselines across visual token budgets while preserving the "train once, deploy anywhere'" paradigm.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
efficiency, elastic inference, vision-language modeling, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_dRBdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PARCEL, an elastic visual token compression architecture that enables a single large vision-language model (LVLM) to operate at multiple inference-time token budgets. The authors examine complementary limitations of Matryoshka Multimodal Models (M3M^{3}) and the Matryoshka Query Transformer (MQT): spatial average pooling can obscure fine-grained details and introduce aliasing, and query-only resampling lacks explicit grid-aligned spatial anchors and can weaken dense spatial grounding. Evaluations on PaliGemma-2 and LLaVA 1.5 report improved aggregate performance retention over the M3M^{3} and MQT baselines at matched token budgets. The reported gains also extend to anchor-only budgets, where no query tokens are used at inference.

Critical strengths
  1. Elastic visual token compression is a valuable research direction for LVLMs, addressing the computational and memory costs of dense visual inputs while allowing a single model to accommodate different deployment budgets.
  2. The paper is well motivated by complementary weaknesses of existing elastic compression methods: spatial pooling can obscure fine-grained details, whereas query-only resampling lacks explicit grid-aligned spatial anchors. The theoretical analysis and the downstream spatial-grounding results provide empirical support for combining spatial anchors with queries that extract complementary information.
  3. The evaluation covers two LVLM backbones, multiple token budgets, and diverse image and video tasks. Ablations support the proposed architectural choices beyond simply increasing connector capacity. The gains at anchor-only budgets further suggest that joint training benefits the pooled pathway even when no queries are used at inference.
Critical weaknesses
  1. The novelty is somewhat incremental. PARCEL largely combines the spatial pooling of M3M^{3} [1] with the learnable queries and nested dropout of MQT [2]. Pool-conditioned query resampling and budget-aware routing refine how these existing components interact, but the overall method remains close to a hybrid of the two prior approaches. The methodological contribution therefore appears to be an incremental integration of established compression mechanisms. The attention-design ablations in Table 4 also show modest gains from the specific PCQR formulation: at 256 tokens, mean retention improves from 95.2% with ViT-only cross-attention and 95.4% with dual cross-attention to 95.6% with PCQR.

  2. The M3M^{3} results in Table 3 are substantially lower on POPE and TextVQA than the original LLaVA-1.5-M3M^{3} results in [1,3] at the same token budgets. At 144 and 36 tokens, respectively, POPE is lower by 2.8 and 2.9 points, and TextVQA is lower by 3.16 and 4.17 points. Appendix E.2 describes changes to the original training recipe, including freezing the visual encoder and sampling one token scale per minibatch instead of training the visual encoder and averaging losses across all scales. The authors should discuss why they perform these operations and mark the results of the reimplementation. Otherwise, the effects of these changes on the baseline performance are not quantified, leaving unclear how much they contribute to the observed gap and affect the reported improvements over M3M^{3}.

[1] Cai, M., Yang, J., Gao, J., & Lee, Y. J. (2025, May). Matryoshka multimodal models. In International Conference on Learning Representations (Vol. 2025, pp. 46254-46272).

[2] Hu, W., Dou, Z. Y., Li, L. H., Kamath, A., Peng, N., & Chang, K. W. (2024). Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37, 50168-50188.

[3] Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024, June). Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26286-26296). IEEE.

Questions for authors

Why are the M3M^{3} POPE and TextVQA scores in Table 3 substantially lower than the original LLaVA-1.5-M3M^{3} results at 144 and 36 tokens? To what extent is this gap attributable to freezing the visual encoder and sampling a single token scale per minibatch? Please quantify the effects of these training changes and clarify whether PARCEL's advantage over M3M^{3} persists when the baseline follows the original training recipe.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dRBd (1/2)

We thank the reviewer for their kind review and positive feedback on our work, along with their acknowledgement of our work being well-motivated, supported by both theoretical analyses and empirical results, and finding our ablations to support our proposed architectural choices.

Below, we address each of the weaknesses raised by the reviewer.


W1: Novelty and Methodological Contribution

PARCEL builds on spatial pooling and query resampling, but its contribution is beyond their mere coexistence: the two pathways are coupled through pool-conditioning, budget-aware routing and joint training, all inside the connector, and PCQR's attention mechanism is only one part of this design. Three findings support this:

  • (1) As the reviewer also acknowledges, PARCEL's gains extend to anchor-only budgets, where the LLM receives only pooled tokens, i.e. with the same inference time input as $M^3$. At 16 tokens, plain 4x4 pooling retains 83.1%, $M^3$ with parameter-matched self-attention 86.8%, and PARCEL 88.3%; at 64 tokens, 90.4% vs. 95.3% (Table 4). The same holds on LLaVA 1.5 (Table 3) and at 448x448 resolution (Table 9). A side-by-side combination of $M^3$ and MQT would not produce this effect, it arises because the pooled pathway is trained together with the query pathway inside a single budget-routed model, and the routing itself matters (a fixed 4x4 anchor grid caps retention at 90.2% at 256 tokens vs. 95.6%, Table 4).

  • (2) The attention-design ablations in Table 4 compare alternatives within PARCEL rather than PARCEL against the baselines. Even the ViT-only variant is not an existing method: it still feeds spatial anchors to the LLM alongside nested queries under budget-aware routing, i.e., it already contains the architecture we propose and differs only in the conditioning step. The 0.4-point difference is therefore the value of that one step, while the architecture it sits on improves over parameter-matched MQT and $M^3$ by +2.3/+3.4 points at 256 tokens and +2.8/+4.9 at 64 (Table 4). Within the design, PCQR remains best or tied at every budget, including over dual cross-attention, which also lets the queries see the anchors.

  • (3) Our spectral analysis explains when pooling and query resampling fail and why their conditioned combination through the PARCEL framework works (Figure 3), which, to our knowledge, prior connectors and training-free methods have not offered for these two paradigms.

AuthorsOpenReview ↗

Response to Reviewer dRBd (2/2)

W2: Effects of Training Changes on $M^3$

As noted by the reviewer, our $M^3$ reproduction differs from the default LLaVA 1.5-$M^3$ settings in two ways, keeping the ViT frozen and training with one scale per batch. Following the reviewer’s suggestion, we will mark the $M^3$ results in Table 3 as our reimplementation, mention it in the main text and detail the core differences in the Appendix. We chose these settings to be able to quantify the effect of the elastic connector as the only changing variable across baselines and PARCEL.

In particular, sampling one scale keeps training compute matched across methods, and accordingly we opted for the one token scale per batch variant, which was described as an alternative in the original $M^3$ work [F]. The reason for this is that averaging the $M^3$ loss over all five scales instead requires five forward passes per sample. Concretely, measured on 200 optimizer steps on 8xA100 chips, this variant takes 5.60x the training time per update compared to the baselines and PARCEL, thereby breaking the fairness of comparisons. Notably, in the $M^3$ work [F] (Table 14) also, the difference of this stage provided only ~2 pts improvements on benchmarks considered, while bringing substantially higher costs. Due to computational constraints this expensive variant brings, we have not been able to finalize an experiment comparing the all-scales per batch variant of $M^3$ and a compute-matched PARCEL yet, we will be doing our best to provide these results as soon as available.

Second, owing to its efficiency, we opted to keep the ViT frozen, following MQT [A] and the standard LLaVA 1.5 recipe [B], which is also commonly adopted across many established efficient connector works [C, D, E].


To quantify the effect of this choice, we retrained both $M^3$ and PARCEL with a trainable ViT. The results show that training the ViT improves both PARCEL and $M^3$. In particular, the mean retention rate of PARCEL improves respectively from 93.0% to 96.7% and 95.9% to 98.9% at budgets 36 and 144, and the mean retention rate of $M^3$ improves respectively from 92.0% to 96.1% and from 95.6% to 98.6% at the same budgets.

For the two benchmarks mentioned by the reviewer, TextVQA score is 52.6 and 54.8 for both PARCEL and $M^3$ with a trainable ViT respectively at budgets 36 and 144. We observe that PARCEL outperforms $M^3$ on POPE at both budgets, even at these improved retention rates (85.1 vs 84.4 and 86.4 vs 86.2 in favor of PARCEL at budgets 36 and 144 respectively), where our $M^3$ values are close to their corresponding reported values in [F]. We note that the remaining TextVQA difference to the reported $M^3$ results (~2 points) is in line with the all-scale averaging gain reported in [F] (Table 14). This difference therefore stems from the training compute budget rather than the connector, and we keep it fixed across methods: MQT, PARCEL, and $M^3$ all sample a single budget per batch in our setup, so the comparison is made under matched compute.

Furthermore, with a trainable ViT, both methods come close to the uncompressed reference on most of these benchmarks. This is consistent with prior findings that many common VQA benchmarks are largely insensitive to visual token compression [F, G, H], and our own findings (Appendix B).

Accordingly, we evaluated both methods also on DocVQA, the most compression-sensitive resolution-sensitive task from our PG-2 suite (Table 2). With a trainable ViT, PARCEL improves over $M^3$ by +2.8 retention points at 36 tokens (72.2% vs. 69.4%) and +2.0 at 144 tokens (81.4% vs. 79.4%). These results mirror our PG-2 results, where the ViT is always trained and PARCEL consistently outperforms $M^3$ at shared budgets across 27 benchmarks.


We hope the trainable-ViT results and the clarification of what the Table 4 ablations compare address the reviewer's concerns. We are glad to clarify any further items during the discussion period if anything remains unclear, and would appreciate it if the reviewer would reconsider their rating in light of these discussions.


[A] Hu, Wenbo, et al. "Matryoshka query transformer for large vision-language models." NeurIPS 2024.

[B] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." CVPR 2024.

[C] Cha, Junbum, et al. "Honeybee: Locality-enhanced projector for multimodal llm." CVPR 2024.

[D] Li, Wentong, et al. "Tokenpacker: Efficient visual projector for multimodal llm." IJCV 2025.

[E] Lou, Haoran, et al. "Llava-sp: Enhancing visual representation with visual spatial tokens for mllms." ICCV 2025.

[F] Cai, Mu, et al. "Matryoshka multimodal models." ICLR 2025.

[G] Liao, Chenfei, et al. "Are we using the right benchmark: An evaluation framework for visual token compression methods." ACL 2026.

[H] Beyer, Lucas, et al. "Paligemma: A versatile 3b vlm for transfer." arXiv:2407.07726 (2024).

Reviewer_YB3eOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presented a new efficient LVLM approach, PARCEL, that aims to address limitations of two typical existing methods under aggressive token compression. In tradition, the rigid spatial downsampling method such as M^3 acts like a low-pass filter that ignores critical visual details, and the other type of methods adopts non-local query sampling, such as MQT, which discarded explicit grid-aligned formulation so that they are limited for grounding or localization tasks. The proposed PARCEL combined both into a PCQR module, eventually achieving the best trade-off. Experimental results show the superiority of PARCEL on 27 benchmarks, and ablation analysis also validates the design choices.

Critical strengths
  • The proposed method is well motivated, especially, the design is empirically grounded in experimental comparison with M^3 and MQT in terms of the spatial frequency.

  • Experimental results showed significant gains over baselines under different visual token budgets. And the superiority is validated on multiple benchmarks across diverse visual understanding tasks.

Critical weaknesses
  • The scope of the paper is limited to beating M^3 and MQT, even though this paper categorized them into two different paradigms. Considering M^3 and MQT are not even the widely-used high-impact baselines, we can hardly see how much impact this paper will contribute to the general audience in the efficient LVLM domain.

  • Technical contribution. The method section does not distinguish the unique technical contributions compared to the M^3 and MQT. The figure 2 even conveys an impression that the PCQR is a simple combination of M^3 and MQT. Thus, what are the unique contributions from this paper?

  • Experiment sections do not show the efficiency improvements like the speedup or memory footage, but only the benchmark accuracy under some budget constraints.

Questions for authors
  • Equation (2) is not properly explained. For example, what does it mean by telling A1 <= B <= Bmax?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YB3e (1/2)

We thank the reviewer for their kind review and positive feedback on our work, along with their acknowledgement of our work being well-motivated, empirically grounded, and finding our method to have validated superiority over diverse visual understanding tasks. Below, we address each of the weaknesses raised by the reviewer.


W1: Scope of the Paper and Effect on Broader Efficient LVLM Audience

PARCEL is an elastic visual token compression method, where a single model serves multiple visual token budgets at inference. In this setting, MQT (NeurIPS 2024, [A]) and $M^3$ (ICLR 2025, [B]) are the representative methods of their two established paradigms, query resampling and spatial pooling, which is why we use them as primary baselines across two LVLM families and 30+ benchmarks.

Like both, PARCEL operates entirely at the bridge between the visual encoder and the LLM. PARCEL does not modify decoder layers or depend on their attention states or on batch-level statistics, so one connector design transfers across LVLMs without decoder-specific changes. We see this as what makes elastic compression practical at foundation-model scale, and the question we address is how much performance can be recovered under this constraint.

Beyond these baselines, our initial submission also compares PARCEL against three well-established visual token compression methods, AIM [C], FastV [D], and PruMerge [E], in Appendix B.3 (Table 10). At matched prefill compute (~1 TFLOP), PARCEL improves mean retention over these methods by +5.9, +23.0, and +7.7 points, respectively (94.7% vs. 88.8%, 71.7%, and 87.0%). We could not discuss these results in the main text due to space limits, but we will move them and the related discussions into the main text.


W2: Unique Contributions Over The Baselines

PARCEL builds on spatial pooling and query resampling, but its contribution is beyond their mere coexistence. Concretely, our work contributes (i) Pool-Conditioned Query Resampling, where queries are conditioned on the pooled anchors before they attend to the ViT features, (ii) the finding that training the two pathways jointly strengthens the pooled pathway itself, so that PARCEL outperforms $M^3$ even at anchor-only budgets where no queries are used at inference, and finally (iii) a spectral analysis explaining why pooling and query resampling fail in complementary ways. Three results show these choices matter beyond a naive combination:

  • (1) PARCEL's gains extend to anchor-only budgets, where the LLM receives only pooled tokens, i.e. with the same inference time input as $M^3$. At 16 tokens, plain 4x4 pooling retains 83.1%, $M^3$ with parameter-matched self-attention 86.8%, and PARCEL 88.3%, and at 64 tokens, 90.4% vs. 95.3% (Table 4). A side-by-side combination of $M^3$ and MQT would not produce this effect: It arises because the pooled pathway is trained together with the query pathway inside a single budget-routed model, and the routing itself matters (a fixed 4x4 anchor grid caps retention at 90.2% at 256 tokens vs. 95.6% of PARCEL, Table 4).
  • (2) Pool-conditioning is the best or tied-best option at every budget, improving over unconditioned queries by +0.4 points at both 256 and 16 tokens (Table 4). The full design gains +2.3/+3.4 points over parameter-matched MQT/$M^3$ at 256 tokens and +2.8/+4.9 at 64, so the gain is structural rather than the added capacity.
  • (3) Our spectral analysis explains when pooling and query resampling fail and why their conditioned combination works: PARCEL's pooled grid concentrates more power at low frequencies than $M^3$'s, while its queries attend to higher-frequency detail beyond the pooled grid (Figure 3). To our knowledge, no prior work offers such an analysis of the two elastic paradigms.

We will revise Fig. 2(c) to show the conditioning step and state the contributions above explicitly in Section 3.


[A] Hu, Wenbo, et al. "Matryoshka query transformer for large vision-language models." Advances in Neural Information Processing Systems 37 (2024): 50168-50188.

[B] Cai, Mu, et al. "Matryoshka multimodal models." International Conference on Learning Representations. Vol. 2025. 2025.

[C] Zhong, Yiwu, et al. "Aim: Adaptive inference of multi-modal llms via token merging and pruning." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[D] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[E] Shang, Yuzhang, et al. "Llava-prumerge: Adaptive token reduction for efficient large multimodal models." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

AuthorsOpenReview ↗

Response to Reviewer YB3e (2/2)

W3: Efficiency Measures in the Experimental Sections

We thank the reviewer for their feedback and agree that reporting the efficiency values is important. We already reported some efficiency measures in Figure 1 (right) of the main text and provided more detailed measures in Appendix C. We will move a compact version of these measurements into the main experimental section and include improved references to the relevant sections.

Here, we also summarize the key numbers for completeness. Since the visual token budget directly sets the LLM's prefill and KV-cache cost, reducing the PaliGemma-2 budget from 256 to 16 tokens lowers 16-frame video prefill from 24.3 to 4.9 TFLOPs and the KV cache from 423 MB to 33 MB (Figure 1). On LLaVA 1.5, PARCEL at 36 tokens requires 0.99 prefill TFLOPs against 8.18 for the uncompressed model (Table 10).

At a matched budget, all elastic methods pass the same number of tokens to the LLM, so KV-cache sizes are identical and only the connector cost differs. Measured on a single A100, the PARCEL connector accounts for 0.4–2.6% (depending on the specific visual token budget) of full-model prefill latency and at most 0.12 GB of peak memory. Relative to MQT, PARCEL is 1.2–1.3% faster at anchor-only budgets and 0.8–1.0% slower at query-active budgets (Appendix C.2). PARCEL therefore improves accuracy at an effectively matched cost.


Q1: Equation Notation Ambiguity

We thank the reviewer for pointing this out. Equation 2 describes how a requested budget $B$ is split between anchors and queries: the router picks the largest anchor grid $A_{j(B)}$ that fits within $B$ and assigns the remaining $B - A_{j(B)}$ tokens to the queries. The condition $A_1 \le B \le B_{\max}$ only defines the range of valid budgets, i.e., at least the smallest anchor grid and at most the largest supported budget ($B_{\max} = 256$ at 224x224 for PaliGemma-2). For example, with A=16,64\mathcal{A} = \\{16, 64\\}, B=256B = 256 gives 64 anchors and 192 queries. We described these equations in detail under Appendix E, though we will also further clarify this in the main text.


We hope these clarifications, together with the efficiency measurements and the Table 10 comparison, address the reviewer's concerns on scope, contribution and efficiency. If so, we would be grateful if the reviewer would consider raising their assessment. We are also happy to clarify any further item within the discussion period.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer YB3e,

We would like to kindly follow up on our response before the discussion period ends. Following the reviewer's suggestions, we have now uploaded a revised draft with all changes marked in blue. In particular, we moved the comparison against the training-free baselines into the main text as Table 5, stated the contributions of PARCEL explicitly in Section 3 and revised Figure 2(c) accordingly, reported the efficiency measures in Section 4, and clarified Equation 2 with an example in Section 3.3.

We remain glad to clarify any further items if anything remains unclear, and would be grateful to hear whether our response and revision address the reviewer's concerns.

Thank you for your time in reviewing our work!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revision

Dear AC and Reviewers,

We thank both reviewers for their kind reviews and constructive feedback, and the AC for handling our submission. We are glad that both reviewers found our work well-motivated. Furthermore, Reviewer YB3e acknowledged our design being empirically grounded through the spectral comparison with $M^3$ and MQT and its validated superiority across diverse visual understanding tasks, and Reviewer dRBd acknowledged the strength of our theoretical analyses, the evaluation across two backbones, multiple budgets and image and video tasks along with our ablations going beyond simply increasing connector capacity, and our gains at anchor-only budgets.

Both reviewers rated our work positively with Weak Accept scores, and both Reviewer dRBd and Reviewer YB3e have kindly confirmed during the discussion that our rebuttal addresses their concerns.

Following the reviewers' suggestions, we have now uploaded a revised PDF with all changes marked in blue. In particular:

  1. Scope and training-free baselines (Reviewer YB3e W1). We moved the comparison against AIM, FastV and PruMerge at matched prefill compute from the Appendix into the main text as Table 5, along with its discussion in Section 4.2, while introducing these baselines in our experimental setup.

  2. Unique contributions and Figure 2 (Reviewer YB3e W2, Reviewer dRBd W1). Section 3 now explicitly states the contributions of PARCEL beyond the mere coexistence of pooling and query resampling. We also revised Figure 2(c) to show that the spatial anchors condition the elastic queries inside PCQR before the queries attend to the original ViT features. Both pathways are optimized within the same model, which strengthens each of them in turn, as also reflected by our gains at anchor-only budgets.

  3. Efficiency measures (Reviewer YB3e W3). We initially detailed these in our Appendix. Following the reviewer’s suggestion, Section 4 now reports the prefill FLOPs and KV-cache reductions of Figure 1 in text, along with the measured connector latency relative to MQT, with further pointers to Appendix C.

  4. Equation 2 (Reviewer YB3e Q1). Section 3.3 now explains the condition $A_1 \leq B \leq B_{\max}$ with a clear example, with the routing rules for each backbone further detailed in Appendix E.

  5. $M^3$ baseline and training recipe (Reviewer dRBd W2). We marked the $M^3$ results in Table 3 as our reimplementation, explained our rationale for matching training compute across methods in Section 4 based on fairness across different methods, and detailed the training differences from the original $M^3$ recipe in Appendix E.2. We also added the trainable-ViT experiment from the discussion as Appendix B.4, where PARCEL retains its lead over $M^3$ and our $M^3$ reproduction comes close to the originally reported POPE scores.

For the revised work, we note that the table numbering of the original submission is preserved for Tables 1-4. The training-free comparison is now Table 5, and the Appendix tables have thus shifted by one.

We thank both reviewers and the AC for their time and efforts in reviewing our work. Following the integration of the reviewers' suggestions, we believe that our presentation is further clarified and our contributions are stated more clearly. The method and the main conclusions remain unchanged and well-supported, as acknowledged by both reviewers in their initial reviews. We remain available for any further questions and appreciate the reviewers’ and the AC’s time and interest.

Authors of Submission #4636

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.