- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PARCEL, an elastic visual token compression architecture that enables a single large vision-language model (LVLM) to operate at multiple inference-time token budgets. The authors examine complementary limitations of Matryoshka Multimodal Models () and the Matryoshka Query Transformer (MQT): spatial average pooling can obscure fine-grained details and introduce aliasing, and query-only resampling lacks explicit grid-aligned spatial anchors and can weaken dense spatial grounding. Evaluations on PaliGemma-2 and LLaVA 1.5 report improved aggregate performance retention over the and MQT baselines at matched token budgets. The reported gains also extend to anchor-only budgets, where no query tokens are used at inference.
Critical strengths
- Elastic visual token compression is a valuable research direction for LVLMs, addressing the computational and memory costs of dense visual inputs while allowing a single model to accommodate different deployment budgets.
- The paper is well motivated by complementary weaknesses of existing elastic compression methods: spatial pooling can obscure fine-grained details, whereas query-only resampling lacks explicit grid-aligned spatial anchors. The theoretical analysis and the downstream spatial-grounding results provide empirical support for combining spatial anchors with queries that extract complementary information.
- The evaluation covers two LVLM backbones, multiple token budgets, and diverse image and video tasks. Ablations support the proposed architectural choices beyond simply increasing connector capacity. The gains at anchor-only budgets further suggest that joint training benefits the pooled pathway even when no queries are used at inference.
Critical weaknesses
-
The novelty is somewhat incremental. PARCEL largely combines the spatial pooling of [1] with the learnable queries and nested dropout of MQT [2]. Pool-conditioned query resampling and budget-aware routing refine how these existing components interact, but the overall method remains close to a hybrid of the two prior approaches. The methodological contribution therefore appears to be an incremental integration of established compression mechanisms. The attention-design ablations in Table 4 also show modest gains from the specific PCQR formulation: at 256 tokens, mean retention improves from 95.2% with ViT-only cross-attention and 95.4% with dual cross-attention to 95.6% with PCQR.
-
The results in Table 3 are substantially lower on POPE and TextVQA than the original LLaVA-1.5- results in [1,3] at the same token budgets. At 144 and 36 tokens, respectively, POPE is lower by 2.8 and 2.9 points, and TextVQA is lower by 3.16 and 4.17 points. Appendix E.2 describes changes to the original training recipe, including freezing the visual encoder and sampling one token scale per minibatch instead of training the visual encoder and averaging losses across all scales. The authors should discuss why they perform these operations and mark the results of the reimplementation. Otherwise, the effects of these changes on the baseline performance are not quantified, leaving unclear how much they contribute to the observed gap and affect the reported improvements over .
[1] Cai, M., Yang, J., Gao, J., & Lee, Y. J. (2025, May). Matryoshka multimodal models. In International Conference on Learning Representations (Vol. 2025, pp. 46254-46272).
[2] Hu, W., Dou, Z. Y., Li, L. H., Kamath, A., Peng, N., & Chang, K. W. (2024). Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37, 50168-50188.
[3] Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024, June). Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26286-26296). IEEE.
Questions for authors
Why are the POPE and TextVQA scores in Table 3 substantially lower than the original LLaVA-1.5- results at 144 and 36 tokens? To what extent is this gap attributable to freezing the visual encoder and sampling a single token scale per minibatch? Please quantify the effects of these training changes and clarify whether PARCEL's advantage over persists when the baseline follows the original training recipe.
2 comments
Response to Reviewer dRBd (2/2)
W2: Effects of Training Changes on $M^3$
As noted by the reviewer, our $M^3$ reproduction differs from the default LLaVA 1.5-$M^3$ settings in two ways, keeping the ViT frozen and training with one scale per batch. Following the reviewer’s suggestion, we will mark the $M^3$ results in Table 3 as our reimplementation, mention it in the main text and detail the core differences in the Appendix. We chose these settings to be able to quantify the effect of the elastic connector as the only changing variable across baselines and PARCEL.
In particular, sampling one scale keeps training compute matched across methods, and accordingly we opted for the one token scale per batch variant, which was described as an alternative in the original $M^3$ work [F]. The reason for this is that averaging the $M^3$ loss over all five scales instead requires five forward passes per sample. Concretely, measured on 200 optimizer steps on 8xA100 chips, this variant takes 5.60x the training time per update compared to the baselines and PARCEL, thereby breaking the fairness of comparisons. Notably, in the $M^3$ work [F] (Table 14) also, the difference of this stage provided only ~2 pts improvements on benchmarks considered, while bringing substantially higher costs. Due to computational constraints this expensive variant brings, we have not been able to finalize an experiment comparing the all-scales per batch variant of $M^3$ and a compute-matched PARCEL yet, we will be doing our best to provide these results as soon as available.
Second, owing to its efficiency, we opted to keep the ViT frozen, following MQT [A] and the standard LLaVA 1.5 recipe [B], which is also commonly adopted across many established efficient connector works [C, D, E].
To quantify the effect of this choice, we retrained both $M^3$ and PARCEL with a trainable ViT. The results show that training the ViT improves both PARCEL and $M^3$. In particular, the mean retention rate of PARCEL improves respectively from 93.0% to 96.7% and 95.9% to 98.9% at budgets 36 and 144, and the mean retention rate of $M^3$ improves respectively from 92.0% to 96.1% and from 95.6% to 98.6% at the same budgets.
For the two benchmarks mentioned by the reviewer, TextVQA score is 52.6 and 54.8 for both PARCEL and $M^3$ with a trainable ViT respectively at budgets 36 and 144. We observe that PARCEL outperforms $M^3$ on POPE at both budgets, even at these improved retention rates (85.1 vs 84.4 and 86.4 vs 86.2 in favor of PARCEL at budgets 36 and 144 respectively), where our $M^3$ values are close to their corresponding reported values in [F]. We note that the remaining TextVQA difference to the reported $M^3$ results (~2 points) is in line with the all-scale averaging gain reported in [F] (Table 14). This difference therefore stems from the training compute budget rather than the connector, and we keep it fixed across methods: MQT, PARCEL, and $M^3$ all sample a single budget per batch in our setup, so the comparison is made under matched compute.
Furthermore, with a trainable ViT, both methods come close to the uncompressed reference on most of these benchmarks. This is consistent with prior findings that many common VQA benchmarks are largely insensitive to visual token compression [F, G, H], and our own findings (Appendix B).
Accordingly, we evaluated both methods also on DocVQA, the most compression-sensitive resolution-sensitive task from our PG-2 suite (Table 2). With a trainable ViT, PARCEL improves over $M^3$ by +2.8 retention points at 36 tokens (72.2% vs. 69.4%) and +2.0 at 144 tokens (81.4% vs. 79.4%). These results mirror our PG-2 results, where the ViT is always trained and PARCEL consistently outperforms $M^3$ at shared budgets across 27 benchmarks.
We hope the trainable-ViT results and the clarification of what the Table 4 ablations compare address the reviewer's concerns. We are glad to clarify any further items during the discussion period if anything remains unclear, and would appreciate it if the reviewer would reconsider their rating in light of these discussions.
[A] Hu, Wenbo, et al. "Matryoshka query transformer for large vision-language models." NeurIPS 2024.
[B] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." CVPR 2024.
[C] Cha, Junbum, et al. "Honeybee: Locality-enhanced projector for multimodal llm." CVPR 2024.
[D] Li, Wentong, et al. "Tokenpacker: Efficient visual projector for multimodal llm." IJCV 2025.
[E] Lou, Haoran, et al. "Llava-sp: Enhancing visual representation with visual spatial tokens for mllms." ICCV 2025.
[F] Cai, Mu, et al. "Matryoshka multimodal models." ICLR 2025.
[G] Liao, Chenfei, et al. "Are we using the right benchmark: An evaluation framework for visual token compression methods." ACL 2026.
[H] Beyer, Lucas, et al. "Paligemma: A versatile 3b vlm for transfer." arXiv:2407.07726 (2024).
Response to Reviewer dRBd (1/2)
We thank the reviewer for their kind review and positive feedback on our work, along with their acknowledgement of our work being well-motivated, supported by both theoretical analyses and empirical results, and finding our ablations to support our proposed architectural choices.
Below, we address each of the weaknesses raised by the reviewer.
W1: Novelty and Methodological Contribution
PARCEL builds on spatial pooling and query resampling, but its contribution is beyond their mere coexistence: the two pathways are coupled through pool-conditioning, budget-aware routing and joint training, all inside the connector, and PCQR's attention mechanism is only one part of this design. Three findings support this:
(1) As the reviewer also acknowledges, PARCEL's gains extend to anchor-only budgets, where the LLM receives only pooled tokens, i.e. with the same inference time input as $M^3$. At 16 tokens, plain 4x4 pooling retains 83.1%, $M^3$ with parameter-matched self-attention 86.8%, and PARCEL 88.3%; at 64 tokens, 90.4% vs. 95.3% (Table 4). The same holds on LLaVA 1.5 (Table 3) and at 448x448 resolution (Table 9). A side-by-side combination of $M^3$ and MQT would not produce this effect, it arises because the pooled pathway is trained together with the query pathway inside a single budget-routed model, and the routing itself matters (a fixed 4x4 anchor grid caps retention at 90.2% at 256 tokens vs. 95.6%, Table 4).
(2) The attention-design ablations in Table 4 compare alternatives within PARCEL rather than PARCEL against the baselines. Even the ViT-only variant is not an existing method: it still feeds spatial anchors to the LLM alongside nested queries under budget-aware routing, i.e., it already contains the architecture we propose and differs only in the conditioning step. The 0.4-point difference is therefore the value of that one step, while the architecture it sits on improves over parameter-matched MQT and $M^3$ by +2.3/+3.4 points at 256 tokens and +2.8/+4.9 at 64 (Table 4). Within the design, PCQR remains best or tied at every budget, including over dual cross-attention, which also lets the queries see the anchors.
(3) Our spectral analysis explains when pooling and query resampling fail and why their conditioned combination through the PARCEL framework works (Figure 3), which, to our knowledge, prior connectors and training-free methods have not offered for these two paradigms.