- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the visual computation reduction in multimodal embedding models. It first conducts an analysis of the representation similarity using CKA and attention mass in current LLM-based models and finds that visual structural similarity remains high in the shallow-to-middle layers, and <emb>-to-visual attention mass displays a U-shaped trend. The analysis motivates the Skip-then-Select framework, which trains a lightweight projector to deliver the visual tokens to the deeper layers where multimodal tokens are interacted. The tables show that the proposed method obtained good results.
Critical strengths
- The motivation experiment displays a consistent and clear insight into the visual token interaction behaviour in multimodal embedding models, providing clear support that transformer-based embedding models have redundant visual computation that may not be necessary for multimodal embedding.
- Comparing with the performance degradation, the acceleration is significant, as shown in Table 1. The performance also surpasses the visual compression baselines on average.
Critical weaknesses
- The design of the proposed method is not closely related to the motivation experiments. The high visual CKA score doesn't mean there is little change to the visual representations between layers, as the visual attention mass doesn't decrease to 0, meaning that there is still an informative mechanism that needs to be explored. The U-shaped trend of attention mass is also far-fetched, as well as the exact layer at which the visual tokens should be reinjected into the model. Based on what the authors have analyzed, why not directly remove the first 16 layers of the model and see if the model can already provide good joint embeddings? I wonder why we should reinject at 15/16 layer and prune at 18/19 layer?
- The ablation study in Table 2 is coarse-grained, as the controlled components also change the projector, trainable parameters, and the adaptation of post-fusion. The improvement of Stage 1 over Direct Reuse doesn't demonstrate the necessity of skipping.
- The paper chooses to compare pruning-based baselines, but lacks direct comparison with deep compression-based methods, including DeepInsert [1], etc.
- Detailed dataset curation can be added to the appendix for reproducibility for the two stages.
- The acceleration is not significant compared with VisionZip [2], according to Table 1.
[1] DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding. In ACL 2026. [2] VisionZip: Longer is Better but Not Necessary in Vision Language Models. In CVPR 2025.
Questions for authors
Please see the Critical Weaknesses.
7 comments
2. Ablation of the Skip module
Thank you for highlighting the need for more controlled ablations. The new Skip Projector variant uses the same shallow visual features as Direct Reuse. Only the projector is trained, with all other model parameters frozen. The results are as blow:
UniME-v2
| Method | Cls | VQA | Ret | Grd | Avg. ↑ |
|---|---|---|---|---|---|
| Direct Reuse | 54.06 | 43.46 | 54.43 | 63.60 | 52.30 |
| Skip Projector | 60.95 | 54.67 | 63.58 | 72.98 | 61.42 |
| Stage 1 Only | 63.80 | 60.34 | 66.09 | 82.20 | 65.65 |
Ops-MM-embedding-v1
| Method | Cls | VQA | Ret | Grd | Avg. ↑ |
|---|---|---|---|---|---|
| Direct Reuse | 60.65 | 33.89 | 58.14 | 58.88 | 52.18 |
| Skip Projector | 63.37 | 45.03 | 63.43 | 65.93 | 58.58 |
| Stage 1 Only | 66.26 | 61.06 | 66.28 | 74.93 | 65.78 |
As shown in the table, its improvement over Direct Reuse supports the benefit of learned feature adaptation. Joint adaptation of the projector and post-fusion layers in Stage 1 Only yields further gains.
3. Comparison with depth-wise compression methods
Following your suggestion, we reproduced Reroute and ViCA using their official code repositories, as DeepInsert’s code is not publicly available. We evaluate both methods on UniME-v2 and Ops-MM-embedding-v1, with results shown below.
UniME-v2
| Method | Cls | VQA | Ret | Grd | Avg. ↑ | FLOPs (T) ↓ | Latency (ms) ↓ |
|---|---|---|---|---|---|---|---|
| Reroute | 64.87 | 58.36 | 70.36 | 83.98 | 67.01 | 5.18 | 41.84 |
| ViCA | 62.86 | 46.60 | 67.20 | 73.78 | 61.00 | 4.79 | 38.11 |
| Ours | 64.42 | 61.13 | 70.53 | 88.60 | 68.23 | 4.67 | 37.29 |
Ops-MM-embedding-v1
| Method | Cls | VQA | Ret | Grd | Avg. ↑ | FLOPs (T) ↓ | Latency (ms) ↓ |
|---|---|---|---|---|---|---|---|
| Reroute | 67.37 | 58.21 | 67.91 | 70.23 | 65.32 | 1.62 | 37.04 |
| ViCA | 66.77 | 48.86 | 68.34 | 72.50 | 62.96 | 1.50 | 33.81 |
| Ours | 67.18 | 62.05 | 68.88 | 79.80 | 67.72 | 1.45 | 35.07 |
On both backbones, our method achieves a higher average MMEB score with fewer FLOPs than either of the two additional baselines. Moreover, our method maintains a substantial advantage on visual grounding tasks. Detailed experimental configurations and comparisons with the full set of baselines are provided in Lines 331–340 and Lines 354–377 of the revised manuscript, respectively.
1. Clarification of the relationship between CKA and layer skipping
We thank the reviewer for the valuable feedback. Below, we provide point-by-point responses to the concerns and questions raised in the review, along with additional experimental results.
Centered kernel alignment (CKA) quantifies the structural similarity between representations. We agree that high CKA similarity alone is insufficient to justify layer skipping. Nevertheless, the high CKA similarity observed between shallow and intermediate visual representations motivates us to explore whether a lightweight skip module can adapt shallow visual features for use in deeper layers. To evaluate this design, we report the MMEB-v1 ablation results from Table 2 below, alongside the original models with 20% visual-token retention for reference.
All configurations in the following two tables retain 20% of the visual tokens after selection. Original performs token selection without layer skipping. All four ablation variants additionally bypass visual computation in the intermediate layers, while text and embedding tokens continue to pass through these layers.
UniME-v2
Ops-MM-embedding-v1
Visual Oracle removes visual tokens from the intermediate layers and injects the original encoder's visual states at the re-entry layer, without fine-tuning. At the same 20% retention ratio, its average scores are 3.76 and 7.81 percentage points below Original (20% retained) on UniME-v2 and Ops-MM-embedding-v1, respectively. These results provide a diagnostic reference for the feature-adaptation comparisons below.
Direct Reuse reinserts the visual states after the post-attention RMSNorm in layer 0. Its average scores fall 11.62 and 9.95 percentage points below Visual Oracle, respectively. This is consistent with our representation analysis: high CKA indicates similar relational structure, while lower cosine similarity indicates changes in the directions of individual visual features. Thus, high structural similarity does not guarantee that shallow features can be directly reused in deeper layers.
Skip Projector trains only the projector to match the original encoder's visual features at the re-entry layer, keeping all other parameters frozen. It improves the average score over Direct Reuse by 9.12 and 6.40 percentage points, respectively, reducing the gaps to Visual Oracle to 2.50 and 3.55 points. These results support the benefit of learned feature adaptation across skipped layers.
Stage 1 Only jointly trains the projector and post-fusion layers through intermediate-state alignment. It improves the average score over Skip Projector by 4.23 and 7.20 percentage points and exceeds Visual Oracle by 1.73 and 3.65 points, respectively. These gains support the value of joint adaptation and suggest that it helps compensate for the omitted intermediate computation.
Thus, CKA serves as a motivation for our architecture, while these ablations provide empirical support for the feasibility of learned feature adaptation and post-fusion within the proposed design.
Removing the first 16 layers. Following your suggestion, we also evaluate an Ops-MM-embedding-v1 variant that removes the first 16 decoder layers entirely, eliminating their computation for visual, text, and embedding tokens. The MMEB-v1 results after fine-tuning are shown below.
The average score remains only 23.56 after fine-tuning, indicating that directly removing these layers substantially compromises embedding quality in this setting. ViS² retains layer 0 and continues processing text and embedding tokens through the intermediate layers, while a learned projector adapts shallow visual features for re-entry. This experiment supports our choice to preserve early-layer processing while selectively bypassing intermediate visual computation. We include these additional results in Appendix A.4.1, Table 8 (Lines 902–917) of the revised manuscript.