acceptodds
Under review as a conference paper at ICLR 2027

Skip-then-Select: Understanding Visual Token Dynamics in Universal Multimodal Embeddings

Abstract

Universal multimodal embedding models harness the capabilities of multimodal large language models to encode diverse inputs into a shared representation space, but generating a single embedding requires costly processing of long visual sequences through the entire language backbone. How visual information evolves across layers and contributes to the final embedding has not been thoroughly investigated in UMEs. We therefore jointly analyze cross-layer visual-state evolution and within-layer embedding-to-visual interaction. This analysis reveals two patterns: visual representations retain similar relational structure across shallow-to-middle layers, while embedding-to-visual attention follows a U-shaped trajectory in total mass and increasingly focuses on task-relevant regions in deep layers. Building on these findings, we propose Visual Skip-then-Select (), which reduces both the depth and sequence length of visual computation. The Skip module preserves early multimodal interaction and routes shallow visual features through a lightweight projector to bypass middle-layer processing. The Select module reintroduces the full visual sequence for interaction with text and embedding tokens in deeper layers, then prunes visual tokens using embedding-to-visual attention. We train the modified encoder in two stages, first aligning intermediate representations with the original model and then distilling its final embeddings. Evaluated on 36 MMEB datasets, retains 95.6% of the original performance while reducing FLOPs by 75.0% and achieving a end-to-end speedup, averaged across two UME models. Our models and code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Universal Multimodal Embeddings, Multimodel Embedding, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uRzeOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the visual computation reduction in multimodal embedding models. It first conducts an analysis of the representation similarity using CKA and attention mass in current LLM-based models and finds that visual structural similarity remains high in the shallow-to-middle layers, and <emb>-to-visual attention mass displays a U-shaped trend. The analysis motivates the Skip-then-Select framework, which trains a lightweight projector to deliver the visual tokens to the deeper layers where multimodal tokens are interacted. The tables show that the proposed method obtained good results.

Critical strengths
  1. The motivation experiment displays a consistent and clear insight into the visual token interaction behaviour in multimodal embedding models, providing clear support that transformer-based embedding models have redundant visual computation that may not be necessary for multimodal embedding.
  2. Comparing with the performance degradation, the acceleration is significant, as shown in Table 1. The performance also surpasses the visual compression baselines on average.
Critical weaknesses
  1. The design of the proposed method is not closely related to the motivation experiments. The high visual CKA score doesn't mean there is little change to the visual representations between layers, as the visual attention mass doesn't decrease to 0, meaning that there is still an informative mechanism that needs to be explored. The U-shaped trend of attention mass is also far-fetched, as well as the exact layer at which the visual tokens should be reinjected into the model. Based on what the authors have analyzed, why not directly remove the first 16 layers of the model and see if the model can already provide good joint embeddings? I wonder why we should reinject at 15/16 layer and prune at 18/19 layer?
  2. The ablation study in Table 2 is coarse-grained, as the controlled components also change the projector, trainable parameters, and the adaptation of post-fusion. The improvement of Stage 1 over Direct Reuse doesn't demonstrate the necessity of skipping.
  3. The paper chooses to compare pruning-based baselines, but lacks direct comparison with deep compression-based methods, including DeepInsert [1], etc.
  4. Detailed dataset curation can be added to the appendix for reproducibility for the two stages.
  5. The acceleration is not significant compared with VisionZip [2], according to Table 1.

[1] DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding. In ACL 2026. [2] VisionZip: Longer is Better but Not Necessary in Vision Language Models. In CVPR 2025.

Questions for authors

Please see the Critical Weaknesses.

7 comments
AuthorsOpenReview ↗

1. Clarification of the relationship between CKA and layer skipping

We thank the reviewer for the valuable feedback. Below, we provide point-by-point responses to the concerns and questions raised in the review, along with additional experimental results.

Centered kernel alignment (CKA) quantifies the structural similarity between representations. We agree that high CKA similarity alone is insufficient to justify layer skipping. Nevertheless, the high CKA similarity observed between shallow and intermediate visual representations motivates us to explore whether a lightweight skip module can adapt shallow visual features for use in deeper layers. To evaluate this design, we report the MMEB-v1 ablation results from Table 2 below, alongside the original models with 20% visual-token retention for reference.

All configurations in the following two tables retain 20% of the visual tokens after selection. Original performs token selection without layer skipping. All four ablation variants additionally bypass visual computation in the intermediate layers, while text and embedding tokens continue to pass through these layers.

UniME-v2

MethodClsVQARetGrdAvg. ↑
Original65.3163.0068.6582.4067.68
Visual Oracle62.3558.0666.7174.1363.92
Direct Reuse54.0643.4654.4363.6052.30
Skip Projector60.9554.6763.5872.9861.42
Stage 1 Only63.8060.3466.0982.2065.65

Ops-MM-embedding-v1

MethodClsVQARetGrdAvg. ↑
Original69.2868.5569.2177.2569.94
Visual Oracle67.3448.8467.6365.8062.13
Direct Reuse60.6533.8958.1458.8852.18
Skip Projector63.3745.0363.4365.9358.58
Stage 1 Only66.2661.0666.2874.9365.78
  • Visual Oracle removes visual tokens from the intermediate layers and injects the original encoder's visual states at the re-entry layer, without fine-tuning. At the same 20% retention ratio, its average scores are 3.76 and 7.81 percentage points below Original (20% retained) on UniME-v2 and Ops-MM-embedding-v1, respectively. These results provide a diagnostic reference for the feature-adaptation comparisons below.

  • Direct Reuse reinserts the visual states after the post-attention RMSNorm in layer 0. Its average scores fall 11.62 and 9.95 percentage points below Visual Oracle, respectively. This is consistent with our representation analysis: high CKA indicates similar relational structure, while lower cosine similarity indicates changes in the directions of individual visual features. Thus, high structural similarity does not guarantee that shallow features can be directly reused in deeper layers.

  • Skip Projector trains only the projector to match the original encoder's visual features at the re-entry layer, keeping all other parameters frozen. It improves the average score over Direct Reuse by 9.12 and 6.40 percentage points, respectively, reducing the gaps to Visual Oracle to 2.50 and 3.55 points. These results support the benefit of learned feature adaptation across skipped layers.

  • Stage 1 Only jointly trains the projector and post-fusion layers through intermediate-state alignment. It improves the average score over Skip Projector by 4.23 and 7.20 percentage points and exceeds Visual Oracle by 1.73 and 3.65 points, respectively. These gains support the value of joint adaptation and suggest that it helps compensate for the omitted intermediate computation.

Thus, CKA serves as a motivation for our architecture, while these ablations provide empirical support for the feasibility of learned feature adaptation and post-fusion within the proposed design.

Removing the first 16 layers. Following your suggestion, we also evaluate an Ops-MM-embedding-v1 variant that removes the first 16 decoder layers entirely, eliminating their computation for visual, text, and embedding tokens. The MMEB-v1 results after fine-tuning are shown below.

ConfigurationClsVQARetGrdAvg. ↑
Remove the first 16 layers (fine-tuned)19.376.5428.4362.0023.56

The average score remains only 23.56 after fine-tuning, indicating that directly removing these layers substantially compromises embedding quality in this setting. ViS² retains layer 0 and continues processing text and embedding tokens through the intermediate layers, while a learned projector adapts shallow visual features for re-entry. This experiment supports our choice to preserve early-layer processing while selectively bypassing intermediate visual computation. We include these additional results in Appendix A.4.1, Table 8 (Lines 902–917) of the revised manuscript.

AuthorsOpenReview ↗

2. Ablation of the Skip module

Thank you for highlighting the need for more controlled ablations. The new Skip Projector variant uses the same shallow visual features as Direct Reuse. Only the projector is trained, with all other model parameters frozen. The results are as blow:

UniME-v2

MethodClsVQARetGrdAvg. ↑
Direct Reuse54.0643.4654.4363.6052.30
Skip Projector60.9554.6763.5872.9861.42
Stage 1 Only63.8060.3466.0982.2065.65

Ops-MM-embedding-v1

MethodClsVQARetGrdAvg. ↑
Direct Reuse60.6533.8958.1458.8852.18
Skip Projector63.3745.0363.4365.9358.58
Stage 1 Only66.2661.0666.2874.9365.78

As shown in the table, its improvement over Direct Reuse supports the benefit of learned feature adaptation. Joint adaptation of the projector and post-fusion layers in Stage 1 Only yields further gains.

AuthorsOpenReview ↗

3. Comparison with depth-wise compression methods

Following your suggestion, we reproduced Reroute and ViCA using their official code repositories, as DeepInsert’s code is not publicly available. We evaluate both methods on UniME-v2 and Ops-MM-embedding-v1, with results shown below.

UniME-v2

MethodClsVQARetGrdAvg. ↑FLOPs (T) ↓Latency (ms) ↓
Reroute64.8758.3670.3683.9867.015.1841.84
ViCA62.8646.6067.2073.7861.004.7938.11
Ours64.4261.1370.5388.6068.234.6737.29

Ops-MM-embedding-v1

MethodClsVQARetGrdAvg. ↑FLOPs (T) ↓Latency (ms) ↓
Reroute67.3758.2167.9170.2365.321.6237.04
ViCA66.7748.8668.3472.5062.961.5033.81
Ours67.1862.0568.8879.8067.721.4535.07

On both backbones, our method achieves a higher average MMEB score with fewer FLOPs than either of the two additional baselines. Moreover, our method maintains a substantial advantage on visual grounding tasks. Detailed experimental configurations and comparisons with the full set of baselines are provided in Lines 331–340 and Lines 354–377 of the revised manuscript, respectively.

Reviewer_gepwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies redundant visual computation in universal multimodal embedding models (UMEs). Its analysis finds that visual representations retain similar relational structure across shallow-to-middle layers, while deep-layer embedding-to-visual (E2V) attention becomes more aligned with task-relevant regions. Motivated by these observations, the authors propose Visual Skip-then-Select (ViS²): preserve early multimodal interaction, route visual states through a projector that bypasses intermediate language-model layers, restore multimodal fusion at deeper layers, and then select visual tokens using E2V attention. Training combines intermediate-state alignment with final-embedding distillation.

Across 36 MMEB datasets, ViS² retains 97.4% and 93.8% of the original average scores on UniME-v2 and Ops-MM-embedding-v1, respectively, while reducing FLOPs by 77.4% and 72.6%. The reported inference speedups are 2.97× and 1.57× on an A100 at batch size 8.

Critical strengths
  1. The problem is practically relevant, and the design has a clear motivation. Processing long visual sequences through an entire language backbone is expensive when the output is a single embedding. Separating temporary computational bypass from permanent token removal, and restoring interaction with the full visual sequence before pruning, is a sensible design.

  2. The results show a competitive performance–efficiency tradeoff on two backbones, with particularly strong visual grounding. In Table 1, ViS² achieves average scores of 68.23 and 67.64, compared with 67.22 and 66.34 for the strongest evaluated compression baselines on the respective backbones. Its grounding scores exceed the best competing grounding results by 4.57 and 7.43 points. These findings provide useful empirical support for preserving visual states for later fusion.

  3. The paper includes informative analyses and supporting experiments. Beyond CKA and attention analyses, it examines projector architectures, training stages, and selection strategies, and reports measured latency at multiple batch sizes. Figure 6 also illustrates how the selected regions change with the question for the same document.

Critical weaknesses

1. The individual contributions of Skip and Select, and their interaction, are not directly established.

High CKA and low intermediate E2V attention provide motivation for reducing computation, but retaining visual tokens does not preserve the feature updates and continued multimodal interaction that would occur in the skipped layers. All Table 2 variants bypass intermediate layers and retain 20% of visual tokens at inference. Consequently, this table mainly establishes the value of adaptation within the proposed architecture, rather than separating the contributions of Skip, Select, and their combination. As the authors acknowledge, Stage 1 jointly learns the projector and adapts the post-fusion layers, so its improvement over Direct Reuse does not isolate either component.

A related concern is whether skipping affects the reliability of subsequent selection. For queries requiring associations between names or roles and visual entities, relationships across multiple regions, or more abstract semantic interpretation, identifying relevant evidence may depend on sufficient multimodal processing. Skip could therefore affect not only the final representation but also the E2V scores used by Select, potentially causing useful evidence to be removed before it has been adequately incorporated. This is a hypothesis to test, not an established failure: the text pathway, projector, and restored fusion may compensate for the omitted computation.

Figure 6 provides encouraging examples, including the largest carrot and a document recipient’s name. However, these examples and the localization analysis in Figure 4 do not directly measure how skipping changes selection quality on more demanding queries. A focused comparison on an identifiable subset of existing evaluation examples involving entity associations or relations would help. With the selection layer and retention rate fixed, the authors could compare relevant-region coverage, where annotations permit, and final retrieval performance with and without Skip, alongside representative failure cases.

The highest-priority ablation is a comparison of Skip-only, late-Select-only, and full ViS² under training conditions matched as closely as possible, reporting both accuracy and cost. Comparing selection at the first re-entry layer where E2V attention is available against selection after several fusion layers would further clarify the value of restored fusion.

2. The advantage over related depth-wise compression methods remains unclear.

Table 1 primarily compares visual-token pruning or compression methods. The paper also discusses closely related approaches that alter visual computation across depth: DeepInsert inserts multimodal tokens into intermediate layers; ViCA bypasses visual self-attention and feed-forward updates while retaining sparse text-to-vision cross-attention; and Reroute allows deferred visual tokens to re-enter later selection stages. Reroute is particularly relevant to the motivation concerning irreversible removal and changing token importance across depth.

These methods also address multimodal computation, although their principal evaluation settings differ from the single-vector embedding and similarity-based retrieval setting studied here. Their existing results therefore do not establish their effectiveness for UMEs. Nevertheless, adapting one representative method to the same embedding task would help determine the advantage of ViS² over related bypass or recoverable-routing mechanisms. The current results establish higher average scores than the tested pruning baselines at the evaluated configurations, but leave this comparison open.

Training budgets also warrant clarification. Section 4.1 states that all baselines receive fine-tuning similar to Stage 2, which is a useful fairness measure. ViS² additionally receives Stage 1 adaptation. Reporting training data, objectives, and total training cost, together with a reasonably budget-matched comparison against a strong baseline, would help distinguish architectural gains from additional adaptation.

3. The scope of the performance–efficiency advantage needs further characterization.

The main results concentrate on configurations retaining 20% of visual tokens, apart from PDrop’s progressive schedule. Equal retention does not imply equal cost: on UniME-v2, ViS² uses 4.67 TFLOPs versus 4.29 for VisionZip. The reported results demonstrate a competitive operating point; a few additional retention settings for the strongest baselines would clarify the range over which the advantage holds.

The paper already reports multiple batch sizes and explicitly attributes the implementation speedup associated with Table 4 to FlashAttention with last-token attention recomputation. Further comparisons should control attention implementations where feasible or explain the optimizations available to each method. In Table 1, the two backbone speedups differ substantially, and the UniME-v2 latency advantage over VisionZip is only 0.40 ms per input at batch size 8. Measurement variability would help establish the robustness of this small difference.

Questions for authors
  1. Under comparable training conditions, how do Skip-only, late-Select-only, and full ViS² compare in accuracy and cost? How much does selection after several fusion layers improve over selection at the first re-entry layer?

  2. For examples involving entity associations, relations across regions, or abstract semantics, does Skip affect subsequent E2V selection quality? With the selection layer and retention rate fixed, can you compare preservation of relevant evidence and final retrieval performance with and without Skip?

  3. Can you provide a comparison with one representative depth-wise compression or recoverable-routing method adapted to the same UME setting, with training budgets specified? Do the overall and grounding advantages persist at comparable FLOPs or latency?

  4. For reproducibility, please clarify the distillation data and validation split used to choose configurations. Also, does the projector receive the complete output of layer 0 or the residual state after its attention sublayer? Equation (3) and the implementation description appear to differ on this point. Do the Table 2 variants use the same feature extraction point?

6 comments
AuthorsOpenReview ↗

1. The individual contributions of Skip and Select, and their interaction, are not directly established.

We thank the reviewer for the valuable feedback. Below, we provide point-by-point responses to the concerns and questions raised in the review, along with additional experimental results.

a. Ablation of Skip-only, Select-only, and Skip-then-Select

Thank you for highlighting this comparison. To address this concern, we conducted additional experiments on Ops-MM-embedding-v1, using the same sampled training data for a fair comparison. We evaluate the following configurations:

  • Skip-then-Select: Our default configuration in paper.
  • Skip-only: Same as our setting in paper without select module.
  • Select-only: Same as our setting in paper without skip module.
  • Skip-only* and Select-only*: For comparison at similar FLOPs, Skip-only* skips to a deeper layer, while Select-only* selects at a shallower layer.
ConfigurationFLOPs (T)Avg.
Skip-only2.6968.94
Select-only4.0670.41
Skip-only*1.4866.13
Select-only*1.5764.71
Skip-then-Select1.4567.72
  1. Compared with Skip-only, Skip-then-Select reduces FLOPs by 46.1%, with a decrease of 1.22 percentage points in the average score. Select-only achieves an average score 2.69 percentage points higher than Skip-then-Select, but requires 2.8 times the FLOPs.

  2. At comparable FLOPs, Skip-then-Select outperforms Skip-only* and Select-only* by 1.59 and 3.01 percentage points, respectively, while using fewer FLOPs.

Ablation figure

We further plot precision against FLOPs to compare Skip-then-Select with Skip-only and Select-only As shown in the figure, Skip-then-Select achieves a more favorable performance–efficiency trade-off than either component alone. Additional details are provided in Lines 416–435 of the revised manuscript.

b. The effectiveness of post-fusion

To evaluate the benefit of post-fusion before token selection, we conduct an ablation on Ops-MM-embedding-v1. We fix the visual re-entry layer and compare token selection at this layer (without additional post-fusion processing) with selection after 3 or 6 post-fusion layers.

ConfigurationClsVQARetGrdAvg.FLOPs (T)
0 post-fusion layer65.0756.7768.2376.6865.111.04
3 post-fusion layers67.1862.0568.8879.8067.721.45
6 post-fusion layers68.1462.8470.3383.3569.091.87

Compared with token selection at the re-entry layer, selection after 3 or 6 post-fusion layers improves the average score by 2.61 and 3.98 percentage points, respectively. These results support the benefit of post-fusion, with a longer fusion stage yielding higher scores across all four task categories. Delaying selection allows the full visual sequence to participate in more multimodal interaction before pruning, but also increases computation. By default, we select tokens after three post-fusion layers to balance performance and computational efficiency.

c. The effect of skipping on visual token selection

We compare the top 20% of visual tokens selected by the original Ops-MM-embedding-v1 model and ViS² across four grounding datasets. For each input, let SorigS_{\mathrm{orig}} and SViS2S_{\mathrm{ViS}^2} denote the two selected token sets, each containing kk retained tokens. We define the overlap ratio as ∣Sorig∩SViS2∣/k×100%|S_{\mathrm{orig}} \cap S_{\mathrm{ViS}^2}| / k \times 100\%. Although their selections differ, the average overlap is 76.62%, indicating substantial agreement with the original model.

MetricMSCOCORefCOCORefCOCO-MatchingVisual7W-PointingAvg.
Overlap ratio (%)81.3175.2075.0674.9276.62

We also visualize the selected regions in both successful and unsuccessful retrieval examples. In successful cases, ViS² retains regions relevant to the query, as illustrated below.

Successful cases.

The failure cases below illustrate two patterns: the selected regions cover query-relevant content but retrieval is incorrect, or the selected regions fail to capture the relevant content.

Failure cases.

Additional visualizations and analysis are provided in the appendix of the revised manuscript (Lines 1080–1130).

AuthorsOpenReview ↗

2. The advantage over related depth-wise compression methods remains unclear.

a. Additional comparisons with depth-wise compression methods

Following your suggestion, we reproduced Reroute and ViCA using their official code repositories, as DeepInsert’s code is not publicly available. We evaluate both methods on UniME-v2 and Ops-MM-embedding-v1, with results shown below.

UniME-v2

MethodClsVQARetGrdAvg. ↑FLOPs (T) ↓Latency (ms) ↓
Reroute64.8758.3670.3683.9867.015.1841.84
ViCA62.8646.6067.2073.7861.004.7938.11
Ours64.4261.1370.5388.6068.234.6737.29

Ops-MM-embedding-v1

MethodClsVQARetGrdAvg. ↑FLOPs (T) ↓Latency (ms) ↓
Reroute67.3758.2167.9170.2365.321.6237.04
ViCA66.7748.8668.3472.5062.961.5033.81
Ours67.1862.0568.8879.8067.721.4535.07

On both backbones, our method achieves a higher average MMEB score with fewer FLOPs than either of the two additional baselines. Moreover, our method maintains a substantial advantage on visual grounding tasks. Detailed experimental configurations and comparisons with the full set of baselines are provided in Lines 331–340 and Lines 354–377 of the revised manuscript, respectively.

b. Clarification of fine-tuning

For the image-based experiments, all compared methods use the same training set of approximately 250K samples drawn from the 20 training datasets in MMEB-v1. Each method sees the same total number of training examples, ensuring consistent data exposure and supporting a fair comparison. We have added these details to the revised manuscript to clarify the training setup and address potential ambiguity regarding the fairness of the comparisons.

AuthorsOpenReview ↗

3. The scope of the performance–efficiency advantage needs further characterization.

a. Comparison with additional retention settings

Using Ops-MM-embedding-v1, we compare ViS² with FastV, VisionZip, and Reroute on MMEB-v1 across five computational budgets, with FLOPs approximately matched across methods. The table below reports the average MMEB-v1 scores.

Method / FLOPs (T)1.301.451.611.761.91
FastV61.5663.8264.8265.7867.34
VisionZip63.6064.5966.0966.8467.52
Reroute62.6663.0665.3266.3667.24
ViS² (Ours)66.2167.7268.3968.6168.76

ViS² achieves the highest average score among the compared methods at all five evaluated budgets, spanning 1.30–1.91 T FLOPs, with gains of 1.24–3.13 percentage points over the strongest baseline at each budget. These results support the effectiveness of our method across the evaluated range of computational budgets.

b. Clarification of implementation and experimental details

  • Differences in speedup between the two backbones. Differences in how the two backbones handle image resolution explain the gap in latency speedup. UniME-v2 uses a fixed input resolution of 384 × 384, upsampling smaller images, whereas Ops-MM-embedding-v1 uses dynamic resolution. The fixed visual-token count in UniME-v2 provides a more consistent amount of visual computation to reduce, helping explain its larger latency speedup.

  • Distillation data and evaluation protocol. For distillation, we use the same training set of approximately 250K samples drawn from the 20 training datasets in MMEB-v1. We evaluate performance across all 36 MMEB-v1 datasets following the standard evaluation protocol. Latency and FLOPs are measured using the same set of 4.3k multimodal inputs sampled from the MMEB-v1 evaluation set.

  • Input to the projector. The projector receives the visual hidden states after the post-attention RMSNorm in layer 0, i.e., the input to the FFN of layer 0. Direct Reuse uses features from this same extraction point without applying a projector. We have revised the notation in Equation (3) and clarified the descriptions of Direct Reuse and the corresponding implementation details to explicitly identify this shared feature extraction point. The revised formulation is consistent with our implementation.

Reviewer_c4cUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first analyzes how visual information evolves across layers and contributes to the final embedding in universal multimodal embedding models. The analysis shows that visual representations remain similar across shallow-to-middle layers, while embedding-to-visual attention follows a U-shaped pattern in deeper layers. Building on these findings, it proposes a method that reduces both the depth and sequence length of visual tokens. Across two MLLM-based embedders and MMEB datasets, it achieves the highest overall score and the lowest inference latency.

Critical strengths
  1. The paper is well-written, starting with a detailed analysis of visual token dynamics in universal multimodal embedding models and providing useful insights into visual representation similarity and embedding-to-visual attention patterns.

  2. The proposed method is intuitive and effectively combines visual layer skipping and token pruning to reduce computational costs while largely preserving embedding performance.

  3. The experiments are conducted on two different multimodal embedding models and include comparisons with several existing visual token compression methods.

Critical weaknesses

I think there are two major concerns:

  1. The paper is missing an ablation study that directly compares Skip-only, Select-only, and Skip-then-Select. Such an ablation would help clarify the individual contributions of these two components and whether combining them provides additional benefits.

  2. The experiments mainly compare against visual token compression methods. However, the paper also discusses several closely related methods involving layer skipping. Since layer skipping is also a key component of the proposed method, I think some related methods should be included as baselines.

Questions for authors
  1. The experiments are limited to the original MMEB benchmark, which mainly focuses on single-image inputs. I am wondering whether the proposed method can be extended to more complex scenarios, such as video and multi-image inputs in MMEB-V2.

  2. The proposed method is motivated by the high CKA similarity across intermediate layers. I am wondering whether these observations are sufficient to justify skipping visual computation in shallow layers.

7 comments
AuthorsOpenReview ↗

1. Ablation of Skip-only, Select-only, and Skip-then-Select

We thank the reviewer for the valuable feedback. Below, we provide point-by-point responses to the concerns and questions raised in the review, along with additional experimental results.

Firstly, thank you for highlighting this comparison. To address this concern, we conducted additional experiments on Ops-MM-embedding-v1, using the same sampled training data for fair comparison. We evaluate the following configurations:

  • Skip-then-Select: Our default configuration in paper.
  • Skip-only: Same as our setting in paper without select module.
  • Select-only: Same as our setting in paper without skip module.
  • Skip-only* and Select-only*: For comparison at similar FLOPs, Skip-only* skips to a deeper layer, while Select-only* selects at a shallower layer.
ConfigurationFLOPs (T) ↓Avg. ↑
Skip-only2.6968.94
Select-only4.0670.41
Skip-only*1.4866.13
Select-only*1.5764.71
Skip-then-Select1.4567.72
  1. Compared with Skip-only, Skip-then-Select reduces FLOPs by 46.1%, with a decrease of 1.22 percentage points in the average score. Select-only achieves an average score 2.69 percentage points higher than Skip-then-Select, but requires 2.8 times the FLOPs.

  2. At comparable FLOPs, Skip-then-Select outperforms Skip-only* and Select-only* by 1.59 and 3.01 percentage points, respectively, while using fewer FLOPs.

Ablation figure

We further plot precision against FLOPs to compare Skip-then-Select with Skip-only and Select-only As shown in the figure, Skip-then-Select achieves a more favorable performance–efficiency trade-off than either component alone. Additional details are provided in Lines 416–435 of the revised manuscript.

AuthorsOpenReview ↗

2. Additional comparisons with layer-skipping methods

Following your suggestion, we reproduced Reroute and ViCA methods by their official repository. We evaluate both methods on UniME-v2 and Ops-MM-embedding-v1, with results shown below.

UniME-v2

MethodClsVQARetGrdAvg. ↑FLOPs (T) ↓Latency (ms) ↓
Reroute64.8758.3670.3683.9867.015.1841.84
ViCA62.8646.6067.2073.7861.004.7938.11
Ours64.4261.1370.5388.6068.234.6737.29

Ops-MM-embedding-v1

MethodClsVQARetGrdAvg. ↑FLOPs (T) ↓Latency (ms) ↓
Reroute67.3758.2167.9170.2365.321.6237.04
ViCA66.7748.8668.3472.5062.961.5033.81
Ours67.1862.0568.8879.8067.721.4535.07

On both backbones, our method achieves a higher average MMEB score with fewer FLOPs than either of the two additional baselines. Detailed experimental configurations and comparisons with the full set of baselines are provided in Lines 331–340 and Lines 354–377 of the revised manuscript, respectively.

AuthorsOpenReview ↗

3. Applicability to video tasks

Following your suggestion, we further evaluated ViS² on the video tasks in MMEB-V2 using Ops-MM-embedding-v1. The results are presented below.

ConfigurationVideo-OverallV-CLSV-QAV-RETV-MRETFLOPs (T)
Original (official)53.7659.6862.2245.7243.2160.11
Original (reproduced)54.0159.7663.0145.9542.8960.11
Ours51.1857.7257.0645.0340.7618.63

Compared with the reproduced baseline, our method retains 94.76% of the Video-Overall score while using 30.99% of the FLOPs. These experiments suggest that our method is also applicable to video tasks and yields promising initial results. Details of these additional experiments are provided in the appendix of the revised manuscript (Lines 995–1017).

Our current work primarily focuses on accelerating UMEs for image-based tasks. In future work, we plan to extend our approach to a broader range of data types and tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.