- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits the problem of evaluating vision encoders without training complete MLLMs. The authors propose RAVEL, a training-free method based on cross-modal nearest-neighbor agreement, enhanced with stabilized PCA whitening and patch-level similarity matching. They conduct an extensive evaluation involving 70 vision encoders, three language backbones, and 210 trained MLLMs. Experiments show that RAVEL achieves substantially higher correlations with downstream MLLM performance than existing unimodal and cross-modal evaluation methods.
Critical strengths
- Comprehensive and well-controlled evaluation. The paper evaluates a diverse collection of 70 vision encoders across three language backbones, using a consistent multimodal training pipeline. This provides convincing evidence for the effectiveness of RAVEL and reveals important limitations of existing cross-modal evaluation metrics.
- Simple and effective method with useful empirical insights. RAVEL achieves substantial improvements over existing methods without requiring additional model training. The analysis of PCA whitening and patch-level matching is particularly informative, showing how representation geometry and feature granularity affect encoder evaluation.
Critical weaknesses
- Limited evaluation of selection among similarly strong encoders. The main results emphasize correlations across a heterogeneous pool of 70 encoders, where performance differences can be substantial. However, practical encoder selection often involves choosing among a small set of competitive candidates. Although the paper includes a selection-based analysis, it remains unclear how reliably RAVEL distinguishes encoders with similar downstream performance.
- Limited generalization across MLLM configurations. All experiments use three relatively small language models, ranging from 1.5B to 1.7B parameters, and the same LLaVA-style training pipeline. It remains unclear whether the strong correlations generalize to larger language models or different multimodal architectures and training strategies. Additionally, much of the improvement comes from PCA whitening, a well-established technique, which somewhat limits the methodological novelty.
Questions for authors
- Could the authors report ranking correlations restricted to the top-performing encoders, such as the top 10 or top 20, and evaluate Top-1 selection regret on randomly sampled competitive candidate pools?
- Does RAVEL maintain its effectiveness with larger language models or different multimodal training recipes? Even a smaller-scale experiment with a different architecture would help clarify its generalizability.
- How sensitive is RAVEL to the distribution of the paired image-text data used for evaluation? Would the ranking remain stable when using a different dataset from the one involved in multimodal training?
- Given that PCA whitening substantially improves several existing metrics, could the authors further analyze the practical benefits of patch-level matching over whitened baselines, particularly when selecting among strong encoders?
5 comments
Response to Reviewer W58w (2/4)
Q1: Could the authors report ranking correlations restricted to the top-performing encoders, such as the top 10 or top 20, and evaluate Top-1 selection regret on randomly sampled competitive candidate pools?
Response. Thank you for this suggestion! We are happy to conduct additional experiments to evaluate RAVEL from a selection-based perspective. However, we would appreciate clarification on the precise definition of "top-1 selection regret," as the intended formulation is not entirely clear to us.
In the meantime, we report an alternative selection-based metric in Table R3.1b, which measures the average downstream performance of the final MLLM using the top-1 encoder selected by each evaluation metric.
In fact, we have already explored this evaluation approach in Appendix K ("ROBUSTNESS UNDER A SELECTION-BASED METRIC"). We chose to emphasize ranking correlations in the main paper because repeated top-1 selection over randomly sampled encoder subsets, followed by averaging, captures information closely related to ranking correlations. We therefore consider correlation-based measures a more comprehensive and straightforward way to evaluate the overall effectiveness of encoder rankings.
Nevertheless, we agree that selection-based evaluations provide a useful complementary perspective and are happy to report additional metrics if the reviewer could clarify the intended definition of "top-1 selection regret."
More specifically, our evaluation protocol for constructing the top-20 encoder pools and computing the selection-based metric is as follows:
- Constructing the top-20 encoder pools. For each language model backbone, we compute the unweighted average of the final MLLM's performance across 11 downstream benchmarks. We then select the 20 encoders with the highest average scores to form a backbone-specific pool of high-performing encoders, using their actual downstream performance as the ground truth.
- Computing the selection-based metric. For each top-20 pool, we enumerate all 15,504 possible five-encoder subsets (), rather than relying on random sampling. Within each subset, each evaluation method selects the encoder with the highest predicted score. We then record the selected encoder's actual downstream performance, measured by the unweighted average across the 11 benchmarks. Finally, we average these scores across all 15,504 subsets to obtain the selection-based performance for each evaluation method.
We report both ranking correlations and selection-based performance within the top-20 encoder pools in Tables R3.1a and R3.1b, respectively. RAVEL consistently achieves the best results under both evaluation criteria across all three language model backbones, demonstrating its effectiveness in distinguishing among similarly high-performing encoders.
Table R3.1a. Correlations within the Top-20 Pools.
| Methods | Qwen3-1.7B Spearman | Qwen3-1.7B Pearson | Qwen2.5-1.5B Spearman | Qwen2.5-1.5B Pearson | SmolLM2-1.7B Spearman | SmolLM2-1.7B Pearson |
|---|---|---|---|---|---|---|
| Linear probing | 0.508 | 0.494 | 0.570 | 0.539 | 0.568 | 0.365 |
| KNN | 0.761 | 0.572 | 0.736 | 0.604 | 0.557 | 0.321 |
| Mid-Training Loss | 0.364 | 0.311 | 0.268 | 0.330 | -0.014 | -0.011 |
| AC Policy | -0.031 | -0.013 | 0.256 | 0.199 | 0.393 | 0.316 |
| Alignment probing | 0.729 | 0.508 | 0.568 | 0.470 | 0.341 | 0.268 |
| RSA | -0.296 | -0.338 | -0.538 | -0.571 | -0.223 | -0.137 |
| CCA | -0.724 | -0.695 | -0.553 | -0.537 | -0.405 | -0.414 |
| MutualNN | -0.048 | -0.062 | -0.083 | -0.096 | -0.033 | 0.039 |
| GW | 0.831 | 0.700 | 0.686 | 0.675 | 0.448 | 0.344 |
| RAVEL | 0.853 | 0.719 | 0.786 | 0.725 | 0.587 | 0.545 |
Table R3.1b. Top-1 Selection Metric within the Top-20 Pools.
| Methods | Qwen3-1.7B Top-1 score | Qwen2.5-1.5B Top-1 score | SmolLM2-1.7B Top-1 score |
|---|---|---|---|
| Linear probing | 50.27 | 52.76 | 43.27 |
| KNN | 50.95 | 53.12 | 43.21 |
| Mid-Training Loss | 50.30 | 52.34 | 42.61 |
| AC Policy | 50.90 | 52.93 | 43.15 |
| Alignment probing | 51.62 | 53.53 | 43.10 |
| RSA | 48.11 | 51.30 | 42.50 |
| CCA | 47.53 | 50.67 | 41.81 |
| MutualNN | 47.83 | 50.70 | 41.45 |
| GW | 52.23 | 53.50 | 43.33 |
| RAVEL | 52.29 | 53.68 | 44.03 |
Response to Reviewer W58w (3/4)
W2: Evaluation uses only three 1.5B–1.7B language models and one LLaVA-style pipeline, leaving generalization to larger models, other architectures, and training recipes unclear. Much of the gain comes from established PCA whitening, raising concerns about methodological novelty.
Q2: Does RAVEL maintain its effectiveness with larger language models or different multimodal training recipes? Even a small-scale experiment with a different architecture would help clarify its generalizability.
Response. Thank you for the comment!
Regarding novelty, we would like to clarify that our contributions lie in adapting and applying two simple yet effective methods to the important task of evaluating vision encoders for MLLMs, which has not been explored in prior work on this problem. We do not claim that the underlying methods themselves are fundamentally novel. Rather, as detailed in Sections 3.2 and 3.3, we introduce necessary modifications to make these methods applicable and effective in the encoder-MLLM evaluation setting.
While the general principles behind these methods have been explored in other domains, their adaptation to encoder-MLLM evaluation represents a novel contribution. We believe these contributions are valuable to the broader research community, particularly in facilitating the development and selection of vision encoders for next-generation MLLMs. We will make this point clearer in the next version of our paper.
Regarding more diverse evaluation settings, thank you for this suggestion! To further evaluate the generalizability of RAVEL, we conduct additional experiments using the publicly released checkpoints from Law of Vision Representation in MLLMs (LAWs) [1], with Vicuna-v1.5-7B as the language model backbone. The LAWs collection comprises 13 vision encoder configurations, including diffusion-based encoders and combinations of features from two different vision encoders. Notably, both the language model backbone and the vision encoder configurations differ substantially from those evaluated in our original experiments, providing a complementary setting for assessing RAVEL's generalizability. Importantly, we directly apply RAVEL to this new evaluation setting using the same hyperparameters as in our original experiments, without any additional hyperparameter tuning or adaptation.
As shown in Table R3.2, RAVEL achieves the highest correlations under both evaluation measures, demonstrating its ability to generalize to larger language model backbones and diverse vision encoder architectures, including diffusion-based and combined representations. These results further highlight RAVEL's robustness and transferability across substantially different model configurations. We will add these results to the next version of our paper.
Table R3.2. Correlations on the LAWs collection.
| Methods | Spearman | Pearson |
|---|---|---|
| Linear probing | 0.845 | 0.800 |
| KNN | 0.804 | 0.867 |
| Alignment probing | 0.817 | 0.903 |
| MutualNN | 0.687 | 0.665 |
| GW | 0.755 | 0.602 |
| RAVEL | 0.851 | 0.923 |
Q3: How sensitive is RAVEL to the distribution of the paired image-text data used for evaluation? Would the ranking remain stable when using a different dataset from the one involved in multimodal training?
Response. Thank you for raising this question!
To investigate RAVEL's sensitivity to the data used for metric estimation, we conduct additional experiments using 1,000 image-text pairs from DOCCI [2], an image captioning dataset that does not overlap with our downstream evaluation benchmarks. We deliberately choose DOCCI to introduce a substantial distribution shift from the data used in our original experiments, thereby providing a more rigorous assessment of RAVEL's robustness to unseen evaluation data. Specifically, we replace only the paired image-text data used for metric estimation, while keeping the original vision encoder collection and their corresponding downstream MLLM performance scores unchanged. This allows us to isolate the effect of changing the evaluation data source. Since this change only affects cross-modal methods that rely on paired image-text data, we restrict this comparison to those methods.
We compute both ranking correlations for each of the three language model backbones. RAVEL consistently achieves the highest correlations under both measures across all three backbones, demonstrating that its effectiveness generalizes to unseen evaluation data.
Table R3.3. Correlations using 1,000 DOCCI data pairs.
| Methods | Qwen3-1.7B Spearman | Qwen3-1.7B Pearson | Qwen2.5-1.5B Spearman | Qwen2.5-1.5B Pearson | SmolLM2-1.7B Spearman | SmolLM2-1.7B Pearson |
|---|---|---|---|---|---|---|
| RSA | 0.115 | 0.123 | 0.044 | 0.101 | 0.511 | 0.592 |
| CCA | 0.230 | 0.202 | 0.315 | 0.178 | 0.412 | 0.169 |
| GW | 0.305 | 0.191 | 0.306 | 0.164 | 0.325 | 0.235 |
| MutualNN | -0.131 | 0.100 | -0.051 | 0.151 | 0.169 | 0.327 |
| RAVEL | 0.756 | 0.773 | 0.860 | 0.836 | 0.860 | 0.879 |
Response to Reviewer W58w (1/4)
We thank the reviewer for the careful assessment and for recognizing the breadth of the evaluation and the usefulness of the representation analyses.
Response. Thank you for raising this important point! We agree that evaluating on a large, diverse encoder pool and a smaller, more competitive pool emphasizes different aspects of encoder evaluation. The former primarily assesses robustness and generalizability across diverse encoder configurations, while the latter focuses on the ability to distinguish between encoders with similar downstream performance.
Our paper primarily focuses on evaluating the generalizability of different metrics across a diverse collection of vision encoders. This contrasts with prior studies, which typically consider a limited number and diversity of encoders, potentially leading to conclusions that do not generalize well across different encoder configurations, as demonstrated in Figure 1.
Meanwhile, we would like to emphasize that the difficulty of an evaluation setting is also metric-dependent. Although RAVEL typically achieves correlations above 0.80 in our main experiments, most existing methods achieve correlations around or below 0.50. This suggests that our large-scale evaluation setting is already challenging for existing metrics, despite RAVEL's strong performance.
Nevertheless, we agree that evaluating RAVEL on a smaller, more competitive encoder subset would provide complementary insights into its ability to distinguish among high-performing encoders. To this end, we conduct additional controlled experiments on the top 20 encoders ranked by their actual downstream MLLM performance, creating a more challenging setting where the differences between candidate encoders are potentially smaller.
The corresponding results are presented in our response to Q1, in Tables R3.1a and R3.1b.