- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the extent to which existing pre-trained large models converge to the same representation of the world and can therefore be related by simple transformations. The authors consider 14 text embedding models and 4 image embedding models. They compare their representations with linear CKA, downstream transfer, representation fidelity, and k-NN retrieval. They find that not all models can be easily translated to one another, and that models coming from the same family generally exhibit greater compatibility.
Critical strengths
- This paper tackles the important question of to what extent different models trained independently learn compatible representations. They cover a large number of text models as well as a smaller number of vision models and use multiple complementary evaluation criteria to assess the compatibility and downstream transfer between models.
Critical weaknesses
-
The interpretation of the observed compatibility results remains largely descriptive. Although the paper includes hierarchical clustering, it does not quantify whether within-family compatibility is systematically higher than between-family compatibility, or assess the stability of this pattern across tasks and anchor protocols. Further analysis of its association with model characteristics, such as embedding dimension, training objective, and training-data similarities, would help explain the observed results.
-
The reported results lack uncertainty estimates across training runs, seeds, or anchor selection. This makes it difficult to assess whether the small differences observed between many heatmap entries are robust.
Questions for authors
- The paper focuses on within modality transfer. Could the same framework be extended to translate between text and image embeddings using paired image-caption anchors?
2 comments
Dear authors, thank you for your answer. The additional experiments provide quantitative support for higher within-family compatibility, addressing part of my concern. However, the observed advantage remains modest, and the additional analyses do not substantially change my overall assessment. I would like to keep my score.
Thank you for the review. Your questions about quantification and stability are closer to the paper's goal, and we address them with new analyses of the existing matrices.
W1. The interpretation is descriptive; within versus between family is not quantified; stability is not assessed
The main observation is family structure. Because it dominates the figures, other factors are visually overshadowed. The paper did not isolate or report them; numerical analysis extracts their effects. We report them and compare them to family.
We define family by architecture group: BERT-like, T5-like, causal decoder, and multimodal VLM. The family contrast is positive in all five text tasks; it is significant for DBpedia-14, AG News, and IMDB under word anchors (p=0.010 to 0.017), and smaller under sentence anchors. The image suite has one same-family pair (Qwen3-VL 2B/8B), so we report that contrast descriptively and note that four image models cannot support significance.
Model size does not help: same-size pairs are 0.528 against 0.586 (-0.058), but the four same-size pairs are mostly cross-family, so size is confounded with family. Within families the sample is too small to isolate size: BERT-like has one same-size pair against 20 different-size pairs, and the causal and T5 families have no same-size pairs. Dimension and pooling are similarly confounded; within families the sample is too small to isolate them (BERT-like 11 vs 10 for dimension; causal 3 vs 3 for pooling). We report this as a limitation, not a null result.
The dendrogram isolates pair structure. Pair identity explains 6.7% (words), 8.1% (sentences), and 13.2% (image) of cross-pair variance; task explains 36.8%, 47.9%, and 52.3%; translator explains 17.2%, 4.4%, and 25.6%. Anchor agreement is in Table 7 (RR 0.820, IRP 0.491, pinv 0.656, SGD 1.000). The family advantage is stable across tasks; the absolute accuracies are task- and anchor-dependent.
Pair rankings across translators correlate from -0.21 to 0.35 on text and from 0.49 to 0.83 on image. The translators agree on the family trend but disagree on individual pairs, which is a stronger form of translator-dependence.
A pair-level regression (n=91 text pairs) confirms the ordering: shared design and family remain significant alongside architecture, while size, dimension, pooling, and tokenizer do not add significant variance.
W2. No uncertainty estimates across runs, seeds, or anchor selection
We will add pair-level bootstrap 95% confidence intervals (resampling unordered pairs):
The text method differences exceed these intervals. The image IRP and pinv intervals overlap, so we will not claim an image IRP-versus-pinv difference. We will add repeated seeds for trained maps and heads where the cached pipeline permits, and state the single-run limitation where it does not.
Question: extending to text-image translation with paired image-caption anchors
This is an important question, and we agree that it is central to the universality claim. The claim is not limited to a single modality: in its strongest form, for example the Platonic representation hypothesis, it holds that the same concept is represented in a shared way regardless of whether it is expressed as text, an image, or another modality. Our benchmark tests the within-modality part of that claim, using paired text inputs for text models and paired image inputs for image models.
A paired image-caption anchor experiment would test the cross-modal part directly: whether a shared anchor set can relate a text representation to an image representation, and whether a translator fitted on caption pairs preserves the target decision interface. That is a natural and important extension.