acceptodds
Under review as a conference paper at ICLR 2027

How Far Do Simple Transformations Translate Across Heterogeneous Text and Image Embedding Models?

Abstract

We investigate whether simple transformations can translate representations across heterogeneous text and image embedding models. Understanding how independently trained models organize semantic information is an enabler for AI-to-AI latent communication without decoding into human-readable text, and for reusing components across models. A recurring hypothesis in the literature is that independently trained models converge to a shared geometry over semantic content, so that a simple, low-supervision transformation should relate their internal representations; we test this hypothesis of latent universality in a realistic setting beyond simplified benchmarks. We study 14 text embedding models and 4 image/multimodal models that differ in architecture, pooling strategy, training objective, and dimension, and evaluate compatibility with linear CKA, downstream transfer, representation fidelity, and -NN retrieval, on full evaluation sets and three anchor protocols. Simple translators recover meaningful shared structure and support transfer for some compatible pairs, but fail sharply for others; same-design models cluster tightly while cross-design pairs degrade in structured ways. Compatibility depends jointly on architecture, training objective, readout/normalization, and data distribution. Overall, the results show that heterogeneous embedding spaces are not universally related by simple mappings as often suggested in the literature.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embedding Models, Text Models, Image Models, Multimodal Models, Embedding Alignment, Stitching, Embedding Translation, Semantic Communications

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x2PuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the extent to which existing pre-trained large models converge to the same representation of the world and can therefore be related by simple transformations. The authors consider 14 text embedding models and 4 image embedding models. They compare their representations with linear CKA, downstream transfer, representation fidelity, and k-NN retrieval. They find that not all models can be easily translated to one another, and that models coming from the same family generally exhibit greater compatibility.

Critical strengths
  • This paper tackles the important question of to what extent different models trained independently learn compatible representations. They cover a large number of text models as well as a smaller number of vision models and use multiple complementary evaluation criteria to assess the compatibility and downstream transfer between models.
Critical weaknesses
  • The interpretation of the observed compatibility results remains largely descriptive. Although the paper includes hierarchical clustering, it does not quantify whether within-family compatibility is systematically higher than between-family compatibility, or assess the stability of this pattern across tasks and anchor protocols. Further analysis of its association with model characteristics, such as embedding dimension, training objective, and training-data similarities, would help explain the observed results.

  • The reported results lack uncertainty estimates across training runs, seeds, or anchor selection. This makes it difficult to assess whether the small differences observed between many heatmap entries are robust.

Questions for authors
  • The paper focuses on within modality transfer. Could the same framework be extended to translate between text and image embeddings using paired image-caption anchors?
2 comments

Thank you for the review. Your questions about quantification and stability are closer to the paper's goal, and we address them with new analyses of the existing matrices.

W1. The interpretation is descriptive; within versus between family is not quantified; stability is not assessed

The main observation is family structure. Because it dominates the figures, other factors are visually overshadowed. The paper did not isolate or report them; numerical analysis extracts their effects. We report them and compare them to family.

ContrastWithinBetweenDifferencep
Text words, same family0.6030.574+0.0280.017
Text sentences, same family0.6730.652+0.0210.058

We define family by architecture group: BERT-like, T5-like, causal decoder, and multimodal VLM. The family contrast is positive in all five text tasks; it is significant for DBpedia-14, AG News, and IMDB under word anchors (p=0.010 to 0.017), and smaller under sentence anchors. The image suite has one same-family pair (Qwen3-VL 2B/8B), so we report that contrast descriptively and note that four image models cannot support significance.

Model size does not help: same-size pairs are 0.528 against 0.586 (-0.058), but the four same-size pairs are mostly cross-family, so size is confounded with family. Within families the sample is too small to isolate size: BERT-like has one same-size pair against 20 different-size pairs, and the causal and T5 families have no same-size pairs. Dimension and pooling are similarly confounded; within families the sample is too small to isolate them (BERT-like 11 vs 10 for dimension; causal 3 vs 3 for pooling). We report this as a limitation, not a null result.

The dendrogram isolates pair structure. Pair identity explains 6.7% (words), 8.1% (sentences), and 13.2% (image) of cross-pair variance; task explains 36.8%, 47.9%, and 52.3%; translator explains 17.2%, 4.4%, and 25.6%. Anchor agreement is in Table 7 (RR 0.820, IRP 0.491, pinv 0.656, SGD 1.000). The family advantage is stable across tasks; the absolute accuracies are task- and anchor-dependent.

Pair rankings across translators correlate from -0.21 to 0.35 on text and from 0.49 to 0.83 on image. The translators agree on the family trend but disagree on individual pairs, which is a stronger form of translator-dependence.

A pair-level regression (n=91 text pairs) confirms the ordering: shared design and family remain significant alongside architecture, while size, dimension, pooling, and tokenizer do not add significant variance.

W2. No uncertainty estimates across runs, seeds, or anchor selection

We will add pair-level bootstrap 95% confidence intervals (resampling unordered pairs):

ProtocolRRIRPLinear (pinv)Linear (SGD)
Text, words0.651 [0.632, 0.668]0.522 [0.511, 0.534]0.478 [0.466, 0.490]0.684 [0.650, 0.717]
Text, sentences0.675 [0.657, 0.694]0.682 [0.674, 0.691]0.595 [0.581, 0.609]0.684 [0.650, 0.717]
Image0.309 [0.251, 0.398]0.507 [0.446, 0.582]0.459 [0.332, 0.594]0.693 [0.655, 0.742]

The text method differences exceed these intervals. The image IRP and pinv intervals overlap, so we will not claim an image IRP-versus-pinv difference. We will add repeated seeds for trained maps and heads where the cached pipeline permits, and state the single-run limitation where it does not.

Question: extending to text-image translation with paired image-caption anchors

This is an important question, and we agree that it is central to the universality claim. The claim is not limited to a single modality: in its strongest form, for example the Platonic representation hypothesis, it holds that the same concept is represented in a shared way regardless of whether it is expressed as text, an image, or another modality. Our benchmark tests the within-modality part of that claim, using paired text inputs for text models and paired image inputs for image models.

A paired image-caption anchor experiment would test the cross-modal part directly: whether a shared anchor set can relate a text representation to an image representation, and whether a translator fitted on caption pairs preserves the target decision interface. That is a natural and important extension.

Dear authors, thank you for your answer. The additional experiments provide quantitative support for higher within-family compatibility, addressing part of my concern. However, the observed advantage remains modest, and the additional analyses do not substantially change my overall assessment. I would like to keep my score.

Reviewer_2JkoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses whether simple transformations can translate representations across heterogeneous text and image embedding models. It evaluates 14 text and 4 image/multimodal encoders using linear mappings, Relative Representations (RR), and Inverse Relative Projection (IRP); and the results indicate that representation compatibility depends on model family, translation method, anchor selection, and downstream task.

Critical strengths
  • The paper examines diverse pretrained encoders and evaluates representation compatibility through multiple complementary metrics, including CKA, retrieval, representation fidelity, and downstream classification
  • The results demonstrate that high geometric similarity does not necessarily imply successful representation translation, providing useful evidence about the limitations of simple mappings
  • The comparison between word and sentence anchors provides useful insights into how anchor selection influences translation effectiveness
Critical weaknesses
  • The paper acknowledges in Section 7 that architecture, training objectives, data, tokenizer, readout, normalization, embedding dimensions, and model scale vary jointly. Although several same-family size comparisons are included, they are insufficient to isolate the effects of individual factors. Consequently, the observed compatibility patterns remain largely descriptive rather than explanatory. This is a major limitation given the paper's central objective of understanding representation universality across heterogeneous models.
  • The study primarily evaluates existing transformation approaches. Without stronger controlled experiments, it provides limited insight into the underlying mechanisms responsible for successful or unsuccessful representation translation.
  • The experiments use fixed anchor counts and protocols without systematically examining anchor selection, supervision budgets, or variability across random seeds. The higher supervision used by Linear (SGD) also complicates direct methodological comparisons.
  • The image evaluation includes only four models, while the broader implications for latent communication and reusable model components are not directly validated on structured intermediate representations.
Questions for authors
  • Can the authors extend the controlled same-family comparisons to isolate the effects of model scale, architecture, training objectives, and representation normalization?
  • Which factors most strongly determine representation compatibility, and can their individual contributions be quantified?
  • How sensitive are the reported conclusions to anchor selection, anchor count, and supervision budget?
1 comment

Thank you for the review.

W1. Jointly varying factors and descriptive patterns

The suite is heterogeneous by design, not factorial. The goal is to test whether a simple map can reconcile representations across models that differ in architecture, objective, readout, dimension, and data, not to isolate a single parameter. We chose maximal heterogeneity to identify the main factor or factors that govern compatibility. Isolating every factor at once would require a much larger, controlled model grid than is feasible to evaluate, and it would change the question from universality across heterogeneity to a single-parameter ablation. We have quantified the factors the suite supports: family is the dominant visible factor, with same-family pairs at 0.603 vs 0.574 on words (+0.028, p=0.017) and 0.673 vs 0.652 on sentences (+0.021, p=0.058). A variance decomposition puts task at 36.8-52.3% of cross-pair variance, translator at 4.4-25.6%, and pair identity at 6.7-13.2%. Size does not help: same-size pairs are 0.528 vs 0.586 (-0.058), but the four same-size pairs are mostly cross-family, so size is confounded. Within families the sample is too small to isolate size or dimension (BERT-like one same-size pair against 20; causal 3 vs 3 for pooling). We report associations, not causal estimates; future work may isolate individual factors, but that is a different study.

W2. Limited insight into mechanisms

The goal is not to propose a new translator. It is to test whether heterogeneous models are similar enough that simple translators from the literature work across them. We use the translators as functional metrics, just as CKA is a structural metric: CKA measures whether two models organize representations similarly, and a translator measures whether a representation from one model can function in another's space. The two are complementary. Future works might include new translator that test whether a more complex map can fit a pair, but that won't answer whether the representations share a geometry.

W3. Fixed anchors, supervision, and seeds

We vary the anchor protocol: IRP top-1 moves from 0.176 to 0.740 and pinv from 0.194 to 0.558, that means that the choice of anchors matters. However, we also tested training a linear map via SGD which is a high-supervision reference fit on 36,718 text and 50,000 image pairs. It still leaves a 0.180 text and 0.102 image gap. Morover, it shows the same trends regarding the compatibity being family-dependet.

W4. Image scale and structured interfaces

The image suite has four models, a limitation, and we report its contrasts descriptively. The reuse claim is validated on the image tasks: ImageNet-1k zero-shot classification, CIFAR zero-shot and frozen-embedding classification, and kNN retrieval all measure whether a target-trained component can consume a translated source representation. The experiments in this study were focused on pooled embeddings, and task that use pooled embeddings. What the paper does not validate is structured intermediate representations such as KV caches, image generation conditioning, etc., those are future work.

Questions

Q1. Extend controlled same-family comparisons to isolate scale, architecture, objectives, and normalization. Future work may isolate individual factors. The present study deliberately trades single-factor isolation for heterogeneity: the question is universality across heterogeneous models, not a single-parameter ablation. A full factorial design would require many more models, or a focus on a single factor (taking models withing same family with different sizes, taking models with same size but different families, etc.).

Q2. Which factors most strongly determine compatibility, and can their contributions be quantified? Family is the dominant visible factor (0.603 vs 0.574, +0.028, p=0.017; 0.673 vs 0.652, +0.021, p=0.058). The variance decomposition quantifies contributions: task 36.8-52.3%, translator 4.4-25.6%, pair identity 6.7-13.2%. Size and dimension do not add significant variance; within-family data are too sparse to isolate them. A pair-level regression confirms family.

Q3. How sensitive are the conclusions to anchor selection, anchor count, and supervision budget? From the 2 anchor protocols we did, we can see that the choice of anchors matters, however, the trend of same-family advantage is robust at least for the two anchor protocols we tested. Future work might focus on optimizing anchor selection, but that is a different study. Supervision is not matched: Linear (SGD) is a high-supervision reference, and still leaves 0.180 text and 0.102 image gaps.

Reviewer_hk7yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether simple transformations can translate representations across different embedding models. The authors evaluate four existing methods (linear pinv, linear SGD, RR, and IRP) across 14 text and 4 image/multimodal embedding models, using geometric similarity, retrieval, representation fidelity and downstream transfer as evaluation criteria. The main findings suggest that translation performance is highly dependent on model pairs, transformation methods, and downstream tasks, and that geometric similarity does not necessarily imply successful translation.

Critical strengths
  1. The paper evaluates a relatively large set of embedding models with different architectures and training objectives, providing a systematic comparison of existing embedding translation methods.
  2. The paper considers geometric similarity, retrieval, representation fidelity and downstream performance, offering a more comprehensive understanding of compatibility across embedding models than relying on a single metric.
  3. The substantial gain of IRP when replacing word anchors with sentence anchors highlights the sensitivity of translation methods to anchor selection.
Critical weaknesses
  1. The paper primarily evaluates existing methods, and most findings, such as better compatibility within model families and performance variations across methods and tasks, are somewhat expected. The analysis remains largely descriptive, without sufficiently investigating why translation succeeds or fails, or deriving actionable insights into how to predict or improve compatibility across embedding models.
  2. The experiments show that translation performance varies substantially across methods and anchor protocols, sometimes even for self-translation. However, the paper provides limited analysis of whether these failures arise from differences between embedding spaces or limitations of the translators themselves. Additional controlled experiments, such as varying the number of anchors and analyzing how self-translation and cross-model translation performance change accordingly, could help explain the observed failures.
  3. The paper discusses several potential applications, but it remains unclear how the current experimental setting relates to these practical scenarios. For example, RR requires training a new classifier in the relative representation space, raising questions about its advantage over directly training a classifier on the source embeddings. More generally, the paper would benefit from demonstrating scenarios where translation offers meaningful advantages over simpler alternatives such as re-encoding data or retraining downstream components.
Questions for authors

see above.

1 comment

Thank you for the review. The paper asks whether models learn the representation and whether components can be reused. It does not propose a translator or aim to beat a tailored head. Section 5.2 calls the translators probes, not competitors.

W1. The findings are expected and the analysis is descriptive

The universality claim we test is that independently trained models share a representation geometry regardless of architecture or training objective, so a simple map can reconcile them. The suite is deliberately heterogeneous across architectures, objectives, readouts, and dimensions. That heterogeneity is the point: it is the most demanding condition, so failure is most informative. If the claim held, it would be strong evidence for universality; if it fails, it marks the boundary.

The main observation is family. Family is the architecture group (BERT-like, T5-like, causal decoder, multimodal VLM). On words, same-family pairs reach 0.603 against 0.574 (+0.028, p=0.017); on sentences, 0.673 against 0.652 (+0.021, p=0.058). Family dominates the figures, so other factors are overshadowed.

Model size does not help: same-size pairs are 0.528 against 0.586 (-0.058), but the four same-size pairs are mostly cross-family, so size is confounded. Within families the sample is too small: BERT-like has one same-size pair against 20; causal and T5 have none. Dimension and pooling are similarly confounded (BERT-like 11 vs 10; causal 3 vs 3). This is a limitation.

This is an empirical test, not a proof. The strong universality claim is not supported in this suite; the broader claim is not disproved.

W2. Limited separation of embedding-space limits from translator limits

The goal is to test whether heterogeneous models are amenable to a simple transformation despite their different architectures and objectives. We test that with several translators, not by searching for the best anchors. Basic English is a compact semantic vocabulary, with sentences as a second protocol. Changing protocols moves IRP top-1 from 0.176 to 0.740 and pinv from 0.194 to 0.558, this is true that the choice of anchors matters. The paper does not claim that any translator is optimal; it claims that the same-family advantage is robust across translators and protocols, and that the strong universality claim is not supported in this suite.

More importantly, Linear (SGD) is trained on 36,718 text and 50,000 image pairs. If the claim held, SGD should recover the map, or a close approximation. Even SGD leaves a 0.180 text and 0.102 image gap, and shows the same trends across classification and kNN retrieval.

The translators can disagree on individual pairs (pair rankings correlate from -0.21 to 0.35 on text), but they agree on the direction of the global family trend: same-family pairs are more compatible, and T5 is almost too compatible in retrieval (IRP 0.96-0.98; SGD 0.99). Downstream results agree with kNN retrieval, and partly with CKA, at the global family level; cosine and reconstruction error are method-specific. Our conclusion still holds: the universality claim might be supported in some cases (especially one models share the same family, components reuse, or latent spaces communication is definitely possible, it is all the case even for some across-family pairs depending on the target task), but it still is too strong to hold in general.

We also believe that a more complex translator with enough expensiveness could map an embedding from one model to another (with limitation of course as weaker models may not have the same information as stronger models). However, this out of scope for this paper, and won't confirm that the representation of model share the same geometry (such preserved inner product, etc.).

W3. Practical advantage over re-encoding or retraining is unclear

The framing does not match the goal. The paper does not aim to beat a tailored head; that would contradict universality and reuse. A head trained on source embeddings is tailored to the source and needs no translation; it is a cost reference. The relevant comparison is a target-trained head on translated source versus native target. We will add the native baseline; Linear (SGD) is 0.14-0.23 below it. Translation is lossy; its advantage is avoiding retraining and target-model access, not accuracy parity.

RR is an existing method we evaluate, not one we propose. We include it because it is one of the methods proposed in the universality literature. Its advantage is reuse: RR trains downstream tasks in a shared relative space, so one relative-space head can serve many models without retraining per model. The constraint is that the head must be trained in relative space; RR cannot reuse a native head, and if tasks are trained on native embeddings, it requires retraining in relative space. That constraint belongs to RR, not to this study; RR is one of four translators, and the same failure appears for pinv, IRP, and SGD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.