acceptodds
Under review as a conference paper at ICLR 2027

Barycentric Alignment for Studying Variation in Representational Convergence Across Inputs

Abstract

Comparing representations across neural networks is challenging because representations admit symmetries, such as arbitrary reordering of units or rotations of activation space, that obscure underlying equivalence between models. We introduce a barycentric alignment framework based on the Generalized Procrustes Algorithm that quotients out these nuisance symmetries to construct a universal embedding space across many models. Unlike existing similarity measures, which summarize relationships over entire stimulus sets, this framework enables similarity to be defined at the level of individual stimuli, revealing inputs that elicit convergent versus divergent representations across models. Using this instance-level notion of similarity, we identify systematic input properties that predict representational convergence versus divergence across vision and language model families. We also construct universal embedding spaces for brain representations across individuals and cortical regions, enabling instance-level comparison of representational agreement across stages of the human visual hierarchy. Finally, we apply the same barycentric alignment framework to purely unimodal vision and language models and find that post-hoc alignment into a shared space yields image–text similarity scores that closely track human cross-modal judgments and approach the performance of contrastively trained vision-language models. This strikingly suggests that independently learned representations already share sufficient geometric structure for human-aligned cross-modal comparison. Together, these results show that resolving representational similarity at the level of individual stimuli reveals phenomena that cannot be detected by set-level comparison metrics.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Representational Alignment, cross modal alignment, neuro AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_uepQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows how fitting a shared representation space across N models allows for the evaluation of alignment consistency at the level of individual samples. Protocol :

  • (step 1) Learn this space by fitting orthogonal maps TiT_i with a GPA-style barycenter iteration factoring out orthogonal transforms.
  • (step 2) Projecting samples onto this shared space enables them to score the consistency of their alignment across the population of models.

They apply this protocol to gain insights into vision, language and brain data. They make four claims :

  • (a) instance-level consistency score (call it ss) is robust to model pool
  • (b) ss correlates with the sample’s difficulty (e.g. prototypicality).
  • (c) cross modal alignment yields retrieval comparable to CLIP, and ss correlates with human judgement
  • (d) on human data, ss is robust across comparable cortical regions, not so much for distant regions. This shows the score ss reflects genuine representational structure rather than purely stimulus driven effects.

Depending on how the authors address my major concerns, I might lean towards a higher score.

Note : I am not able to provide detailed review of the biological data part, which I mostly leave to other reviewers.

Critical strengths

S1 : The particular choices in step 1 and 2 instantiating the general protocol are elegant and well motivated.

S2 : The instance-level consistency score (call it ss) is validated for its robustness to various choices and baselines.

S3 : ss is shown to correlate with interesting aspects of both data (claims (b) and (c)) and representation structure (claim (d)).

S4 : The appendices report negative or unflattering results.

Critical weaknesses

Major :

W1 : Novelty overclaiming. The authors are very careful in stating that the choices in step 1 (quotienting by orthogonal maps using GPA) and step 2 (average cosim) are not novel. However, it seems to claim that the overall protocol is, by focusing the introduction on methods that are different. Different instances of the same protocol already exist, and the paper cites some. The novelty lies not in the protocol but in its applications. This is a genuine contribution but has to be clarified to avoid giving the impression of overclaiming.

W2 : Cross-modal results are under-controlled. People have been learning CLIP-style encoders by aligning unimodal models since at least 2022. “These findings reveal that unimodal vision and language models already encode, implicitly and independently, much of the structure needed for cross-modal matching” is therefore not a contribution of this work. The figures and tables in the main text have to compare against baselines. They are currently only discussed in the appendix, but they have to be referenced in the main text. Authors should reformulate this claim as “matching” (or approaching, or outperforming, depending on the exact result) known alignment performances while revealing something new (i.e. that ss correlates with human judgement).

W3 : Methodological inconsistency. The cross-modal result seems to rely on an extra alignment step defined in appendix A.4. (l1080-1083). This extra step goes against the whole motivation of a “shared quotient space”, quotienting by orthogonal maps. This is done without any reference to appendix A.4. anywhere in the main body, and while still claiming that the resulting transforms give “a shared quotient space” (l354). As shallowly presented in the appendix, it sounds like the GPA map should be close to a no-op after ridge pre-maps. This silently breaks the “geometry preserving, post hoc” story inherent to the paper. Section 2.1 motivates orthogonal maps because they are isometries that preserve geometry within each model. A ridge regression is a general linear map that can stretch, shrink and project the geometry. Section 3.3 then interprets the result as “shallow, geometry-preserving” alignment and concludes that unimodal models already share structure. That interpretation depends on the pre-map being benign, which isn’t shown. Authors should at least report (1) pre-alignment alone, (2) GPA alignment alone and (3) both together.

W4 : Result inconsistency. Figure 4 displays cross-modal retrieval of ≈0.5\approx 0.5 in top-1. Figure 8 is much lower (≈0.1\approx 0.1). Without explanation this inconsistency is too big to be ignored and severely harms the trust in the rest of the results.

W5 : Overfitting. Take the case of language pools : models have dimensionalities in the thousands, and the appendix reports 1200 training sentences. There, orthogonal procrustes can nearly interpolate. Table 1 even suggests that this happens, with huge mismatches between train residuals (near perfect) and test residuals (very high).

Minor :

  • “tracks stimulus difficulty” should be reframed, e.g. as “tracks model-defined difficulty”, since the score correlates only with other scores based on the same models, and not with humans (figure 17).
  • reference “Figures 16-??” (l1483) is broken
  • table 8 : the RMS of PA, presented as a (in this case) lower bound over PF and PB is ≈16×\approx 16 \times higher than the second worst.
  • Using the chance level at 0.01 in table 2 one can infer that the train-test split is 900-100, but this should be stated to avoid unnecessary mental load.
  • The text uses the term hyperalignment without reference to haxby or explanation of the term.
  • The introduction claims that the method is independent of the stimulus set … at inference. - This is a pretty important point that appears to be swept under the rug. Given the appendix showing the strong dependence on the training stimulus set, I’d like the authors to emphasise this more.
Questions for authors

Q1 : Can the authors make the presentation of the novel contribution clearer ?

Q2 : Can the authors give baseline comparisons for their cross-modal alignment result ?

Q3 : Can the authors explain the ridge pre-mapping, justify it, and report results (1) with ride pre-maps alone, (2) GPA maps alone and (3) both ?

Q4 : Can the authors explain the severe discrepancy between figure 4 and 8 ?

Q5 : Can the authors address overfitting concerns and provide insights into statistical significance of the alignment under small training sets for the TiT_i ?

5 comments
AuthorsOpenReview ↗

Response to Reviewer uepQ (1/3)

We thank the reviewer for their thoughtful comments and feedback, as well as for their encouraging assessment of our work. We have uploaded the revised manuscript PDF, with all changes highlighted in blue, and included the anonymised source code as a ZIP file in the supplementary material.


W3, Q3: Clarification of the cross-modal ridge regression pre-processing step.

1. What the step is.

On rereading the description in Appendix A.4 of the submission, we realized that the term ‘shared item-level target space’ was misleading and apologize for the confusion. We fit the ridge regression entirely within each modality as a denoising step to address the many-to-many relation between images and captions. On the language side, our goal was to capture what a caption says about a scene while reducing sensitivity to how it is phrased. For example, ‘A kid is riding a bicycle’ and ‘A child is pedaling a bike’ can describe the same visual event equally well. MS-COCO gives us five independently written captions for each image. We take one caption’s embedding and train a ridge regression to predict the same language model’s average embedding of the other four captions. We treat the average across different descriptions as the ‘meaning’ representation for the language model. At test time, the fitted map estimates this from a single caption.

We also apply the same idea to vision. A single caption can describe many visually different images, so we generate four images per caption. From one image embedding, the ridge map predicts the vision model’s average embedding of other renderings of that description. This pre-mapping step was inspired by He et al. (2025) who found that averaging the embeddings of several captions describing the same image improved alignment between vision and language models. They observed a similar benefit when they averaged the embeddings of different images generated from a single caption. Importantly, in our setup, in both modalities, the input and target of the regression come from the same model. We bring vision and language into shared coordinates only in the subsequent Procrustes step, when we fit orthogonal transforms to matched training images and captions (pipeline see Appendix Figure 38). We have corrected this description in the revised manuscript (revised Section 3.3 and Appendix A.4.3)

2. Contribution of orthogonal alignment vs. within-modality ridge denoising: ablation results (new Appendix A.5.9, Appendix Figures 39–42).

We conducted an additional ablation under five conditions: (i) orthogonal alignment with no ridge denoising; (ii) denoising of captions only, (iii) of images only, or (iv) of both, followed by alignment; and (v) denoising of both with no orthogonal alignment. Top-1 results are below (top-5/10 in Figures 39–40; "1,000 / 800" = ridge denoising map fit on all 1,000 items or on the 800 training items only).

Within-modality ridge denoisingOrthogonal alignmentText→Image top-1 (%, chance 0.5) 1,000 / 800Image→Text top-1 (%, chance 0.5) 1,000 / 800Flickr8k τ 1,000 / 800
NoneYes31.023.00.51
Captions onlyYes30.0 / 28.045.0 / 40.50.53 / 0.53
Images onlyYes40.5 / 36.522.5 / 21.50.51 / 0.51
Captions + imagesYes54.0 / 38.541.5 / 38.00.54 / 0.54
Captions + imagesNo0.5 / 1.00.5 / 0.50.02 / 0.01
CLIP ViT-B/32 (same test pairs)—52.558.00.57

We can draw three takeaways based on this. First, orthogonal alignment alone recovers substantial cross-modal structure. On raw embeddings, barycentric alignment reaches 31% text→image and 23% image→text top-1 retrieval accuracy (chance 0.5%). Denoising alone, without orthogonal alignment, yields none (0.5%-1% top-1), as expected.

Second, within-modality denoising improves cross-modal retrieval: denoising captions improves image→text retrieval (23.0→45.0%), denoising images improves text→image retrieval (31.0→40.5%), and both together improve both directions (54.0% text→image, 41.5% image→text).

Third, denoising adds little to human judgement agreement (τ = 0.51 without denoising and 0.53-0.54 with it, CLIP: 0.57; new Figure 41), which is obtained almost entirely by orthogonal alignment of the raw embeddings.

Together, these show that the cross-modal correspondence we report is produced by the orthogonal alignment and not by the ridge denoising step. Ridge denoising raises retrieval toward CLIP, but it does not meaningfully improve agreement with human judgments. Because the maps are within-modality, Figure 4 fits them on all 1,000 items; we also report the fit on the 800 training items only, which leaves the no-denoising and Flickr8k results unchanged and gives 38.5 / 38.0% top-1 with both modalities denoised.

AuthorsOpenReview ↗

Response to Reviewer uepQ (2/3)

3. Comparison with baselines.

The ridge denoising step is used only in Figure 4 (main results) and Figures 34–36 (compare across model pools). In the baseline comparisons (Figures 8–9: barycenter, pairwise Procrustes, RelRep, ASIF), as well as in the depth (Figures 25–26) and training-size (Figures 28–29) analyses, no method receives the ridge denoising step: our barycentric alignment and all baselines are run on the same raw embeddings, with the same train/test splits and depths, and differ only in their alignment method. The baseline results therefore do not depend on the ridge stage in any way, and the contribution of that stage is isolated separately by the ablation experiment above.

In the new revision, we now state this in Appendix A.4.3, in the captions of Figures 8–9, and 28–29, in the “baseline processing” paragraph of Appendix A.4, and, as Reviewer uepQ requests, Figures 8–9 are referenced from Section 3.3.


W1, Q1, Minor Question 5: Contribution

Our contribution is to use the classical generalized procrustes analysis (GPA) to measure representational convergence across an entire population of systems for a single held-out input, not the introduction of the algorithm itself. Based on the reviewer’s suggestion, we have added changes to our introduction to highlight this contribution (lines 90-122, pages 1-2). To our knowledge, this is the first systematic use of Procrustes barycenter to identify which inputs elicit convergence or divergence across pools of vision models, language models and visual cortical recordings and to relate that variation to properties of the inputs themselves.

Hyperalignment established that a shared functional space can support decoding across participants (Haxby et al., 2011). We use that same principle to ask a different question: for which stimuli do the systems agree most closely once aligned? We have also now explicitly described how our method relates to existing methods in the introduction (page 2, lines 72-96) like Relative Representations (Moschella et al., 2023), ASIF (Norelli et al. (2023)), and independently fitted pairwise Procrustes alignment (Maiorca et al.,2023). Although some of these approaches can compare instance level similarity, none provides an explicit shared coordinate space with fixed per-model transforms that can be applied to held-out stimuli. We have clarified these distinctions in the revised paper.


W2, Q2:

we have also included comparison of the barycentric alignment algorithm to several existing baselines: anchor based approaches (Moschella et al., 2023), ASIF Norelli et al. (2023) and independently optimized pairwise Procrustes alignment (Maiorca et al.,2023) in Appendix section A.5.1. We have also referenced this in the main text (page 8, lines 257-261 and 398-402). Further, we have modified our wording in section 3.3.1 to reflect that the cross modal alignment reiterates well known results, and that cross modal alignment obtained via the barycentric alignment approach beats other baselines, except pairwise procrustes alignment which gives an upper limit on the best possible performance (lines 398-402).


W4, Q4: Methodology

We apologize for the confusion, and for the lack of details.

In Figure 4, the evaluation is performed only on 200 image-caption pairs, whereas in Appendix Figure 8, the evaluation is performed on 1000 image-caption pairs. Since retrieval is performed against the full set of candidate images or captions, the larger evaluation set introduces more competing candidates, making it less likely for the correct match to rank among the top retrieved results. Thus, the higher retrieval performance in Figure 4 is expected and should be interpreted in the context of the different evaluation set sizes, rather than as a discrepancy between the two results.

We have now further clarified the implementation details about all the experiments in Appendix section A.4 (changed highlighted in blue).

AuthorsOpenReview ↗

Response to Reviewer uepQ (3/3)


W5,Q5: Sensitivity of the language-pool alignment to training-set size

We thank the reviewer for this point. With 1,200 training sentences and embedding dimensions in the thousands, an orthogonal map can nearly interpolate the training set, so the training residual in Table 1 is indeed not informative enough on its own. To test whether the small training set affects our conclusion, we re-learned the alignment on nested subsets of an independent corpus, the Brown corpus [1] (which like the Tuckute set spans a wide range of genres; 300 to 15,188 sentences, de-duplicated against the Tuckute set), and evaluated every run on the same 300 held-out Tuckute sentences used in the paper. The new results are now included in the revision (new Appendix 5.10, Figure 43–47), and we also point to the new analysis from Section 3.2.2.

The key result is that the instance-level consistency scores, on which the analyses in Section 3.2 rest, are stable with respect to the training set: the per-sentence ranking changes little with training-set size (Spearman ρ with the reported scores 0.68-0.74 at every size from 300 to 15,188 sentences; Figure 46), and the sentence-property profile of Figure 3D is reproduced whether the alignment is learned on 1,200 or 15,188 sentences (correlation with the reported partial correlations 0.86 and 0.80; Figure 47), although no Tuckute sentence was used to learn it. In other words, the consistency scores are not an artifact of overfitting to the small training set and are largely insensitive to the sentences used to fit the alignment. At the pool level, held-out performance saturates after a few thousand training sentences and converges to the Table 1 values, and the train/held-out residual gap closes as the training set grows (Figure 43–45; details in Appendix 5.10).


Minor inconsistencies:

  • “tracks stimulus difficulty” should be reframed, e.g. as “tracks model-defined difficulty”, since the score correlates only with other scores based on the same models, and not with humans (figure 17). We have now made this change to the header of Section 3.2.

  • reference “Figures 16-??” (l1483) is broken. We have now fixed this.

  • table 8 : the RMS of PA, presented as a (in this case) lower bound over PF and PB is higher than the second worst. PA or Procrustes alignment aligns each pair of models in the pool using an optimal orthogonal Procrustes transformation, providing an upper bound (not lower) on alignment performance under pairwise rotational transformations and serving as a reference for the best achievable performance. Note that PA results for a model group, is the averaged result for every pair of models in the pool. In Appendix table 8, PA does have the highest possible performance amongst PA, PB and PF.

  • Using the chance level at 0.01 in table 2 one can infer that the train-test split is 900-100, but this should be stated to avoid unnecessary mental load. We have now added the train test split in the main text (line 473-474, page 9).

  • The text uses the term hyperalignment without reference to haxby or explanation of the term. Thank you for pointing this out. We have connected our work to Haxby et al (2011) who used GPA (termed hyperalignment) to align cortical response patterns across subjects, enabling cross-subject decoding (page 2, line 95-99). We use that same principle to ask a different question: which inputs elicit the most convergent or divergent representations across different systems? To keep the terminology consistent, we have used GPA throughout in the revised manuscript instead of hyperalignment.

  • The introduction claims that the method is independent of the stimulus set … at inference. - This is a pretty important point that appears to be swept under the rug. Given the appendix showing the strong dependence on the training stimulus set, I’d like the authors to emphasise this more. We want to clarify that the instance-wise measure for a particular stimulus during evaluation is independent of other stimuli present in the inference set, and we have clarified this in the introduction. We agree that an input’s consistency score depends on the training distribution used to fit the barycentric alignment, as our category-exclusion analyses show. We have clarified in the Discussion (page 10, lines 522-526) and methods (lines 197-199, pages 4) that this interpretation is conditional on the fitted alignment (even though each input can be scored independently of other test inputs).


[1]: W. N. Francis and H. Kučera. Brown Corpus Manual: Manual of Information to Accompany a Standard Corpus of Present-Day Edited American English, for Use with Digital Computers. Brown University, Providence, RI, 1964; revised 1979. Accessed via NLTK (S. Bird, E. Klein and E. Loper, Natural Language Processing with Python, O'Reilly, 2009).

Reviewer_ER3DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed method aligns latent representations across models using the classical Procrustes algorithm from factor analysis. That is, it learns an orthogonal rotation for each model that maps its representations into a shared space in a way that minimizes RMS error (Frobenius norm). Application to vision and language models and human neural data reveals an array of results on which model families are more or less alignable and which individual stimuli are more or less consistently represented.

Critical strengths

Applicable to individual (out of sample) stimuli. This opens up several new lines of investigation, including factors that determine consistency of an example's representation, and direct decoding from one model to another.

Informative and sensible pattern of empirical results. The agreement between unimodal models (vision and language) is especially intriguing.

Critical weaknesses

Requires a global parametric fit, unlike RSA-type methods that are tolerant of distortions at the scale of the whole stimulus space that don’t affect local structure

The choice of orthogonal maps (isometric embeddings) might be too limited (most models apply general linear transformations at the output, so isometric information may not be meaningful) and at the same time too flexible (it loses the coordinate basis, which single-unit nonlinearities act on).

Questions for authors

Is the matrix from each model standardized (i.e., sphered or whitened) prior to analysis? This seems important to do, since dynamic range could vary across models and orthogonal maps cannot eliminate this.

Why use cosine similarity at test when the training objective is Euclidean distance?

Is the correlation between consistency and prototypicality (fig 3c) an artifact of the restriction to linear embeddings? If there are nonlinear distortions between models at a large scale (i.e., with low spatial frequency in stimulus space), linear maps will align the centroids at the cost of the periphery.

Likewise with DDD. This is a more compelling result, but a stimulus that is far from everything else will have low DDD and will be more susceptible to nonlinearities as above. What happens if you control for stimulus centrality?

The sentence at 369, especially “while remaining close”, suggests the universal space underperforms CLIP at top-5 and top-10 (and that the percentages given in the text list CLIP first, as in the next sentence), but fig 4b shows the reverse. Please check or adjust the wording.

6 comments
AuthorsOpenReview ↗

Response to Reviewer ER3D (1/2)

We thank the reviewer for their thoughtful comments and feedback, as well as for their encouraging assessment of our work. Changes in the current pdf are highlighted in blue.

W1: Global parametric fit vs. RSA-style tolerance to large-scale distortions.

We agree this is a genuine trade-off. The global map is what provides (a) one explicit coordinate system for N models simultaneously, (b) projection of new stimuli without recomputing kernels, and (c) a per-stimulus score that does not depend on which other stimuli are in the evaluation set. Neighborhood- and kernel-based measures (RSA, PNKA, mutual-kNN) gain tolerance to global distortions by defining each stimulus relative to the evaluation set, which is the dependence we aim to remove (Sec. 1).

W2: Orthogonal maps may be too limited and too flexible.

We deliberately use orthogonal alignment because it provides a principled middle ground between two extremes. As discussed by Williams et al. (2021, Generalized Shape Metrics on Neural Representations), the choice of alignment determines which transformations are treated as irrelevant. Permutation alignment preserves the unit basis, whereas orthogonal alignment allows rotations and reflections while preserving the underlying geometry (inner products, angles, and pairwise distances). More general linear mappings relax these constraints further, potentially altering the geometry being compared.

Too restrictive. Although more flexible mappings may capture additional shared structure, Kapoor et al. (NeurIPS 2025) show that orthogonal alignment recovers nearly all the alignment achievable with unconstrained linear maps across several model families. Moreover, unconstrained linear maps are ill-posed in our multi-model barycentric setting, as they can trivially minimize the objective by contracting representations toward the template. Orthogonality prevents this degeneracy while preserving each model's internal geometry.

Too flexible. Orthogonal alignment does discard the original unit basis. However, our model pools span different architectures, widths, and even fMRI populations, where individual units have no natural correspondence. Permutation alignment would therefore impose an assumption that is generally not justified as is also shown in Kapoor et al. (NeurIPS 2025) where permutation matching generally performs much worse than procrustes or linear alignment. .

Importantly, our results do not imply that arbitrary systems must share orthogonally alignable representations. Rather, they quantify shared structure under this specific equivalence.

Q1: Is the matrix from each model standardized?

Apologies for not making this clear in the manuscript. Before the barycentric alignment step, each model’s representation matrix is mean-centered, and each stimulus vector is then divided by its L2 norm. The same transform is applied to held-out test stimuli. This puts all models on a common scale, so differences in dynamic range across models do not affect the alignment. We have clarified this in Section 2.2 (highlighted in blue).

Q2: Why use cosine similarity at test when the training objective is Euclidean distance?

Cosine similarity and Euclidean distance are closely related measures of representational similarity. For L2-normalized representations, they are mathematically equivalent up to a linear transformation, as (|x-y|_2^2 = 2(1-\cos(x,y))), and therefore produce identical pairwise rankings. Even without exact normalization, they remain highly correlated when representation norms are similar. Empirically, we find that both metrics yield nearly identical results, and using either does not change our conclusions. We have now added a scatterplot showing the correlation between universal scores calculated using cosine distance and euclidean distance in Appendix figure 48.

AuthorsOpenReview ↗

Response to Reviewer ER3D (2/2)

Q3: Is the correlation between consistency and prototypicality (fig 3c) an artifact of the restriction to linear embeddings? If there are nonlinear distortions between models at a large scale (i.e., with low spatial frequency in stimulus space), linear maps will align the centroids at the cost of the periphery.

Likewise with DDD. This is a more compelling result, but a stimulus that is far from everything else will have low DDD and will be more susceptible to nonlinearities as above. What happens if you control for stimulus centrality?

Our representation matrices are already normalized, ruling out differences in overall scale as an explanation for the observed relationship. More fundamentally, our objective is to quantify how similarly different systems represent individual stimuli,and we do so via orthogonal alignment which does not alter the internal geometry of individual representation spaces. The reviewer raises the possibility that nonlinear distortions between systems may affect alignment differently across stimuli. We clarify that this is not something our framework seeks to correct: if nonlinear differences prevent certain stimuli from being consistently aligned, this reflects lower representational agreement under the geometric constraints of our framework. Thus, our results characterize stimulus-level agreement under orthogonal equivalence, rather than agreement that could potentially be recovered through arbitrary nonlinear transformations.

Q4: The sentence at 369, especially “while remaining close”, suggests the universal space underperforms CLIP at top-5 and top-10 (and that the percentages given in the text list CLIP first, as in the next sentence), but fig 4b shows the reverse. Please check or adjust the wording.

We have now fixed this (highlighted in blue) in section 3.3.1.

Thanks for the replies. Regarding Q3, I don’t think you’re taking this question on directly. Normalization and scale differences are not relevant; a set of normalized embeddings will still have items at the periphery of the distribution, and their alignment will be more sensitive to errors in the mapping. So the finding as currently presented is unsurprising and does not necessarily indicate anything about agreement between models. As I suggested in my review, this should be easy to fix by controlling for distance to the grand mean. This distance is correlated with but distinct from your predictors of interest (class-specific prototypicality and DDD).

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer wp9m (1/2)

We thank the reviewer for their thoughtful comments and feedback, as well as for their encouraging assessment of our work. We have uploaded the revised manuscript PDF, with all changes highlighted in blue, and included the anonymised source code as a ZIP file in the supplementary material.


W1, Q1, Q2: Clarification of the cross-modal ridge regression pre-processing step.

1. What the step is.

On rereading the description in Appendix A.4 of the submission, we realized that the term ‘shared item-level target space’ was misleading and apologize for the confusion. We fit the ridge regression entirely within each modality as a denoising step to address the many-to-many relation between images and captions. On the language side, our goal was to capture what a caption says about a scene while reducing sensitivity to how it is phrased. For example, ‘A kid is riding a bicycle’ and ‘A child is pedaling a bike’ can describe the same visual event equally well. MS-COCO gives us five independently written captions for each image. We take one caption’s embedding and train a ridge regression to predict the same language model’s average embedding of the other four captions. We treat the average across different descriptions as the ‘meaning’ representation for the language model. At test time, the fitted map estimates this from a single caption.

We also apply the same idea to vision. A single caption can describe many visually different images, so we generate four images per caption. From one image embedding, the ridge map predicts the vision model’s average embedding of other renderings of that description. This pre-mapping step was inspired by He et al. (2025) who found that averaging the embeddings of several captions describing the same image improved alignment between vision and language models. They observed a similar benefit when they averaged the embeddings of different images generated from a single caption. Importantly, in our setup, in both modalities, the input and target of the regression come from the same model. We bring vision and language into shared coordinates only in the subsequent Procrustes step, when we fit orthogonal transforms to matched training images and captions (pipeline see Appendix Figure 38). We have corrected this description in the revised manuscript (revised Section 3.3 and Appendix A.4.3)

2. Contribution of orthogonal alignment vs. within-modality ridge denoising: ablation results (new Appendix A.5.9, Appendix Figures 39–42).

We conducted an additional ablation under five conditions: (i) orthogonal alignment with no ridge denoising; (ii) denoising of captions only, (iii) of images only, or (iv) of both, followed by alignment; and (v) denoising of both with no orthogonal alignment. Top-1 results are below (top-5/10 in Figures 39–40; "1,000 / 800" = ridge denoising map fit on all 1,000 items or on the 800 training items only).

Within-modality ridge denoisingOrthogonal alignmentText→Image top-1 (%, chance 0.5) 1,000 / 800Image→Text top-1 (%, chance 0.5) 1,000 / 800Flickr8k τ 1,000 / 800
NoneYes31.023.00.51
Captions onlyYes30.0 / 28.045.0 / 40.50.53 / 0.53
Images onlyYes40.5 / 36.522.5 / 21.50.51 / 0.51
Captions + imagesYes54.0 / 38.541.5 / 38.00.54 / 0.54
Captions + imagesNo0.5 / 1.00.5 / 0.50.02 / 0.01
CLIP ViT-B/32 (same test pairs)—52.558.00.57

We can draw three takeaways based on this. First, orthogonal alignment alone recovers substantial cross-modal structure. On raw embeddings, barycentric alignment reaches 31% text→image and 23% image→text top-1 retrieval accuracy (chance 0.5%). Denoising alone, without orthogonal alignment, yields none (0.5%-1% top-1), as expected.

Second, within-modality denoising improves cross-modal retrieval: denoising captions improves image→text retrieval (23.0→45.0%), denoising images improves text→image retrieval (31.0→40.5%), and both together improve both directions (54.0% text→image, 41.5% image→text).

Third, denoising adds little to human judgement agreement (τ = 0.51 without denoising and 0.53-0.54 with it, CLIP: 0.57; new Figure 41), which is obtained almost entirely by orthogonal alignment of the raw embeddings.

Together, these show that the cross-modal correspondence we report is produced by the orthogonal alignment and not by the ridge denoising step. Ridge denoising raises retrieval toward CLIP, but it does not meaningfully improve agreement with human judgments. Because the maps are within-modality, Figure 4 fits them on all 1,000 items; we also report the fit on the 800 training items only, which leaves the no-denoising and Flickr8k results unchanged and gives 38.5 / 38.0% top-1 with both modalities denoised.

AuthorsOpenReview ↗

Response to Reviewer wp9m (2/2)

3. Comparison with baselines.

The ridge denoising step is used only in Figure 4 (main results) and Figures 34–36 (compare across model pools). In the baseline comparisons (Figures 8–9: barycenter, pairwise Procrustes, RelRep, ASIF), as well as in the depth (Figures 25–26) and training-size (Figures 28–29) analyses, no method receives the ridge denoising step: our barycentric alignment and all baselines are run on the same raw embeddings, with the same train/test splits and depths, and differ only in their alignment method. The baseline results therefore do not depend on the ridge stage in any way, and the contribution of that stage is isolated separately by the ablation experiment above.

In the new revision, we now state this in Appendix A.4.3, in the captions of Figures 8–9, and 28–29, in the “baseline processing” paragraph of Appendix A.4, and, as Reviewer uepQ requests, Figures 8–9 are referenced from Section 3.3.

We have also included the anonymised source code as a ZIP file in the revised supplementary material so reviewers can check the implementation directly.


Q1: Disclosure and protocols.

We have now included details about training and validation sets and other implementation details in Appendix section A.4 highlighted in blue. We have also ensured that all Appendix sections, figures and tables are referenced in the main text. The two cross-modal protocols the reviewer identifies are now stated in the revised “train/test splits” paragraph of A.4, including data sources and candidate-set sizes. For the same no-denoising condition, the findings are consistent across the two protocols: on Flickr8k-Expert (both evaluate on the same 5,822 pairs), Kendall's τ is 0.50–0.53 with alignment learned on 5k–20k pairs (Figure 29) and 0.51 under the Figure 4 protocol (Figure 41).


W2: Reframing Primary Contributions and baseline comparisons

Our contribution is to use the classical generalized procrustes analysis (GPA) to measure representational convergence across an entire population of systems for a single held-out input, not the introduction of the algorithm itself. Based on the reviewer’s suggestion, we have added changes to our introduction to highlight this contribution (lines 90-122, pages 1-2). To our knowledge, this is the first systematic use of Procrustes barycenter to identify which inputs elicit convergence or divergence across pools of vision models, language models and visual cortical recordings and to relate that variation to properties of the inputs themselves.

Hyperalignment established that a shared functional space can support decoding across participants (Haxby et al., 2011). We use that same principle to ask a different question: for which stimuli do the systems agree most closely once aligned? We have also now explicitly described how our method relates to existing methods in the introduction (page 2, lines 72-96) like Relative Representations (Moschella et al., 2023), ASIF (Norelli et al. (2023)), and independently fitted pairwise Procrustes alignment (Maiorca et al.,2023). Although some of these approaches can compare instance level similarity, none provides an explicit shared coordinate space with fixed per-model transforms that can be applied to held-out stimuli. We have clarified these distinctions in the revised paper. We also include quantitative comparisons of the barycentric alignment algorithm to these baselines in Appendix A.5.1.

Lastly, we agree that an input’s consistency score depends on the training distribution used to fit the barycentric alignment, as our category-exclusion analyses show. The advantage over methods like PNKA lies in how directly our score measures convergence on the input itself. PNKA asks whether two models give a target the same pattern of similarities to the other test inputs (Kolling et al., 2025). Even with a fixed reference set, a low PNKA score leaves open whether the models differ in their representation of the input, the references, or both. Barycentric alignment gives us a more direct test. After using training stimuli to account for rotations and reflections, we ask whether the models place the same held-out input in the same part of the shared space. A high score means its normalized representations cluster tightly across models. This makes the score particularly well suited to our question of which inputs elicit agreement across a population of systems. We have clarified in the Discussion (page 10, lines 522-526) that this interpretation is conditional on the fitted alignment (even though each input can be scored independently of other test inputs). We have also acknowledged that our choice of comparing models at similar representational depths may not necessarily match the processing stages across models optimally (page 10, lines 526-529).

AuthorsOpenReview ↗

Follow-up on our response to Reviewer wp9m

Thank you again for your thoughtful critiques and the thorough, constructive review, which have significantly improved our work. If there are any remaining questions or concerns, we would be happy to clarify them during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.