- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows how fitting a shared representation space across N models allows for the evaluation of alignment consistency at the level of individual samples. Protocol :
- (step 1) Learn this space by fitting orthogonal maps with a GPA-style barycenter iteration factoring out orthogonal transforms.
- (step 2) Projecting samples onto this shared space enables them to score the consistency of their alignment across the population of models.
They apply this protocol to gain insights into vision, language and brain data. They make four claims :
- (a) instance-level consistency score (call it ) is robust to model pool
- (b) correlates with the sample’s difficulty (e.g. prototypicality).
- (c) cross modal alignment yields retrieval comparable to CLIP, and correlates with human judgement
- (d) on human data, is robust across comparable cortical regions, not so much for distant regions. This shows the score reflects genuine representational structure rather than purely stimulus driven effects.
Depending on how the authors address my major concerns, I might lean towards a higher score.
Note : I am not able to provide detailed review of the biological data part, which I mostly leave to other reviewers.
Critical strengths
S1 : The particular choices in step 1 and 2 instantiating the general protocol are elegant and well motivated.
S2 : The instance-level consistency score (call it ) is validated for its robustness to various choices and baselines.
S3 : is shown to correlate with interesting aspects of both data (claims (b) and (c)) and representation structure (claim (d)).
S4 : The appendices report negative or unflattering results.
Critical weaknesses
Major :
W1 : Novelty overclaiming. The authors are very careful in stating that the choices in step 1 (quotienting by orthogonal maps using GPA) and step 2 (average cosim) are not novel. However, it seems to claim that the overall protocol is, by focusing the introduction on methods that are different. Different instances of the same protocol already exist, and the paper cites some. The novelty lies not in the protocol but in its applications. This is a genuine contribution but has to be clarified to avoid giving the impression of overclaiming.
W2 : Cross-modal results are under-controlled. People have been learning CLIP-style encoders by aligning unimodal models since at least 2022. “These findings reveal that unimodal vision and language models already encode, implicitly and independently, much of the structure needed for cross-modal matching” is therefore not a contribution of this work. The figures and tables in the main text have to compare against baselines. They are currently only discussed in the appendix, but they have to be referenced in the main text. Authors should reformulate this claim as “matching” (or approaching, or outperforming, depending on the exact result) known alignment performances while revealing something new (i.e. that correlates with human judgement).
W3 : Methodological inconsistency. The cross-modal result seems to rely on an extra alignment step defined in appendix A.4. (l1080-1083). This extra step goes against the whole motivation of a “shared quotient space”, quotienting by orthogonal maps. This is done without any reference to appendix A.4. anywhere in the main body, and while still claiming that the resulting transforms give “a shared quotient space” (l354). As shallowly presented in the appendix, it sounds like the GPA map should be close to a no-op after ridge pre-maps. This silently breaks the “geometry preserving, post hoc” story inherent to the paper. Section 2.1 motivates orthogonal maps because they are isometries that preserve geometry within each model. A ridge regression is a general linear map that can stretch, shrink and project the geometry. Section 3.3 then interprets the result as “shallow, geometry-preserving” alignment and concludes that unimodal models already share structure. That interpretation depends on the pre-map being benign, which isn’t shown. Authors should at least report (1) pre-alignment alone, (2) GPA alignment alone and (3) both together.
W4 : Result inconsistency. Figure 4 displays cross-modal retrieval of in top-1. Figure 8 is much lower (). Without explanation this inconsistency is too big to be ignored and severely harms the trust in the rest of the results.
W5 : Overfitting. Take the case of language pools : models have dimensionalities in the thousands, and the appendix reports 1200 training sentences. There, orthogonal procrustes can nearly interpolate. Table 1 even suggests that this happens, with huge mismatches between train residuals (near perfect) and test residuals (very high).
Minor :
- “tracks stimulus difficulty” should be reframed, e.g. as “tracks model-defined difficulty”, since the score correlates only with other scores based on the same models, and not with humans (figure 17).
- reference “Figures 16-??” (l1483) is broken
- table 8 : the RMS of PA, presented as a (in this case) lower bound over PF and PB is higher than the second worst.
- Using the chance level at 0.01 in table 2 one can infer that the train-test split is 900-100, but this should be stated to avoid unnecessary mental load.
- The text uses the term hyperalignment without reference to haxby or explanation of the term.
- The introduction claims that the method is independent of the stimulus set … at inference. - This is a pretty important point that appears to be swept under the rug. Given the appendix showing the strong dependence on the training stimulus set, I’d like the authors to emphasise this more.
Questions for authors
Q1 : Can the authors make the presentation of the novel contribution clearer ?
Q2 : Can the authors give baseline comparisons for their cross-modal alignment result ?
Q3 : Can the authors explain the ridge pre-mapping, justify it, and report results (1) with ride pre-maps alone, (2) GPA maps alone and (3) both ?
Q4 : Can the authors explain the severe discrepancy between figure 4 and 8 ?
Q5 : Can the authors address overfitting concerns and provide insights into statistical significance of the alignment under small training sets for the ?
5 comments
Response to Reviewer uepQ (2/3)
3. Comparison with baselines.
The ridge denoising step is used only in Figure 4 (main results) and Figures 34–36 (compare across model pools). In the baseline comparisons (Figures 8–9: barycenter, pairwise Procrustes, RelRep, ASIF), as well as in the depth (Figures 25–26) and training-size (Figures 28–29) analyses, no method receives the ridge denoising step: our barycentric alignment and all baselines are run on the same raw embeddings, with the same train/test splits and depths, and differ only in their alignment method. The baseline results therefore do not depend on the ridge stage in any way, and the contribution of that stage is isolated separately by the ablation experiment above.
In the new revision, we now state this in Appendix A.4.3, in the captions of Figures 8–9, and 28–29, in the “baseline processing” paragraph of Appendix A.4, and, as Reviewer uepQ requests, Figures 8–9 are referenced from Section 3.3.
W1, Q1, Minor Question 5: Contribution
Our contribution is to use the classical generalized procrustes analysis (GPA) to measure representational convergence across an entire population of systems for a single held-out input, not the introduction of the algorithm itself. Based on the reviewer’s suggestion, we have added changes to our introduction to highlight this contribution (lines 90-122, pages 1-2). To our knowledge, this is the first systematic use of Procrustes barycenter to identify which inputs elicit convergence or divergence across pools of vision models, language models and visual cortical recordings and to relate that variation to properties of the inputs themselves.
Hyperalignment established that a shared functional space can support decoding across participants (Haxby et al., 2011). We use that same principle to ask a different question: for which stimuli do the systems agree most closely once aligned? We have also now explicitly described how our method relates to existing methods in the introduction (page 2, lines 72-96) like Relative Representations (Moschella et al., 2023), ASIF (Norelli et al. (2023)), and independently fitted pairwise Procrustes alignment (Maiorca et al.,2023). Although some of these approaches can compare instance level similarity, none provides an explicit shared coordinate space with fixed per-model transforms that can be applied to held-out stimuli. We have clarified these distinctions in the revised paper.
W2, Q2:
we have also included comparison of the barycentric alignment algorithm to several existing baselines: anchor based approaches (Moschella et al., 2023), ASIF Norelli et al. (2023) and independently optimized pairwise Procrustes alignment (Maiorca et al.,2023) in Appendix section A.5.1. We have also referenced this in the main text (page 8, lines 257-261 and 398-402). Further, we have modified our wording in section 3.3.1 to reflect that the cross modal alignment reiterates well known results, and that cross modal alignment obtained via the barycentric alignment approach beats other baselines, except pairwise procrustes alignment which gives an upper limit on the best possible performance (lines 398-402).
W4, Q4: Methodology
We apologize for the confusion, and for the lack of details.
In Figure 4, the evaluation is performed only on 200 image-caption pairs, whereas in Appendix Figure 8, the evaluation is performed on 1000 image-caption pairs. Since retrieval is performed against the full set of candidate images or captions, the larger evaluation set introduces more competing candidates, making it less likely for the correct match to rank among the top retrieved results. Thus, the higher retrieval performance in Figure 4 is expected and should be interpreted in the context of the different evaluation set sizes, rather than as a discrepancy between the two results.
We have now further clarified the implementation details about all the experiments in Appendix section A.4 (changed highlighted in blue).
Response to Reviewer uepQ (3/3)
W5,Q5: Sensitivity of the language-pool alignment to training-set size
We thank the reviewer for this point. With 1,200 training sentences and embedding dimensions in the thousands, an orthogonal map can nearly interpolate the training set, so the training residual in Table 1 is indeed not informative enough on its own. To test whether the small training set affects our conclusion, we re-learned the alignment on nested subsets of an independent corpus, the Brown corpus [1] (which like the Tuckute set spans a wide range of genres; 300 to 15,188 sentences, de-duplicated against the Tuckute set), and evaluated every run on the same 300 held-out Tuckute sentences used in the paper. The new results are now included in the revision (new Appendix 5.10, Figure 43–47), and we also point to the new analysis from Section 3.2.2.
The key result is that the instance-level consistency scores, on which the analyses in Section 3.2 rest, are stable with respect to the training set: the per-sentence ranking changes little with training-set size (Spearman ρ with the reported scores 0.68-0.74 at every size from 300 to 15,188 sentences; Figure 46), and the sentence-property profile of Figure 3D is reproduced whether the alignment is learned on 1,200 or 15,188 sentences (correlation with the reported partial correlations 0.86 and 0.80; Figure 47), although no Tuckute sentence was used to learn it. In other words, the consistency scores are not an artifact of overfitting to the small training set and are largely insensitive to the sentences used to fit the alignment. At the pool level, held-out performance saturates after a few thousand training sentences and converges to the Table 1 values, and the train/held-out residual gap closes as the training set grows (Figure 43–45; details in Appendix 5.10).
Minor inconsistencies:
-
“tracks stimulus difficulty” should be reframed, e.g. as “tracks model-defined difficulty”, since the score correlates only with other scores based on the same models, and not with humans (figure 17). We have now made this change to the header of Section 3.2.
-
reference “Figures 16-??” (l1483) is broken. We have now fixed this.
-
table 8 : the RMS of PA, presented as a (in this case) lower bound over PF and PB is higher than the second worst. PA or Procrustes alignment aligns each pair of models in the pool using an optimal orthogonal Procrustes transformation, providing an upper bound (not lower) on alignment performance under pairwise rotational transformations and serving as a reference for the best achievable performance. Note that PA results for a model group, is the averaged result for every pair of models in the pool. In Appendix table 8, PA does have the highest possible performance amongst PA, PB and PF.
-
Using the chance level at 0.01 in table 2 one can infer that the train-test split is 900-100, but this should be stated to avoid unnecessary mental load. We have now added the train test split in the main text (line 473-474, page 9).
-
The text uses the term hyperalignment without reference to haxby or explanation of the term. Thank you for pointing this out. We have connected our work to Haxby et al (2011) who used GPA (termed hyperalignment) to align cortical response patterns across subjects, enabling cross-subject decoding (page 2, line 95-99). We use that same principle to ask a different question: which inputs elicit the most convergent or divergent representations across different systems? To keep the terminology consistent, we have used GPA throughout in the revised manuscript instead of hyperalignment.
-
The introduction claims that the method is independent of the stimulus set … at inference. - This is a pretty important point that appears to be swept under the rug. Given the appendix showing the strong dependence on the training stimulus set, I’d like the authors to emphasise this more. We want to clarify that the instance-wise measure for a particular stimulus during evaluation is independent of other stimuli present in the inference set, and we have clarified this in the introduction. We agree that an input’s consistency score depends on the training distribution used to fit the barycentric alignment, as our category-exclusion analyses show. We have clarified in the Discussion (page 10, lines 522-526) and methods (lines 197-199, pages 4) that this interpretation is conditional on the fitted alignment (even though each input can be scored independently of other test inputs).
[1]: W. N. Francis and H. Kučera. Brown Corpus Manual: Manual of Information to Accompany a Standard Corpus of Present-Day Edited American English, for Use with Digital Computers. Brown University, Providence, RI, 1964; revised 1979. Accessed via NLTK (S. Bird, E. Klein and E. Loper, Natural Language Processing with Python, O'Reilly, 2009).
Response to Reviewer uepQ (1/3)
We thank the reviewer for their thoughtful comments and feedback, as well as for their encouraging assessment of our work. We have uploaded the revised manuscript PDF, with all changes highlighted in blue, and included the anonymised source code as a ZIP file in the supplementary material.
W3, Q3: Clarification of the cross-modal ridge regression pre-processing step.
1. What the step is.
On rereading the description in Appendix A.4 of the submission, we realized that the term ‘shared item-level target space’ was misleading and apologize for the confusion. We fit the ridge regression entirely within each modality as a denoising step to address the many-to-many relation between images and captions. On the language side, our goal was to capture what a caption says about a scene while reducing sensitivity to how it is phrased. For example, ‘A kid is riding a bicycle’ and ‘A child is pedaling a bike’ can describe the same visual event equally well. MS-COCO gives us five independently written captions for each image. We take one caption’s embedding and train a ridge regression to predict the same language model’s average embedding of the other four captions. We treat the average across different descriptions as the ‘meaning’ representation for the language model. At test time, the fitted map estimates this from a single caption.
We also apply the same idea to vision. A single caption can describe many visually different images, so we generate four images per caption. From one image embedding, the ridge map predicts the vision model’s average embedding of other renderings of that description. This pre-mapping step was inspired by He et al. (2025) who found that averaging the embeddings of several captions describing the same image improved alignment between vision and language models. They observed a similar benefit when they averaged the embeddings of different images generated from a single caption. Importantly, in our setup, in both modalities, the input and target of the regression come from the same model. We bring vision and language into shared coordinates only in the subsequent Procrustes step, when we fit orthogonal transforms to matched training images and captions (pipeline see Appendix Figure 38). We have corrected this description in the revised manuscript (revised Section 3.3 and Appendix A.4.3)
2. Contribution of orthogonal alignment vs. within-modality ridge denoising: ablation results (new Appendix A.5.9, Appendix Figures 39–42).
We conducted an additional ablation under five conditions: (i) orthogonal alignment with no ridge denoising; (ii) denoising of captions only, (iii) of images only, or (iv) of both, followed by alignment; and (v) denoising of both with no orthogonal alignment. Top-1 results are below (top-5/10 in Figures 39–40; "1,000 / 800" = ridge denoising map fit on all 1,000 items or on the 800 training items only).
We can draw three takeaways based on this. First, orthogonal alignment alone recovers substantial cross-modal structure. On raw embeddings, barycentric alignment reaches 31% text→image and 23% image→text top-1 retrieval accuracy (chance 0.5%). Denoising alone, without orthogonal alignment, yields none (0.5%-1% top-1), as expected.
Second, within-modality denoising improves cross-modal retrieval: denoising captions improves image→text retrieval (23.0→45.0%), denoising images improves text→image retrieval (31.0→40.5%), and both together improve both directions (54.0% text→image, 41.5% image→text).
Third, denoising adds little to human judgement agreement (τ = 0.51 without denoising and 0.53-0.54 with it, CLIP: 0.57; new Figure 41), which is obtained almost entirely by orthogonal alignment of the raw embeddings.
Together, these show that the cross-modal correspondence we report is produced by the orthogonal alignment and not by the ridge denoising step. Ridge denoising raises retrieval toward CLIP, but it does not meaningfully improve agreement with human judgments. Because the maps are within-modality, Figure 4 fits them on all 1,000 items; we also report the fit on the 800 training items only, which leaves the no-denoising and Flickr8k results unchanged and gives 38.5 / 38.0% top-1 with both modalities denoised.