acceptodds
Withdrawn from ICLR 2027

Alignment-Free Geometry Of Representational Trajectories Across The Brain And Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce an alignment-free Riemannian framework for comparing representational trajectories across systems that share neither coordinates, dimensionality, nor a common axis of evolution. We validate it by analyzing the response of the visual cortex and foundation models on common images. We study EEG and MEG responses to natural images from THINGS-EEG2 and THINGS-MEG, together with a label-fed language model, Llama-3-8B, and an image-fed vision model, DINOv2. For each system, an image is represented by a trajectory of neighborhood covariances on the symmetric positive-definite manifold. Each system's covariance trajectories are mapped to the tangent space at that system's Fr\'echet mean, velocity directions are computed along the system-specific axis, and systems are compared only through the pairwise angular geometry of per-image velocity directions. This statistic is invariant to an independent invertible congruence of each system's PCA-reduced covariances and to per-image rescaling. Across both EEG and MEG, cortical trajectories share positive angular geometry with both Llama and DINOv2. Order-sensitive tests separate two kinds of correspondence: the best-matching depth does not advance steadily in either model, while a rate-free monotone test finds cortical time and model depth ordered against the label-fed model in both cohorts and not against the image-fed one. Category analysis shows the correspondence is structured by graded neighborhood relations rather than reducible to discrete category labels, while scaffold controls reveal a substantial fraction of slice-wise concordance is induced by the shared stimulus-neighborhood graph. Methodologically, the framework separates coordinate-free geometric correspondence from order-preserving dynamical correspondence, providing a general tool to test when two representational systems share structure and trajectory.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Representational geometry, representational dynamics, brain–model alignment, Riemannian geometry, foundation models, neural representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qpfVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds, for each image, a trajectory of neighborhood covariance matrices over time (EEG/MEG) or over layers (Llama-3-8B, DINOv2). It compares systems through the angles between per-image velocity directions, without fitting any map between them. Brain and both models share angle structure, and only Llama shows a rate-free time-to-depth ordering.

Critical strengths
  • The method needs no cross-system mapping and comes with invariance proofs.
  • It is tested on both EEG and MEG and 2 models across modalities (language and vision)
  • The controls are honest. Category partialling and the noise-feature scaffold show how much of the effect comes from the setup.
Critical weaknesses
  • The headline ordering result is weak. The “Llama carries the time-depth order” claim rests on one test in half the subjects, peak depths flip between EEG and MEG, why?
  • It is not shown what the velocity view adds over simple RSA on the states.
Questions for authors
  • The order results are weak and do not replicate across EEG and MEG, can you analyse this further?
  • The added value over simpler methods is unclear. Can you compare directly to RSA on states (same images, same pair subsets) and show results across k, d and window size?
  • The shared CLIP graph builds in part of the match. Can you define neighborhoods from each system’s own responses, or from a different embedding, and report how much of the effect remains?
Reviewer_hRqSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study addresses the problem of comparing representations between two systems. It argues for a coordinate-free and relational comparison (in the style of RSA) that is also explicitly dynamical (in the style of DSA). Concretely, each system's response to an image is represented as a trajectory of covariances (of neighborhoods from CLIP space) on the symmetric positive-definite (SPD) manifold. These are mapped to the tangent space at each system's Fréchet mean, where a velocity direction is computed per image. Pairwise angle matrices of size (# stimuli × # stimuli) are then used to compare systems. One matrix is produced per time point (brain) or layer (model), and the systems are compared via correlation of these across slices. This statistic is provably invariant to image re-scaling and to independent invertible congruence of either system's covariances. Two order-sensitive statistics (ridge and dynamic time warping) test whether brain time and model depth correspond in order.

The method is tested on MEG and EEG responses and on two foundation models (Llama-3-8B, fed object labels, and DINOv2, fed images). Results show a positive angular concordance between visual cortex and both foundation models. Using DTW, an ordered correspondence is found between cortical time and Llama depth but not for DINOv2; the ridge test finds no ordering for either model. Controls indicate that category membership accounts for a minority of the concordance and that the shared CLIP neighborhoods already reproduce part of it.

Novelty: The paper positions itself as complementary to DSA and as a departure from the encoding-model paradigm common in the LLM representation literature, in that it compares representational geometry without any learned alignment.

Critical strengths
  • Creative combination of SPD-manifold geometry and DTW to compare representations dynamically across time or layer depth. It provides a way to check whether two systems successively change their representations in the same order, which standard metrics such as RSA and CKA do not provide.

  • Several controls are used to test whether the concordance can be explained by trivial factors (e.g., category membership, the shared CLIP neighborhood graph, random features).

Critical weaknesses
  • Treatment of the graph-scaffold and random-feature controls. The main text reports the isotropic null for Llama only (35% of the observed concordance in EEG, 24% in MEG). The spectrum-matched null in Table 2 is substantially larger — 41–56% for Llama and 60–71% for DINOv2 across the two cohorts — and is not discussed outside the appendix. More importantly, the authors' own random-features control (App. D, Fig. 7C) shows that a model with no features of its own, routed through the shared CLIP graph, reproduces a large share of the concordance and in fact exceeds the real Llama layers under the rank concordance in every EEG and MEG subject (0.330 vs. 0.235 EEG; 0.286 vs. 0.270 MEG). This is acknowledged in App. D but not in the main text or Discussion, and it raises a concern about how much of the reported brain–Llama geometry is attributable to Llama rather than to the shared pipeline. A related phenomenon has been described in the RSA literature (e.g., Dujmović et al., 2023, on shared stimulus confounds producing high similarity between dissimilar systems); here the shared structure is built into the pipeline itself.

  • Clarity of the main findings. Beyond the now-standard result that brain and deep networks share geometry, the specific findings were hard to follow, even in the abstract. Key sentences rely on terminology that is not defined before use, for example "rate-free monotone test", "label-fed", "scaffold controls", and "slice-wise concordance":

    (i) "the best-matching depth does not advance steadily in either model, while a rate-free monotone test finds cortical time and model depth ordered against the label-fed model in both cohorts and not against the image-fed one";

    (ii) "Category analysis shows the correspondence is structured by graded neighborhood relations rather than reducible to discrete category labels, while scaffold controls reveal a substantial fraction of slice-wise concordance is induced by the shared stimulus-neighborhood graph".

    Section 4 could also be reworked: the results are dense with numbers and provide little interpretation.

Questions for authors
  1. Fig. 1C. What is this panel supposed to represent, two similar or two dissimilar matrices? The mean r is positive, yet the angle patterns in the two matrices appear quite different from the color bar.

  2. Graph-scaffold null in the main results. Table 2 suggests that a large share of the concordance (41–71% under the spectrum-matched null) comes from the shared CLIP graph alone, and App. D shows that random features on the same graph match cortex more strongly than the real Llama layers do. How do the authors reconcile this with the claim that cortex and Llama share geometry? It would also help to report the main results as margins above the spectrum-matched null rather than above zero.

  3. Per-system PCA (Section 3.2). What data is used to fit the per-system PCA? Is a single basis estimated per system and shared across all images and time windows or layers, or is the basis refit at each slice? This detail does not seem to be in the appendix either. If the basis is refit per slice, consecutive 8×8 covariances are expressed in different coordinates, and the velocity of Eq. 3 would mix a change of basis with the change in representation. A rotation common to all images could then contribute to the angular concordance independently of either system's representation.

  4. Code availability. Could the authors make the code available? This would help with reproducibility and with understanding the computational details of the approach, including the point raised above.

  5. Computational cost. Could the authors report the computational cost of the full pipeline, e.g., runtime per subject and the hardware used?

  6. Relation to RSA in the main text. App. L states that with a Pearson correlation, Eq. 7 is exactly an RSA comparison, and locates the novelty in (i) using velocity directions of covariance trajectories rather than state dissimilarities and (ii) the order test on the window×depth grid. Could this contrast be stated more clearly in the main text, rather than only in the appendix? It would help the reader place the contribution relative to the most closely related method.

Reviewer_UT6JOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a method to compare representational trajectories across systems, using intrinsic geometry to be invariant to what they deem as irrelevant differences between systems. This metric is based on the angles between velocity vectors, accounting for covariant derivatives on the positive semi-definite manifold of covariance matrices, where the response covariances are estimated over a family of inputs. Their proposal allows time-warping, unlike Dynamic Similarity Analysis, and compares trajectories (across time, or layers, etc), unlike static views of Representational Similarity Analysis.

Critical strengths

The problem the paper addresses seems interesting and important, and they seem to have identified a valid gap in current methods to compare different systems.

Critical weaknesses

The paper is terribly written. They use extensive jargon, including undefined acronyms, and jump quickly between concepts. They do a poor job explaining what to observe in their plots. If their results are sound, then I hope they are able to communicate these results much more clearly. I could write a long list but I don't think this would be useful, because the problems are pervasive, in every section. For example, Section 3.5 is called Angular Concordance but nowhere in that section do you define what angular concordance is. Maybe you just mean "angular distance" (5) but this is a microcosm of the many writing problems.

Questions for authors

I apologize but I have too many questions to list. Some of these are clarification questions because of my inadequate familiarity with specific methods in differential geometry. Others are questions about the choices and implications of those choices. For example, how did you compute acceleration? You say "A pairwise angle is invariant to any rotation of a system’s frame" which is correct, but then follow it with "so the comparison requires no common basis and no learned map" — overly general since angles depend on the metric and the metric changes with, e.g., different scaling of a basis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.