acceptodds
Under review as a conference paper at ICLR 2027

The Wittgensteinian Representation Hypothesis: Is Language the Effective Reference Geometry of Multimodal Alignment?

Abstract

Understanding why independently trained neural networks from different modalities exhibit shared representational structures, and what governs this cross-modal alignment, remains an open question in representation learning. Existing evidence relies on symmetric similarity measures, which can detect modal alignment but are structurally blind to its direction. We introduce directional modal alignment analysis using cycle-kNN, an asymmetric alignment measure, and apply it to dozens of independently trained unimodal models spanning point clouds, vision, and language. We uncover a consistent directional asymmetry: non-language modalities align with the neighborhood structure of language significantly more than language aligns with theirs. This pattern holds across all model families and scales, yet remains entirely invisible to symmetric measures. Mechanistic analysis traces this directionality to feature-density asymmetry, whereby language representations occupy the most compact regions of representational space. The Information Bottleneck framework provides a principled interpretation: optimization under compression promotes alignment with the discrete, compositional structures characteristic of language. We formalize this insight as the Wittgensteinian Representation Hypothesis: the semantic structure of language constitutes the effective reference geometry for cross-modal representation alignment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Cross-Modal Alignment, Directional Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8cW6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper analyses the result of an asymmetric alignment metric (cycle-KNN) for latent geometry across modalities. It selects ~50 unimodal models over point clouds, vision and language --- modalities chosen for their increasing abstract nature. They find a Δ≈+0.01\Delta \approx +0.01 to +0.05+0.05 (corresponding to a relative difference between 1% and 6%) in the direction of the more “abstract” modality.

This effect is linked to a density measure D of the latent space, and to the information bottleneck. The paper makes the following claims :

  • C1 : The asymmetry of cycle-KNN consistently points towards language.
  • C2 : This effect is scale-invariant.
  • C3 : The underlying geometric cause is feature density.
  • C4 : language geometry is most stable.
  • C5 : WRH : cross-modal geometric convergence is directional rather than mutual, and points towards more compact representation and semantic coherence. The empirical target is language geometry.
Critical strengths

S1 : Novelty. To the best of my knowledge, the directionality of alignment has indeed never been investigated significantly in prior work. The paper is transparent on the fact that this metric is not novel, and that what is being studied here is its asymmetry.

S2 : Large empirical sweep. The paper reports an analysis that spans 10610^6 to 101110^11 parameters, across 29 language models, 22 vision and 7 point cloud models, with ablations and selection protocol (e.g. for layer selection). The reported p−p-values show that, though small, the asymmetry measure is significant.

S3 : Strong appendices. The appendices provide additional controls, ablations, and sensitivity. They also report unflattering results.

S4 : Claims C1 and C2 are well supported by the empirical findings.

S5 : Claim C4 is a good attempt at falsifying the assumption. I’d be careful with the framing, it currently reads as a confirmation, where it’s only a failed falsification. The fact that language geometry is stable doesn’t mean it’s the only possible one. See the weaknesses for a discussion on whether C4 is sufficiently supported though.

Critical weaknesses

The core of the paper, but also where it falls short, is the explanation and interpretation of the observed effect. Let’s take the WRH.

Directional convergence. The WRH mentions “convergence”. But convergence with respect to what ? This is very obscure, and seems to be convergence as modality become more “abstract” (cf “We assemble 50 independently trained models spanning three modalities along an abstraction hierarchy” l143 144). This notion is not defined, and an attempt at formalising it might not produce a partial order, let alone a pointed one where language could serve as a candidate top. Moreover, this makes the statement of directionality tautological. If there were no directionality in “modality” term (as opposed to, say, scale or training step), there would be no convergence. Moreover, an arbitrary directed graph between three points is transitive 6 out of 8 times; and the largest gap (PC - vision) does not involve language. The number of modalities has to be higher for this claim to be significant.

Geometric explanation. The authors correlate convergence with compactness, or density, as measured by DD, the average pairwise distance between the embeddings. Correlation (*) is not causation, and a trivial counter example shows that DD cannot be the geometric explanation behind the asymmetry in cycle-KNN. See counter-example below (**) disproving claim C3C3.

Based on the nearest-neighbor aspect of your metric, I’d say that more you could look at something like normalised local density : D(X)=∑i∑j∈NN(i)∥xi−xj∥D(X) = \sum_{i} \sum_{j \in NN(i)} \|x_i - x_j\|, where XX is standardized, or maybe hubness. There might be some link to neural collapse. This is just an intuition of course but I’m curious what the authors think about this discussion.

(*) note : the authors have to report the DD at the target layer. Currently, the only trace that the DD are smaller for language then vision then PC is a promise, not a number. You have to show a graph of the diff (or ratio) in D on one axis, and Δ\Delta on the other axis, and show that there is indeed a correlation. I’d also like this graph for all pairs of layers, not just for the selected ones, to see if the correlation survives there.

(**) Let XX, YY be unitary, L2L2 normalised embeddings, XX living in a plane of dimension d−1d-1, orthogonal to bb. Assume D(X)>D(Y)D(X) > D(Y). Now take X’≔X+αbX’ \coloneqq X + \alpha b, and X’’X’’ the L2L2 normalized counterpart. The nearest neighbor geometry of X’’X’’ is exactly the same as that of XX since embeddings will simply occupy a tighter d−1d-1 dimensional sphere Therefore, Δ(X’’→Y)=Δ(X→Y)\Delta(X’’ \to Y) = \Delta(X \to Y). For α\alpha large enough, we have D(X)<D(Y)D(X) < D(Y).

Semantic coherence is not defined despite being key in the WRH. Maybe something like the DD proposed above might be a candidate definition ? At least this is how I understand it based on your paper. I might be mistaken.

Conclusion. Any attempt at better formalising the WRH with the current experiment fails to do anything else than restate C1C1. Claiming a new representation hypothesis based on such a small and unexplained gap in one metric is a massive leap…


Less critical points :

  • How to interpret absolute values of cycle-KNN (S) ? It seems that S(X→X)S(X \to X) is not even guaranteed to be 11. E.g., isolated or OOD points will not be in their neighbors’ neighborhood. This seems to create a confound that might bias the results and explain the reported asymmetry. The paper should report values for S(X→X)S(X\to X) per modality.
  • equation 2 is stated as a proposition. But, once again, there is no definition of “coherent”. Even though I get the intuitive idea you are trying to convey, this should be formalised for such a statement to even make sense, let alone be proven. This seems to be the core lack in this paper, as this would likely provide the missing geometric characterisation
  • equation 3 : once again, DD is a measure about global geometry, and cycleKNN is about local geometry. (a) A uniform blob and (b) two tight blobs separated by a large gap will share the same DD but have massively different NN relationships.
  • figure 2 : showing the confidence intervals would greatly help visualise the significance of the Δ\Delta. This is minor since you give p−p-values in the caption.
  • figure 4 : CKA is known to have massive confounds like dimensionality. The paper doesn’t report whether it used a normalised version of CKA or whether all dimensionalities are the same. Dimensionality alone could explain the differences.
  • The number of points per modality is only 1024 pairs for vision-text. This is smaller or in the same order of magnitude than latent dimensionality. This cannot approximate the latent distribution enough to make stable measures of alignment.
  • “WRH differs from PRH in providing an operationally testable endpoint—the most compressed, semantically coherent modality [in the sense of D]” (l400) Once again, this claim is false in the sense of “compressed”, and undefined in the sense of “semantically coherent”.
Questions for authors

Q1 : figure 4 and table 2 : isn’t the instability in point clouds fully explained by architectural choices of the models, or some other factor ? The bottom left corner seems rather stable to me.

Q2 : The number of models reported seems inconsistent. You report both (5, 16, 29) and (7, 22, 29) models for (PC, vision, language). This is likely only a typo. Could you precise which is the correct number ?

4 comments
AuthorsOpenReview ↗

(1/4) Responses to reviewer 8cW6

We thank the reviewer for their careful evaluation of our manuscript and for recognizing the novelty of our systematic study of alignment directionality, the breadth of our empirical investigation across model families and scales, the strength and transparency of the appendices and sensitivity analyses, the empirical support for Claims C1 and C2, and the value of C4 as an attempt to falsify the proposed interpretation. We address the reviewer's concerns below:

(1) Directional convergence. What does “convergence” mean in this paper? Is the proposed abstraction hierarchy undefined or tautological?

We DO NOT introduce a NEW definition of “convergence” in this paper. In fact, the meaning of “convergence” is clear: the definition of “convergence” in this paper follows PRH [1] — a positive association between competence/scale and convergence degree observed at the cross-sectional level:

PRH defines convergence as "alignment toward Z increases toward a convergent point as a function of scale" (Sec. 1 [1]), where the axis is competence/scale, not training time. All of PRH's evidence is cross-sectional: 78 vision models aligned by VTAB competence (Fig. 2 [1]), 20 LLMs aligned by language modeling score (Fig. 3 [1]) — no training trajectories, no movement through representation space. Our evidence is structurally identical: 58 independently trained models across 3 modalities, analyzed as a function of model capability rather than training dynamics.

In short, the definition and empirical evidence of "convergence" are already established by PRH's extensive experiments; the focus of this paper is the modal directionality of this "convergence". Our contribution is to exploit cycle-kNN's inherent asymmetry: Δcyc(A,B)=cycle⁡(A→B)−cycle⁡(B→A)\Delta_{\mathrm{cyc}}(A,B)=\operatorname{cycle}(A\to B)-\operatorname{cycle}(B\to A) reveals cross-modal directional asymmetry that PRH's symmetric metrics cannot capture.

(2) Geometric explanation. The paper associates convergence with compactness, measured by the global mean pairwise distance DD, but correlation is not causation. The counterexample shows that DD can change while nearest-neighbor geometry—and hence cycle-kNN—remains unchanged, so DD cannot by itself explain the asymmetry or establish Claim C3.

The counterexample establishes that the scalar DD is not a sufficient statistic for cycle-kNN; it does not establish that feature density is irrelevant to the asymmetry in semantically organized learned representations. Claim C3 concerns a geometric association in this restricted empirical setting, not a universal identity in which DD uniquely determines Δ\Delta for arbitrary point configurations.

We separates these claims through several interventions: row permutation preserves DD while destroying semantic correspondence, DD-matched Gaussians reproduce compactness without reproducing the directional gap, and centroid collapse and random projection produce qualitatively different behavior from the real representations. These controls show that compactness alone is insufficient, while the observed direction is associated with compactness only when accompanied by semantically organized neighborhood structure. We will therefore revise any wording that presents the global scalar DD as a unique causal explanation and state the supported conclusion more precisely: DD is a modality-level summary of compactness and one component of the proposed mechanism, whereas the directional statistic also depends on local semantic topology that DD alone does not encode.

The paper does not claim that a space serves as a more effective reference simply because it is more compact. Rather, it claims that the more compact and semantically coherent space serves as the effective reference (see Section 5).

AuthorsOpenReview ↗

(2/4) Responses to reviewer 8cW6

(3) Given that cycle-kNN is based on nearest neighbors, would normalized local density or hubness be a more appropriate geometric diagnostic? Could the phenomenon be related to Neural Collapse?

Local density and hubness would be useful complementary diagnostics, but they are not replacements for DD, because the quantities serve different purposes. Cycle-kNN is already a local statistic: it directly measures whether instance-level neighborhoods can be recovered through another representation space. In contrast, DD is used only as a global summary of modality-level compactness; the paper does not claim that DD alone determines the local cycle-kNN topology. The appendix also goes beyond global DD: among the eight synthetic generators, the folded-manifold experiment explicitly controls local density through fold amplitude, and the directional gap follows the induced neighborhood-density asymmetry for k≥5k\geq5. Together with the row-permutation and DD-matched Gaussian controls—where compactness without semantic correspondence produces no stable gap—this supports the paper's stated joint account of compactness and semantically coherent local neighborhoods. A hubness analysis could further characterize which points dominate neighbor sets, but its absence does not invalidate the measured directional asymmetry.

The existing class-centroid-collapse control argues against simple within-class collapse as the explanation of our result. Replacing each representation with its class centroids maximizes within-class compactness, yet it produces a qualitatively different signature: both absolute cycle-kNN scores decrease substantially while ∣Δ∣|\Delta| is inflated to approximately 0.20.2, about an order of magnitude larger than the real effects. The observed asymmetry instead occurs while absolute cross-modal alignment remains high.

(4) Semantic coherence is not defined despite being key in the WRH.

As with “convergence,” “semantic coherence” is not introduced by WRH as a new undefined primitive. We follow the standard PRH-style setting in which semantically corresponding samples are expected to preserve meaningful neighborhood relations across independently trained representation spaces. In our paper, this notion is also given an explicit and independently measurable operational definition in the appendix: psemp_{\mathrm{sem}}, the fraction of a sample's kNN neighbors that share its ground-truth category. Under this definition, language reaches psem≈0.7p_{\mathrm{sem}}\approx0.7, whereas vision is approximately 0.20.2–0.50.5 at comparable DD. Thus, the reviewer's suggestion that DD might define semantic coherence conflates two quantities that the paper deliberately separates: DD measures global compactness, whereas psemp_{\mathrm{sem}} measures the semantic organization of local neighborhoods. This distinction is empirically necessary: row permutation preserves DD exactly while destroying semantic correspondence, and DD-matched random representations likewise fail to produce a stable directional gap.

Equation 2 is a hypothesis/prediction, not a theorem, and it was not intended as a logical derivation of semantic coherence from Δ\Delta. The empirical quantities on both sides are independently operationalized: Δ\Delta by directional recovery, compactness by DD, and semantic coherence by psemp_{\mathrm{sem}}. We will relabel the statement and spell out these definitions, but the absence of a theorem proof is not a defect for a falsifiable empirical hypothesis.

(5) Is proposing WRH from a small and insufficiently explained effect measured by a single metric too large a leap?

Calling WRH a “hypothesis” is already the appropriate epistemic framing; hypotheses are proposed from systematic, falsifiable evidence rather than only after becoming laws. The evidence is not one small number: it spans 50 models, three pair types, model-pair sign matrices, k-sensitivity, seven layer protocols, semantic-destruction controls, granularity controls, and independent Information Imbalance agreement on V–L. Moreover, under the aggregation-aware permutation calibration of Gröger et al. (2026), the V→L gap remains +0.0104+0.0104, with the predicted sign in 80.7% of 638 model pairs (p=1.1×10−63p=1.1\times10^{-63}), while the PC→L gap remains +0.0269+0.0269, with the predicted sign in all 203 pairs (p=4.6×10−35p=4.6\times10^{-35}). Thus, the directional pattern is not an artifact of reporting uncalibrated maxima over layer pairs. We will distinguish the measured effect from the proposed mechanism more sharply, but the breadth and consistency of the pattern justify proposing WRH.

AuthorsOpenReview ↗

(3/4) Responses to reviewer 8cW6

(6) What are the S(X→X)S(X\to X) values for each modality, and could variation in self-cycle recovery confound the reported asymmetry?

S(X→X)S(X\to X) is not guaranteed to equal 1. Because the query itself is excluded from its kNN list, S(X→X)S(X\to X) measures the fraction of samples with at least one reciprocal k-nearest neighbor in XX. Anti-hubs—samples that never occur in another sample's neighborhood—therefore reduce the self-score. More generally, S(X→Y)S(X\to Y) closes its return cycle in the source graph XX and is upper-bounded by RXR_X, the fraction of samples with non-zero k-occurrence in that graph. Since the two directions in Δ=S(X→Y)−S(Y→X)\Delta=S(X\to Y)-S(Y\to X) close their cycles in different source graphs, modality differences in neighborhood reciprocity could in principle contribute to the raw directional gap.

We quantified this effect explicitly. At the selected layers, the self-scores are all high—0.9220.922 to 0.9770.977, compared with a chance level of 0.0940.094—and anti-hubs account for only 1.2%1.2\%–3.4%3.4\% of samples:

ModalityS(X→X)S(X\to X), selected layersS(X→X)S(X\to X), all-layer meanRXR_XAnti-hubs
Point cloud0.977±0.0080.977\pm0.0080.9720.9720.9880.9881.2%1.2\%
Vision0.966±0.0140.966\pm0.0140.9430.9430.9850.9851.5%1.5\%
Language0.950±0.0170.950\pm0.0170.8540.8540.9740.9742.6%2.6\%

The selected-layer self-score gap favors the reported direction by +0.045+0.045 for PC→L, and +0.016+0.016 for V→L. We therefore recomputed Δ\Delta under three controls: (i) both directions evaluated on the identical set of anchors with non-zero k-occurrence in both graphs; (ii) a stricter common-anchor set requiring a reciprocal neighbor in both graphs; and (iii) each directional score normalized by its source-graph ceiling RXR_X. We also regressed Δ\Delta on the self-score gap and evaluated the intercept at equal reciprocity.

DirectionRaw Δ\DeltaMatched reachable anchorsCeiling-normalizedMatched reciprocal anchorsRegression intercept
PC→L+0.0243+0.0243+0.0130+0.0130+0.0099+0.0099+0.0026+0.0026+0.0062+0.0062 (p=6.5×10−3p=6.5\times10^{-3})
PC→V+0.0406+0.0406+0.0284+0.0284+0.0239+0.0239+0.0143+0.0143+0.0299+0.0299 (p=4.9×10−20p=4.9\times10^{-20})
V→L+0.0103+0.0103+0.0044+0.0044+0.0036+0.0036+0.0023+0.0023+0.0086+0.0086 (p=2.9×10−50p=2.9\times10^{-50})

All controlled gaps remain positive and statistically significant (Wilcoxon signed-rank p≤3×10−5p\leq3\times10^{-5}). Thus, differences in neighborhood reciprocity do not create the directional asymmetry: the effect persists when both directions use exactly the same anchors and when the source-graph ceilings are normalized.

(7) Figure 4 does not specify the CKA variant or representation dimensionalities, so dimensionality could explain the reported differences.

The CKA computation in Figure 4 follows the same standard symmetric CKA implementation and representation-comparison protocol used by PRH. It is not the supplementary directional CKA analyzed separately in our appendix.

The stronger claim that dimensionality alone explains Figure 4 is not supported by the results. Figure 4's ordering is reproduced by mutual kNN, which compares the overlap of neighborhood identities and does not require the two representations to have matched feature dimensions: Language 0.562±0.1000.562\pm0.100 >> Vision 0.502±0.0790.502\pm0.079 >> Point Cloud 0.248±0.2360.248\pm0.236. This is the same ordering produced by CKA: Language 0.904±0.0550.904\pm0.055 >> Vision 0.857±0.0510.857\pm0.051 >> Point Cloud 0.440±0.3240.440\pm0.324. Thus, representation dimensionality may affect the absolute magnitude of CKA and should be documented, but it cannot by itself account for an ordering independently recovered by a neighborhood-based metric. We will clarify the CKA implementation and feature dimensions and present mutual kNN as the dimension-mismatch-robust corroboration rather than relying on CKA alone.

AuthorsOpenReview ↗

(4/4) Responses to reviewer 8cW6

(8) With only N=1,024N=1{,}024 paired samples for vision–language—comparable to or smaller than the latent dimensionality—are the alignment estimates stable?

N=1,024N=1{,}024 is not an arbitrary sample size: it directly follows the cross-modal evaluation protocol of PRH, whose Appendix C.2 computes k=10k=10 nearest-neighbor alignment over 1,024 WIT image–text pairs. Using the same dataset size preserves direct comparability with the work on which our analysis builds. Moreover, the observation that NN is comparable to the latent dimensionality does not by itself imply that cycle-kNN is unstable. Cycle-kNN is a rank- and neighborhood-based statistic; it does not estimate a full dd-dimensional density or covariance matrix and therefore does not require N>dN>d as an identifiability condition. Empirically, the direction is evaluated across 638 vision–language model pairs, with 530/638 pairs (83.1%) showing the predicted sign, and remains consistent across kk, layers, and layer-selection protocols. Subsampling is still a useful finite-sample robustness check, but its absence is not evidence that the reported estimate is unstable.

Q1: Could the apparent instability of point-cloud representations be explained by model architecture or another factor rather than modality?

Architectural diversity may contribute to the lower point-cloud consensus, but it is not an alternative explanation that nullifies a modality-level result: the evaluated modality is instantiated precisely through its current architectures and objectives. Moreover, language and vision pools are also architecturally diverse, yet the same two metrics give the ordered consensus Language > Vision > Point Cloud.

Q2: The manuscript inconsistently reports (5,16,29)(5,16,29) and (7,22,29)(7,22,29) point-cloud, vision, and language models. Which count is correct?

The correct counts are 5 point-cloud models, 16 vision models, and 29 language models, for 50 models in total.

Reviewer_qhepOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether cross-modal representational convergence is asymmetric. Previous studies have tested for convergence by comparing representation spaces with similarity measures such as CKA, mutual kNN and RSA. Because these measures are symmetric by construction, they can show how closely two spaces align but not the direction of that alignment. To capture direction as well as degree, the authors use cycle-kNN, the only asymmetric measure among the eight catalogued by Huh et al. (2024). They define a directional gap Δ\Delta as the difference between cycle-kNN computed from A to B and from B to A, and compute it over 50 unimodal models from three modalities, i.e. point clouds (PC), vision and language. The paper reports Δ>0\Delta>0 for PC --> language and vision --> language, which the authors read as alignment toward language. PC --> vision also gives Δ>0\Delta>0, in this case toward vision. The authors tie this effect to a “feature-density asymmetry”: at the alignment-optimal layers, the pairwise mean distance on l2l2-normalized features is lowest for language. They interpret the pattern through the Information Bottleneck framework and summarize the findings as the Wittgensteinian Representation Hypothesis (WRH). The appendix gives detailed numerical results, along with a proof that cycle-kNN is symmetric at k=1k=1 and can become asymmetric for k≥2k \ge 2.

Critical strengths

As the authors point out, previous studies of cross-modal convergence have relied almost entirely on symmetric measures. Thus, asking whether convergence has a direction is a new and meaningful question. Cycle-kNN is a sensible choice for this purpose as it is easy to compute and asymmetric by construction. In the appendix, the authors prove that cycle-kNN is symmetric at k=1k=1 and show that it can become asymmetric for k≥2k \ge 2. They also report detailed numerical experiments, including row-permutation and D-matched Gaussian controls and a comparison of seven layer-selection protocols. In my view, the main contribution of this work is the question it raises. The authors claim that the asymmetry points toward language. If this claim holds, its implications would not be limited to computation but it could bear on human cognition and on language development, which is presumably why the authors named the hypothesis after Wittgenstein’s view of language. For these reasons, I see real merit in the direction this paper takes.

Critical weaknesses

As noted in the strengths section, this paper’s implications would be considerable if its main claim holds. For the same reason, its numerical results need to be read with particular care. The larger the implications, the more it matters whether the asymmetry is statistically significant and free of confounds.

The central claim is that the asymmetry points toward language, and this is what gives the hypothesis its name. It is not clear if the data single out language. The authors report Δ>0\Delta >0 for PC --> language and vision --> language, but also for PC --> vision, a pair that does not involve language at all. In all three cases, Δ\Delta seems to point toward whichever space is more compact. The paper notes this pattern in Section 4.1, and the formal statement of WRH is itself phrased in terms of compactness. If so, the evidence supports an ordering by density, and language comes first only because it is the most compact of the three modalities studied. Whether language plays any special role beyond this, as the name suggests, has not been tested. Note also that Appendix C shows density alone, without semantic correspondence, produces no gap, so density cannot be the full explanation either.

This work belongs to a line of research that began with the Platonic Representation Hypothesis (Huh et al., ICML 2024). Gr"oger et al. revisited that hypothesis in “Revisiting the Platonic Representation Hypothesis: An Aristotelian View” (ICML 2026). They showed that model scale and layer-wise max aggregation inflate raw similarity scores, and that several trends reported in 2024 disappear under their aggregation-aware permutation calibration. Any measure used in this setting should be calibrated the same way. The present paper does not do this: all scores are raw maxima over layer pairs, and each direction selects its own best layer pair. Without calibration, it is hard to judge how much of Δ\Delta , or of the scale-invariance in Figure 5, would survive. The paper also does not cite Gr\oger et al. (2026). The Gr"oger et al., 2025 in its reference list is a different paper.

The values of Δ\Delta are critical, yet the effect is small, and on the main axis it overlaps with the paper’s own null. For vision --> language, Δ=0.010\Delta = 0.010, while the between-pair SD of the score is about 0.07. With N=1,024N = 1,024, this amounts to roughly ten stimuli. Appendix E bounds the double-selection bias at ±0.008\pm 0.008 using a language <--/-->language null and calls this “an order of magnitude below the reported effects (0.02–0.05).” The headline vision--> language effect, however, is 0.010, not 0.02 - 0.05, so it is about the size of the bias bound. The vision–language result, which motivates the hypothesis’s name, turns out to be the weakest of the three.

The paper states that “no prior work employs an asymmetric measure to investigate the direction of this alignment.” Acevedo et al. (2025), which the authors cite in their own related work, already reported a text --> vision asymmetry using Information Imbalance. The novelty claim should be revised accordingly.

Questions for authors

Gr"oger et al. (2026) applied their aggregation-aware permutation calibration to every measure they compared. WRH, by contrast, reports raw maximum scores and even selects layers separately for each direction. Under aggregation-aware permutation calibration, with one correspondence permutation shared across all layer pairs and both directions, what are Δ and its per-pair sign rate for V --> L, PC --> L and PC --> V? Does the scale-invariance in Figure 5 still hold for the calibrated Δ\Delta?

Cycle-kNN is a purely local measure. Does the directional signal also appear at a global scale, or only within local kNN neighborhoods? For example, would a directional measure based on global relational structure, such as edges of the minimum spanning tree, show the same asymmetry?

Mean-pooled LLM representations often contain a few outlier dimensions with very large values. Some recent work clips features at, for instance, the 95th percentile of absolute values before l2l2 normalization. WRH does not apply such a step, so these outlier dimensions could be driving both the low D and Δ\Delta for language. Does Δ\Delta remain after outlier clipping? The language<--/--> language null bounds the selection bias at +/- 0.008, and the V --> L effect is +0.010. On what basis should the V --> L effect be considered distinguishable from this null?

3 comments
AuthorsOpenReview ↗

(1/2) Responses to reviewer qhep

We thank the reviewer for the time and effort devoted to reviewing our manuscript. We also appreciate the reviewer's recognition of the novelty and significance of asking whether cross-modal convergence has a direction, the suitability of cycle-kNN for studying this question, the value of our theoretical analysis of its asymmetry, and the breadth of our numerical experiments and controls. We address the reviewer's concerns point by point below:

**(1) It is not clear if the data single out language. The authors report Δ>0\Delta>0 for PC → Language and Vision → Language, but also for PC → Vision, a pair that does not involve language. In all three cases, Δ\Delta seems to point toward whichever space is more compact. **

First, we would like to clarify that the paper does not claim that a space serves as a more effective reference simply because it is more compact. Rather, it claims that the more compact and semantically coherent space serves as the effective reference (see Section 5). In other words, effective reference geometry is jointly determined by compactness and semantic coherence, not by compactness alone. This question therefore presupposes a stronger claim than WRH actually makes. The paper's formal claim is conditional: among the three evaluated modalities, the more compact and semantically coherent space acts as the effective reference, with language occupying the extremal position in this measured ordering. PC→Vision is not counterevidence; it is the predicted intermediate comparison that makes the ordering nontrivial.

(2) Whether language plays any special role beyond this, as the name suggests, has not been tested. Appendix C also shows that density alone, without semantic correspondence, produces no gap; therefore, density cannot be the full explanation.

As the reviewer notes, Appendix C shows that compactness without semantic correspondence produces no directional gap. This result is consistent with, rather than contrary to, our claim: WRH posits that effective reference geometry depends jointly on compactness and semantic coherence. Language also differs structurally from perceptual modalities such as images and point clouds: its discrete and compositional symbolic structure abstracts away low-level perceptual variation while preserving high-level semantics. Empirically, among the three modalities examined, language occupies the extremal position not only in compactness but also in semantic-neighborhood coherence and cross-model consensus. These properties—and language's measured extremal position—motivate the name WRH.

We do not claim that language has an untested ontological privilege beyond them. In fact, the paper explicitly leaves open whether a more compressed and semantically coherent modality could displace language as the effective reference.

(3) and Q1: The paper reports raw maxima over layer pairs and selects the best layer pair separately for each direction. Without the aggregation-aware permutation calibration of Gröger et al. (2026), it is unclear how much of Δ\Delta, or of the scale invariance in Figure 5, would survive. The paper also does not cite Gröger et al. (2026).

We thank the reviewer for bringing this to our attention, and we recognize the importance of the recent work by Gröger et al. (2026). We would like to clarify that, in addition to reporting results obtained by selecting the best layer pair separately for each direction, the paper also reports permutation tests, seven layer-selection protocols—including direction-neutral, last-layer, all-layer-mean, and all-layer-median rules—and complete layer-pair heatmaps. These results show that the positive sign is not confined to the selected maximum (see Section 3.1, Appendix E, and Appendix H.3).

We have also added the aggregation-aware permutation calibration proposed by Gröger et al. (2026) and included the corresponding citation, with the following results:

Direction (n pairs)Raw Δ (% Δ>0)Calibrated Δ (% Δ>0)Wilcoxon pCalibrated score, fwd / bwd
V→L (638)+0.0103 (83.1%)+0.0104 (80.7%)1.1e-630.514 / 0.504
PC→L (203)+0.0243 (100%)+0.0269 (100%)4.6e-350.644 / 0.618

Under the aggregation-aware permutation calibration proposed by Gröger et al. (2026) (K=200K=200, α=0.05\alpha=0.05, smax⁡=1s_{\max}=1), we generate one correspondence permutation for each null replicate, apply the same permutation to all L×LL\times L layer pairs and both directions, and take the maximum over the entire layer grid. After calibration, Δ\Delta is +0.0104+0.0104 for V→L, with a positive sign for 80.7% of the 638 model pairs, and +0.0269+0.0269 for PC→L, with a positive sign for all 203 model pairs. Both effects remain statistically significant (Wilcoxon p<10−26p<10^{-26}). Thus, applying aggregation-aware permutation calibration does not change our conclusions.

AuthorsOpenReview ↗

(2/2) Responses to reviewer qhep

(4) and Q4: The V→L effect is small: Δ=0.010\Delta=0.010, the between-pair SD is about 0.07, and with N=1,024N=1{,}024 the mean gap corresponds to roughly ten recovered stimuli. It is also close to the language↔language double-selection bound of ±0.008\pm0.008 and is the weakest of the three effects. On what basis is V→L distinguishable from this null?

The absolute V→L gap is modest, and “an order of magnitude below the reported effects” is inaccurate for V→L, we will correct that wording.

However, the SD of approximately 0.07 measures heterogeneity across model pairs, not uncertainty of the paired mean, and “ten stimuli” describes the mean difference for one set of 1,024 queries while ignoring that the direction is evaluated across 638 model pairs from many independently trained families. Among them, 530/638 pairs (83.1%) have the predicted sign, and the gap remains positive under direction-neutral and all-layer aggregation protocols. More importantly, the language↔language value of ±0.008\pm0.008 is a descriptive upper bound on double-selection bias; it is not a confidence interval, a significance threshold, or the null distribution of the V→L statistic. Therefore, the numerical proximity of +0.010+0.010 to +0.008+0.008 does not itself test whether the V→L effect is significant. The relevant inferential result is the paired Wilcoxon test across the forward and reverse scores of the V→L model pairs, which rejects zero directional difference (p<0.05p<0.05), together with the consistent predicted sign in 530/638 pairs. Thus, the effect is modest in magnitude but systematic and statistically significant.

(5) The paper states that no prior work employs an asymmetric measure to investigate alignment direction, but Acevedo et al. (2025)—already cited in the related work—reported a text → vision asymmetry using Information Imbalance. The novelty claim should be revised.

Acevedo et al. (2025) and our work ask different research questions: Acevedo et al. focus on quantifying semantic information in deep text and image representations using Information Imbalance; their text→vision asymmetry is an empirical observation within that analysis, rather than a test of whether cross-modal convergence has a preferred direction or which modality acts as an effective reference geometry.

Our work begins with precisely this latter question, formulates a directional reference-geometry hypothesis before measurement, and tests its predicted ordering across point cloud, vision, and language representations. Acevedo et al. therefore provide relevant precedent for an asymmetric measurement, but they neither formulate nor test the WRH research question.

Q2: Cycle-kNN is a local measure. Does the directional signal also appear at a global scale—for example, under a directional measure based on minimum-spanning-tree edges—or only within local kNN neighborhoods?

Cycle-kNN is intentionally local because WRH's measured phenomenon is instance-level neighborhood recoverability; the hypothesis does not require every global statistic to encode the same asymmetry. Nevertheless, the appendix already provides complementary evidence beyond cycle-kNN: Information Imbalance, a rank-based measure, reproduces the V–L direction on identical inputs, and Acevedo et al. report the same V–L direction on a different dataset and model set. Agreement is weaker for PC–L, while directional CKA has a known feature-dimensionality confound; we will present this metric specificity transparently. We do not introduce an ad hoc “directional MST” score because an MST is undirected and would require a separately justified asymmetric comparison rule. Constructing and validating a principled global directional statistic is useful future work, but it is not a prerequisite for the local geometric claim tested here.

Q3: Mean-pooled LLM representations may contain a few outlier dimensions. Because WRH does not clip these dimensions before ℓ2\ell_2 normalization, they could drive both low DD and language-ward Δ\Delta. Does Δ\Delta remain after outlier clipping?

We already apply this step. Following the PRH codebase (Huh et al., 2024), every alignment score in the paper is computed after clamping each model's features at the 95th percentile of absolute values and then L2-normalizing each layer. The reported Δ, including the calibrated Δ in our W4 response, is therefore already computed after outlier clipping.

Thank you for the authors' detailed explanation. It has clarified the questions I raised in my initial review. I think the directional investigation of structural similarity is a valuable aspect of this work. I would like to maintain my score of 3.

Reviewer_BvRkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the directionality of cross-modal representation alignment, extending the Platonic Representation Hypothesis. Using cycle-kNN as an asymmetric alignment measure, the authors analyze representations from independently trained models across point cloud, vision, and language modalities. They observe a consistent directional asymmetry toward language, alongside higher intra-modality representational consistency among language models. The authors further associate this asymmetry with differences in representational compactness and semantic coherence, offering an interpretation through the Information Bottleneck framework. Based on these findings, they propose the Wittgensteinian Representation Hypothesis, which posits that language serves as the effective reference geometry for cross-modal alignment.

Critical strengths
  1. An interesting and thought-provoking research question. The paper explores a largely overlooked aspect of cross-modal representation alignment by questioning whether alignment exhibits directional asymmetry. The conceptual framing is insightful and opens up interesting directions for understanding the geometric relationships between representations across modalities.

  2. A systematic empirical investigation. The authors examine directional asymmetry across diverse model families, architectures, and scales, complemented by synthetic experiments and several control analyses. This provides useful empirical observations and a foundation for further investigating the underlying mechanisms.

Critical weaknesses
  1. Insufficient justification for directional alignment and its underlying mechanism. A positive directional gap, Δ = S(X→Y) − S(Y→X), establishes asymmetric round-trip recoverability, but the paper does not justify why this identifies Y as the reference geometry for alignment. Moreover, representational compactness is defined by mean pairwise Euclidean distance, whereas cycle-kNN depends on nearest-neighbor relationships. The synthetic experiments alter both compactness and neighborhood structure, leaving their respective contributions unresolved. Thus, the proposed mechanism remains largely correlational, and the central theoretical interpretation is insufficiently supported.

  2. Insufficient control over dataset-dependent neighborhood structure. Cycle-kNN is inherently sensitive to sample composition, semantic granularity, and hierarchical relationships, yet these factors are not systematically examined. For example, 1,000 instances of the same category with detailed descriptions may exhibit very different neighborhood structures from 1,000 distinct categories represented by single-word labels. Similarly, with 100 categories and 10 closely related instances per category (k=10), within-category neighborhoods could yield nearly perfect round-trip recovery in both directions, regardless of the proposed modality hierarchy. Without controlling category distributions, within-category diversity, and semantic hierarchy, it is unclear whether the observed asymmetry reflects intrinsic modality properties or dataset-specific structure.

Questions for authors

Please see critical weaknesses.

1 comment
AuthorsOpenReview ↗

Responses to reviewer BvRk

We thank the reviewer for the time and effort devoted to reviewing our manuscript. We are encouraged that the reviewer recognized the importance and thought-provoking nature of the research question, the insight offered by our conceptual framing, and the breadth of our systematic empirical investigation across model families, architectures, and scales, including the synthetic experiments and control analyses. We address the reviewer's concerns below by clarifying the operational meaning of “reference geometry” and distinguishing the measured directional phenomenon from its proposed geometric mechanism.

(1) A positive gap establishes asymmetric round-trip recoverability, but why does this identify YY as the reference geometry? Moreover, compactness is measured by global mean pairwise distance DD, whereas cycle-kNN depends on local neighborhoods; the synthetic experiments change both, so the proposed mechanism remains correlational.

Positive Δ\Delta identifies YY as the effective reference by the paper's explicit operational definition: in S(X→Y)S(X\to Y), YY's kNN graph supplies the first-hop relational structure, and systematically higher round-trip recovery means that YY's neighborhoods provide the more effective intermediate geometry.

We agree that the scalar DD cannot uniquely determine a local cycle-kNN statistic, but the paper's claim is not such a universal identity. The evidence instead supports a joint account of compactness and semantically organized local topology:

  1. The folded-manifold experiment explicitly varies local density through fold amplitude, and the directional gap tracks the induced neighborhood-density asymmetry for k≥5k\geq5.

  2. Row permutation preserves DD exactly while destroying correspondence, and DD-matched Gaussians reproduce compactness without producing a stable gap; compactness alone is therefore insufficient.

  3. Class-centroid collapse and random one-dimensional projection yield qualitatively different signatures—depressed absolute alignment and abnormally inflated gaps—rather than reproducing the real-data regime.

  4. After removing source-graph reciprocity differences by evaluating both directions on identical reachable anchors, the gaps remain positive: +0.0130+0.0130 for PC→L, +0.0284+0.0284 for PC→V, and +0.0044+0.0044 for V→L, with all controlled gaps significant at Wilcoxon p≤3×10−5p\leq3\times10^{-5}. Ceiling normalization and the stricter reciprocal-anchor control give the same qualitative conclusion.

(2) Could the result be caused by dataset composition and semantic granularity?

The submission directly tests the proposed semantic-granularity mechanism. ShapeNet uses 1,024 unique instance-level Cap3D captions averaging 49.9 words, not repeated category labels. Shortening the descriptions from approximately 50 words to instance-specific phrases of approximately 5 words changes Δ\Delta by only −0.005-0.005, within the empirical noise floor. A qualitative inflation occurs only for bare one-word class names (Δ\Delta=+0.179, roughly ten times the paper's operating effect), which are never used in the main experiment:

Text levelMean Δ\DeltaFraction with Δ>0\Delta>0 / Δ>0\Delta>0Mean words
Full description (used in the paper)+0.017+0.0171.0050
First sentence+0.019+0.0191.0017
Brief instance-level description+0.012+0.0120.775
Class name+0.179+0.1790.051

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.