acceptodds
Under review as a conference paper at ICLR 2027

Sensorimotor Alignment: A Training-Free Predictor of Robot Policy Success

Abstract

Robotic foundation models are typically evaluated through costly closed-loop rollouts, which motivates cheap offline proxies for model capability. The Platonic Representation Hypothesis (PRH) suggests that as representations improve, different modalities such as vision and language converge toward a shared statistical model of reality. We ask whether motion trajectories form another such modality: as robot models become more capable, should their sensory representations converge toward the structure of physical action? To examine this, we propose **Sensorimotor Alignment** (), a training-free offline metric that measures how well a model’s sensory representation aligns with the geometry of physical action by comparing sensory and trajectory similarity structures. Crucially, consistently and strongly correlates with downstream robot task success across embodied model families, making it a low-cost evaluation proxy. It can also identify promising VLM backbones even before costly VLA fine-tuning. Moreover, we show that , as a diagnostic lens, can explain how design choices shape where control-relevant structure is represented in the models. More broadly, offers to bring PRH into embodiment, turning sensorimotor convergence into a measurable object of study.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, robot learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d5YjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Sensorimotor Alignment (S²), a training-free offline metric for assessing embodied representations. Given paired sensory observations and demonstration trajectories, it compares a cosine-similarity structure over model features with a trajectory-based motor structure using SO(3)-aware dynamic time warping and Centered Kernel Nearest-Neighbor Alignment. The authors report positive associations between S² and downstream success for fine-tuned VLAs, raw VLM backbones subsequently fine-tuned into policies, WAMs, and policies with published real-robot evaluations. Additional analyses examine sensitivity to reference datasets and implementation choices, as well as relationships with feature modality, vision-encoder fine-tuning, denoising progress, and network depth. The intended contribution is a low-cost representation-based screening signal complementary to, rather than a replacement for, closed-loop evaluation.

Critical strengths

1. A practically relevant evaluation question supported by promising empirical associations.

The paper addresses an important practical problem: how to prioritize candidate embodied models before investing in expensive policy training or closed-loop evaluation. Comparing sensory representations with demonstrated motor structure provides a concrete, action-grounded alternative to generic perception or generation metrics.

The association across 12 VLAs is encouraging, with Pearson r = 0.717 and Spearman ρ = 0.802 under the default setting. The raw-VLM experiment is particularly useful because the downstream fine-tuning recipe is held fixed across the eight backbones, reducing an important source of variation. These results support further investigation of S² as an offline screening tool.

2. The paper includes meaningful robustness analyses rather than relying on a single favorable correlation.

The reference-dataset comparison, fixed-anchor horizon sweep, component and pooling ablations, and architecture-group removal analysis provide useful evidence that the VLA result is not confined to one narrow implementation choice.

I also appreciate the additional frozen-backbone study and the explicit reporting of less favorable cases. In particular, the relationship weakens as the number of behavior-cloning demonstrations increases and is not positive in the reported reinforcement-learning settings. Reporting these results helps clarify the regimes in which the metric may or may not be useful and makes the empirical study more informative.

Critical weaknesses

1. The incremental value of trajectory-based alignment over simpler state- or task-based controls is not sufficiently established.

The central interpretation of S² is that it captures how sensory representations organize episodes according to the physical behaviors they require. However, the current experiments do not sufficiently distinguish this from alignment with the current robot configuration, task identity, or scene grouping.

The reference trajectories include absolute end-effector poses and gripper states. Notably, the correlation is already 0.692 at h = 1, compared with 0.717 at the default h = 7. In addition, removing DTW at h = 7 leaves the correlation nearly unchanged at 0.713. These results do not imply that future trajectory information is unnecessary, but they make an anchor-state-only motor baseline particularly informative.

I appreciate the anchor-relative and component-weight ablations in Appendix C.1, but these do not directly test whether the trajectory-based metric provides predictive information beyond current pose, gripper state, or task/scene identity. Comparisons with VQA and video-quality metrics address a different question.

Without these more direct controls, it remains unclear how much of the proposed metric's utility specifically comes from motor-trajectory structure. This matters both for establishing its incremental value over simpler baselines and for interpreting the proposed notion of sensorimotor alignment.

2. The broader model-selection claim requires stronger validation and clearer reconciliation of the WAM results.

The VLA correlations are promising, but several of the broader conclusions rely on very small model cohorts: six WAMs, five real-robot policies, and only three WAMs in the comparison against video-quality metrics. In particular, a near-perfect correlation across only three models does not by itself establish reliable superiority over alternative metrics.

The architecture-group removal study is useful, but it mainly evaluates the stability of the pooled VLA association after removing one model family. It does not directly demonstrate model-selection performance on an unseen model group under a fixed evaluation protocol.

There is also an important dependence on the reference dataset and feature-extraction setting that should be clarified. Figure 5 reports an ALOHA-based ranking that is broadly consistent with WAM success, whereas the RoboTwin analysis in Table 7 assigns Vidar a higher final-layer-normalized S² than Motus and LingBot-VA at every denoising point, despite Vidar having the lowest success rate in Figure 5.

These analyses may use different reference datasets, feature-extraction procedures, or normalization settings, so I do not interpret this as a direct contradiction. However, the paper should explain the source of this difference and clarify under which fixed reference and feature-extraction conditions S² reliably supports cross-model ranking. This is important for its proposed use as a practical model-selection proxy.

Questions for authors

1. What predictive information is added by the trajectory?

Could the authors compare S² against an otherwise matched motor kernel constructed using only the anchor-frame end-effector pose and gripper state? A task- or scene-controlled analysis, for example by evaluating within matched groups, would further help determine whether the observed relationship depends on motor structure beyond coarse episode identity. Does the trajectory-based metric consistently improve over these simpler controls?

2. How stable is WAM model ranking across reference datasets and feature-extraction settings?

Please clarify the relationship between the ALOHA-based scores reported in Figures 3 and 5 and the RoboTwin-based scores in Table 7. When feature layer, denoising point, trajectory horizon, and neighborhood size are held fixed, do the same WAMs retain a ranking consistent with downstream success across both reference datasets? In particular, what explains Vidar's higher final-layer-normalized S² in Table 7 despite its lower success rate in Figure 5?

3. What is the exact computation of the motor kernel and the final alignment score?

Section 2 defines the sensory kernel using cosine similarity, while the motor kernel is written directly in terms of DTW distance. As written, larger values therefore have opposite interpretations on the two sides. Could the authors specify the exact distance-to-similarity transformation, if any, the direction used for nearest-neighbor selection, and the centering and normalization applied in the final alignment computation?

An explicit formula or pseudocode, together with a precise description of how the bimanual trajectories are constructed, would make the metric substantially easier to reproduce independently.

4. Could the statistical significance in Figure 5 be clarified?

Figure 5 reports r = 0.999 and p = 0.001 based on three WAMs. Given the very small number of models, could the authors clarify how the p-value was computed and what was treated as the independent sampling unit?

3 comments
AuthorsOpenReview ↗

Response to Reviewer d5Yj (1/3): W1 and Q1 — horizons and DTW

W1 + Q1

We thank the reviewer for these questions.

1. Does extending the horizon and adding DTW raise S2S^2?

Could the authors compare S² against an otherwise matched motor kernel constructed using only the anchor-frame end-effector pose and gripper state?

Yes. At h=1h=1 our metric is already the matched anchor-state-only kernel the reviewer asks for. Here hh is the number of trajectory steps, so at h=1h=1 the motor reference is only the anchor-frame end-effector pose and gripper state, and Dynamic Time Warping (DTW) reduces to the distance between two anchor states.

Extending the horizon and adding DTW both raise S2S^2. Moving from this single anchor state to a long trajectory raises sensorimotor alignment (S2S^2) by up to 3.25×3.25\times on average across the 12 VLAs on DROID, and adding DTW on top of the trajectory raises S2S^2 by up to a further 34.134.1% over the same trajectory matched without DTW. The table below reports S2S^2 across horizons with and without DTW.

DROID sensorimotor alignment S2S^2 (12 fine-tuned VLAs, samples N=6,000N=6{,}000, neighbors k=10k=10)

h=1h=1h=3h=3h=7h=7h=15h=15h=25h=25h=40h=40h=75h=75h=110h=110h=150h=150h=220h=220
Mean S2S^2 (with DTW)0.02680.02680.02800.03140.03420.04100.05370.06530.07580.0870
Relative to h=1h=1 (with DTW)1.00×1.00×1.05×1.17×1.28×1.53×2.01×2.44×2.83×3.25×
Mean S2S^2 (without DTW)0.02680.02690.02770.03020.03170.03650.04420.05330.06020.0649
DTW change vs. without DTW+0.0%-0.6%+1.2%+4.1%+7.7%+12.4%+21.7%+22.4%+25.9%+34.1%

2. Does the trajectory improve the correlation with task success?

What predictive information is added by the trajectory? ... it remains unclear how much of the proposed metric's utility specifically comes from motor-trajectory structure.

Yes, in part. The trajectory adds predictive information for task success. The clearest evidence is on BridgeDataV2: the anchor-only state at h=1h=1 does not reach significance at one-tailed p=0.078p=0.078, while every multi-step trajectory with h≥3h \ge 3 does.

As the reviewer noticed, on DROID the correlation rr barely changes when we shorten the horizon or drop DTW. The reason is that each change lowers sensorimotor alignment by a similar proportion across all 12 models. A common scaling across models leaves Pearson rr unchanged.

We report the horizon and DTW ablations on DROID and BridgeDataV2 below, and have added them to Appendix C of the revised manuscript.

DROID correlation with task success (12 fine-tuned VLAs, samples N=6,000N=6{,}000, neighbors k=10k=10)

h=1h=1h=3h=3h=7h=7h=15h=15h=25h=25h=40h=40h=75h=75h=110h=110h=150h=150h=220h=220
Pearson rr (with DTW)0.6920.6990.7170.6960.6890.6830.6470.6460.6290.610
One-tailed pp (with DTW)0.0060.0060.0040.0060.0070.0070.0120.0120.0140.018
Pearson rr (without DTW)0.6920.7000.7130.7100.7110.6800.6630.6610.6330.617

BridgeDataV2 correlation with task success (12 fine-tuned VLAs, samples N=3,245N=3{,}245, neighbors k=10k=10; trajectories available up to h=40h=40)

h=1h=1h=3h=3h=7h=7h=15h=15h=25h=25h=40h=40
Mean S2S^20.0340.0480.0480.0480.0560.056
Relative to h=1h=11.00×1.42×1.41×1.41×1.64×1.65×
Pearson rr0.4370.6130.6690.7000.6870.681
One-tailed pp0.0780.0170.0090.0060.0070.007
AuthorsOpenReview ↗

Response to Reviewer d5Yj (2/3): W1/Q1 continued and W2/Q2

W1 + Q1 (continued)

3. Is the correlation driven by real motor structure or by coarse episode identity?

A task- or scene-controlled analysis, for example by evaluating within matched groups, would further help determine whether the observed relationship depends on motor structure beyond coarse episode identity.

The correlation comes from real motor structure beyond coarse episode identity. We thank the reviewer for suggesting a matched-group analysis and ran a within-group trajectory shuffle study.

Following the main VLA evaluation in our paper, we compute sensorimotor alignment (S2S^2) for all 12 fine-tuned VLAs on DROID. We group DROID episodes by shared lab, scene, task, or both scene and task, and shuffle motor trajectories across episodes within each group while keeping each episode's image and text fixed. This shuffle breaks each episode's actual motion while keeping its coarse lab, scene, or task identity intact. Recomputing S2S^2 on these shuffled pairs gives the shuffled S2S^2: a higher value means the model gets alignment just from matching the shared group label, while a lower value means coarse group identity contributes little. Taking original S2S^2 - shuffled S2S^2 removes this group-level baseline and leaves the alignment from each episode's own trajectory.

DROID correlation with task success (Pearson rr)

Matched groupShuffled S2S^2 (coarse identity)Original S2S^2 - shuffled S2S^2 (episode motion)
Lab0.1070.680
Scene0.2200.741
Task0.0220.679
Scene and task0.3290.755

We check what still predicts task success after we scramble the motion. Shuffling pairs each episode with a random episode's motion from the same group, so the model can only rely on the shared lab, scene, or task. On these shuffled pairs, S2S^2 barely correlates with success at r=0.022r = 0.022 to 0.3290.329: knowing the group alone tells us almost nothing about which VLA is better. The gap between the original and shuffled S2S^2 is the extra alignment that appears only when each episode keeps its own real motion, and this gap correlates strongly with success in every group, at r=0.679r = 0.679 to 0.7550.755. The predictive signal comes from real motor structure, not from coarse episode identity.

We have added this matched-group analysis to Appendix B.2 of the revised manuscript.

W2 + Q2

The broader model-selection claim requires stronger validation and clearer reconciliation of the WAM results. How stable is WAM model ranking across reference datasets and feature-extraction settings?

... There is also an important dependence on the reference dataset and feature-extraction setting that should be clarified. Figure 5 reports an ALOHA-based ranking that is broadly consistent with WAM success, whereas the RoboTwin analysis in Table 7 assigns Vidar a higher final-layer-normalized S² than Motus and LingBot-VA at every denoising point, despite Vidar having the lowest success rate in Figure 5. ... The paper should explain the source of this difference and clarify under which fixed reference and feature-extraction conditions S² reliably supports cross-model ranking. ... In particular, what explains Vidar's higher final-layer-normalized S² in Table 7 despite its lower success rate in Figure 5?

Answer

We thank the reviewer for raising the gap between Figure 5 and Table 7. As the reviewer noted, this difference comes from the reference dataset: to rank models, we compute S2S^2 on a diverse, broad-coverage reference dataset such as ALOHA in Figures 3c and 5, not on demonstrations from the target task.

Table 7 instead computes S2S^2 on narrow RoboTwin rollouts solely as a within-model diagnostic across denoising steps and layers, so its scores, including Vidar's final-layer-normalized score, are not meant for comparison across models. We have clarified both this usage rule and Table 7's role in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer d5Yj (3/3): Q3 and Q4

Q3

What is the exact computation of the motor kernel and the final alignment score?

Section 2 defines the sensory kernel using cosine similarity, while the motor kernel is written directly in terms of DTW distance. As written, larger values therefore have opposite interpretations on the two sides.

We thank the reviewer for catching this. We have corrected it in the revised Section 2, which now writes both sides as distances, so a smaller value always means closer neighbors.

Could the authors specify the exact distance-to-similarity transformation, if any, the direction used for nearest-neighbor selection, and the centering and normalization applied in the final alignment computation? An explicit formula or pseudocode ... would make the metric substantially easier to reproduce independently.

We compute S2S^2 by ranking neighbors from smallest to largest distance in sensory and motor space, taking the overlap of the two kk-nearest-neighbor sets, and centering and normalizing that overlap with the CKNNA estimator (k=10k=10 unless stated otherwise).

We first build the two representations. The sensory representation zi\mathbf{z}_i is the image and text features of episode ii from the model's vision and text encoders, average-pooled into one vector. The motor representation τi\boldsymbol{\tau}_i stacks the end-effector position, orientation, and gripper state over the next hh steps. For the bimanual ALOHA dataset, τi\boldsymbol{\tau}_i instead stacks the recorded 14-dimensional joint position per step, six joints and one gripper per arm.

Using the distances

dsensory(i,j)=1−cos⁡(zi,zj),dmotor(i,j)=DTW⁡(τi,τj),d_{\mathrm{sensory}}(i,j)=1-\cos(\mathbf{z}_i,\mathbf{z}_j), \qquad d_{\mathrm{motor}}(i,j)=\operatorname{DTW}(\boldsymbol{\tau}_i,\boldsymbol{\tau}_j),

we select for each episode ii the kk nearest neighbors Nsensory(i)\mathcal{N}_{\mathrm{sensory}}(i) and Nmotor(i)\mathcal{N}_{\mathrm{motor}}(i) by smallest distance, and compute their mean overlap:

mNN=1Nk∑i=1N∣Nsensory(i)∩Nmotor(i)∣.m_{\mathrm{NN}} = \frac{1}{Nk}\sum_{i=1}^{N}\left|\mathcal{N}_{\mathrm{sensory}}(i)\cap\mathcal{N}_{\mathrm{motor}}(i)\right|.

In practice, our reported S2S^2 uses a centered, normalized variant of the KNN overlap above (the CKNNA estimator). We give the full formulation, including the centering and normalization, in the revised Section 2 and Appendix A, and we will release our full code upon acceptance.

Q4

Could the statistical significance in Figure 5 be clarified?

Figure 5 reports r = 0.999 and p = 0.001 based on three WAMs. Given the very small number of models, could the authors clarify how the p-value was computed and what was treated as the independent sampling unit?

Answer

We thank the reviewer for this question. In Figure 5, the independent sampling unit is the model (n=3n=3 WAMs), and the pp-values came from a one-tailed Pearson correlation test. We agree that 3 samples are too few for a reliable significance test or a stable estimate of rr: Figure 5 now reports Pearson rr without pp-values, and its caption and Section 3.2 state that each point is one model and that the comparison shows a trend.

Figure 5 includes only 3 of the 6 open-source WAMs in our study because video-quality metrics require decoded future video, and only Motus, LingBot-VA, and Vidar generate future video during closed-loop evaluation; the other 3 decode actions directly without generating video.

Reviewer_2qUDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Sensorimotor Alignment (S²), an offline metric that measures the alignment between sensory representations and demonstrated motion trajectories. It constructs a sensory similarity matrix from vision-language features and a trajectory relationship matrix using SO(3)-aware DTW, then compares them with CKNNA. Experiments examine the association between S² and task success across VLAs, raw VLMs, WAMs, and policies evaluated on real robots. The paper also analyzes how S² changes with fine-tuning, modality, denoising stage, and network depth.

The main contribution is a simple, training-free signal for assessing robot-relevant representations, particularly for choosing VLM backbones before robot fine-tuning. The results are promising, but the added value of full trajectory alignment over simpler state-based measures remains unclear.

Critical strengths

S1. The paper addresses a practical evaluation problem with a simple approach.

Closed-loop robot evaluation is expensive. Computing a representation metric on fixed offline demonstrations could help prioritize candidate models without training an additional evaluator. Applying it to raw VLMs before robot fine-tuning is particularly useful.

S2. The main VLA result has substantial supporting analysis.

Across 12 VLAs, the paper reports a Pearson correlation of 0.717 and a Spearman correlation of 0.802. The authors examine different reference datasets, trajectory horizons, neighborhood sizes, component weights, rotation distances, and pooling methods. These checks show that the positive association is not confined to one narrow configuration.

S3. The raw-VLM experiment provides a more controlled comparison.

The common VLM4VLA fine-tuning recipe makes the association between pre-fine-tuning S² and downstream success more informative than a comparison of policies trained under different procedures. The appendix also reports weaker correlations with more BC data and no positive correlation in some RL settings, helping establish the metric's scope.

Critical weaknesses

W1. The added value of future motion over the current state is unclear.

Table 3 reports a correlation of 0.692 at h = 1, compared with 0.717 at h = 7. If h = 1 contains only the anchor state, much of the predictive signal is already present without future motion. The anchor-relative result in Table 2 is helpful, but retains absolute orientation and gripper state. These results support the usefulness of S², while leaving open how much it gains from future trajectories. Clarifying the h = 1 definition and discussing this comparison would help establish what the metric captures and where its main contribution lies.

W2. The DTW-to-CKNNA computation is not fully specified.

Section 2 defines the sensory matrix using cosine similarity and the motor matrix using DTW distance. The paper should explain how these matrices enter CKNNA, particularly how their opposite similarity conventions are handled in neighbor selection and score computation. Appendix C.1 provides the distance components and neighborhood size, but does not resolve this step. A formula or short pseudocode would address this reproducibility issue.

W3. Some interpretations go beyond what the comparisons establish.

The visual/text-token analysis measures alignment at different token positions; it does not directly measure how much a policy relies on vision or language during execution. The controlled freezing and denoising experiments provide useful behavioral evidence, but do not establish alignment as the mechanism behind the performance changes. Separately, crowded and overlapping labels in Figures 3, 6, and 8 make some results difficult to read.

Questions for authors

Q1. Does h = 1 contain only the anchor state? If so, what additional predictive value do future trajectories provide?

Q2. How is the DTW distance matrix incorporated into CKNNA, including neighbor selection and score computation?

Q3. What behavioral evidence supports interpreting visual/text-token alignment as a measure of dependence on vision versus language?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2qUD (1/2): W1 and Q1

W1 + Q1

We thank the reviewer for this question.

1. Does h=1h=1 contain only the anchor state?

Does h = 1 contain only the anchor state? ... Clarifying the h = 1 definition ... would help establish what the metric captures and where its main contribution lies.

Yes. At h=1h=1 the motor reference is only the single anchor-frame end-effector pose and gripper state, and Dynamic Time Warping (DTW) reduces to the distance between two anchor states.

2. What additional predictive value do future trajectories provide?

If so, what additional predictive value do future trajectories provide? ... These results support the usefulness of S², while leaving open how much it gains from future trajectories.

Switching from the anchor state to a future trajectory adds predictive information for task success. The clearest evidence is on BridgeDataV2: the anchor-only state at h=1h=1 does not reach significance at one-tailed p=0.078p=0.078, while every multi-step trajectory with h≥3h \ge 3 does.

BridgeDataV2 (12 fine-tuned VLAs, samples N=3,245N=3{,}245, neighbors k=10k=10; trajectories available up to h=40h=40)

h=1h=1h=3h=3h=7h=7h=15h=15h=25h=25h=40h=40
Mean S2S^20.0340.0480.0480.0480.0560.056
Relative to h=1h=11.00×1.42×1.41×1.41×1.64×1.65×
Pearson rr0.4370.6130.6690.7000.6870.681
One-tailed pp0.0780.0170.0090.0060.0070.007

3. Why does the DROID correlation barely move from h=1h=1 to h=7h=7?

Table 3 reports a correlation of 0.692 at h = 1, compared with 0.717 at h = 7. If h = 1 contains only the anchor state, much of the predictive signal is already present without future motion.

On DROID the correlation rr barely changes as we extend the horizon from h=1h=1 to h=7h=7. The reason is that a longer horizon raises sensorimotor alignment by a similar proportion across all 12 models. A common scaling across models leaves Pearson rr unchanged.

DROID (12 fine-tuned VLAs, samples N=6,000N=6{,}000, neighbors k=10k=10)

h=1h=1h=3h=3h=7h=7h=15h=15h=25h=25h=40h=40h=75h=75h=110h=110h=150h=150h=220h=220
Mean S2S^20.0270.0270.0280.0310.0340.0410.0540.0650.0760.087
Relative to h=1h=11.00×1.00×1.05×1.17×1.28×1.53×2.01×2.44×2.83×3.25×
Pearson rr0.6920.6990.7170.6960.6890.6830.6470.6460.6290.610
One-tailed pp0.0060.0060.0040.0060.0070.0070.0120.0120.0140.018
AuthorsOpenReview ↗

Response to Reviewer 2qUD (2/2): W2–W3 and Q2–Q3

W2 + Q2

The DTW-to-CKNNA computation is not fully specified. How is the DTW distance matrix incorporated into CKNNA, including neighbor selection and score computation?

Section 2 defines the sensory matrix using cosine similarity and the motor matrix using DTW distance. The paper should explain how these matrices enter CKNNA, particularly how their opposite similarity conventions are handled in neighbor selection and score computation. ... A formula or short pseudocode would address this reproducibility issue.

Answer

We thank the reviewer for catching this. We have corrected it in the revised Section 2, which now writes both sides as distances, so a smaller value always means closer neighbors.

We compute S2S^2 by ranking neighbors from smallest to largest distance in sensory and motor space, taking the overlap of the two kk-nearest-neighbor sets, and centering and normalizing that overlap with the CKNNA estimator (k=10k=10 unless stated otherwise).

We first build the two representations. The sensory representation zi\mathbf{z}_i is the image and text features of episode ii from the model's vision and text encoders, average-pooled into one vector. The motor representation τi\boldsymbol{\tau}_i stacks the end-effector position, orientation, and gripper state over the next hh steps.

Using the distances

dsensory(i,j)=1−cos⁡(zi,zj),dmotor(i,j)=DTW⁡(τi,τj),d_{\mathrm{sensory}}(i,j)=1-\cos(\mathbf{z}_i,\mathbf{z}_j), \qquad d_{\mathrm{motor}}(i,j)=\operatorname{DTW}(\boldsymbol{\tau}_i,\boldsymbol{\tau}_j),

we select for each episode ii the kk nearest neighbors Nsensory(i)\mathcal{N}_{\mathrm{sensory}}(i) and Nmotor(i)\mathcal{N}_{\mathrm{motor}}(i) by smallest distance, and compute their mean overlap:

mNN=1Nk∑i=1N∣Nsensory(i)∩Nmotor(i)∣.m_{\mathrm{NN}} = \frac{1}{Nk}\sum_{i=1}^{N}\left|\mathcal{N}_{\mathrm{sensory}}(i)\cap\mathcal{N}_{\mathrm{motor}}(i)\right|.

In practice, our reported S2S^2 uses a centered, normalized variant of the KNN overlap above (the CKNNA estimator). We give the full formulation, including the centering and normalization, in the revised Section 2 and Appendix A, and we will release our full code upon acceptance.

W3 + Q3

Some interpretations go beyond what the comparisons establish. What behavioral evidence supports interpreting visual/text-token alignment as a measure of dependence on vision versus language?

The visual/text-token analysis measures alignment at different token positions; it does not directly measure how much a policy relies on vision or language during execution. The controlled freezing and denoising experiments provide useful behavioral evidence, but do not establish alignment as the mechanism behind the performance changes. ...

Answer

We thank the reviewer for raising this point. The reviewer is right that token-level S2S^2 does not directly measure how much a policy relies on vision or language during execution, and we have reframed this result as a representation-level finding.

We find that across the 12 fine-tuned VLAs, vision-motor alignment is relatively strong: S2S^2 computed on the visual tokens correlates with task success at r=0.694r=0.694. Language-motor alignment is weaker: S2S^2 on the text tokens does not correlate, at r=0.237r=0.237, which is not significant. So motor structure related to task success sits mainly in the visual tokens rather than the instruction tokens, which is consistent with prior reports of weak instruction grounding in VLAs. In the revised manuscript, we updated Section 4.1 (now titled "In fine-tuned VLAs, the correlation with task success comes mainly from visual features"), the Section 4 overview, and the Figure 8 caption, and removed the sentence stating that this pattern explains weak instruction grounding.

Thank you for the clarifications. Section 2 and Appendix A.3 now explain how DTW distances are used to select motor neighbors and how those neighbors enter the CKNNA score. This resolves my concern in W2. The revised discussion in Section 4.1 also addresses my main concern in W3. It makes clear what the visual- and text-token correlations tell us about the representations and acknowledges the limits of using them to infer how a policy relies on vision or language during execution.

For W1, I appreciate the clarification that h=1h = 1 uses only the anchor state. The BridgeDataV2 results are helpful, but on DROID the anchor state alone already gives a correlation with task success close to that obtained with a seven-frame trajectory (r=0.692r = 0.692 and 0.7170.717, respectively). I therefore still have some reservations about how much future motion adds beyond the current state, and I maintain my original score.

Reviewer_2pPbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Inspired by the Platonic Representation Hypothesis, this paper treats robot trajectories as another projection of physical reality and asks whether capable embodied models organize their sensory representations according to the structure of physical action. It proposes Sensorimotor Alignment (S²), a training-free metric that applies CKNNA between a cosine kernel over model features and an SO(3)-aware DTW kernel over ground-truth end-effector trajectories on offline datasets such as DROID. The paper reports positive correlations between S² and downstream success for fine-tuned VLAs, raw VLMs, WAMs, and real-robot policies, and uses S² to give representation-level explanations of several phenomena reported in prior work.

Critical strengths
  1. The problem is practically important. Closed-loop evaluation is expensive and noisy, and a cheap, training-free offline proxy grounded in robot behavior would be useful to the community.

  2. The study covers a broad range of model families (VLAs, raw VLMs, WAMs, frozen vision backbones, and real-robot policies) and includes several robustness checks, such as cross-dataset transfer, trajectory horizons, and leave-one-out over architecture groups.

  3. The finding that VQA scores show no correlation with downstream VLA success, while a behavior-grounded metric does, is an informative result.

Critical weaknesses

The idea is interesting, but the statistical evidence is weak and the key comparisons are missing.

  1. The statistical evidence is thin. Apart from the 12 VLAs, the correlations rely on very few models: 8 VLMs (one-tailed p = 0.031), 6 WAMs (p = 0.054, not significant), 5 real-robot policies (p = 0.041), and only 3 WAMs in Figure 5, where r = +0.999 carries little statistical meaning. All tests are one-tailed. In addition, the absolute S² values for VLAs (about 0.015–0.045) are only slightly above the shuffled baseline reported in the appendix (about 0.012), so the signal itself is weak.

  2. Strong baselines are missing. The paper compares S² only with VQA scores and video-quality metrics, which are not designed for control and are therefore weak baselines. More natural offline proxies, such as open-loop action prediction error on held-out episodes, linear probing of trajectories from the same features, other kernel-alignment metrics (e.g., CKA), or simply model size and pretraining data scale, are not evaluated. Without these comparisons, it is unclear what S² adds.

  3. The claimed practical use is not directly validated. The paper argues that S² can prioritize models before closed-loop evaluation, but it does not run an actual model-selection experiment, such as top-k selection accuracy, ranking regret, or the evaluation cost saved. Correlation across a handful of models does not by itself show that S² is a reliable selection tool.

  4. The representation-level explanations in Section 4 are largely post hoc. The frozen-encoder analysis compares only two models, and the observation that S² is already high at early denoising steps restates the conclusion of Fast-WAM rather than giving a new prediction. Together with the stretched PRH framing (the paper itself notes that the scaling claim cannot be tested in robotics), the technical contribution mainly amounts to applying CKNNA to a trajectory kernel.

Questions for authors

See Weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2pPb (1/2): W1 and W2

W1-1

The statistical evidence is thin. Apart from the 12 VLAs, the correlations rely on very few models: 8 VLMs (one-tailed p = 0.031)...

Answer

During the rebuttal, we evaluated 7 more VLMs, so the raw-VLM analysis now covers 15 VLMs and correlates with downstream task success at Pearson r=0.853r = 0.853 with one-tailed p=7.2×10−5p = 7.2\times10^{-5}. We detail these 7 VLMs in our response to W3 and have added the combined analysis to Appendix B.3 of the revised manuscript.

W1-2

...6 WAMs (p = 0.054, not significant), 5 real-robot policies (p = 0.041), and only 3 WAMs in Figure 5, where r = +0.999 carries little statistical meaning.

Answer

We agree that 3 samples are too few for a reliable significance test or a stable estimate of rr: Figure 5 now reports Pearson rr without pp-values, and its caption and Section 3.2 state that each point is one model and that the comparison shows a trend. Figure 5 includes only 3 of the 6 open-source WAMs in our study because video-quality metrics require decoded future video, and only Motus, LingBot-VA, and Vidar generate future video during closed-loop evaluation; the other 3 decode actions directly without generating video.

All tests are one-tailed.

Answer

We use one-tailed tests because our hypothesis is directional: we predict that higher S2S^2 goes with higher task success. On the 12 fine-tuned VLAs, the correlation stays significant under a two-tailed test at p=0.009p = 0.009, compared with p=0.004p = 0.004 one-tailed.

W1-3

In addition, the absolute S² values for VLAs (about 0.015–0.045) are only slightly above the shuffled baseline reported in the appendix (about 0.012), so the signal itself is weak.

Answer

The absolute VLA S2S^2 values are many times higher than their matched random baseline. The 0.0120.012 figure the reviewer cites comes from a 500500-episode subset, while the evaluation in our main paper uses a much larger set of 6,0006{,}000 episodes. Random neighbor overlap shrinks as the episode pool grows, and we measure the random baseline for VLAs at about 0.0010.001. So VLA S2S^2 values of about 0.0150.015 to 0.0450.045 are 9×9\times to 40×40\times above baseline across horizons. We have clarified how the random baseline scales with dataset size in Appendix A.3 of the revised manuscript.

W2

Strong baselines are missing. The paper compares S² only with VQA scores and video-quality metrics ... More natural offline proxies, such as open-loop action prediction error on held-out episodes, linear probing of trajectories from the same features, other kernel-alignment metrics (e.g., CKA), or simply model size and pretraining data scale, are not evaluated. Without these comparisons, it is unclear what S² adds.

Answer

We thank the reviewer for the suggestion.

We first keep the sensorimotor-alignment measure and change only the kernel from CKNNA, the nearest-neighbor alignment that S2S^2 uses, to linear or RBF CKA. Following the main evaluation in the paper, we compute each score on the 12 fine-tuned VLAs with the DROID reference and correlate it with their SimplerEnv task success rate. Sensorimotor alignment keeps a strong correlation with task success under every kernel.

Kernel (12 fine-tuned VLAs, DROID)Pearson rrSpearman ρ\rho
CKNNA (S2S^2, ours)+0.717+0.802
Linear CKA+0.663+0.662
RBF CKA+0.697+0.620

We then compare S2S^2 with the other three proxies the reviewer suggests: open-loop action error, a linear trajectory probe, and model size. None of the three predicts task success, while S2S^2 does.

Offline proxyPearson rrSpearman ρ\rho
S2S^2 (ours)+0.948+0.929
Open-loop action error (negated)-0.27-0.12
Linear trajectory probe+0.05-0.15
Model size-0.26-0.07

For S2S^2, we score each of the 7 raw VLM backbones before any robot training and correlate the score with the task success rate of the VLA fine-tuned from it.

For open-loop action error, we measure all 12 fine-tuned VLAs by comparing their predicted actions with the logged actions on held-out BridgeDataV2 episodes. We then test whether a lower open-loop error predicts a higher task success rate, and we find the correlation does not hold.

For the linear trajectory probe, we train a linear model to predict the robot's future trajectory from each raw VLM's features, the same features that S2S^2 uses, and record how accurately it predicts trajectories on held-out episodes. We then test whether a VLM with a more accurate probe gives a higher task success rate after fine-tuning, and we find the correlation does not hold.

For model size, we test whether a VLM with more parameters gives a higher task success rate after fine-tuning, and we find the correlation does not hold.

We have added these comparisons to Appendix B of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 2pPb (2/2): W3 and W4

W3

The claimed practical use is not directly validated. The paper argues that S² can prioritize models before closed-loop evaluation, but it does not run an actual model-selection experiment, such as top-k selection accuracy, ranking regret, or the evaluation cost saved. Correlation across a handful of models does not by itself show that S² is a reliable selection tool.

Answer

We thank the reviewer for this suggestion. We ran a controlled VLM backbone-selection experiment across 7 pretrained backbones and confirmed that offline S2S^2 prioritizes the strongest candidates before VLA fine-tuning and closed-loop evaluation.

Following the raw-VLM evaluation setup in our paper, we first measure S2S^2 offline across 7 pretrained VLM backbones on the DROID reference dataset before any robot training. We then pair all 7 backbones with the same GR00T flow-matching action head, fine-tune them end-to-end on BridgeDataV2 under identical settings, and report each model's best closed-loop task success rate across the four SimplerEnv WidowX manipulation tasks:

VLM BackboneRank by VLM S2S^2Rank by VLA Success RateVLM S2S^2VLA Best Success Rate (%)
Florence-2-Large110.02262172.92
Qwen3-VL-4B230.02163765.62
Molmo2-4B320.02159866.67
Qwen3.5-2B450.02111557.29
Cosmos-Reason2-2B540.02025163.54
PaliGemma 2-3B660.01943354.17
Qwen2.5-VL-3B770.01752639.58

Across the 7 backbones, raw-VLM S2S^2 correlates with downstream VLA task success rate at Pearson r=0.948r = 0.948 with one-tailed p=0.0006p = 0.0006.

We also report the three selection metrics raised by the reviewer:

  • Top-kk selection accuracy: This is the fraction of the top-kk backbones ranked by S2S^2 that are also in the true top-kk VLAs by task success. Top-1 (1/1) and top-3 (3/3) accuracy are both 100%.
  • Ranking regret: Regret measures the drop in best VLA success rate when we train only the top-kk backbones ranked by S2S^2 instead of all 7 candidates. Since Florence-2-Large ranks first in both offline S2S^2 and VLA success rate (72.92%), regret is 0% for every k≥1k \ge 1. Picking one backbone at random loses about 13% on average.
  • Training and evaluation cost saved: Fine-tuning and evaluating one VLA takes about 200 H100 GPU hours, while scoring one raw VLM offline with S2S^2 takes about 20 minutes on one GPU. Training only the top-1 backbone skips 6 of the 7 runs and saves about 86% of the compute, roughly 1,200 GPU hours. Training the top-3 skips 4 runs and saves about 57%, roughly 800 GPU hours..

We have added this controlled backbone-selection experiment to Appendix B.3 of the revised manuscript, and leave selection on larger model pools and other model families, such as WAMs and real-robot policies, to future work, as noted in the Limitations section.

W4-2

The representation-level explanations in Section 4 are largely post hoc. [...] the observation that S² is already high at early denoising steps restates the conclusion of Fast-WAM rather than giving a new prediction. Together with the stretched PRH framing (the paper itself notes that the scaling claim cannot be tested in robotics), the technical contribution mainly amounts to applying CKNNA to a trajectory kernel.

Answer

We thank the reviewer for this comment. We agree that S2S^2 builds on the CKNNA estimator and that the closed-loop behaviors analyzed in Section 4 were already reported in prior work. Our contribution is the two findings this metric enables:

  1. A training-free predictor of robot policy success. Computed offline from model representations and reference trajectories, S2S^2 tracks closed-loop task success across VLAs, raw VLMs, and WAMs without robot training or rollouts. Our new backbone-selection experiment tests this directly: S2S^2 measured on 7 raw VLMs before any robot fine-tuning correlates with downstream VLA task success at Pearson r=0.948r = 0.948.
  2. A diagnostic tool for embodied models. S2S^2 explains at the representation level why those closed-loop behaviors hold. For example, Fast-WAM showed that skipping iterative video denoising barely lowers task success, and S2S^2 shows why: WAM features already align strongly with robot trajectories at the earliest denoising step.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.