- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Sensorimotor Alignment (S²), a training-free offline metric for assessing embodied representations. Given paired sensory observations and demonstration trajectories, it compares a cosine-similarity structure over model features with a trajectory-based motor structure using SO(3)-aware dynamic time warping and Centered Kernel Nearest-Neighbor Alignment. The authors report positive associations between S² and downstream success for fine-tuned VLAs, raw VLM backbones subsequently fine-tuned into policies, WAMs, and policies with published real-robot evaluations. Additional analyses examine sensitivity to reference datasets and implementation choices, as well as relationships with feature modality, vision-encoder fine-tuning, denoising progress, and network depth. The intended contribution is a low-cost representation-based screening signal complementary to, rather than a replacement for, closed-loop evaluation.
Critical strengths
1. A practically relevant evaluation question supported by promising empirical associations.
The paper addresses an important practical problem: how to prioritize candidate embodied models before investing in expensive policy training or closed-loop evaluation. Comparing sensory representations with demonstrated motor structure provides a concrete, action-grounded alternative to generic perception or generation metrics.
The association across 12 VLAs is encouraging, with Pearson r = 0.717 and Spearman ρ = 0.802 under the default setting. The raw-VLM experiment is particularly useful because the downstream fine-tuning recipe is held fixed across the eight backbones, reducing an important source of variation. These results support further investigation of S² as an offline screening tool.
2. The paper includes meaningful robustness analyses rather than relying on a single favorable correlation.
The reference-dataset comparison, fixed-anchor horizon sweep, component and pooling ablations, and architecture-group removal analysis provide useful evidence that the VLA result is not confined to one narrow implementation choice.
I also appreciate the additional frozen-backbone study and the explicit reporting of less favorable cases. In particular, the relationship weakens as the number of behavior-cloning demonstrations increases and is not positive in the reported reinforcement-learning settings. Reporting these results helps clarify the regimes in which the metric may or may not be useful and makes the empirical study more informative.
Critical weaknesses
1. The incremental value of trajectory-based alignment over simpler state- or task-based controls is not sufficiently established.
The central interpretation of S² is that it captures how sensory representations organize episodes according to the physical behaviors they require. However, the current experiments do not sufficiently distinguish this from alignment with the current robot configuration, task identity, or scene grouping.
The reference trajectories include absolute end-effector poses and gripper states. Notably, the correlation is already 0.692 at h = 1, compared with 0.717 at the default h = 7. In addition, removing DTW at h = 7 leaves the correlation nearly unchanged at 0.713. These results do not imply that future trajectory information is unnecessary, but they make an anchor-state-only motor baseline particularly informative.
I appreciate the anchor-relative and component-weight ablations in Appendix C.1, but these do not directly test whether the trajectory-based metric provides predictive information beyond current pose, gripper state, or task/scene identity. Comparisons with VQA and video-quality metrics address a different question.
Without these more direct controls, it remains unclear how much of the proposed metric's utility specifically comes from motor-trajectory structure. This matters both for establishing its incremental value over simpler baselines and for interpreting the proposed notion of sensorimotor alignment.
2. The broader model-selection claim requires stronger validation and clearer reconciliation of the WAM results.
The VLA correlations are promising, but several of the broader conclusions rely on very small model cohorts: six WAMs, five real-robot policies, and only three WAMs in the comparison against video-quality metrics. In particular, a near-perfect correlation across only three models does not by itself establish reliable superiority over alternative metrics.
The architecture-group removal study is useful, but it mainly evaluates the stability of the pooled VLA association after removing one model family. It does not directly demonstrate model-selection performance on an unseen model group under a fixed evaluation protocol.
There is also an important dependence on the reference dataset and feature-extraction setting that should be clarified. Figure 5 reports an ALOHA-based ranking that is broadly consistent with WAM success, whereas the RoboTwin analysis in Table 7 assigns Vidar a higher final-layer-normalized S² than Motus and LingBot-VA at every denoising point, despite Vidar having the lowest success rate in Figure 5.
These analyses may use different reference datasets, feature-extraction procedures, or normalization settings, so I do not interpret this as a direct contradiction. However, the paper should explain the source of this difference and clarify under which fixed reference and feature-extraction conditions S² reliably supports cross-model ranking. This is important for its proposed use as a practical model-selection proxy.
Questions for authors
1. What predictive information is added by the trajectory?
Could the authors compare S² against an otherwise matched motor kernel constructed using only the anchor-frame end-effector pose and gripper state? A task- or scene-controlled analysis, for example by evaluating within matched groups, would further help determine whether the observed relationship depends on motor structure beyond coarse episode identity. Does the trajectory-based metric consistently improve over these simpler controls?
2. How stable is WAM model ranking across reference datasets and feature-extraction settings?
Please clarify the relationship between the ALOHA-based scores reported in Figures 3 and 5 and the RoboTwin-based scores in Table 7. When feature layer, denoising point, trajectory horizon, and neighborhood size are held fixed, do the same WAMs retain a ranking consistent with downstream success across both reference datasets? In particular, what explains Vidar's higher final-layer-normalized S² in Table 7 despite its lower success rate in Figure 5?
3. What is the exact computation of the motor kernel and the final alignment score?
Section 2 defines the sensory kernel using cosine similarity, while the motor kernel is written directly in terms of DTW distance. As written, larger values therefore have opposite interpretations on the two sides. Could the authors specify the exact distance-to-similarity transformation, if any, the direction used for nearest-neighbor selection, and the centering and normalization applied in the final alignment computation?
An explicit formula or pseudocode, together with a precise description of how the bimanual trajectories are constructed, would make the metric substantially easier to reproduce independently.
4. Could the statistical significance in Figure 5 be clarified?
Figure 5 reports r = 0.999 and p = 0.001 based on three WAMs. Given the very small number of models, could the authors clarify how the p-value was computed and what was treated as the independent sampling unit?
3 comments
Response to Reviewer d5Yj (2/3): W1/Q1 continued and W2/Q2
W1 + Q1 (continued)
3. Is the correlation driven by real motor structure or by coarse episode identity?
A task- or scene-controlled analysis, for example by evaluating within matched groups, would further help determine whether the observed relationship depends on motor structure beyond coarse episode identity.
The correlation comes from real motor structure beyond coarse episode identity. We thank the reviewer for suggesting a matched-group analysis and ran a within-group trajectory shuffle study.
Following the main VLA evaluation in our paper, we compute sensorimotor alignment () for all 12 fine-tuned VLAs on DROID. We group DROID episodes by shared lab, scene, task, or both scene and task, and shuffle motor trajectories across episodes within each group while keeping each episode's image and text fixed. This shuffle breaks each episode's actual motion while keeping its coarse lab, scene, or task identity intact. Recomputing on these shuffled pairs gives the shuffled : a higher value means the model gets alignment just from matching the shared group label, while a lower value means coarse group identity contributes little. Taking original - shuffled removes this group-level baseline and leaves the alignment from each episode's own trajectory.
DROID correlation with task success (Pearson )
| Matched group | Shuffled (coarse identity) | Original - shuffled (episode motion) |
|---|---|---|
| Lab | 0.107 | 0.680 |
| Scene | 0.220 | 0.741 |
| Task | 0.022 | 0.679 |
| Scene and task | 0.329 | 0.755 |
We check what still predicts task success after we scramble the motion. Shuffling pairs each episode with a random episode's motion from the same group, so the model can only rely on the shared lab, scene, or task. On these shuffled pairs, barely correlates with success at to : knowing the group alone tells us almost nothing about which VLA is better. The gap between the original and shuffled is the extra alignment that appears only when each episode keeps its own real motion, and this gap correlates strongly with success in every group, at to . The predictive signal comes from real motor structure, not from coarse episode identity.
We have added this matched-group analysis to Appendix B.2 of the revised manuscript.
W2 + Q2
The broader model-selection claim requires stronger validation and clearer reconciliation of the WAM results. How stable is WAM model ranking across reference datasets and feature-extraction settings?
... There is also an important dependence on the reference dataset and feature-extraction setting that should be clarified. Figure 5 reports an ALOHA-based ranking that is broadly consistent with WAM success, whereas the RoboTwin analysis in Table 7 assigns Vidar a higher final-layer-normalized S² than Motus and LingBot-VA at every denoising point, despite Vidar having the lowest success rate in Figure 5. ... The paper should explain the source of this difference and clarify under which fixed reference and feature-extraction conditions S² reliably supports cross-model ranking. ... In particular, what explains Vidar's higher final-layer-normalized S² in Table 7 despite its lower success rate in Figure 5?
Answer
We thank the reviewer for raising the gap between Figure 5 and Table 7. As the reviewer noted, this difference comes from the reference dataset: to rank models, we compute on a diverse, broad-coverage reference dataset such as ALOHA in Figures 3c and 5, not on demonstrations from the target task.
Table 7 instead computes on narrow RoboTwin rollouts solely as a within-model diagnostic across denoising steps and layers, so its scores, including Vidar's final-layer-normalized score, are not meant for comparison across models. We have clarified both this usage rule and Table 7's role in the revised manuscript.
Response to Reviewer d5Yj (3/3): Q3 and Q4
Q3
What is the exact computation of the motor kernel and the final alignment score?
Section 2 defines the sensory kernel using cosine similarity, while the motor kernel is written directly in terms of DTW distance. As written, larger values therefore have opposite interpretations on the two sides.
We thank the reviewer for catching this. We have corrected it in the revised Section 2, which now writes both sides as distances, so a smaller value always means closer neighbors.
Could the authors specify the exact distance-to-similarity transformation, if any, the direction used for nearest-neighbor selection, and the centering and normalization applied in the final alignment computation? An explicit formula or pseudocode ... would make the metric substantially easier to reproduce independently.
We compute by ranking neighbors from smallest to largest distance in sensory and motor space, taking the overlap of the two -nearest-neighbor sets, and centering and normalizing that overlap with the CKNNA estimator ( unless stated otherwise).
We first build the two representations. The sensory representation is the image and text features of episode from the model's vision and text encoders, average-pooled into one vector. The motor representation stacks the end-effector position, orientation, and gripper state over the next steps. For the bimanual ALOHA dataset, instead stacks the recorded 14-dimensional joint position per step, six joints and one gripper per arm.
Using the distances
we select for each episode the nearest neighbors and by smallest distance, and compute their mean overlap:
In practice, our reported uses a centered, normalized variant of the KNN overlap above (the CKNNA estimator). We give the full formulation, including the centering and normalization, in the revised Section 2 and Appendix A, and we will release our full code upon acceptance.
Q4
Could the statistical significance in Figure 5 be clarified?
Figure 5 reports
r = 0.999andp = 0.001based on three WAMs. Given the very small number of models, could the authors clarify how the p-value was computed and what was treated as the independent sampling unit?
Answer
We thank the reviewer for this question. In Figure 5, the independent sampling unit is the model ( WAMs), and the -values came from a one-tailed Pearson correlation test. We agree that 3 samples are too few for a reliable significance test or a stable estimate of : Figure 5 now reports Pearson without -values, and its caption and Section 3.2 state that each point is one model and that the comparison shows a trend.
Figure 5 includes only 3 of the 6 open-source WAMs in our study because video-quality metrics require decoded future video, and only Motus, LingBot-VA, and Vidar generate future video during closed-loop evaluation; the other 3 decode actions directly without generating video.
Response to Reviewer d5Yj (1/3): W1 and Q1 — horizons and DTW
W1 + Q1
We thank the reviewer for these questions.
1. Does extending the horizon and adding DTW raise S2?
Yes. At h=1 our metric is already the matched anchor-state-only kernel the reviewer asks for. Here h is the number of trajectory steps, so at h=1 the motor reference is only the anchor-frame end-effector pose and gripper state, and Dynamic Time Warping (DTW) reduces to the distance between two anchor states.
Extending the horizon and adding DTW both raise S2. Moving from this single anchor state to a long trajectory raises sensorimotor alignment (S2) by up to 3.25× on average across the 12 VLAs on DROID, and adding DTW on top of the trajectory raises S2 by up to a further 34.1% over the same trajectory matched without DTW. The table below reports S2 across horizons with and without DTW.
DROID sensorimotor alignment S2 (12 fine-tuned VLAs, samples N=6,000, neighbors k=10)
2. Does the trajectory improve the correlation with task success?
Yes, in part. The trajectory adds predictive information for task success. The clearest evidence is on BridgeDataV2: the anchor-only state at h=1 does not reach significance at one-tailed p=0.078, while every multi-step trajectory with h≥3 does.
As the reviewer noticed, on DROID the correlation r barely changes when we shorten the horizon or drop DTW. The reason is that each change lowers sensorimotor alignment by a similar proportion across all 12 models. A common scaling across models leaves Pearson r unchanged.
We report the horizon and DTW ablations on DROID and BridgeDataV2 below, and have added them to Appendix C of the revised manuscript.
DROID correlation with task success (12 fine-tuned VLAs, samples N=6,000, neighbors k=10)
BridgeDataV2 correlation with task success (12 fine-tuned VLAs, samples N=3,245, neighbors k=10; trajectories available up to h=40)