- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes learning transferable behavior embeddings from unlabeled robot manipulation videos to support policy training. The method discovers coherent video segments at multiple temporal granularities and jointly trains next-frame prediction, endpoint reconstruction, and status scoring objectives to capture motion, reached configurations, and execution status. A frozen encoder then generates offline targets for auxiliary prediction from policy action features, with the additional modules removed at inference. Experiments on RoboTwin 2.0 and Mobile ALOHA demonstrate improvements in task performance and under the tested instruction and appearance shifts, complemented by representation probes and ablations.
Critical strengths
-
The empirical results support the practical value of the approach. Improvements across multiple policy configurations and both simulated and real-world tasks suggest that the learned embeddings provide useful, transferable supervision. The results under instruction recombinations and appearance changes further strengthen this contribution.
-
The method offers a coherent and practical integration with existing policies. Auxiliary representation prediction introduces no additional inference-time modules. Comparisons with alternative visual representations, together with probes and ablations, provide useful evidence for understanding the method’s benefits.
Critical weaknesses
-
Limited policy-level validation of the core design choices. Endpoint and status ablations are evaluated through representation probes rather than downstream policy performance. The benefits of segmentation based on learned representations at multiple temporal granularities are also not directly isolated, leaving the contribution of these components to the policy gains unclear.
-
Insufficient downstream comparison with closely related representations. Policy comparisons focus on general image and video representations, while the UniVLA latent-action comparison is limited to action-label probes. A downstream comparison using related action representations as auxiliary targets under matched policy-training conditions would better establish the approach’s added value. Although the paper discusses related approaches, clearer methodological distinctions from the closest methods would strengthen the novelty argument.
-
Limitations in the representation evaluation. The normalized-time probe does not isolate possible temporal cues introduced by forward-window extraction and end-of-episode padding, limiting its interpretation as evidence of semantic execution progress. The spatial probe uses the same validation episodes for checkpoint selection and reporting, a limitation explicitly acknowledged in Appendix A.6.5, so its performance estimates may be optimistic. These concerns limit the strength of the conclusions drawn from these probes, while the separate skill and grasp-completion evaluations provide complementary evidence.
Questions for authors
-
The endpoint and status ablations are evaluated through representation probes. Are policy-level results available that isolate their contributions to task performance? A matched comparison with single-granularity or fixed-length segmentation would also help clarify the value of multi-granularity subprocess discovery.
-
Could you clarify the key methodological differences from the closest latent-action and subprocess-representation methods discussed in Section 2? Beyond the UniVLA action-label probes, is a downstream comparison available using a related action representation as an auxiliary target under matched policy-training conditions?
-
The normalized-time probe uses embeddings extracted from forward windows with end-of-episode padding. Does Full retain its advantage over the ablations when controlling for these temporal cues, for example by excluding padded windows or varying the extraction horizon? This would help distinguish semantic execution-progress information from episode-position cues.
-
Appendix A.6.5 acknowledges that the same validation episodes are used for checkpoint selection and reporting. Do the spatial decoding gains in Table 18 persist when these steps use separate data, such as an inner validation split for checkpoint selection and an untouched outer fold for evaluation?
We thank the reviewers for their thoughtful feedback. We have added experiments and revised the manuscript to address their concerns. Key manuscript revisions are marked in dark orange; references below use the revised numbering.
1. Policy-level ablations and segmentation controls
jyoC W1/Q1; RfL8 W2/Q1; RrMA W1/Q1.
We added matched real-robot w/o End and w/o Status policy comparisons on placement and nesting, and next-frame-only, fixed-tolerance and fixed-length controls on nesting (Tables 24–25, Appendix A.9). These results show which gains persist after removing objectives and where Full provides further improvements. Full exceeds the simpler prediction and segmentation controls in mean success in every nesting condition.
2. Related representations and methodological distinctions
jyoC W2/Q2; RfL8 W4/Q4.
We added UniVLA and a video-based HiMaCon adaptation as auxiliary targets under matched policy-training conditions; Full achieves higher mean success in every nesting condition (Table 25). We also clarified the inherited components and extensions relative to HiMaCon, and the distinctions from HiLAM and FLARE, covering representation design, training scale and frozen-encoder transfer (Section 2, Appendix A.1.5).
3. Independent real-world policy-training runs
RfL8 W6/Q5; RrMA W2/Q2.
We now report mean ± sample SD across three independently trained real-world policies per evaluated variant/task, holding each representation encoder fixed (Table 2). We also clarified the checkpoint counts, approximate matching of physical initial configurations, scripted resets, and trial-order and cooling procedures (Appendices A.3.3 and A.5).
4. Temporal-cue controls and held-out spatial evaluation
jyoC W3/Q3–Q4; RfL8 W6; RrMA Q3.
We added a shorter extraction horizon, reducing forward context and padding exposure while retaining padded windows. Full retains its normalized-time advantage over both ablations on all evaluated tasks (Table 15, Appendix A.7.2).
We replaced spatial validation results with validation-based checkpoint selection and separate held-out test evaluation. Full's lower spatial errors persist across channel settings (Table 19, Appendix A.7.5). Figure 5 presents held-out examples with matching supplementary clips.
5. Training-to-target explanation and policy-feature analysis
RfL8 W1/Q2 and W3/Q3.
We clarified how subprocess training shapes the exported transition embeddings, how changing-reference status scores are coupled, and how policy targets are extracted, with a measured status-score example (Section 3, Appendices A.1/A.3, Figure 6). We also added matched Base–Full inference-time policy-feature readouts on one held-out observation/action chunk, providing qualitative evidence consistent with the proposed explanation of instruction generalization (Appendix A.6, Figure 7).
6. Strong auxiliary targets under randomization
RfL8 W5/Q5.
Our response to RfL8 reports SigLIP2† and V-JEPA2.1 under the same randomization settings on four tasks. Ours achieves the highest success on each task in this subset. We acknowledge the similar fractions of clean success retained by Base and Ours and emphasize better task execution under the changed scene conditions. This additional comparison is reported in the rebuttal.