acceptodds
Under review as a conference paper at ICLR 2027

Transferable Behavior Embeddings from Unlabeled Videos for Policy Learning

Abstract

A robot policy chooses better actions when it knows how the scene is changing, where that change leads, and how far the current step has progressed. We learn this information from unlabeled manipulation videos as behavior embeddings that describe subprocesses: coherent video segments discovered from the embeddings at several granularities. Each embedding combines a motion channel describing ongoing motion with a scene configuration channel describing the configuration a subprocess reaches and its relation to the current scene. Three tasks give these channels their meaning: next-frame prediction, endpoint reconstruction, and status scoring. Learning requires no action, language, subtask, or progress annotations. The frozen encoder then computes embeddings for policy demonstrations. Predicting these embeddings from the policy’s action features regularizes those features with no additional inference cost. Experiments show consistent improvements in task success across vision–language–action and world-action models. On a real robot, behavior supervision also improves task success and generalization to unseen instruction combinations and appearance changes. Probes also show that motion, reached configuration, and execution status are readable from the embeddings.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic, Manipulation, Generalization, Concept Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_jyoCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes learning transferable behavior embeddings from unlabeled robot manipulation videos to support policy training. The method discovers coherent video segments at multiple temporal granularities and jointly trains next-frame prediction, endpoint reconstruction, and status scoring objectives to capture motion, reached configurations, and execution status. A frozen encoder then generates offline targets for auxiliary prediction from policy action features, with the additional modules removed at inference. Experiments on RoboTwin 2.0 and Mobile ALOHA demonstrate improvements in task performance and under the tested instruction and appearance shifts, complemented by representation probes and ablations.

Critical strengths
  1. The empirical results support the practical value of the approach. Improvements across multiple policy configurations and both simulated and real-world tasks suggest that the learned embeddings provide useful, transferable supervision. The results under instruction recombinations and appearance changes further strengthen this contribution.

  2. The method offers a coherent and practical integration with existing policies. Auxiliary representation prediction introduces no additional inference-time modules. Comparisons with alternative visual representations, together with probes and ablations, provide useful evidence for understanding the method’s benefits.

Critical weaknesses
  1. Limited policy-level validation of the core design choices. Endpoint and status ablations are evaluated through representation probes rather than downstream policy performance. The benefits of segmentation based on learned representations at multiple temporal granularities are also not directly isolated, leaving the contribution of these components to the policy gains unclear.

  2. Insufficient downstream comparison with closely related representations. Policy comparisons focus on general image and video representations, while the UniVLA latent-action comparison is limited to action-label probes. A downstream comparison using related action representations as auxiliary targets under matched policy-training conditions would better establish the approach’s added value. Although the paper discusses related approaches, clearer methodological distinctions from the closest methods would strengthen the novelty argument.

  3. Limitations in the representation evaluation. The normalized-time probe does not isolate possible temporal cues introduced by forward-window extraction and end-of-episode padding, limiting its interpretation as evidence of semantic execution progress. The spatial probe uses the same validation episodes for checkpoint selection and reporting, a limitation explicitly acknowledged in Appendix A.6.5, so its performance estimates may be optimistic. These concerns limit the strength of the conclusions drawn from these probes, while the separate skill and grasp-completion evaluations provide complementary evidence.

Questions for authors
  1. The endpoint and status ablations are evaluated through representation probes. Are policy-level results available that isolate their contributions to task performance? A matched comparison with single-granularity or fixed-length segmentation would also help clarify the value of multi-granularity subprocess discovery.

  2. Could you clarify the key methodological differences from the closest latent-action and subprocess-representation methods discussed in Section 2? Beyond the UniVLA action-label probes, is a downstream comparison available using a related action representation as an auxiliary target under matched policy-training conditions?

  3. The normalized-time probe uses embeddings extracted from forward windows with end-of-episode padding. Does Full retain its advantage over the ablations when controlling for these temporal cues, for example by excluding padded windows or varying the extraction horizon? This would help distinguish semantic execution-progress information from episode-position cues.

  4. Appendix A.6.5 acknowledges that the same validation episodes are used for checkpoint selection and reporting. Do the spatial decoding gains in Table 18 persist when these steps use separate data, such as an inner validation split for checkpoint selection and an untouched outer fold for evaluation?

Reviewer_RrMAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper learns behavior embeddings from robot manipulation videos without action, language, subtask, or progress annotations during representation training. A bidirectional video encoder produces motion and scene-configuration channels. Motion similarity defines temporal subprocesses at sampled granularities; next-frame prediction, endpoint reconstruction, and status scoring train the embeddings. A frozen encoder then supplies auxiliary prediction targets during policy fine-tuning, with the encoder and auxiliary head removed at inference.

Critical strengths
  1. The method improves performance across several policies. It raises success rates on all 11 clean simulation tasks for π₀, π₀.₅, and FastWAM. It also improves or matches π₀.₅ under scene randomization. The gains on FastWAM are useful evidence because that policy already learns to predict future visual features (Tables 1 and 21).
  2. The real-world tests show useful generalization without extra inference cost. The robot must place familiar objects in new destinations, which requires changing its behavior. The method succeeds in 49/80 trials, compared with 0/80 for the base policy. The extra training components are removed before deployment (section 3.3; Table 2).
Critical weaknesses

It is unclear which parts of the method improve policy performance. The authors remove endpoint and status supervision, but test these versions only with representation probes. They do not test whether the resulting policies perform worse. They also do not compare against fixed video segments or a simpler video-prediction objective. The full method helps, but these experiments do not establish why (section 4.3; Tables 3, 18, and 23).

The gains over the strongest alternatives need uncertainty estimates. The paper does not report variation across independent policy-training runs. On unseen placements, the method succeeds in 49/80 trials, compared with 40/80 for SigLIP2. On patterned cloth, the errors are 3.5 and 4.0 cm. Without uncertainty estimates, it is difficult to judge whether these differences are reliable (Table 2; Appendices A.3.3, A.5, and A.8).

Questions for authors

Do policy gains persist when targets come from the w/o End and w/o Status encoders? A representative downstream comparison would establish whether these objectives improve control, rather than only probe accessibility, and could change my recommendation.

How many independently trained policy checkpoints underlie each result? Were evaluation initial states shared across variants, and how were real-world trial order and resets controlled? Please provide available variability estimates, especially against the strongest auxiliary-target baselines.

How much normalized-time accuracy remains after excluding padded windows or controlling for the amount of repeated terminal imagery? This would distinguish behavior-related progress information from remaining-horizon cues in the extraction procedure.

Reviewer_RfL8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns behavior embeddings from unlabeled DROID videos and uses them as auxiliary targets during policy fine-tuning. A video encoder produces motion and scene-configuration channels. Motion coherence defines subprocesses at sampled tolerances; next-frame prediction, endpoint reconstruction, and status scoring train the representation. A frozen encoder then labels policy demonstrations, and a prediction head supervises action features without adding inference cost. Experiments report gains across π0, π0.5, and FastWAM, plus real-robot improvements under held-out instruction pairings and appearance changes.

Critical strengths

S1. The policy gains cover several architectures and settings. On 11 RoboTwin2.0 tasks, mean clean success improves by 10.5, 13.0, and 14.4 percentage points for π0, π0.5, and FastWAM, respectively. The same frozen DROID encoder supplies all behavior targets. On Mobile ALOHA, held-out placement success reaches 49/80 versus 0/80 for Base, although SigLIP2† leads on one of the two pairings.

S2. The auxiliary supervision adds no inference cost. Representation learning requires no action, language, or progress annotations, and the encoder and auxiliary head are removed at inference. The policy comparisons share data, optimization, head-construction rules, and adaptive loss weighting. The V-JEPA2.1 comparison is useful because it also uses temporal context.

Critical weaknesses

W1. The connection between subprocess training and policy targets is hard to follow. I had trouble following the transition from Sec. 3.2 to Sec. 3.3. Endpoint and status training use a shared motion prototype, but the policy predicts each transition's own embedding. The paper should explain this connection more directly, including how supervision at different tolerances shapes the exported target. I also do not understand why comparing status scores with different scene-configuration references in Eq. 5 measures progress. One example following a video segment through these steps would help.

W2. The policy experiments do not isolate the proposed representation-learning components. Endpoint and status ablations are evaluated through embedding probes, without corresponding policy-success results. Thus, the paper shows that the full target is useful, but does not establish whether these objectives account for the policy improvement. The multi-tolerance design is illustrated only qualitatively, without a fixed-tolerance policy comparison. My priority would be policy results using w/o End, w/o Status, and next-frame-only encoders, followed by a single-tolerance control, under otherwise matched training settings.

W3. The explanation of reduced reliance on familiar pairings is not directly tested. The held-out placement results demonstrate useful instruction generalization. However, the placement demonstrations used to generate targets still pair each object with only one destination. Predicting their embeddings might help the pretrained policy use instructions, but it does not show whether the policy relies less on the training pairings. The probes measure frozen encoder outputs rather than the policy features used at inference. Comparing object, destination, and status readouts from Base and Ours on held-out pairings would help test the proposed explanation.

W4. The differences from prior methods need to be clearer. HiMaCon (Liu et al., 2025) discovers coherent segments from visual–proprioceptive data, predicts endpoints, and supervises policies. The auxiliary head in Sec. 3.3 also follows this work. The input modalities differ, but the paper should explain which components extend that design. FLARE (Zheng et al., 2025) is cited for future-feature supervision without much discussion of how the proposed targets differ. What do endpoint and status supervision add over these approaches? The V-JEPA2.1 comparison provides a useful temporal-context control, but leaves this question open.

W5. The robustness comparison is incomplete. For π0.5, randomized success is 18.2% for Base and 25.6% for Ours, but the fractions of clean success retained are similar: approximately 0.49 and 0.51. Ours performs better in the randomized scenes; the retention figures suggest little change in sensitivity to the shift. Its robustness relative to other auxiliary targets also remains unclear, since those targets are evaluated only in clean simulation. Please evaluate the strongest auxiliary-target baselines under the same randomization.

W6. Some reported differences need uncertainty estimates. Policy tables give rollout counts but no variation across independently trained policies. Some advantages are small: seen placement is 70/80 versus 67/80 for V-JEPA2.1, and nesting with changed colors is 20/40 versus 17/40. Repeated training runs would help assess these differences. The spatial probes in Sec. 4.3 also report validation RMSE; I would prefer results on a separate test split.

Questions for authors
  1. Which policy gains remain when endpoint, status, or multi-tolerance supervision is removed?
  2. Why does the changing-reference order loss support the intended progress interpretation? Could one example show the scores and the final exported target?
  3. What do the policy features and auxiliary predictions reveal about object–destination information on held-out pairings, relative to Base?
  4. What do endpoint and status supervision add over HiMaCon-style supervision and simpler future-feature targets?
  5. How stable are the policy results across training seeds, and do the strongest auxiliary-target baselines retain their ranking under randomization?

Comments on the paper

2 comments

We thank the reviewers for their thoughtful feedback. We have added experiments and revised the manuscript to address their concerns. Key manuscript revisions are marked in dark orange; references below use the revised numbering.

1. Policy-level ablations and segmentation controls

jyoC W1/Q1; RfL8 W2/Q1; RrMA W1/Q1.

We added matched real-robot w/o End and w/o Status policy comparisons on placement and nesting, and next-frame-only, fixed-tolerance and fixed-length controls on nesting (Tables 24–25, Appendix A.9). These results show which gains persist after removing objectives and where Full provides further improvements. Full exceeds the simpler prediction and segmentation controls in mean success in every nesting condition.

2. Related representations and methodological distinctions

jyoC W2/Q2; RfL8 W4/Q4.

We added UniVLA and a video-based HiMaCon adaptation as auxiliary targets under matched policy-training conditions; Full achieves higher mean success in every nesting condition (Table 25). We also clarified the inherited components and extensions relative to HiMaCon, and the distinctions from HiLAM and FLARE, covering representation design, training scale and frozen-encoder transfer (Section 2, Appendix A.1.5).

3. Independent real-world policy-training runs

RfL8 W6/Q5; RrMA W2/Q2.

We now report mean ± sample SD across three independently trained real-world policies per evaluated variant/task, holding each representation encoder fixed (Table 2). We also clarified the checkpoint counts, approximate matching of physical initial configurations, scripted resets, and trial-order and cooling procedures (Appendices A.3.3 and A.5).

4. Temporal-cue controls and held-out spatial evaluation

jyoC W3/Q3–Q4; RfL8 W6; RrMA Q3.

We added a shorter extraction horizon, reducing forward context and padding exposure while retaining padded windows. Full retains its normalized-time advantage over both ablations on all evaluated tasks (Table 15, Appendix A.7.2).

We replaced spatial validation results with validation-based checkpoint selection and separate held-out test evaluation. Full's lower spatial errors persist across channel settings (Table 19, Appendix A.7.5). Figure 5 presents held-out examples with matching supplementary clips.

5. Training-to-target explanation and policy-feature analysis

RfL8 W1/Q2 and W3/Q3.

We clarified how subprocess training shapes the exported transition embeddings, how changing-reference status scores are coupled, and how policy targets are extracted, with a measured status-score example (Section 3, Appendices A.1/A.3, Figure 6). We also added matched Base–Full inference-time policy-feature readouts on one held-out observation/action chunk, providing qualitative evidence consistent with the proposed explanation of instruction generalization (Appendix A.6, Figure 7).

6. Strong auxiliary targets under randomization

RfL8 W5/Q5.

Our response to RfL8 reports SigLIP2† and V-JEPA2.1 under the same randomization settings on four tasks. Ours achieves the highest success on each task in this subset. We acknowledge the similar fractions of clean success retained by Base and Ours and emphasize better task execution under the changed scene conditions. This additional comparison is reported in the rebuttal.

Area_Chair_o1F7OpenReview ↗

Please review the authors’ rebuttal and update your assessment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.