acceptodds
Under review as a conference paper at ICLR 2027

Distilling World Understanding from Video Generators for Robot Control

Abstract

Video generation models have shown promising results in robot control through future scene generation supported by their implicit understanding of the physical world. However, generating realistic textures does not guarantee an accurate representation of physical properties such as depth and motion, which are important for robot control. To leverage **K**nowledge for **R**obot control from the implicit **U**nderstanding in video **G**enerators, we propose **KRUG**, a two-stage framework based on generation as understanding. To activate this implicit world understanding, we apply multimodal instruction tuning to a pretrained video generator using a small amount of multimodal data, yielding a single World Understanding Expert that generates depth, surface normals, segmentation masks, or optical flow conditioned on modality-specific instructions. The challenge is to transfer this understanding to a student policy that uses RGB as its only visual input. We introduce World-to-Action Distillation (W2AD), which uses the expert as a frozen teacher and aligns features retrieved from the teacher and student using the same action queries to guide action prediction. For model training, we construct a multimodal embodied AI dataset by annotating RGB videos from nine sources, with 600,000 training clips and over 1,200 hours counted across available modalities. **KRUG** achieves strong performance on RoboTwin 2.0 and advances the **state of the art** on DexJoCo with an average success rate of 60.4%.

Primary area:
generative models
Keywords:
Generative World Understanding, Robot, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jyDJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KRUG, a framework that distills world understanding from video generators into robot policies. It first trains a video generator to predict multiple modalities, including depth, surface normals, segmentation, and optical flow, and then transfers this knowledge to an action policy. The method is evaluated extensively in simulation, with additional real-world demonstrations.

Critical strengths
  • Large-scale data curation. The authors curate a large and diverse dataset from multiple sources, with rich multimodal annotations. This is a valuable resource for training robot policies with more comprehensive world understanding.
  • Extensive simulation experiments. The method is evaluated across a broad range of tasks and settings, with comparisons to several existing approaches and ablation studies. This provides a reasonably comprehensive evaluation of the proposed framework.
Critical weaknesses
  • The training pipeline is quite complicated and expensive. The framework involves multiple stages, including multimodal instruction tuning, action-query tuning, and joint distillation training. This makes the overall training process lengthy and introduces additional complexity.
  • The real-world evaluation is mostly qualitative. The real-robot experiments provide visual demonstrations, but lack quantitative success rates and comparisons with relevant baselines. As a result, it is difficult to assess how much the proposed data and multimodal world understanding actually improve real-world robot performance. This is particularly important because collecting large-scale multimodal data in real-world settings is challenging. If the benefits are mainly demonstrated in simulation, the practical impact of the approach for real-world robotics remains unclear.
Questions for authors
  • What is the main reason for introducing the distilation stage? According to the ablation results, distillation appears to contribute the largest performance improvement. Could the authors explain why this step is necessary?

  • Under the same rand-obj setting, why are the DexJoCo results for DP-T and π0.5 approximately 17 and 9 percentage points lower than their official results?

5 comments

We sincerely thank you for your constructive review and recognition of our dataset and extensive simulation evaluation. To address your concern about training complexity and cost, we clarify the role of each training stage, describe the sources of auxiliary annotations, and explain the single-stage fine-tuning procedure for downstream real-robot tasks. We provide quantitative real-robot evaluations with baseline comparisons and ablations demonstrating the benefits of multimodal pretraining. We also explain the motivation for distillation and quantify the resulting performance gain. Finally, we clarify the sources of the DexJoCo baseline values and distinguish the single-task and multi-task results. We have revised the manuscript accordingly, with modifications highlighted in blue.


W1: The training pipeline is quite complicated and expensive. The framework involves multiple stages, including multimodal instruction tuning, action-query tuning, and joint distillation training. This makes the overall training process lengthy and introduces additional complexity.

Response: Thank you for raising this concern. As for training complexity, KRUG follows a typical pipeline for robot policies, with pre-training followed by post-training. For example, Motus uses three stages. It first adapts its video generator to embodied data, then pretrains the model with latent actions, and finally fine-tunes it on trajectories from the target robot. LingBot-VA uses two phases: pre-training on videos and robot action data, followed by post-training on task-specific demonstrations. KRUG uses its training stages to extract world knowledge from the video generator and transfer this knowledge to the action policy.

As for training cost, the multimodal videos we use to capture geometry, semantics, and motion require additional data preprocessing. The auxiliary targets come from existing dataset annotations, simulators, or pretrained estimators. This requires no additional manual labeling. Fig. 6 in Appendix C.2 of the revised manuscript summarizes the annotation sources for each dataset. Once the multimodal Wan from Stage 1 is available, adapting KRUG to a new real-robot task requires only a single fine-tuning stage (Section 4.3).

(1) Contribution of each training stage. Each stage serves a specific role in learning, retrieving, or transferring the world knowledge used for action prediction. The full model achieves 92.76% / 91.82% success on RoboTwin 2.0 under the Clean / Random settings. Table 6 of the revised manuscript demonstrates the contribution of each stage:

  • Multimodal instruction tuning (Stage 1) develops the video generator’s world understanding through multimodal prediction. Removing it reduces success by 16.41 / 17.51 percentage points.
  • Action-Query Tuning trains the Action Expert to retrieve action-relevant features across modalities. Removing it reduces success by 7.14 / 7.24 percentage points.
  • World-to-Action Distillation (W2AD) aligns features retrieved from the multimodal teacher and the RGB student using the same action queries. Removing this alignment lowers success by 5.53 / 5.34 percentage points.

(2) Real-world post-training. For each real-robot task, we initialize the video expert from the Stage 1 checkpoint and jointly fine-tune it with the Action Expert on demonstrations for that task. This step requires no auxiliary-modality annotations. After fine-tuning, KRUG succeeded in 38 of 50 trials (76%) across five real-robot tasks (detailed in W2).


W2: The real-world evaluation is mostly qualitative. The real-robot experiments provide visual demonstrations, but lack quantitative success rates and comparisons with relevant baselines. As a result, it is difficult to assess how much the proposed data and multimodal world understanding actually improve real-world robot performance. This is particularly important because collecting large-scale multimodal data in real-world settings is challenging. If the benefits are mainly demonstrated in simulation, the practical impact of the approach for real-world robotics remains unclear.

Response: Thank you for your detailed suggestions about the real-world evaluation. We have added quantitative results for five real-robot tasks in Section 4.3 of the revised manuscript. Table 3 compares KRUG with VPP and π0.5, and Table 4 presents ablation results on the contributions of RGB pretraining and multimodal supervision. We also present qualitative demonstrations of the real-robot tasks in the supplementary video real-robot-rollouts.mp4.

(1) Real-world quantitative evaluation. We evaluate five tasks on the Galaxea R1 Lite robot. We collect 30 demonstrations per task and train a separate KRUG policy for each task following the RGB post-training procedure described in Section 4.3. We evaluate each method over 10 trials per task.

KRUG achieves an average success rate of 76%, compared with 46% for VPP and 42% for π0.5. These results support the practical effectiveness of our design on these real-world tasks.

TaskVPPπ0.5KRUG
Bottle to Bin405080
Tissue to Drawer403070
Transfer Milk607090
Stack Ladles302060
Cup to Coaster604080
Average464276

Success rates (%) over ten trials per task.

(2) Challenge of collecting multimodal data for real-world tasks. Our Stage 1 combines multimodal data from real-world and simulated environments. We automatically generate any additional auxiliary labels, without manual labeling. For downstream real-robot tasks, no auxiliary-modality annotations are required for post-training. We describe the annotation sources and post-training procedure in our response to W1.

(3) Ablation study on real-world tasks. To assess the contributions of additional pretraining data and multimodal supervision, we compare three settings: without Stage 1, RGB-only Stage 1, and multimodal Stage 1 (KRUG). The RGB-only and multimodal variants each use approximately 600 hours of Stage 1 training data from the same nine source datasets.

RGB-only Stage 1 raises the average success rate from 44% to 60%. With the same pretraining data scale, multimodal supervision raises it further to 76%, a gain of 16 percentage points over RGB-only pretraining.

Taskw/o Stage 1RGB-only Stage 1KRUG
Bottle to Bin407080
Tissue to Drawer506070
Transfer Milk607090
Stack Ladles204060
Cup to Coaster506080
Average446076

Success rates (%) over ten trials per task.


Q1: What is the main reason for introducing the distilation stage? According to the ablation results, distillation appears to contribute the largest performance improvement. Could the authors explain why this step is necessary?

Response: Thank you for this question. The main purpose of distillation is to transfer action-relevant multimodal world knowledge from the teacher to the RGB student. While Stage 1 activates the video generator’s world understanding, our goal is to transfer this knowledge to action prediction, rather than focus on the fidelity of multimodal prediction.

(1) Distilling action-relevant knowledge from video representations. Video generators can encode physical knowledge through learning to predict future scenes. Stage 1 makes this knowledge explicit through multimodal tuning on auxiliary tasks. During distillation, the frozen teacher provides action-relevant physical knowledge as training targets, and the RGB student learns to recover the corresponding information from its own visual representations. Section 3.3 describes the two steps below:

  • Where to distill from. Action-Query Tuning trains the Action Expert to attend to action-relevant information across modalities.
  • What to distill. We use the same action queries to retrieve features from the frozen multimodal teacher and the RGB student and then align the student’s retrieved features with the teacher’s.

(2) Quantitative evidence. We report the distillation ablation in Table 6 (ID 4 vs. ID 6). Both variants use multimodal Stage 1 and Action-Query Tuning. Adding distillation improves success by 5.53 / 5.34 percentage points under Clean / Random. These results support the benefit of explicitly aligning the RGB student’s retrieved features with the teacher’s action-relevant representations.

Stage 1Action-Query TuningDistillationClean (%) ↑Random (%) ↑
Multimodal✓✗87.2386.48
Multimodal✓✓92.7691.82

Q2: Under the same rand-obj setting, why are the DexJoCo results for DP-T and π0.5 approximately 17 and 9 percentage points lower than their official results?

Response: Thank you for pointing this out. DexJoCo uses the rand-obj setting for both single-task and multi-task policies. Our paper quotes the official multi-task results from DexJoCo v1 (https://arxiv.org/pdf/2605.16257v1). The differences you mention appear to correspond to a comparison with the single-task results in DexJoCo v2 (https://arxiv.org/pdf/2605.16257v2).

(1) Source and experimental settings. The DP-T and π0.5 results in our paper are taken directly from the multi-task columns of DexJoCo v1, Table 3, which report average success rates of 33.2% and 45.5%, respectively.

Following the same multi-task training setting, we jointly train a single policy on 1,100 demonstrations covering six single-arm and five bimanual tasks and evaluate it separately on each task under rand-obj.

(2) Comparison with single-task results. DexJoCo v2, Table II reports single-task success rates of 50.4% for DP-T and 54.1% for π0.5, where a separate policy is trained for each task. Comparing these with the v1 multi-task averages quoted in our paper gives differences of 17.2 percentage points for DP-T (50.4 − 33.2) and 8.6 percentage points for π0.5 (54.1 − 45.5).

(3) Updated multi-task results in v2. We also note that DexJoCo v2, Table IV updates the official multi-task results:

  • π0.5: The reported average decreases from 45.5% to 35.0%.
  • DP-T: The reported average is revised from 33.2% to 28.2%, while all individual task scores remain unchanged.

We have clarified the baseline sources in Table 2 of the revised manuscript.

Reviewer_yYT5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper posits that video generators contain implicit physical-world knowledge, but high-fidelity RGB output does not guarantee the depth/motion representations needed for control, so that knowledge must be explicitly activated and distilled into a policy.

Critical strengths

Clear thesis with a complete, deployment-aware pipeline. The framing that physical understanding must be activated then distilled is operational end-to-end, and the staged ablation quantifies each choice; the action-query mechanism concretely answers where the student should attend.

Substantial data contribution. Curating nine sources and auto-generating depth/normal/segmentation/flow annotations at 1,200 hours / 600K clips with reusable pipelines is a resource the community can build on; DexJoCo shows cross-task breadth.

Critical weaknesses

The causal link is not isolated, and the teacher's geometry is not state-of-the-art. The World Expert underperforms specialist geometry models on depth/normal, yet the Stage-1 ablation is large. Without an equal-scale control using non-physical auxiliary tasks, the gains could be multi-task/regularization effects rather than transferred physics. The SOTA margins themselves are thin.

Weak real-robot evidence and annotation-heavy training. The Galaxea R1 results are qualitative, and all depth/flow supervision comes from external estimators whose errors under hand/object occlusion propagate into the teacher; the 1,200-hour scale versus baselines' undisclosed data makes the comparison hard to judge.

Questions for authors

see weakness

7 comments

Thank you for your thoughtful review and for recognizing the clarity of our framework and the value of our multimodal dataset. To assess the contribution of multimodal supervision, we distill each auxiliary modality separately and compare KRUG with an equal-scale RGB-only pipeline. In our multi-task regularization control, we replace the four auxiliary modalities with four video style-transfer tasks while keeping the training-data scale fixed. KRUG outperforms this control by over 5 percentage points in both RoboTwin 2.0 settings. Across five tasks on the Galaxea R1 Lite robot, KRUG achieves an average success rate of 76%, compared with 46% for VPP and 42% for π0.5\pi_{0.5}. For a fair comparison, we also evaluate a LingBot-VA baseline with approximately 600 hours of additional RGB pretraining data from the same sources as KRUG. All changes are highlighted in blue in the revised manuscript.


W1: The causal link is not isolated, and the teacher's geometry is not state-of-the-art. The World Expert underperforms specialist geometry models on depth/normal, yet the Stage-1 ablation is large.

Response: We thank you for this insightful comment. KRUG and the specialist models in Table 5 perform different tasks. The key distinction is that KRUG predicts future modalities from the current RGB frame, while specialists such as MoGe-2 estimate the geometry of the frame they observe. Despite this additional challenge, KRUG achieves lower depth error than MoGe-2 and VGGT-Ω on both RoboTwin and DexJoCo. These results provide evidence that KRUG captures knowledge of the physical world when predicting future scenes.

Video generators can acquire implicit world knowledge through generative pretraining. Our goal is to transfer this knowledge to action prediction. To this end, KRUG uses a shared backbone to predict auxiliary modalities. Below, we discuss the geometry evaluation setting, the contribution of individual modalities to action learning, and the effect of removing Stage 1.

(1) Geometry evaluation setting. As described in Section 4.4, we evaluate all geometry outputs against simulator ground truth at time t+1. KRUG predicts these outputs from the RGB observation at time t. Because the specialist models estimate geometry from an observed frame, we give them the ground-truth RGB frame at t+1 so that their estimates correspond to the same evaluation frame. Under this setting, KRUG achieves lower depth error than MoGe-2 on both RoboTwin and DexJoCo, while MoGe-2 achieves lower normal error (Table 5).

(2) Distilling individual modalities for action learning. To assess how each auxiliary modality contributes to action learning, we separately distill depth, surface normals, segmentation, and optical flow into the RGB student. The four settings use the same multimodal Stage 1 teacher and Action-Query Tuning procedure and differ only in the auxiliary modality used for W2AD. We compare each setting with the no-distillation baseline and with full KRUG, which uses all four auxiliary modalities for distillation.

Distillation modalityClean (%) ↑Random (%) ↑
None (Table 6, ID 4)87.2386.48
Depth90.2889.63
Surface normals88.0286.95
Segmentation89.2688.39
Optical flow89.5188.94
All four (KRUG)92.7691.82

Distilling each modality individually improves the RGB student’s success rate over the no-distillation baseline under both the Clean and Random settings on RoboTwin 2.0, and joint distillation performs best. Although KRUG has higher surface-normal error than MoGe-2, surface-normal supervision still benefits action learning.

(3) Results of the Stage 1 ablation. We remove multimodal instruction tuning while retaining Action-Query Tuning and W2AD (Section 4.6, Table 6, ID 1). As shown below, removing Stage 1 reduces the success rate by 16.41 percentage points on Clean and 17.51 points on Random relative to full KRUG.

MethodClean (%) ↑Random (%) ↑
Without Stage 1 (Table 6, ID 1)76.3574.31
KRUG (Table 6, ID 6)92.7691.82

Stage 1 aims to activate the video generator’s implicit world understanding through multimodal instruction tuning. During the subsequent Action-Query Tuning step, the video backbone is frozen and only the Action Expert is updated. Without Stage 1, the Action Expert may therefore struggle to retrieve the action-relevant physical knowledge that remains implicit in the frozen generator.


W2: Without an equal-scale control using non-physical auxiliary tasks, the gains could be multi-task/regularization effects rather than transferred physics.

Response: Thank you for raising this important point. In Appendix E.3 of the revised manuscript, we present an equal-scale appearance-based auxiliary-task control. This comparison helps assess the contribution of explicit physical supervision beyond the benefits of multi-task training, including its potential regularization effects. We also compare KRUG with an equal-scale RGB-only baseline to evaluate the contribution of multimodal supervision.

(1) Appearance-based auxiliary-task control. We retain RGB prediction and replace depth, surface normals, segmentation, and optical flow with four auxiliary prediction tasks based on video style transfer. Each task predicts a future video in one of four styles (colored pencil, cartoon, sketch, or oil painting) from the original current RGB frame. These targets change the visual appearance of RGB videos without providing explicit physical supervision. We generate the targets by applying pretrained CAP-VSTNet [1] to the corresponding RGB videos, using one fixed reference image per style. Figures 11 and 12 in Appendix E.3 of the revised manuscript show the reference images and stylized-video examples, respectively.

The appearance-based model uses the same source clips, training-data scale, and architecture as KRUG and follows the full Stage 1, Action-Query Tuning, and W2AD pipeline. During W2AD, the teacher uses the stylized-video targets and the student uses the corresponding RGB targets. We report the RoboTwin 2.0 results in Table 7 of the revised manuscript.

Auxiliary supervisionClean (%) ↑Random (%) ↑
Four appearance styles87.4286.65
Physical modalities (KRUG)92.7691.82

KRUG outperforms the appearance-based model by 5.34 percentage points on Clean and 5.17 points on Random.

(2) Equal-scale RGB-only ablation. To further evaluate the contribution of multimodal supervision within the full training pipeline, we replace the auxiliary-modality samples with additional RGB samples while keeping the amount of Stage 1 data fixed. The baseline uses RGB-only Stage 1, RGB-only Action-Query Tuning, and RGB-to-RGB distillation. We report this baseline in Table 6 (ID 5) of the revised manuscript.

Stage 1Action-Query TuningDistillationClean (%) ↑Random (%) ↑
RGB-only✓†✓†88.0787.19
Multimodal✓✓92.7691.82

† denotes RGB-only Action-Query Tuning and RGB-to-RGB distillation.

KRUG outperforms the equal-scale RGB-only baseline by 4.69 percentage points on Clean and 4.63 points on Random. Together, the two comparisons show that KRUG achieves higher success rates than both the equal-scale RGB-only pipeline and multi-task training with non-physical targets.

[1] Wen L, Gao C, Zou C. CAP-VSTNet: Content affinity preserved versatile style transfer. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.


W3: The SOTA margins themselves are thin.

Response: Thank you for this observation. We agree that the margins on RoboTwin 2.0 are small (Table 1). On DexJoCo, KRUG achieves an average success rate of 60.4%, exceeding the strongest baseline, DECO (56.4%), by 4.0 percentage points. The margins over π0.5\pi_{0.5} (45.5%) and the reported DP-T average (33.2%) are 14.9 and 27.2 points, respectively (Table 2). We also conducted additional quantitative evaluations on the Galaxea R1 Lite robot. KRUG achieved higher success rates than both VPP and π0.5\pi_{0.5} on all five tasks, as detailed in W4.


W4: Weak real-robot evidence. The Galaxea R1 results are qualitative.

Response: Thank you for raising this important point. We agree that quantitative evaluation is essential for assessing real-world policy performance. We have added quantitative results for five tasks on the Galaxea R1 Lite robot in Section 4.3 (Table 3) of the revised manuscript. We collect 30 demonstrations per task and train a separate KRUG policy for each task using RGB post-training. We evaluate each method over 10 trials per task. We also include qualitative demonstrations of the real-robot tasks in the supplementary video real-robot-rollouts.mp4.

TaskVPPπ0.5\pi_{0.5}KRUG
Bottle to Bin405080
Tissue to Drawer403070
Transfer Milk607090
Stack Ladles302060
Cup to Coaster604080
Average464276

Success rates (%) over ten trials per real-robot task.

KRUG achieves the highest observed success rate on all five tasks, succeeding in 38 of 50 trials (76%), compared with 23 of 50 (46%) for VPP and 21 of 50 (42%) for π0.5\pi_{0.5}. These correspond to improvements of 30 and 34 percentage points, respectively.


W5: Annotation-heavy training. All depth/flow supervision comes from external estimators whose errors under hand/object occlusion propagate into the teacher.

Response: Thank you for raising this important question. We address this concern in two parts below, focusing on annotation cost and the hand–object occlusion.

(1) Auxiliary supervision and annotation cost. KRUG uses auxiliary modalities to activate the video generator’s implicit world understanding, and our pipeline obtains these targets without additional manual labeling. Several advanced methods use annotated video data for robotic pretraining. For example, Motus learns latent actions from optical flow, while VPP2 [2] annotates manipulation videos with detailed captions for video pretraining. Fig. 6 in Appendix C.2 of the revised manuscript summarizes the annotation sources for each dataset. For example, depth comes directly from the simulator for RoboTwin and DexJoCo and from the source dataset for AgiBot. We use pretrained estimators for other targets, including DPFlow for optical flow.

Moreover, our real-world post-training requires no auxiliary modality annotations for the target tasks. For the five real-robot tasks in Section 4.3, we use only RGB observations, robot states, task instructions, and action labels for post-training.

(2) Hand–object occlusion and annotation quality. Hand–object occlusion is a natural part of manipulation and can reflect the relative positions and motion of the hand and object. Therefore, we do not regard hand–object occlusion itself as an annotation error. For example, when a hand partially occludes an object, the overlap indicates that the hand lies in front of the occluded part of the object from the camera’s perspective. Moreover, as described in Appendix C.2, we exclude clips with severe, implausible camera occlusion and check the remaining annotations for valid depth coverage and plausible optical flow.

[2] Guo Y, Yan H, Zhong Z, et al. Video Prediction Policy 2: Predict Better, Act Better. arXiv preprint arXiv:2610.10270, 2026.

Reviewer_UqUNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KRUG, a two-stage framework built around the idea - “generation as understanding.” In Stage 1, authors finetune a pretrained video generator with multimodal instructions to obtain a World Understanding Expert, which can predict depth, surface normals, segmentation, and optical flow. In Stage 2, authors propose W2AD, which uses shared action-token queries to align features between teacher model and the RGB-only student model. For this idea, they also build a multimodal embodied dataset with around 600k clips and 1,200+ hours from nine sources. Experiments on RoboTwin 2.0, DexJoCo, and a real Galaxea robot show strong results: 60.4% average success on DexJoCo and 92.76%/91.82% on RoboTwin 2.0 under Clean/Random.

Critical strengths

S1. KRUG points out an important limitation of video generation models: good-looking videos do not necessarily mean physically reasonable, so directly relying on raw generated videos may inherit these physical errors. KRUG solves this by making the generator to predict depth, normals, segmentation, and optical flow, and then transferring these physical cues rather than unreliable pixels to the policy, which is a compelling direction.

S2. The dataset combines nine sources across simulated and real-world manipulation, single-arm and bimanual tasks, dexterous manipulation, and human–object interaction. Its scale and diversity make it a useful resource for embodied learning.

Critical weaknesses

W1. Authors claim that generated videos may look realistic while still having unreliable motion, which may hurt action prediction. However, I don't find the corresponding visualization. A qualitative example showing realistic RGB predictions alongside clearly wrong motion would make the motivation much more convincing.

W2. Section 3.2 argues that directly appending modality prompts to language instructions may contaminate the instruction representation, which motivates using separate modality embeddings. This claim is plausible, but the paper does not prove it. An ablation would be helpful.

W3. Despite strong average performance, KRUG underperforms some baselines on individual tasks, such as Hammer Nail, Water Plant. I hope authors can clarify whether they come from task characteristics, data imbalance, or something else. These points do not invalidate the overall gains, but they leave the method’s robustness and limitations insufficiently understood.

Questions for authors

Could the authors clarify why KRUG underperforms some baselines on specific tasks such as Hammer Nail and Water Plant—is this due to task characteristics, data imbalance, or other factors?

2 comments

Thank you for your thoughtful review and constructive suggestions. We appreciate your positive assessment of KRUG’s motivation and our dataset’s scale and diversity. In response to your comments, we add a supplementary video that shows physically inconsistent motion in realistic-looking predictions and compares the outputs of the original Wan2.2-5B model and our Stage 1 multimodal Wan. We also clarify why KRUG encodes modality prompts separately from task instructions and report an ablation of this design. Finally, we present one representative failure rollout for each of the four DexJoCo tasks where KRUG underperforms at least one baseline and discuss factors that may contribute to these gaps. All changes in the revised manuscript are highlighted in blue.


W1: Authors claim that generated videos may look realistic while still having unreliable motion, which may hurt action prediction. However, I don't find the corresponding visualization. A qualitative example showing realistic RGB predictions alongside clearly wrong motion would make the motivation much more convincing.

Response: Thank you for this helpful suggestion. We clarify our motivation and have added motivation.mp4 to the supplementary material to show that visually realistic videos do not always exhibit reliable motion and interactions. We believe these visualizations make our motivation more convincing.

WAMs have attracted growing interest for robot control. Video generators acquire implicit world understanding through generative pretraining, and we aim to activate this knowledge for action learning. However, we find that while the original Wan2.2-5B model can generate visually realistic videos, its predictions still exhibit unreliable motion and deformation. For example, in the left panel of the first slide in motivation.mp4, the basket rises even though the gripper does not appear to grasp it. We further evaluate our Stage 1 multimodal Wan model on the same scene. As shown in the right panel, the gripper appears to maintain a firm grasp on the basket as they rise together. This more consistent motion supports our motivation for multimodal pretraining and illustrates its benefit.

To further illustrate our motivation, we use MiniMax-H3 (33B), a powerful video generation model, to generate videos of manipulation tasks. The final two slides present examples from simulated and real-world scenes, respectively. For example, in the middle panel of the last slide, the rope looks realistic in individual frames but deforms unnaturally. These examples illustrate the gap between visual realism and physical consistency.


W2: Section 3.2 argues that directly appending modality prompts to language instructions may contaminate the instruction representation, which motivates using separate modality embeddings. This claim is plausible, but the paper does not prove it. An ablation would be helpful.

Response: Thank you for raising this important point. We clarify our motivation for encoding modality prompts separately from language instructions and provide an ablation of this design.

(1) Clarification of our Unified Task Modulation. Our motivation is to maintain a consistent instruction representation across output modalities. During action learning, appending a modality prompt to the task instruction produces different text representations for the same manipulation objective. For example:

  • “Pick up the cup. Predict a depth video.”
  • “Pick up the cup. Predict an optical flow video.”

The same intended action thus corresponds to different text representations depending on the requested output modality. Our design encodes the task instruction and modality prompt separately and injects the modality embedding through timestep modulation. The task-instruction encoding therefore remains consistent across target modalities. At deployment, the policy receives only the task instruction as text input.

(2) Ablation of modality conditioning. We conduct an ablation on RoboTwin 2.0, comparing our design against a baseline that appends the modality prompt to the task instruction and jointly encodes the resulting text. The baseline follows the same training stages as KRUG, including Stage 1, action-query tuning, and distillation. This comparison quantifies the contribution of separate modality conditioning to downstream action learning.

MethodClean (%) ↑Random (%) ↑
Append modality prompt88.1487.26
KRUG92.7691.82

W3 & Q1 : Despite strong average performance, KRUG underperforms some baselines on individual tasks, such as Hammer Nail, Water Plant. I hope authors can clarify whether they come from task characteristics, data imbalance, or something else. These points do not invalidate the overall gains, but they leave the method’s robustness and limitations insufficiently understood; Could the authors clarify why KRUG underperforms some baselines on specific tasks such as Hammer Nail and Water Plant—is this due to task characteristics, data imbalance, or other factors?

Response: Thank you for highlighting these task-specific performance gaps. The DexJoCo training set is balanced across all 11 tasks, with 100 training samples per task. We evaluated checkpoints taken at the same training step across multiple training runs. The average success rate across the 11 tasks remained stable, while success rates on individual tasks varied across runs. In the paper, we report results from the checkpoint with the highest average success rate across all 11 tasks. It achieves 60.4% on DexJoCo, the highest average among the methods in Table 2.

The supplementary video failed-rollouts.mp4 shows one representative failure for each of the four tasks on which KRUG underperforms the best-performing baseline. In these rollouts, the robot fails to complete an interaction that requires precise contact.

  • Hammer Nail (KRUG 80.0% vs. π0.5 86.7%). The robot keeps its grasp on the hammer and makes repeated striking attempts, but fails to drive the nail in far enough.
  • Water Plant (KRUG 76.0% vs. DECO 88.0%). The robot brings the spray bottle near the plant but does not complete watering, which requires holding the trigger.
  • Click Mouse (KRUG 80.0% vs. DECO 90.0%). The robot moves the mouse toward the pad but fails to register a click.
  • Photograph/B (KRUG 40.0% vs. DECO 76.0%). The robot holds and adjusts the camera but fails to take a photograph, which requires shutter contact.

We visualize the auxiliary modalities in failed rollouts and find that small actions during contact produce subtle changes in depth. For example, in Photograph/B, pressing the shutter involves a small fingertip movement with little corresponding change in depth.

During distillation, we randomly sample the teacher’s auxiliary modality from depth, surface normals, segmentation, and optical flow. In the single-modality ablation reported in our response to R2–W1, depth supervision yielded the largest gain on RoboTwin 2.0. If depth was sampled more often during training for the reported DexJoCo checkpoint, KRUG may rely more heavily on depth features. This reliance may make fine contact actions such as pressing the shutter harder to predict. These observations will help us improve the policy for large movements and fine manipulation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.