- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes KRUG, a framework that distills world understanding from video generators into robot policies. It first trains a video generator to predict multiple modalities, including depth, surface normals, segmentation, and optical flow, and then transfers this knowledge to an action policy. The method is evaluated extensively in simulation, with additional real-world demonstrations.
Critical strengths
- Large-scale data curation. The authors curate a large and diverse dataset from multiple sources, with rich multimodal annotations. This is a valuable resource for training robot policies with more comprehensive world understanding.
- Extensive simulation experiments. The method is evaluated across a broad range of tasks and settings, with comparisons to several existing approaches and ablation studies. This provides a reasonably comprehensive evaluation of the proposed framework.
Critical weaknesses
- The training pipeline is quite complicated and expensive. The framework involves multiple stages, including multimodal instruction tuning, action-query tuning, and joint distillation training. This makes the overall training process lengthy and introduces additional complexity.
- The real-world evaluation is mostly qualitative. The real-robot experiments provide visual demonstrations, but lack quantitative success rates and comparisons with relevant baselines. As a result, it is difficult to assess how much the proposed data and multimodal world understanding actually improve real-world robot performance. This is particularly important because collecting large-scale multimodal data in real-world settings is challenging. If the benefits are mainly demonstrated in simulation, the practical impact of the approach for real-world robotics remains unclear.
Questions for authors
-
What is the main reason for introducing the distilation stage? According to the ablation results, distillation appears to contribute the largest performance improvement. Could the authors explain why this step is necessary?
-
Under the same rand-obj setting, why are the DexJoCo results for DP-T and π0.5 approximately 17 and 9 percentage points lower than their official results?
5 comments
W2: The real-world evaluation is mostly qualitative. The real-robot experiments provide visual demonstrations, but lack quantitative success rates and comparisons with relevant baselines. As a result, it is difficult to assess how much the proposed data and multimodal world understanding actually improve real-world robot performance. This is particularly important because collecting large-scale multimodal data in real-world settings is challenging. If the benefits are mainly demonstrated in simulation, the practical impact of the approach for real-world robotics remains unclear.
Response:
Thank you for your detailed suggestions about the real-world evaluation. We have added quantitative results for five real-robot tasks in Section 4.3 of the revised manuscript. Table 3 compares KRUG with VPP and π0.5, and Table 4 presents ablation results on the contributions of RGB pretraining and multimodal supervision. We also present qualitative demonstrations of the real-robot tasks in the supplementary video real-robot-rollouts.mp4.
(1) Real-world quantitative evaluation. We evaluate five tasks on the Galaxea R1 Lite robot. We collect 30 demonstrations per task and train a separate KRUG policy for each task following the RGB post-training procedure described in Section 4.3. We evaluate each method over 10 trials per task.
KRUG achieves an average success rate of 76%, compared with 46% for VPP and 42% for π0.5. These results support the practical effectiveness of our design on these real-world tasks.
| Task | VPP | π0.5 | KRUG |
|---|---|---|---|
| Bottle to Bin | 40 | 50 | 80 |
| Tissue to Drawer | 40 | 30 | 70 |
| Transfer Milk | 60 | 70 | 90 |
| Stack Ladles | 30 | 20 | 60 |
| Cup to Coaster | 60 | 40 | 80 |
| Average | 46 | 42 | 76 |
Success rates (%) over ten trials per task.
(2) Challenge of collecting multimodal data for real-world tasks. Our Stage 1 combines multimodal data from real-world and simulated environments. We automatically generate any additional auxiliary labels, without manual labeling. For downstream real-robot tasks, no auxiliary-modality annotations are required for post-training. We describe the annotation sources and post-training procedure in our response to W1.
(3) Ablation study on real-world tasks. To assess the contributions of additional pretraining data and multimodal supervision, we compare three settings: without Stage 1, RGB-only Stage 1, and multimodal Stage 1 (KRUG). The RGB-only and multimodal variants each use approximately 600 hours of Stage 1 training data from the same nine source datasets.
RGB-only Stage 1 raises the average success rate from 44% to 60%. With the same pretraining data scale, multimodal supervision raises it further to 76%, a gain of 16 percentage points over RGB-only pretraining.
| Task | w/o Stage 1 | RGB-only Stage 1 | KRUG |
|---|---|---|---|
| Bottle to Bin | 40 | 70 | 80 |
| Tissue to Drawer | 50 | 60 | 70 |
| Transfer Milk | 60 | 70 | 90 |
| Stack Ladles | 20 | 40 | 60 |
| Cup to Coaster | 50 | 60 | 80 |
| Average | 44 | 60 | 76 |
Success rates (%) over ten trials per task.
Q1: What is the main reason for introducing the distilation stage? According to the ablation results, distillation appears to contribute the largest performance improvement. Could the authors explain why this step is necessary?
Response: Thank you for this question. The main purpose of distillation is to transfer action-relevant multimodal world knowledge from the teacher to the RGB student. While Stage 1 activates the video generator’s world understanding, our goal is to transfer this knowledge to action prediction, rather than focus on the fidelity of multimodal prediction.
(1) Distilling action-relevant knowledge from video representations. Video generators can encode physical knowledge through learning to predict future scenes. Stage 1 makes this knowledge explicit through multimodal tuning on auxiliary tasks. During distillation, the frozen teacher provides action-relevant physical knowledge as training targets, and the RGB student learns to recover the corresponding information from its own visual representations. Section 3.3 describes the two steps below:
- Where to distill from. Action-Query Tuning trains the Action Expert to attend to action-relevant information across modalities.
- What to distill. We use the same action queries to retrieve features from the frozen multimodal teacher and the RGB student and then align the student’s retrieved features with the teacher’s.
(2) Quantitative evidence. We report the distillation ablation in Table 6 (ID 4 vs. ID 6). Both variants use multimodal Stage 1 and Action-Query Tuning. Adding distillation improves success by 5.53 / 5.34 percentage points under Clean / Random. These results support the benefit of explicitly aligning the RGB student’s retrieved features with the teacher’s action-relevant representations.
| Stage 1 | Action-Query Tuning | Distillation | Clean (%) ↑ | Random (%) ↑ |
|---|---|---|---|---|
| Multimodal | ✓ | ✗ | 87.23 | 86.48 |
| Multimodal | ✓ | ✓ | 92.76 | 91.82 |
Q2: Under the same rand-obj setting, why are the DexJoCo results for DP-T and π0.5 approximately 17 and 9 percentage points lower than their official results?
Response: Thank you for pointing this out. DexJoCo uses the rand-obj setting for both single-task and multi-task policies. Our paper quotes the official multi-task results from DexJoCo v1 (https://arxiv.org/pdf/2605.16257v1). The differences you mention appear to correspond to a comparison with the single-task results in DexJoCo v2 (https://arxiv.org/pdf/2605.16257v2).
(1) Source and experimental settings. The DP-T and π0.5 results in our paper are taken directly from the multi-task columns of DexJoCo v1, Table 3, which report average success rates of 33.2% and 45.5%, respectively.
Following the same multi-task training setting, we jointly train a single policy on 1,100 demonstrations covering six single-arm and five bimanual tasks and evaluate it separately on each task under rand-obj.
(2) Comparison with single-task results. DexJoCo v2, Table II reports single-task success rates of 50.4% for DP-T and 54.1% for π0.5, where a separate policy is trained for each task. Comparing these with the v1 multi-task averages quoted in our paper gives differences of 17.2 percentage points for DP-T (50.4 − 33.2) and 8.6 percentage points for π0.5 (54.1 − 45.5).
(3) Updated multi-task results in v2. We also note that DexJoCo v2, Table IV updates the official multi-task results:
- π0.5: The reported average decreases from 45.5% to 35.0%.
- DP-T: The reported average is revised from 33.2% to 28.2%, while all individual task scores remain unchanged.
We have clarified the baseline sources in Table 2 of the revised manuscript.
We sincerely thank you for your constructive review and recognition of our dataset and extensive simulation evaluation. To address your concern about training complexity and cost, we clarify the role of each training stage, describe the sources of auxiliary annotations, and explain the single-stage fine-tuning procedure for downstream real-robot tasks. We provide quantitative real-robot evaluations with baseline comparisons and ablations demonstrating the benefits of multimodal pretraining. We also explain the motivation for distillation and quantify the resulting performance gain. Finally, we clarify the sources of the DexJoCo baseline values and distinguish the single-task and multi-task results. We have revised the manuscript accordingly, with modifications highlighted in blue.
W1: The training pipeline is quite complicated and expensive. The framework involves multiple stages, including multimodal instruction tuning, action-query tuning, and joint distillation training. This makes the overall training process lengthy and introduces additional complexity.
Response: Thank you for raising this concern. As for training complexity, KRUG follows a typical pipeline for robot policies, with pre-training followed by post-training. For example, Motus uses three stages. It first adapts its video generator to embodied data, then pretrains the model with latent actions, and finally fine-tunes it on trajectories from the target robot. LingBot-VA uses two phases: pre-training on videos and robot action data, followed by post-training on task-specific demonstrations. KRUG uses its training stages to extract world knowledge from the video generator and transfer this knowledge to the action policy.
As for training cost, the multimodal videos we use to capture geometry, semantics, and motion require additional data preprocessing. The auxiliary targets come from existing dataset annotations, simulators, or pretrained estimators. This requires no additional manual labeling. Fig. 6 in Appendix C.2 of the revised manuscript summarizes the annotation sources for each dataset. Once the multimodal Wan from Stage 1 is available, adapting KRUG to a new real-robot task requires only a single fine-tuning stage (Section 4.3).
(1) Contribution of each training stage. Each stage serves a specific role in learning, retrieving, or transferring the world knowledge used for action prediction. The full model achieves 92.76% / 91.82% success on RoboTwin 2.0 under the Clean / Random settings. Table 6 of the revised manuscript demonstrates the contribution of each stage:
(2) Real-world post-training. For each real-robot task, we initialize the video expert from the Stage 1 checkpoint and jointly fine-tune it with the Action Expert on demonstrations for that task. This step requires no auxiliary-modality annotations. After fine-tuning, KRUG succeeded in 38 of 50 trials (76%) across five real-robot tasks (detailed in W2).