- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SIP-WAM adds an auxiliary branch to Cosmos Policy to jointly predict current locations and future trajectories of task-relevant points. The branch is removed at inference. Experiments cover simulation and real-world manipulation, including co-training with human videos.
Critical strengths
-
The auxiliary supervision adds no inference overhead, making the approach practical to deploy.
-
The human-video experiments show promising gains in task completion, suggesting that the method can make better use of human demonstrations.
Critical weaknesses
-
Limited novelty. Point-trajectory supervision has already been explored in JOPAT and Pri4R, and Pri4R also removes the auxiliary head at inference. Jointly predicting current locations and future trajectories of task-relevant points seems like a modest extension.
-
Limited validation of the core design. Adding current-location prediction improves performance over temporal-only supervision by just 0.5 percentage points, with no variability reported for this ablation. The grid comparison also changes the number of points, making the benefit of interaction-focused sampling harder to isolate.
Questions for authors
-
For LIBERO-Plus, was the RGB-only baseline also fine-tuned from the same checkpoint for the same 2,500 steps?
-
Are the four real-world tasks trained jointly or separately? What training budgets and human/robot sampling ratios are used, and are they matched between methods?
-
How are action and proprioception slots handled when training on human videos, and how are human hands represented in the interaction targets?
-
How do the real-world evaluation conditions differ from training, and how are the 24 trials distributed across conditions? Could you also report full-task success rates?
5 comments
Q1: Fine-tuning budget for the RGB-only baseline on LIBERO-Plus.
A1: SIP-WAM is fine-tuned from the original Cosmos Policy checkpoint on LIBERO. We have additionally fine-tuned the RGB-only Cosmos Policy baseline from the same checkpoint for the same 2,500 steps, using the same training data. Its success rate on LIBERO-Plus remains 80.6%, compared with 81.5% for SIP-WAM.
Q2: Joint training, training budgets, and human/robot sampling ratios.
A2: All four real-world tasks are trained jointly using a single policy. Cosmos Policy and SIP-WAM follow the same training budgets and sampling schedules:
- Robot-only: 20,000 steps of shared RGB-only robot training, followed by 6,000 steps of method-specific robot training.
- Human-video co-training: 14,000 steps of shared RGB-only robot training, followed by 6,000 steps of co-training with 60% human / 40% robot samples, and then 6,000 steps of co-training with 30% human / 70% robot samples.
Both settings therefore use 26,000 optimization steps in total, with a batch size of 448. The higher initial human sampling fraction helps the WAM adapt to future visual prediction on human manipulation videos. We subsequently increase the proportion of robot demonstrations to prioritize robot action learning while retaining human-video supervision. This progression is consistent with the staged training strategy described in GR00T 2 [1], which places greater emphasis on robot data in its final training stage.
We have clarified joint training and the matched training settings in the main-paper experimental setup (Sec. 4.4), with the detailed training protocol provided in Appendix B.2.
[1] NVIDIA et al. GR00T 2: Scaling Robot Foundation Models for Generalization, Dexterity, and Adaptation. NVIDIA Whitepaper, 2026.
Q3: Human-video slots and hand interaction targets.
A3: For human samples, both methods preserve the unified latent sequence layout and exclude the action, current and future proprioception, and value slots from the corresponding prediction losses. Future visual prediction supervision is retained.
In SIP-WAM, human hands serve as the manipulation subject, analogous to robot grippers. Within the supervised third-person view, we uniformly sample eight points on each hand and the instruction-specified target, and jointly predict their current locations and future trajectories. Points from each interaction participant form a separate matching group, following the same group-wise Hungarian matching procedure as in the simulation experiments. We have clarified these details in Appendix B.2.
Q4: Real-world evaluation conditions, trial allocation, and full-task success rates.
A4: Evaluation conditions and trial allocation. Each method is evaluated over 24 trials per task, with task-specific conditions:
- Pick-and-place and Open drawer: 12 in-distribution (ID) and 12 out-of-distribution (OOD) trials per task. ID trials use backgrounds, object positions, and object orientations similar to training. OOD trials introduce background perturbations, including additional distractors, together with object positions and orientations not encountered during training.
- Fold shirt: 24 ID trials.
- Write letter: 16 ID and 8 OOD trials. OOD trials introduce background perturbations through additional distractor objects.
The evaluation conditions and trial allocations are identical across methods and training settings.
Full-task success rates. We additionally report the percentage of trials completing all required task stages, corresponding to a task completion score of 100.
| Task | Cosmos Policy w/o Human | Cosmos Policy w/ Human | SIP-WAM w/o Human | SIP-WAM w/ Human |
|---|---|---|---|---|
| Pick-and-place | 58.3 | 54.2 | 62.5 | 66.7 |
| Open drawer | 66.7 | 50.0 | 70.8 | 79.2 |
| Fold shirt | 12.5 | 16.7 | 25.0 | 37.5 |
| Write letter | 54.2 | 54.2 | 58.3 | 62.5 |
| Average | 47.9 | 43.8 | 54.2 | 61.5 |
We now present both task completion scores and full-task success rates in Fig. 5 of the revised manuscript, with their definitions and the corresponding analysis included in the main-paper real-world experiments (Sec. 4.4).
We sincerely thank you again for your time and constructive feedback. We greatly appreciate your thoughtful questions and suggestions, which have helped us clarify our contributions and strengthen the experimental validation. We have uploaded the revised manuscript, with the relevant revisions highlighted in blue, and have also included a real-world demonstration video covering all four tasks in the supplementary materials.
We hope that the clarifications of our contributions, the additional controlled comparisons, and the more complete real-world evaluation details address your concerns. If so, we would greatly appreciate it if you could consider a more favorable rating in light of these clarifications and additional results.
We sincerely thank you for your careful evaluation and constructive feedback, and for recognizing the practical value of our zero-inference-overhead design and the benefits of human-video co-training. We greatly appreciate your thoughtful questions and valuable suggestions, and address your concerns point by point below.
W1: Novelty relative to JOPAT and Pri4R.
A1: SIP-WAM introduces interaction-aware spatiotemporal prediction for WAMs. We explain why this constitutes more than a modest extension of existing point-trajectory supervision through three aspects.
Explicit supervision for interaction-aware spatial grounding. In contrast to methods such as Pri4R and JOPAT, SIP-WAM is, to our knowledge, the first work to explicitly treat both current and future point coordinates of interaction participants as prediction targets for WAMs, jointly supervising interaction-aware spatial grounding and temporal interaction dynamics. Our prediction-target ablation (Table 3(a)) shows that joint supervision improves success from 71.6% to 72.1% over temporal-only supervision.
Interaction-aware spatiotemporal prediction. JOPAT uses predefined grid points across the image, while SIP-WAM focuses on the interaction participants: the manipulation subject and the instruction-specified target. Our comparisons with commonly used trajectory-supervision schemes, including Grid-style and ATM-style supervision (Table 3(b)), support the effectiveness of this formulation, with gains of 1.3 and 1.4 percentage points, respectively. The action-to-video attention visualizations in Fig. 3 further illustrate increased focus on interaction-relevant regions.
Layerwise coupling and group-wise supervision. We propose layerwise asymmetric joint attention to couple the Interaction DiT and Video-Action DiT while retaining the original WAM objectives and inference pathway. This architecture improves success by 1.3 and 2.2 percentage points over sparse-layer and all-layer fusion, respectively (Table 3(c)). We also use group-wise Hungarian matching to allow flexible assignment within each interaction participant. These designs support the joint prediction formulation and help the WAM benefit more effectively from the proposed supervision.
W2: Validation of spatial grounding and interaction-focused sampling.
A2: Multi-seed prediction-target ablation. We evaluate the Temporal only and Spatial + Temporal variants across the same evaluation seeds. Following Table 3(a), both variants exclude Hungarian matching. We summarize success rates (%) as mean ± SD below.
Joint spatiotemporal prediction yields a paired improvement of 0.57 ± 0.06 percentage points, computed within each evaluation seed and reported as mean ± SD. The improvement holds across all evaluation seeds, supporting the stability of these performance gains. We have added these results and corresponding analysis to Appendix A.2.
Grid-style comparison with a comparable point count. We additionally evaluate Grid-style supervision using a 5 × 5 grid (25 points), comparable to the 24 predicted points per view in SIP-WAM. Other training and evaluation settings follow Table 3(b).
SIP-WAM exceeds Grid-style supervision by 1.7 percentage points, supporting the effectiveness of interaction-aware spatiotemporal prediction with comparable point counts.