acceptodds
Under review as a conference paper at ICLR 2027

SIP-WAM: Enhancing World Action Models with Interaction-aware Spatiotemporal Prediction

Abstract

World Action Models (WAMs) have demonstrated strong potential for robot policy learning by jointly modeling future visual observations and actions. However, existing RGB-centric WAMs remain limited in learning interaction-aware spatiotemporal representations. Temporally sparse visual prediction provides insufficient supervision for fine-grained interaction dynamics, whereas temporally dense RGB prediction incurs substantial computational overhead. Meanwhile, full-frame RGB objectives do not explicitly prioritize task-relevant interaction regions, limiting interaction-aware spatial grounding. To address these limitations, we introduce SIP-WAM, a framework that enhances WAMs with Interaction-aware Spatiotemporal Prediction. Specifically, alongside the WAM backbone, an Interaction DiT jointly predicts the current spatial locations and temporally dense future trajectories of a sparse set of points uniformly sampled on interaction participants, providing efficient and control-relevant interaction supervision for both spatial grounding and fine-grained temporal dynamics. To avoid additional inference overhead, we couple the two branches through layerwise asymmetric joint attention, enabling the Interaction DiT to guide representation learning during training and be removed at inference. Extensive experiments on RoboCasa, LIBERO-Plus, and real-world single-arm and bimanual manipulation demonstrate strong policy performance and generalization, validating the effectiveness of interaction-aware spatiotemporal supervision for world action modeling.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, interaction-aware Spatiotemporal prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eMTYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SIP-WAM adds an auxiliary branch to Cosmos Policy to jointly predict current locations and future trajectories of task-relevant points. The branch is removed at inference. Experiments cover simulation and real-world manipulation, including co-training with human videos.

Critical strengths
  1. The auxiliary supervision adds no inference overhead, making the approach practical to deploy.

  2. The human-video experiments show promising gains in task completion, suggesting that the method can make better use of human demonstrations.

Critical weaknesses
  1. Limited novelty. Point-trajectory supervision has already been explored in JOPAT and Pri4R, and Pri4R also removes the auxiliary head at inference. Jointly predicting current locations and future trajectories of task-relevant points seems like a modest extension.

  2. Limited validation of the core design. Adding current-location prediction improves performance over temporal-only supervision by just 0.5 percentage points, with no variability reported for this ablation. The grid comparison also changes the number of points, making the benefit of interaction-focused sampling harder to isolate.

Questions for authors
  1. For LIBERO-Plus, was the RGB-only baseline also fine-tuned from the same checkpoint for the same 2,500 steps?

  2. Are the four real-world tasks trained jointly or separately? What training budgets and human/robot sampling ratios are used, and are they matched between methods?

  3. How are action and proprioception slots handled when training on human videos, and how are human hands represented in the interaction targets?

  4. How do the real-world evaluation conditions differ from training, and how are the 24 trials distributed across conditions? Could you also report full-task success rates?

5 comments

We sincerely thank you for your careful evaluation and constructive feedback, and for recognizing the practical value of our zero-inference-overhead design and the benefits of human-video co-training. We greatly appreciate your thoughtful questions and valuable suggestions, and address your concerns point by point below.

W1: Novelty relative to JOPAT and Pri4R.

A1: SIP-WAM introduces interaction-aware spatiotemporal prediction for WAMs. We explain why this constitutes more than a modest extension of existing point-trajectory supervision through three aspects.

Explicit supervision for interaction-aware spatial grounding. In contrast to methods such as Pri4R and JOPAT, SIP-WAM is, to our knowledge, the first work to explicitly treat both current and future point coordinates of interaction participants as prediction targets for WAMs, jointly supervising interaction-aware spatial grounding and temporal interaction dynamics. Our prediction-target ablation (Table 3(a)) shows that joint supervision improves success from 71.6% to 72.1% over temporal-only supervision.

Interaction-aware spatiotemporal prediction. JOPAT uses predefined grid points across the image, while SIP-WAM focuses on the interaction participants: the manipulation subject and the instruction-specified target. Our comparisons with commonly used trajectory-supervision schemes, including Grid-style and ATM-style supervision (Table 3(b)), support the effectiveness of this formulation, with gains of 1.3 and 1.4 percentage points, respectively. The action-to-video attention visualizations in Fig. 3 further illustrate increased focus on interaction-relevant regions.

Layerwise coupling and group-wise supervision. We propose layerwise asymmetric joint attention to couple the Interaction DiT and Video-Action DiT while retaining the original WAM objectives and inference pathway. This architecture improves success by 1.3 and 2.2 percentage points over sparse-layer and all-layer fusion, respectively (Table 3(c)). We also use group-wise Hungarian matching to allow flexible assignment within each interaction participant. These designs support the joint prediction formulation and help the WAM benefit more effectively from the proposed supervision.

W2: Validation of spatial grounding and interaction-focused sampling.

A2: Multi-seed prediction-target ablation. We evaluate the Temporal only and Spatial + Temporal variants across the same evaluation seeds. Following Table 3(a), both variants exclude Hungarian matching. We summarize success rates (%) as mean ± SD below.

Prediction targetSuccess rate (%), mean ± SD
Temporal only70.6 ± 0.9
Spatial + Temporal71.1 ± 0.9

Joint spatiotemporal prediction yields a paired improvement of 0.57 ± 0.06 percentage points, computed within each evaluation seed and reported as mean ± SD. The improvement holds across all evaluation seeds, supporting the stability of these performance gains. We have added these results and corresponding analysis to Appendix A.2.

Grid-style comparison with a comparable point count. We additionally evaluate Grid-style supervision using a 5 × 5 grid (25 points), comparable to the 24 predicted points per view in SIP-WAM. Other training and evaluation settings follow Table 3(b).

Supervision schemePredicted points per viewSuccess rate (%)
Grid-style, 5 × 5 grid2570.4
SIP-WAM2472.1

SIP-WAM exceeds Grid-style supervision by 1.7 percentage points, supporting the effectiveness of interaction-aware spatiotemporal prediction with comparable point counts.

Q1: Fine-tuning budget for the RGB-only baseline on LIBERO-Plus.

A1: SIP-WAM is fine-tuned from the original Cosmos Policy checkpoint on LIBERO. We have additionally fine-tuned the RGB-only Cosmos Policy baseline from the same checkpoint for the same 2,500 steps, using the same training data. Its success rate on LIBERO-Plus remains 80.6%, compared with 81.5% for SIP-WAM.

Q2: Joint training, training budgets, and human/robot sampling ratios.

A2: All four real-world tasks are trained jointly using a single policy. Cosmos Policy and SIP-WAM follow the same training budgets and sampling schedules:

  • Robot-only: 20,000 steps of shared RGB-only robot training, followed by 6,000 steps of method-specific robot training.
  • Human-video co-training: 14,000 steps of shared RGB-only robot training, followed by 6,000 steps of co-training with 60% human / 40% robot samples, and then 6,000 steps of co-training with 30% human / 70% robot samples.

Both settings therefore use 26,000 optimization steps in total, with a batch size of 448. The higher initial human sampling fraction helps the WAM adapt to future visual prediction on human manipulation videos. We subsequently increase the proportion of robot demonstrations to prioritize robot action learning while retaining human-video supervision. This progression is consistent with the staged training strategy described in GR00T 2 [1], which places greater emphasis on robot data in its final training stage.

We have clarified joint training and the matched training settings in the main-paper experimental setup (Sec. 4.4), with the detailed training protocol provided in Appendix B.2.

[1] NVIDIA et al. GR00T 2: Scaling Robot Foundation Models for Generalization, Dexterity, and Adaptation. NVIDIA Whitepaper, 2026.

Q3: Human-video slots and hand interaction targets.

A3: For human samples, both methods preserve the unified latent sequence layout and exclude the action, current and future proprioception, and value slots from the corresponding prediction losses. Future visual prediction supervision is retained.

In SIP-WAM, human hands serve as the manipulation subject, analogous to robot grippers. Within the supervised third-person view, we uniformly sample eight points on each hand and the instruction-specified target, and jointly predict their current locations and future trajectories. Points from each interaction participant form a separate matching group, following the same group-wise Hungarian matching procedure as in the simulation experiments. We have clarified these details in Appendix B.2.

Q4: Real-world evaluation conditions, trial allocation, and full-task success rates.

A4: Evaluation conditions and trial allocation. Each method is evaluated over 24 trials per task, with task-specific conditions:

  • Pick-and-place and Open drawer: 12 in-distribution (ID) and 12 out-of-distribution (OOD) trials per task. ID trials use backgrounds, object positions, and object orientations similar to training. OOD trials introduce background perturbations, including additional distractors, together with object positions and orientations not encountered during training.
  • Fold shirt: 24 ID trials.
  • Write letter: 16 ID and 8 OOD trials. OOD trials introduce background perturbations through additional distractor objects.

The evaluation conditions and trial allocations are identical across methods and training settings.

Full-task success rates. We additionally report the percentage of trials completing all required task stages, corresponding to a task completion score of 100.

TaskCosmos Policy w/o HumanCosmos Policy w/ HumanSIP-WAM w/o HumanSIP-WAM w/ Human
Pick-and-place58.354.262.566.7
Open drawer66.750.070.879.2
Fold shirt12.516.725.037.5
Write letter54.254.258.362.5
Average47.943.854.261.5

We now present both task completion scores and full-task success rates in Fig. 5 of the revised manuscript, with their definitions and the corresponding analysis included in the main-paper real-world experiments (Sec. 4.4).

We sincerely thank you again for your time and constructive feedback. We greatly appreciate your thoughtful questions and suggestions, which have helped us clarify our contributions and strengthen the experimental validation. We have uploaded the revised manuscript, with the relevant revisions highlighted in blue, and have also included a real-world demonstration video covering all four tasks in the supplementary materials.

We hope that the clarifications of our contributions, the additional controlled comparisons, and the more complete real-world evaluation details address your concerns. If so, we would greatly appreciate it if you could consider a more favorable rating in light of these clarifications and additional results.

Reviewer_axcyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a plug-in module, i.e., the Interaction DiT, for the Video-Action-Models (though named WAM) to jointly predict the 2D positions and the 2D point trajectories of the end-effectors (human hands & grippers) and the manipulated object. The effectiveness of the model is validated on both simulation benchmarks and real-world robot experiments.

Critical strengths
  1. This paper is well motivated to address the poor learning efficiency of dense pixel-level future prediction by incorporating an additional branch to predict sparse spatial locations and temporal position shifts for 2D points sampled from end-effectors (grippers and hands) and the interacted objects.
  2. The proposed method surpasses the baseline methods and the presentation is easy to follow.
Critical weaknesses
  1. The results are not presented with error bars, which undermines the reliability of some conclusions, especially for the minor improvement from Hungarian matching (0.2% in Tab.4.D), is it really necessary?
  2. The value of λint\lambda_{\text{int}}, λvis\lambda_{\text{vis}}, λFM\lambda_{\text{FM}} are not reported. As for the ablation part, there are no comparisons isolating the contribution of visibility prediction in Eq.7.
Questions for authors
  1. How well can this method be applied to occluded scenarios? In practice, the cameras may suffer from occlusions caused by the gripper itself and the robot arms.
  2. A natural direction would be to directly predict 3D point trajectories with the proposed method.
5 comments
AuthorsOpenReview ↗

Response to Reviewer axcy (1/3)

We sincerely thank you for your careful evaluation and constructive feedback, and for recognizing the motivation, effectiveness, and clear presentation of our method. We greatly appreciate your thoughtful questions and valuable suggestions concerning the experimental validation and future directions. We address these points below.

W1: Reliability of ablation gains and the contribution of Hungarian matching.

A1: We have added results for SIP-WAM with and without Hungarian matching across the same evaluation seeds, reporting success rates (%) as mean ± SD. These results are included in Appendix A.2 (Table 7).

VariantSuccess rate (%), mean ± SD
SIP-WAM, w/o Hungarian71.1 ± 0.9
SIP-WAM, w/ Hungarian71.4 ± 0.8

Hungarian matching enables more flexible assignment within each participant group, relaxing the fixed-order constraint between predictions and ground-truth point sequences, and yields a paired improvement of 0.23 ± 0.06 percentage points, computed within each evaluation seed and reported as mean ± SD. This modest improvement holds across all three evaluation seeds, supporting the stability of the performance gain.

W2: Loss weights and the contribution of visibility prediction.

A2: Loss weights. The coefficients in Eq. 7 are specified below.

SettingλFM\lambda_{\mathrm{FM}}λvis\lambda_{\mathrm{vis}}λint\lambda_{\mathrm{int}}
RoboCasa0.20.11.0
LIBERO-Plus0.10.051.0
Real-world experiments0.20.11.0

We use the same loss weights for RoboCasa and the real-world experiments. For LIBERO training, we halve both λFM\lambda_{\mathrm{FM}} and λvis\lambda_{\mathrm{vis}} to balance the gradient contribution of interaction-aware spatiotemporal prediction relative to the original WAM objective, while preserving the same 2:1 ratio between the two coefficients. We keep λint=1.0\lambda_{\mathrm{int}}=1.0 in all settings. We have added these values to Table 5 for the simulation experiments and Appendix B.2 for the real-world experiments.

Visibility prediction ablation. We conduct this ablation using the full SIP-WAM configuration, including group-wise Hungarian matching. To isolate the contribution of visibility prediction, we set λvis=0\lambda_{\mathrm{vis}}=0 while keeping all other training settings unchanged. The results are summarized below.

VariantSuccess rate (%)
SIP-WAM, w/o visibility loss71.7
Full SIP-WAM72.3

We have added this ablation and the corresponding analysis to the main-paper ablation studies (Table 3(f)).

AuthorsOpenReview ↗

Response to Reviewer axcy (2/3)

Q1: Performance in occluded scenarios.

A1: RoboCasa already contains complex occlusions during manipulation, including target objects partially blocked by the gripper and robot arms. We select several representative tasks with frequent occlusions and visualize examples in Fig. 6, with gripper masks overlaid in red and target-object masks in green.

SIP-WAM improves over Cosmos Policy* on all four illustrated tasks:

TaskCosmos Policy*SIP-WAMImprovement
OpenDrawer82.088.0+6.0
PnPMicrowaveToCounter40.052.0+12.0
TurnOffSinkFaucet84.094.0+10.0
PnPCounterToSink72.082.0+10.0

Success rates are reported in %, and improvements in percentage points.

Q2: Extending SIP-WAM to 3D point trajectory prediction.

A2: We sincerely thank you for this valuable suggestion. We conducted a preliminary experiment on RoboCasa using SpatialTrackerV2 [1] to generate 3D point trajectory pseudo-labels and train a 3D prediction variant of SIP-WAM. Without Hungarian matching, this variant achieves 71.8% success rate, compared with 72.1% for the 2D variant.

We believe that estimation errors in the 3D pseudo-labels may limit the benefits of 3D supervision in this experiment. Due to the limited rebuttal period, we have not been able to investigate this direction more extensively. Nevertheless, we consider prediction of 3D point trajectories a promising direction for future work, particularly with more accurate 3D supervision.

[1] Xiao Y, Wang J, Xue N, et al. Spatialtrackerv2: Advancing 3d point tracking with explicit camera motion[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 6726-6737.

AuthorsOpenReview ↗

Response to Reviewer axcy (3/3)

We sincerely thank you again for your positive assessment and constructive feedback. We greatly appreciate your thoughtful questions and suggestions, which have helped us strengthen the experimental validation and improve the presentation of our work.

We have uploaded the revised manuscript, with the relevant revisions highlighted in blue, and have also included a real-world demonstration video covering all four tasks in the supplementary materials. We hope that the additional results and clarifications address your questions and further strengthen your confidence in our work. We would be very grateful for your continued support.

Reviewer_fSRgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SIP-WAM, which augments a World Action Model backbone (Cosmos Policy) with an auxiliary Interaction DiT trained via flow matching and Hungarian matching to jointly predict the current 2D spatial locations and dense future trajectories of sparse points sampled on the robot gripper and target object. The two branches are coupled during training via layerwise asymmetric joint attention (where only the Interaction DiT attends to Video-Action tokens), allowing the Interaction DiT to be discarded at inference with zero runtime overhead. SIP-WAM is evaluated on RoboCasa, LIBERO-Plus, and 4 real-world manipulation tasks (including human-video co-training).

Critical strengths
  1. Zero Inference Overhead Design: Coupling the auxiliary Interaction DiT via asymmetric joint attention (Eq. 5) provides dense spatiotemporal supervision during training without increasing test-time latency.

  2. Effective Cross-Embodiment Co-Training: Applying interaction-aware point trajectory supervision to action-free human videos yields consistent gains across all 4 real-world tasks (+8.5% over robot-only training, Fig. 5), whereas the RGB-only Cosmos Policy struggles to benefit (+2.5%).

  3. Clean Formulation: Unifying current spatial grounding (Pte\mathbf{P}t^e) and future motion dynamics (Pt+1:t+He\mathbf{P}{t+1:t+H}^e) with permutation-invariant group-wise Hungarian matching (Eqs. 4, 6) avoids needing test-time query point detectors.

Critical weaknesses
  1. Marginal Overall Gains & Mixed Task-Level Performance: On LIBERO-Plus (Table 2), SIP-WAM improves overall success over Cosmos Policy by only +0.9% (81.5% vs. 80.6%) and over FoMoVLA by +1.0% (81.5% vs. 80.5%), while lagging far behind GE-Act on Robot perturbations (54.9% vs. 77.0%, -22.1%) and FoMoVLA on Background/Language perturbations (86.5% vs. 96.2%). On RoboCasa (Table 1), nearly half of the gain over original Cosmos Policy (67.1% →\to 72.3%) comes from dataset replay (Cosmos Policy reaches 69.5%), leaving a +2.8% net gain; across the 24 tasks (Table 7), SIP-WAM shows no gain or regresses on 10 tasks (e.g., -6.0% on PnPStoveToCounter and TurnOnStove, -4.0% on CoffeeSetupMug and TurnOnMicrowave).

  2. Negligible Ablation Margins & Unchanged Visual World Modeling: In Table 4, key design choices yield marginal differences that fall within cross-seed variance (Table 6): Spatial+Temporal (72.1%) beats Temporal-only (71.6%) by just +0.5%, and Hungarian matching (Table 4d) adds only +0.2% (72.1% →\to 72.3%). Moreover, Table 3 shows that SIP-WAM does not improve future visual prediction quality over RGB-only WAM (LPIPS 0.128 vs. 0.129, FID 22.92 vs. 23.34), suggesting limited impact on the backbone's visual dynamics representation.

  3. Privileged Simulator Masks vs. Unablated Foundation-Model Noise: Simulation experiments (Sec. 3.2, A.1) rely on ground-truth simulator-rendered segmentation masks rather than SAM 3, creating a gap between privileged simulation supervision and noisy real-world SAM 3 + CoTracker3 pseudo-labels. The paper lacks ablations using SAM 3 masks in simulation or analyzing sensitivity to segmentation/tracking failures in cluttered multi-object scenes.

  4. Under-Specified Multi-View Formulation & Missing Hyperparameters: While the policy takes 3 camera views (left, right, wrist in Sec. A.1/B.1), Eq. 4 defines 2D coordinates Pte∈[0,1]Ne×2\mathbf{P}t^e \in [0,1]^{N_e \times 2} without explaining how point sampling, cross-view consistency, and Hungarian matching (Eq. 6) are handled across multiple views. Additionally, the loss weights (λFM,λvis,λint\lambda{\text{FM}}, \lambda_{\text{vis}}, \lambda_{\text{int}} in Eq. 7) and the offline preprocessing cost of SAM 3 + CoTracker3 are omitted from Table 5 and the appendix.

Questions for authors
  1. Privileged Simulator Masks vs. SAM 3: How does SIP-WAM perform on RoboCasa when trained with SAM 3 + CoTracker3 masks instead of ground-truth simulator segmentation masks?

  2. Multi-View Coordinate Formulation: Given 3 RGB camera inputs (including wrist cameras where the target object frequently enters/leaves the field of view), are the Ne=8N_e = 8 points sampled and matched independently per camera view or only in a single third-person view?

  3. Statistical Significance of Ablations: Given that Spatial+Temporal over Temporal-only (+0.5%, Table 4a) and Hungarian matching (+0.2%, Table 4d) are smaller than inter-seed variance (Table 6), do these gains hold across all 3 evaluation seeds?

  4. Missing Hyperparameters & Preprocessing Cost: What are the exact values of λFM,λvis,λint\lambda_{\text{FM}}, \lambda_{\text{vis}}, \lambda_{\text{int}} in Eq. 7, and what is the GPU wall-clock time required to run SAM 3 and CoTracker3 preprocessing vs. DiT training?

  5. Real-World Trajectory Baseline: How does a trajectory-supervised baseline (e.g., Grid-style / FoMoVLA or ATM-style from Table 4b) perform in the real-world human-video co-training setting (Fig. 5)?

7 comments

Thank you for your careful evaluation and constructive feedback, and for recognizing the zero-inference-overhead design, the benefits of human-video co-training, and our formulation. We also appreciate your valuable questions, which have helped us identify aspects that were not fully addressed in the original submission. We address your concerns point by point below.

W1: Overall improvements and task-level performance.

A1: LIBERO-Plus. SIP-WAM achieves a strong overall success rate of 81.5%. Our goal is to establish the value of interaction-aware spatiotemporal prediction through a simple and clean architecture, rather than pursuing SOTA. Therefore, we also did not perform targeted hyperparameter tuning for LIBERO-Plus. Achieving large additional gains over an already strong baseline is challenging on LIBERO-Plus. Nevertheless, against its already strong direct RGB-only baseline, Cosmos Policy, SIP-WAM improves in 6/7 categories and ties in the remaining category. It also outperforms GE-Act in 5/7 categories and FoMoVLA in 4/7. Substantial gains include Camera (+12.9 percentage points) and Language (+12.4 points) over GE-Act, and Camera (+9.6 points) and Noise (+6.9 points) over FoMoVLA. Beyond LIBERO-Plus, SIP-WAM achieves consistent improvements on RoboCasa and substantial improvements in the real-world experiments.

RoboCasa. Against its matched baseline, Cosmos Policy*, SIP-WAM improves success from 69.5% to 72.3%, yielding a 2.8-percentage-point net gain after accounting for dataset replay, which is not modest. This aggregate improvement holds across all three evaluation seeds: +3.0, +2.8, and +3.4 percentage points (Table 6). Although gains vary across tasks, SIP-WAM improves performance across multiple manipulation types, including pick-and-place, door and drawer manipulation, and faucet operation. Examples in Table 8 of the revised manuscript include +12 points on PnPMicrowaveToCounter, +12 on CloseDoubleDoor, +6 on OpenDrawer, +12 on TurnOnSinkFaucet, and +10 on TurnOffSinkFaucet.

W2: Reliability of ablation gains and future visual prediction quality.

A2: Multi-seed ablation results. The gains from joint spatiotemporal prediction and Hungarian matching remain consistent across all evaluation seeds, as summarized below. We summarize success rates as mean ± sample SD below.

VariantSuccess rate (%)
Temporal only, w/o Hungarian70.6 ± 0.9
Spatial + Temporal, w/o Hungarian71.1 ± 0.9
Spatial + Temporal, w/ Hungarian71.4 ± 0.8

Without Hungarian matching, Spatial + Temporal outperforms Temporal only by 0.57 ± 0.06 percentage points, and Hungarian matching provides a further improvement of 0.23 ± 0.06 points. These gains are computed within each evaluation seed and reported as mean ± sample SD. Both improvements hold across all three evaluation seeds, supporting the stability of these performance gains. We have added these results and corresponding analysis to Appendix A.2 (Table 7) of the revised manuscript.

Future visual prediction. Substantial improvements in visual generation often stem from stronger generative foundation models or training datasets specifically curated for visual generation. Our RGB-only WAM baseline already benefits from a strong pretrained video foundation model and has been adapted to future RGB prediction on the RoboCasa domain. With the foundation model and training demonstrations held fixed, our proposed interaction-aware spatiotemporal supervision still achieves modest improvements in both visual prediction metrics (Table 4). This supervision also improves policy performance, which is our primary objective, while maintaining comparable future visual prediction quality, with modest improvements in both metrics.

W3: SAM 3 pseudo-label supervision in simulation.

A3: Thank you for this thoughtful and valuable suggestion concerning the gap between simulator-rendered masks and foundation-model pseudo-labels. We have added an experiment on RoboCasa using SAM 3 masks instead of simulator-rendered segmentation masks. This experiment is conducted without Hungarian matching, keeping the RGB training data, initialization, training budget, and other training settings unchanged. Further details of the SAM 3 + CoTracker3 preprocessing pipeline are provided in our response to W4.

Trained with these pseudo-labels, SIP-WAM achieves 71.6% success rate, compared with 72.1% using simulator masks and 69.5% for its matched RGB-only baseline, Cosmos Policy*. SIP-WAM retains a 2.1-percentage-point gain over the RGB-only baseline. These results suggest that the benefits of interaction-aware spatiotemporal prediction persist under SAM 3 pseudo-label noise. In this setting, we believe that more accurate segmentation and tracking models could further improve the quality of the interaction targets and lead to larger policy gains.

We have added this experiment to Table 3(e) and the corresponding Q5 discussion in the revised manuscript.

W4: Multi-view formulation, loss weights, and offline preprocessing cost.

A4: We sincerely thank you again for these thoughtful and detailed questions, and for the time and care devoted to examining our formulation and implementation. We greatly appreciate your constructive feedback and valuable suggestions, which have helped us strengthen the clarity and completeness of our presentation.

Multi-view formulation. The Video-Action DiT receives multi-view RGB observations, and the Interaction DiT jointly predicts current and future 2D point coordinates in third-person views: the left and right third-person views for RoboCasa, and a single third-person view for LIBERO and the real-world experiments.

Point sampling and tracking are performed independently in each supervised view. Within each view, we uniformly sample eight points on each interaction participant and treat the points from each participant as a separate matching group. Group-wise Hungarian matching (Eq. 6) is performed independently within each such group. We construct a pairwise cost matrix whose entries measure coordinate discrepancies between predicted and ground-truth point sequences over the complete current-to-future horizon. The Hungarian algorithm then finds the one-to-one assignment that minimizes the total matching cost. We do not establish explicit point correspondences across views or use a cross-view consistency loss. We have clarified these details in Sec. 3.2 and Appendices A.1 and B.2.

Loss weights. The coefficients in Eq. 7 are specified below.

SettingλFM\lambda_{\mathrm{FM}}λvis\lambda_{\mathrm{vis}}λint\lambda_{\mathrm{int}}
RoboCasa0.20.11.0
LIBERO-Plus0.10.051.0
Real-world experiments0.20.11.0

We use the same loss weights for RoboCasa and the real-world experiments. For LIBERO training, we halve both λFM\lambda_{\mathrm{FM}} and λvis\lambda_{\mathrm{vis}} to balance the gradient contribution of interaction-aware spatiotemporal prediction relative to the original WAM objective, while preserving the same 2:1 ratio between the two coefficients. We keep λint\lambda_{\mathrm{int}} in all settings. We have added these values to Table 5 for the simulation experiments and Appendix B.2 for the real-world experiments.

Offline preprocessing cost. Because RoboCasa contains complex, cluttered scenes, directly applying SAM 3 alone does not provide sufficiently reliable segmentation. We therefore use an automated Codex + SAM 3 pipeline for the left and right third-person videos: Codex selects anchor frames and provides bounding-box and point prompts, SAM 3 propagates the masks, and CoTracker3 tracks points uniformly sampled from each interaction participant’s mask to obtain trajectories and visibility labels.

SAM 3 segmentation, CoTracker3 tracking, and joint DiT training each use eight NVIDIA A800 80GB GPUs. The wall-clock times for our full RoboCasa training dataset (approximately 1,200 demonstrations) are summarized below.

StageTime (hours)
Codex anchor selection and prompt preparation12.2
SAM 3 segmentation and mask propagation5.1
Point sampling and CoTracker3 tracking8.0
Total offline preprocessing25.3
Joint Video-Action DiT and Interaction DiT training40.0

We also reviewed recent segmentation approaches that could strengthen our annotation pipeline [1, 2]. We regret that time constraints during the rebuttal period prevented us from validating these alternatives in our setting. Nevertheless, our real-world experiments already demonstrate substantial policy gains using SAM 3 + CoTracker3 pseudo-labels. We believe that more accurate segmentation and tracking pipelines could further improve the quality of the interaction targets and lead to larger policy gains, with the potential to support more effective learning from larger datasets.

[1] Mei H, Huang Q, Ci H, et al. Robotseg: A model and dataset for segmenting robots in image and video[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 14346-14356.

[2] Du T, Li H, Fan Z, et al. SAM-veteran: An MLLM-based human-like SAM agent for reasoning segmentation[C]//The Fourteenth International Conference on Learning Representations. 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank the AC and all reviewers for the time and care you have devoted to evaluating our work and for your constructive feedback. We have uploaded the revised manuscript, with relevant revisions highlighted in blue, and a real-world demonstration video covering all four tasks in the supplementary materials. Below, we summarize the strengths highlighted in the reviews and our responses to the main concerns.

Strengths highlighted by the reviewers

  • Zero inference overhead: The Interaction DiT is removed at inference, preserving the original deployment cost (fSRg, eMTY).
  • Benefits of human-video co-training: Promising gains from human manipulation videos (fSRg, eMTY).
  • Motivation and formulation: Clear motivation and presentation (axcy) and a clean formulation of joint spatial and temporal prediction (fSRg).

Responses to concerns shared across reviewers

ConcernReviewersResponse and supporting evidence
Reliability of ablation gainsfSRg, axcy, eMTYWe provide multi-seed evaluation of the spatiotemporal prediction target and Hungarian-matching ablations. Their paired gains are 0.57 ± 0.06 and 0.23 ± 0.06 percentage points, respectively (mean ± sample SD), with positive gains across all three seeds (Table 7).
Loss weights and visibility predictionfSRg, axcyWe report the loss weights in Table 5 and Appendix B.2. Removing visibility loss from the full configuration reduces success from 72.3% to 71.7% (Table 3(f)).
Real-world comparisons and evaluation detailsfSRg, eMTYWe clarify joint training, matched budgets and sampling schedules, and ID/OOD evaluation protocols. With human-video co-training, SIP-WAM exceeds Grid-style and ATM-style supervision by 5.2 and 5.7 points in average completion score.

Additional concerns raised by individual reviewers

ConcernReviewerResponse and supporting evidence
Overall improvements and future visual prediction qualityfSRgSIP-WAM achieves 81.5% on LIBERO-Plus, improving over its direct baseline in 6/7 categories and tying in the remaining category. On RoboCasa, success improves from 69.5% to 72.3% over Cosmos Policy*, consistently across three evaluation seeds and multiple manipulation types. With the same foundation model and demonstrations, future visual prediction quality remains comparable, with modest improvements in both metrics (Table 4).
Novelty and interaction-focused samplingeMTYTo our knowledge, SIP-WAM is the first work to explicitly treat both current and future point coordinates of interaction participants as prediction targets for WAMs, jointly supervising interaction-aware spatial grounding and temporal interaction dynamics. Layerwise asymmetric joint attention retains the original WAM objectives and inference pathway. An additional 5×5 Grid-style comparison yields 70.4% vs. 72.1% for SIP-WAM, supporting interaction-focused sampling with comparable point counts.
Dependence on simulator-rendered masksfSRgUsing SAM 3 pseudo-labels, SIP-WAM achieves 71.6%, compared with 72.1% using simulator masks, while retaining a 2.1-point gain over Cosmos Policy*. Both mask variants exclude Hungarian matching (Table 3(e)).

We also provide a detailed multi-view formulation, clarify offline preprocessing cost, and discuss occluded scenarios and future directions for 3D prediction in our detailed responses.

We sincerely appreciate your consideration and hope these additional experiments and clarifications strengthen your confidence in our contributions. We respectfully ask that this additional evidence be considered in the final assessment of our work.

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.