- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents STARRY, a spatial-temporal world action model that jointly predicts future scene representations and robot actions using diffusion. The model combines RGB, depth, and end-effector trajectories in a unified latent representation. It also introduces Geometry-Aware Selective Attention Modulation (GASAM), which uses predicted 3D geometry to guide the attention between actions and video features. On 50 RoboTwin 2.0 tasks, STARRY achieves 93.82% and 93.30% success under Clean and Randomized settings, respectively. Real-world bimanual manipulation experiments also show promising results, with an average success rate of 67.1% compared with 35.0% for π0.5.
Critical strengths
-
The paper presents an interesting combination of spatial-temporal world modeling and geometry-guided action prediction. Rather than predicting RGB frames alone, STARRY jointly models future appearance, depth, and end-effector motion. I find the GASAM module particularly relevant here, as it uses the predicted 3D relationships to influence action generation through attention modulation (Eqs. 5–9). This provides an explicit connection between geometric prediction and action generation, which is a useful design choice for manipulation tasks.
-
The ablation studies provide reasonable evidence for the effectiveness of the main components. In Table 3(a), adding GASAM improves performance from 88.82% to 93.30% under the Randomized setting with Full-ST modeling. Similarly, when GASAM is enabled, moving from RGB-only prediction to Full-ST modeling improves success from 86.96% to 93.30%. Table 3(b) further examines the contributions of depth and trajectory information separately. These results suggest that the spatial-temporal representation and geometry-aware attention provide complementary benefits, rather than the improvement coming solely from the pretrained model.
-
The evaluation is relatively extensive, and the real-world results are encouraging. The authors evaluate STARRY against several competitive baselines on 50 RoboTwin 2.0 tasks. The improvements are especially noticeable on geometry-sensitive tasks such as Hanging Mug and Handover Mic (Table 1). I also appreciate that the physical experiments include repeated evaluations rather than just a small number of demonstrations. According to Appendix B.2, the authors conduct four evaluation batches with 80 trials per task-stage setting. STARRY achieves 67.1 ± 3.1% success compared with 35.0 ± 5.6% for π0.5, with improvements across all six task-stage settings. This provides useful evidence that the method can transfer to real robotic manipulation.
Critical weaknesses
-
The methodological novelty, particularly the specific design of GASAM, could be better justified. Joint video-action diffusion has already been explored in Unified World Models (Zhu et al., RSS 2025) and Motus (Bi et al., CVPR 2026). GeoPredict (Qian et al., CVPR 2026) also incorporates predicted geometric and kinematic information into manipulation policies. The main distinction of STARRY appears to be its use of predicted geometry to modulate attention during inference. However, the formulation in Eqs. 7–9 is essentially a distance-based bias added to the attention logits. While Table 3 shows that GASAM improves performance, it is not clear whether this particular design offers an advantage over simpler alternatives, such as directly conditioning on geometric features, using a learned spatial attention bias, or applying a basic distance prior. I think such comparisons would help establish what is specifically gained from GASAM beyond existing geometry-conditioning approaches.
-
There is limited direct evidence that GASAM actually improves geometric grounding. The module uses future metric depth and end-effector positions predicted by the Geometry Expert (Eq. 5), but the paper does not quantitatively evaluate the accuracy of these predictions. It is also unclear whether the resulting attention weights consistently focus on the correct interaction regions. Figure 3 provides some qualitative visualizations, and the ablations in Table 3 demonstrate improvements in task success. However, these results alone do not establish that the gains come from more accurate future geometry or better spatial alignment. They could also partly result from the additional geometric supervision. Since geometry-guided action generation is a central contribution of the paper, I would like to see more direct evidence supporting this mechanism.
-
The comparisons with competing methods leave some uncertainty about the source and significance of the improvements. Although the authors use the same RoboTwin adaptation budget, the evaluated methods have different pretrained backbones and pretraining resources, as described in Appendix A.3. STARRY also makes use of depth information and geometric supervision that may not be available to all baselines. This makes it difficult to attribute the improvements entirely to the proposed architecture. Moreover, the gaps over the strongest baseline are fairly small: 93.82% versus 92.93% under Clean settings and 93.30% versus 91.55% under Randomized settings (Table 1). Without variability across simulation runs or independent training seeds, it is hard to judge how reliable these differences are. More controlled comparisons regarding input modalities and supervision would make the empirical claims stronger.
-
The inference latency may limit the practicality of the method for closed-loop manipulation. Appendix A.4 reports around 2.8 seconds of end-to-end inference time on an A100 GPU. However, according to Appendix A.1, a 16-action chunk takes only 1.6 seconds to execute at the nominal 10 Hz control frequency, and the paper mentions additional synchronous inference time between updates. This raises a concern about how frequently the policy can actually respond to new observations during execution, especially when the target or environment changes. Although GASAM itself introduces only 5.1% additional overhead, the overall inference time seems more relevant to practical deployment. I think the paper should discuss this limitation more explicitly and clarify how inference latency affects closed-loop responsiveness.
Questions for authors
-
Could the authors compare GASAM with simpler ways of incorporating geometry, such as directly conditioning the Action Expert on predicted depth and end-effector features, or using a standard learned spatial attention bias? It would be helpful to keep the training setup and geometric supervision the same.
-
Can the authors provide quantitative results for future depth and pose prediction accuracy? It would also be interesting to evaluate whether GASAM's attention weights align with ground-truth interaction regions. This could help clarify whether the improvement comes from better geometric grounding or mainly from the additional supervision.
-
How many evaluation rollouts and independent training seeds were used for the RoboTwin 2.0 experiments? Could the authors report uncertainty estimates for the relatively small gains over LingBot-VA? I would also appreciate clarification on whether the baselines receive comparable input modalities and geometric supervision.
-
How is the approximately 2.8-second inference time handled during real-world execution? What is the effective policy update frequency, and can inference run asynchronously while the robot executes the previous action chunk?