acceptodds
Withdrawn from ICLR 2027

STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial-temporal prediction and geometric grounding are increasingly critical for high-precision robotic manipulation, yet remain challenging for world action models (WAMs). While these models capture temporal context through future prediction, RGB-only modeling does not explicitly represent the 3D geometry required for precise control. Consequently, visually plausible futures may still fail to expose action-relevant spatial constraints, limiting their ability to improve action generation. We propose STARRY, a world action model that aligns future spatial-temporal prediction with action generation through a unified diffusion process, enabling actions to be informed by the predicted evolution of scene appearance, spatial geometry, and end-effector motion. Meanwhile, to make more effective use of the predicted spatial-temporal information for action generation, we introduce Geometry-Aware Selective Attention Modulation (GASAM), which converts geometric cues derived from the predicted spatial-temporal variables into token-aligned weights, selectively directing action attention toward regions relevant to future interactions. On RoboTwin 2.0, STARRY achieves 93.82% / 93.30% average success under Clean and Randomized settings across 50 bimanual tasks. Real-world experiments show that STARRY improves average success from 35.0% to 67.1% across four evaluation batches compared with . These results demonstrate that spatial-temporal action-centric world modeling improves robotic manipulation by aligning future spatial-temporal prediction with action generation and selectively guiding action attention with predicted geometry.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Action-Centric World Modeling, Spatial-Temporal Representations, Geometry-Aware Attention Modulation, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R7vJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents STARRY, a spatial-temporal world action model that jointly predicts future scene representations and robot actions using diffusion. The model combines RGB, depth, and end-effector trajectories in a unified latent representation. It also introduces Geometry-Aware Selective Attention Modulation (GASAM), which uses predicted 3D geometry to guide the attention between actions and video features. On 50 RoboTwin 2.0 tasks, STARRY achieves 93.82% and 93.30% success under Clean and Randomized settings, respectively. Real-world bimanual manipulation experiments also show promising results, with an average success rate of 67.1% compared with 35.0% for π0.5.

Critical strengths
  1. The paper presents an interesting combination of spatial-temporal world modeling and geometry-guided action prediction. Rather than predicting RGB frames alone, STARRY jointly models future appearance, depth, and end-effector motion. I find the GASAM module particularly relevant here, as it uses the predicted 3D relationships to influence action generation through attention modulation (Eqs. 5–9). This provides an explicit connection between geometric prediction and action generation, which is a useful design choice for manipulation tasks.

  2. The ablation studies provide reasonable evidence for the effectiveness of the main components. In Table 3(a), adding GASAM improves performance from 88.82% to 93.30% under the Randomized setting with Full-ST modeling. Similarly, when GASAM is enabled, moving from RGB-only prediction to Full-ST modeling improves success from 86.96% to 93.30%. Table 3(b) further examines the contributions of depth and trajectory information separately. These results suggest that the spatial-temporal representation and geometry-aware attention provide complementary benefits, rather than the improvement coming solely from the pretrained model.

  3. The evaluation is relatively extensive, and the real-world results are encouraging. The authors evaluate STARRY against several competitive baselines on 50 RoboTwin 2.0 tasks. The improvements are especially noticeable on geometry-sensitive tasks such as Hanging Mug and Handover Mic (Table 1). I also appreciate that the physical experiments include repeated evaluations rather than just a small number of demonstrations. According to Appendix B.2, the authors conduct four evaluation batches with 80 trials per task-stage setting. STARRY achieves 67.1 ± 3.1% success compared with 35.0 ± 5.6% for π0.5, with improvements across all six task-stage settings. This provides useful evidence that the method can transfer to real robotic manipulation.

Critical weaknesses
  1. The methodological novelty, particularly the specific design of GASAM, could be better justified. Joint video-action diffusion has already been explored in Unified World Models (Zhu et al., RSS 2025) and Motus (Bi et al., CVPR 2026). GeoPredict (Qian et al., CVPR 2026) also incorporates predicted geometric and kinematic information into manipulation policies. The main distinction of STARRY appears to be its use of predicted geometry to modulate attention during inference. However, the formulation in Eqs. 7–9 is essentially a distance-based bias added to the attention logits. While Table 3 shows that GASAM improves performance, it is not clear whether this particular design offers an advantage over simpler alternatives, such as directly conditioning on geometric features, using a learned spatial attention bias, or applying a basic distance prior. I think such comparisons would help establish what is specifically gained from GASAM beyond existing geometry-conditioning approaches.

  2. There is limited direct evidence that GASAM actually improves geometric grounding. The module uses future metric depth and end-effector positions predicted by the Geometry Expert (Eq. 5), but the paper does not quantitatively evaluate the accuracy of these predictions. It is also unclear whether the resulting attention weights consistently focus on the correct interaction regions. Figure 3 provides some qualitative visualizations, and the ablations in Table 3 demonstrate improvements in task success. However, these results alone do not establish that the gains come from more accurate future geometry or better spatial alignment. They could also partly result from the additional geometric supervision. Since geometry-guided action generation is a central contribution of the paper, I would like to see more direct evidence supporting this mechanism.

  3. The comparisons with competing methods leave some uncertainty about the source and significance of the improvements. Although the authors use the same RoboTwin adaptation budget, the evaluated methods have different pretrained backbones and pretraining resources, as described in Appendix A.3. STARRY also makes use of depth information and geometric supervision that may not be available to all baselines. This makes it difficult to attribute the improvements entirely to the proposed architecture. Moreover, the gaps over the strongest baseline are fairly small: 93.82% versus 92.93% under Clean settings and 93.30% versus 91.55% under Randomized settings (Table 1). Without variability across simulation runs or independent training seeds, it is hard to judge how reliable these differences are. More controlled comparisons regarding input modalities and supervision would make the empirical claims stronger.

  4. The inference latency may limit the practicality of the method for closed-loop manipulation. Appendix A.4 reports around 2.8 seconds of end-to-end inference time on an A100 GPU. However, according to Appendix A.1, a 16-action chunk takes only 1.6 seconds to execute at the nominal 10 Hz control frequency, and the paper mentions additional synchronous inference time between updates. This raises a concern about how frequently the policy can actually respond to new observations during execution, especially when the target or environment changes. Although GASAM itself introduces only 5.1% additional overhead, the overall inference time seems more relevant to practical deployment. I think the paper should discuss this limitation more explicitly and clarify how inference latency affects closed-loop responsiveness.

Questions for authors
  1. Could the authors compare GASAM with simpler ways of incorporating geometry, such as directly conditioning the Action Expert on predicted depth and end-effector features, or using a standard learned spatial attention bias? It would be helpful to keep the training setup and geometric supervision the same.

  2. Can the authors provide quantitative results for future depth and pose prediction accuracy? It would also be interesting to evaluate whether GASAM's attention weights align with ground-truth interaction regions. This could help clarify whether the improvement comes from better geometric grounding or mainly from the additional supervision.

  3. How many evaluation rollouts and independent training seeds were used for the RoboTwin 2.0 experiments? Could the authors report uncertainty estimates for the relatively small gains over LingBot-VA? I would also appreciate clarification on whether the baselines receive comparable input modalities and geometric supervision.

  4. How is the approximately 2.8-second inference time handled during real-world execution? What is the effective policy update frequency, and can inference run asynchronously while the robot executes the previous action chunk?

Reviewer_KSPtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STARRY, a world action model for bimanual robotic manipulation that jointly denoises future action sequences and spatial-temporal latent variables (multi-view RGB, JET-rendered depth, and 2D-projected end-effector trajectories). To guide action generation with 3D geometry, the authors introduce a Geometry Expert and Geometry-Aware Selective Attention Modulation (GASAM), which predicts future depth and 3D end-effector poses to reweight action-to-video attention logits based on 3D Euclidean distance. STARRY is evaluated across 50 simulated tasks on RoboTwin 2.0 and 3 real-world bimanual tasks on an ARX R5 robot.

Critical strengths

Well-Motivated Problem Formulation: The paper clearly identifies two practical bottlenecks in current WAMs: (i) predicting visually plausible RGB futures does not guarantee physically valid 3D contact/alignment geometry, and (ii) global cross-attention between action queries and 2D visual tokens struggles to prioritize metrically proximate interaction regions.

Comprehensive Simulation Benchmark & Ablation Coverage: Evaluating across all 50 bimanual tasks in RoboTwin 2.0 (Clean and Randomized) provides broad empirical coverage. The authors also transparently report task-level breakdowns (Tables 11–12), non-saturated task subsets (Table 10), and sensitivity to estimated depth via Depth Anything V3 (Table 8).

Strong Gains on Geometrically Constrained Multi-Stage Tasks: On tasks requiring fine bimanual coordination or 3D spatial alignment—such as Hanging Mug (+34 points over Motus in Randomized) and Handover Mic (99% vs. Motus 63%)—incorporating depth and trajectory priors yields clear empirical benefits.

Critical weaknesses
  1. Confounded GASAM Ablation & Negligible Logit Bias: In Appendix A.2 and Table 7, GASAM weights w∈[0.9,1.1]w \in [0.9, 1.1] with λ=1\lambda = 1 add a tiny logit bias λlog⁡(w)∈[−0.105,+0.095]\lambda \log(w) \in [-0.105, +0.095] (at most a 1.22×1.22\times attention preference, Eq. 9), which can hardly explain the +10.92% (Action-only) and +4.48% (Full ST) gains in Table 3(a). Because the "−- GASAM" baseline removes both attention modulation and the auxiliary supervision Lgeo\mathcal{L}{\text{geo}} (Eq. 12), the gains likely stem from Lgeo\mathcal{L}{\text{geo}} regularizing intermediate features rather than GASAM modulation; notably, Table 3(c) tests λ∈0.5,1,2\lambda \in 0.5, 1, 2 but omits λ=0\lambda = 0.

  2. Prohibitive Inference Latency (~2.8s/query): Table 6 reports an end-to-end latency of 2800.1 ms (41.58 GiB VRAM) on an A100 GPU for a 1.6s action chunk with synchronous inference (Appendix A.1). Pausing 2.8s after every 1.6s of execution (>63% dead time) prevents closed-loop reactivity, and latency comparisons against baselines are missing.

  3. Visual Occlusion from Image-Space Trajectory Overlays: Encoding depth via non-linear JET colormaps and drawing 2D trajectories directly onto RGB images (Eq. 14) occludes fine-grained terminal visual cues. Consequently, STARRY suffers severe drops on terminal-precision tasks compared to LingBot-VA/Motus (Table 10/11), including Handover Block (48% vs. 78%, -30%), Open Microwave (61% vs. 91%, -30%), and Blocks Ranking Size (74% vs. 96%, -22%).

  4. Weak Real-World Baseline & Unexplained Depth Scale Calibration: While STARRY's simulation margin over LingBot-VA is small (+0.89% Clean, +1.75% Randomized), real-world experiments (Table 2) only compare against the weakest baseline (π0.5\pi_{0.5}, 60.30%) rather than Motus or LingBot-VA. Moreover, the paper does not explain how scale/shift ambiguity in monocular Depth Anything V3 (Table 8) is resolved when unprojecting multi-view depths into a common 3D metric frame (Eq. 6).

Questions for authors
  1. Decoupling Lgeo\mathcal{L}_{\text{geo}} from GASAM: What is the success rate of the Full-ST checkpoint in Table 3(c) at inference-time λ=0\lambda = 0, and how does training with Lgeo\mathcal{L}_{\text{geo}} but without GASAM modulation compare to the 88.82% baseline?

  2. Narrow Weight Bounds: Why restrict [wmin⁡,wmax⁡][w_{\min}, w_{\max}] to [0.9,1.1][0.9, 1.1] (max 1.22×1.22\times weight ratio), and how does a wider range (e.g., [0.1,1.0][0.1, 1.0]) perform?

  3. Inference Efficiency: What are the latency and VRAM costs of π0.5\pi_{0.5}, X-VLA, Motus, and LingBot-VA compared to STARRY (2.8s), and can fewer denoising steps eliminate the 2.8s synchronous pause?

  4. Avoiding RGB Occlusion: Does overlaying projected trajectories only on the depth views Dtc\mathbf{D}_t^c while keeping the RGB views Itc\mathbf{I}_t^c clean in Eq. 14 recover the 22–30% performance drops on Handover Block, Open Microwave, and Blocks Ranking Size?

  5. Real-World WAM Comparison & Metric Calibration: How do Motus or LingBot-VA perform on the real-world setup (Table 2), and how are multi-view Depth Anything V3 predictions metrically aligned for 3D unprojection in Eq. 6?

Reviewer_hCXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STARRY jointly predicts future visual/spatial content and robot actions. Its Geometry Expert estimates future depth and end-effector motion, and GASAM uses those estimates to bias attention toward the expected interaction region. The paper evaluates on RoboTwin 2.0 and three two-stage physical tasks.

Critical strengths

The physical study is a real strength. Across six task-stage settings, STARRY averages 67.1% versus 35.0% for π0.5, with four batches of 20 rollouts per setting and per-setting intervals (Table 2; Appendix B.2). Gains appear in every setting. That is a more useful test of multi-stage manipulation than a few selected successful videos, and it gives the paper practical weight.

The ablation is informative within this system. On RoboTwin Randomized, RGB future prediction without GASAM scores 85.80%, full spatial-temporal prediction without GASAM 88.82%, and the complete model 93.30% (Table 3). This is a clearer component test than comparing entirely different pretrained systems. Estimated depth still yields 91.26% versus 93.30% with simulator depth (Appendix B.1), so the method does not collapse when depth is imperfect.

Critical weaknesses

The main weakness is attribution. The physical experiment compares complete STARRY with π0.5, but has no physical STARRY-minus-GASAM or STARRY-minus-future-geometry result. The 32.1-point physical mean gap is the paper's most striking result, yet this experiment cannot tell us which new component produced it. I can accept that the full system works in these settings without accepting that the proposed geometry mechanism caused the gain.

A second weakness is the comparison and mechanism evidence in simulation. The RoboTwin averages exceed LingBot-VA by only 0.89 / 1.75 points on Clean / Randomized, and the baseline checkpoints have different training resources (Appendix A.3). The paper visualizes future predictions, but I could not find quantitative future-depth or end-effector errors, nor a test linking those errors to rollout success. GASAM may benefit from a spatial prior even if its predicted geometry is inaccurate. The current results do not separate these explanations.

Finally, the deployment cost is easy to miss behind the relative 5.1% GASAM overhead. A full query takes 2.80 seconds and 41.58 GiB on an A100 (Appendix A.4), but the effective action update frequency on the real robot is not clear. This matters for tasks that require quick corrective actions. Without that timing, it is hard to judge how far the physical result extends beyond the tested two-stage tasks.

Questions for authors
  1. Is there a physical ablation of STARRY without GASAM or future geometry on the same tasks? Even neutralizing GASAM weights on the trained checkpoint would help identify its contribution.
  2. Can you quantify future-depth and end-effector prediction error and relate it to rollout success? How sensitive is GASAM to deliberately degraded predictions?
  3. What was the effective action update frequency on the robot, including the 2.80-second query and action-chunk execution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.