- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SWAN, a long-horizon robotic world model built upon the pretrained InfinityStar framework. The method introduces scale-aligned action conditioning to improve action-guided visual prediction and a time-scale memory mechanism to preserve historical states under bounded memory constraints. It further incorporates scale-wise dream forcing and on-policy visual alignment to reduce error accumulation and improve the consistency of generated dynamics. Experiments on long-horizon video prediction, simulated policy evaluation, and real-world robotic manipulation demonstrate the effectiveness of the proposed framework.
Critical strengths
-
The paper addresses an important problem in robotic world modeling, particularly the challenges of action-following accuracy, long-horizon consistency, and efficient memory management.
-
The proposed framework integrates action-conditioned generation, hierarchical temporal memory, and rollout-based refinement into a unified world model, providing a practical approach to long-horizon robotic simulation.
-
The experimental evaluation covers multiple aspects, including generation quality, action following, long-horizon consistency, and real-world robotic manipulation. The matched simulation-to-real rollout evaluation further provides useful evidence of the model's potential for robotic policy assessment.
Critical weaknesses
-
The ablation studies do not sufficiently disentangle the contributions of the proposed architectural components from the additional domain-specific training. Since SWAN is initialized from InfinityStar and further trained on extensive robotic data, it is unclear how much of the performance improvement comes from the proposed modules rather than domain adaptation. A more controlled comparison using the same pretrained backbone, training data, and training budget, with individual components progressively introduced, would help establish their respective contributions.
-
The ablation study only compares the full model with a variant without action conditioning, which demonstrates the benefit of action information but does not establish the effectiveness of the proposed scale-aligned design. A comparison with standard action-conditioning strategies, such as feature concatenation or cross-attention without scale alignment, is needed to justify this architectural contribution.
-
The ablation study does not sufficiently validate the necessity of scale-aligned action conditioning. The authors should compare the proposed multi-scale action conditioning with a standard single-scale conditioning strategy under identical settings to demonstrate the specific benefits of scale alignment.
-
The related work provides only a high-level discussion of existing action-conditioned world models, without adequately comparing their action-conditioning mechanisms with the proposed scale-aligned design. The authors should clarify how their approach differs from prior action-conditioning strategies and what specific limitations of these methods motivate scale alignment.
Questions for authors
Refer to Weaknesses
Response to Reviewer fKFE
We thank Reviewer fKFE for the thoughtful review and constructive feedback. Below, we address W1–W4 in detail.
Q1 (W1):
Comparing pretrained InfinityStar against progressive SWAN variants trained on the exact same robotic dataset directly separates domain adaptation from our proposed modules:
Domain adaptation alone (Pretrained InfinityStar → SWAN w/o Action) improves Image Quality (0.602→0.683) and unconditioned Trajectory Accuracy (0.251→0.418), but leaves Action Following near zero (0.019) and policy ranking weak (0.660 Spearman, 0.112 MAE).
SWAN's architectural and alignment modules on the exact same robotic dataset drive the gains in controllable simulation and policy evaluation—lifting Action Following from 0.019→0.264→0.290, Trajectory Accuracy from 0.418→0.512→0.539, and policy-ranking Spearman from 0.660→0.831→0.876 (with component-wise ablations in our response to Reviewer 8FsE Q2 further isolating time-scale memory and dream forcing under the same robotic training set).
Q2 (W2 & W3):
We clarify what SWAN changes relative to standard concatenation or vanilla cross-attention and why multi-scale conditioning matches next-scale generation:
Mechanism vs. standard concatenation and cross-attention. Standard concatenation or vanilla cross-attention exposes a uniform, single-resolution sequence of raw action tokens to all visual queries, requiring attention layers to implicitly infer 3D-to-2D camera projection, clip boundaries, and spatial scale alignment at once. SWAN instead projects 3D actions into each camera's 2D coordinate frame across 8 spatial scales (s0–s7). For visual queries at clip c, camera v, and visual scale ℓ∈{0,…,11}, SWAN restricts attention to the matched (c,v) window and routes queries to the finest action scale κ(ℓ)≤ℓ (using s0–s7 for ℓ≤7 and reusing s7 for s8–s11).
Why multi-scale over single-scale representations. In next-scale autoregression, coarse scales determine global arm motion direction and scene layout, while fine scales refine local gripper-object contact geometry. Matching the spatial resolution of the action prefix κ(ℓ) to each visual scale ℓ avoids granularity mismatch across coarse-to-fine prediction steps and directly motivates our choice of a next-scale autoregressive backbone.
Q3 (W4):
Prior action-conditioned world models align actions and video predictions at two primary granularities: (1) Sequence-level conditioning (e.g., UniSim), which conditions generation on global language or low-level control vectors; and (2) Frame-aligned single-scale conditioning (e.g., IRASim, WorldGym, Ctrl-World), which aligns per-frame action tokens with latent video frames inside diffusion or autoregressive transformers operating on a single fixed spatial latent grid. SWAN addresses an orthogonal dimension unique to next-scale autoregression: as generation progresses from coarse layout to fine detail within the same camera view and physical time window, combining 3D-to-camera projection with explicit scale routing κ(ℓ) ensures that each visual scale reads action features at a matched spatial resolution.