- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Pelican-Sim 1.0, a unified action-conditioned video world model for embodied intelligence. The method combines numerical robot configurations with camera-aligned whole-arm skeleton videos through interleaved conditioning in a video diffusion transformer. It further incorporates sparse MoE layers and few-step autoregressive distillation for efficient rollouts. Experiments demonstrate strong video prediction quality across multiple robot datasets and promising downstream results in data augmentation, policy evaluation, action selection, and policy optimization.
Critical strengths
- The combination of numerical configurations and visual action representations is well motivated, and the proposed framework provides a unified approach to modeling different robot embodiments.
- The experimental results are strong across multiple datasets and evaluation metrics. The authors also provide useful ablations on action representations, conditioning strategies, and model capacity.
- The paper goes beyond video generation metrics by evaluating downstream applications, including policy learning and action selection, demonstrating the potential utility of the learned world model.
Critical weaknesses
- The proposed visual action representation is closely related to Visual Action Prompts (VAP), while combining numerical and visual action conditions has also been explored in EnerVerse-AC and ViPSim. Although Pelican-Sim introduces interleaved conditioning, the extent of its architectural advantage over these prior approaches remains insufficiently isolated.
- Although the baselines are retrained on the same dataset, they use different action representations. Pelican-Sim benefits from explicit whole-arm geometry derived from URDF and camera metadata, which may provide richer information than the conditions available to some baselines. This makes it difficult to isolate the architectural improvements from the benefits of additional geometric information.
- The proposed method relies on accurate URDF models, joint configurations, and camera calibration to construct visual action conditions. Although the paper describes data cleaning and alignment correction, its robustness to noisy or inaccurate geometric parameters remains unclear.
- Although the paper reports strong video-quality and motion-related metrics, these measurements do not fully establish the physical accuracy of the predicted interactions. In particular, quantitative evaluations of object motion, contact consistency, and accumulated errors during long-horizon closed-loop rollouts remain limited.
Questions for authors
- Can the authors more clearly distinguish the technical contributions from VAP and ViPSim, and provide controlled comparisons under matched action representations and backbones?
- What proportion of the world-model training data consists of failure trajectories? Can the authors evaluate failure prediction accuracy, particularly whether the model incorrectly generates successful outcomes for unsuccessful action sequences?
- Can the authors compare against baselines equipped with the same visual action conditions, or otherwise control for the amount of geometric information provided?
- Can the authors evaluate sensitivity to perturbations in camera calibration, joint states, and URDF parameters?
8 comments
Official Comment by Authors
Question 2: Failure trajectories and false-success predictions
Failure trajectories constitute 0% of Pelican-Sim's large-scale world-model pretraining data. The 32.3% failure fraction refers to a separate evaluation pool. On that pool, Pelican-Sim has a 5.73% false-success rate, compared with 37.77% for EnerVerse-AC, a reduction of 32.04 percentage points (Table R9).
We collected 2,000 simulated executions across five RoboTwin tasks using policy checkpoints from different training steps: 1,354 successes and 646 failures. For each initial state and action sequence, RoboTwin supplies the ground-truth outcome label, while a VLM judges success or failure in the generated world-model rollout. We compare these labels trajectory by trajectory on the same evaluation set.
Among the 646 simulator failures, 37 Pelican-Sim rollouts and 244 EnerVerse-AC rollouts are judged successful. Pelican-Sim also improves success recall by 13.81 percentage points and balanced accuracy by 22.93 percentage points, as summarized in Table R9. This indicates more faithful prediction of simulator outcomes within the evaluated world-model-plus-VLM pipeline, while identifying a remaining false-success rate of 5.73%.
Table R9. Failure prediction against RoboTwin outcome labels.
| Model | False-success rate ↓ | Failure recall ↑ | Success recall ↑ | Balanced accuracy ↑ | Overall accuracy ↑ |
|---|---|---|---|---|---|
| Pelican-Sim | 5.73% | 94.27% | 99.41% | 96.84% | 97.75% |
| EnerVerse-AC | 37.77% | 62.23% | 85.60% | 73.91% | 78.05% |
Note: Metrics are pooled over the same 2,000 paired executions. False-success rate is the fraction of the 646 simulator failures whose generated rollouts the VLM judges successful. Balanced accuracy is the mean of success and failure recall; overall accuracy uses all 2,000 trajectories. The metrics assess the combined world-model and VLM evaluation pipeline.
Official Comment by Authors
Question 3: Controlling the geometric information in action conditions
The added controls show that Pelican's hierarchical conditioning retains a benefit with identical geometric inputs, while explicit whole-arm geometry also contributes to prediction quality. In Table R10, Pelican improves PSNR by 1.3 dB and reduces FVD by 4.3 with identical whole-arm skeletons and numerical joint inputs (Groups 2→4). With identical end-effector maps, it improves PSNR by 1.1 dB and reduces FVD by 1.8 (Groups 1→3). Richer visual inputs alone therefore do not explain the observed advantage.
Explicit whole-arm geometry provides a further gain. With injection fixed, replacing end-effector maps with whole-arm skeletons improves PSNR by 0.8/1.0 dB and reduces FVD by 1.6/4.1 under ViPSim-style/Pelican conditioning. Every group receives the same 28-dimensional joint configuration, so this comparison tests the value of explicitly representing intermediate joints and links.
Original Table 1 compares systems using their native conditioning interfaces. The new controls fix the Cosmos backbone and initialization, MoE configuration, 28-dimensional joint input, data, training budget, and generation settings. Only the visual action representation and injection mechanism change; other observation conditions remain the same.
Table R10. Matched-geometry controls on AgiBotWorld Beta.
| Group | Visual action representation | Injection mechanism | PSNR ↑ (dB) | FVD ↓ |
|---|---|---|---|---|
| 1 | End-effector projection maps | ViPSim-style adaptation | 20.1 | 16.9 |
| 2 | Whole-arm skeleton | ViPSim-style adaptation | 20.9 | 15.3 |
| 3 | End-effector projection maps | Pelican interleaving | 21.2 | 15.1 |
| 4 | Whole-arm skeleton | Pelican interleaving | 22.2 | 11.0 |
Note: These are the same independent controlled re-evaluation results as B1–B4 in Table R8. ViPSim-style adapts the conditioning mechanism to the common backbone and inputs, rather than reproducing the full original system. The comparisons separately support the representation and interaction design within this controlled setting.
Official Comment by Authors
Question 1: Technical distinction from VAP and ViPSim
Controlled comparisons support both explicit whole-arm geometry and the hierarchical interaction design: Pelican retains a quality gain when action representations and the backbone are matched. Tables R7–R8 separate native method characteristics from results obtained after adapting conditioning mechanisms to common inputs and a common backbone.
VAP uses visual skeleton prompts without a separate numerical action branch. ViPSim already combines numerical and visual conditions through visual latent fusion and numerical cross-attention. Pelican-Sim focuses on hierarchical interaction between joint-level numerical configurations and whole-arm visual geometry: synchronized skeletons explicitly represent intermediate-joint and link motion, while numerical scale-shift modulation and visual residual updates operate on shared hidden states. Interleaving is the default schedule.
Table R7. Native method characteristics and conditioning inputs.
For the controlled comparisons in Table R8, we fix the Cosmos backbone and initialization, MoE configuration, data, training budget, and generation settings. Both visual-only variants disable numerical conditioning; B1–B4 receive the same 28-dimensional joint configuration. Other observations are unchanged.
With geometry matched, Pelican gains 1.1 dB and reduces FVD by 1.8 for end-effector maps (B1→B3), and gains 1.3 dB and reduces FVD by 4.3 for whole-arm skeletons (B2→B4). Thus, richer visual inputs alone do not explain the conditioning advantage.
With injection fixed, whole-arm skeletons improve PSNR by 0.8/1.0 dB and reduce FVD by 1.6/4.1 under ViPSim-style/Pelican conditioning (B1→B2/B3→B4). Since all four variants already receive the same joint configurations, this tests the value of explicitly rendering whole-arm geometry.
With visual inputs matched and numerical branches disabled, Pelican raises PSNR by 0.9 dB and reduces FVD by 2.1 relative to VAP-style (A1→A2). Adding numerical conditioning to Pelican further improves PSNR by 1.0 dB and reduces FVD by 3.3 (A2→B4), supporting complementary numerical and visual information.
Table R8. Controlled comparison results on AgiBotWorld Beta.
Note: “Style” denotes adaptation of a conditioning mechanism to the shared backbone and inputs, not the complete original system. In particular, ViPSim-style receives the common 28-dimensional joint input. Full and visual-only Pelican use the same training/evaluation protocol. These are independent controlled re-evaluation results, reported to one decimal place, distinct from the main-table 22.276 dB/10.85 FVD results.