acceptodds
Withdrawn from ICLR 2027

ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present ShadowDancer, an approach to any-action, frame-level control of interactive video world models. The obstacle is representational: existing interfaces either encode an action loosely, leaving how it unfolds for the model to improvise, or encode it exactly through structured signals that serve one family and are hard to acquire, so precise control across diverse dynamics remains impractical. Demonstration videos are the natural remedy, specifying any dynamics frame by frame; yet a video shows its dynamics only through one particular appearance, a single shadow of the underlying dynamics, so actions learned from demonstrations transfer poorly to new scenes. ShadowDancer addresses this with two key innovations: (1) shadow pairs, video pairs that replay the same dynamics under independently resampled appearance, constructed at scale by our Shadow Library, so that a dynamics family becomes controllable exactly when such pairs can be constructed for it; and (2) cross-shadow prediction, which learns actions by predicting one shadow from the other, so that whatever the pairing resamples is discarded by construction and whatever it preserves becomes the action, yielding a unified dynamics representation that drives a block-causal world model. Any demonstrated clip thus becomes a reusable action asset, replayed in new environments without action labels, motion estimators, or fine-tuning. Experiments show improved action transfer and long action rollout over strong latent-action and interactive world model baselines across diverse dynamics families, with an average blinded rollout win rate of 86%. We strongly encourage readers to watch supplementary videos.

Primary area:
generative models
Keywords:
world models, interactive video generation, latent action models, action transfer, video diffusion, representation learning, dynamics representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BwhvOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ShadowDancer, a video world model controlled by demonstration clips rather than a fixed action vocabulary or task-specific motion labels. It constructs synchronized shadow pairs that replay the same dynamics under changed appearance, trains a latent action model by predicting across the pair, and conditions a block-causal video generator on the resulting action representation. The Shadow Library spans human motion, robotics, gameplay, and camera movement. On held-out paired-render transfer tests and long-horizon rollouts, the authors report gains over Olaf-World and interactive world-model baselines, including a high blinded preference rate.

Critical strengths
  1. The shadow-pair objective provides a clear and useful supervision signal for separating transferable dynamics from appearance. This directly addresses a central failure mode of latent-action learning and is supported by cross-appearance transfer tests.
  2. The paper builds a broad data and evaluation pipeline spanning several control families, and tests both frame-aligned transfer and chained rollouts. This makes the proposed interface more consequential than a result confined to one action domain.
Critical weaknesses
  1. The central identifiability premise depends on pairing protocols that preserve exactly the intended dynamics while resampling all other relevant factors. That is plausible in curated simulators but is not generally available for arbitrary real videos; moreover, some protocols deliberately preserve correlated elements such as weapons or effects. Thus, the broad "any action" framing exceeds the demonstrated scope, which is bounded by the pairing definitions and curated sources.
  2. Much of the evidence is generated or evaluated in controlled renderings. The strongest quantitative transfer test relies on paired synthetic/rendered targets, while long-rollout quality is judged by a VLM with only a subset human-audited. The evidence supports controlled transfer, but less clearly establishes reliable control on unconstrained real-world demonstrations or independently specified user intent. Related work such as Olaf-World[1] already targets transferable latent actions; the paper's distinction is its paired cross-render supervision, whose practical coverage is therefore central to the contribution.

[1] Jiang, Yuxin, et al. "Olaf-World: Orienting Latent Actions for Video World Modeling." arXiv preprint arXiv:2602.10104 (2026).

Questions for authors
  1. What fraction of the Shadow Library and held-out transfer results uses independently rendered matched pairs versus self-pairs or other weaker pairing protocols? Please report performance broken down by protocol and identify which dynamics families rely on manually specified preservation masks.
  2. How does the method behave when a shadow changes a factor that is causally coupled to the action (for example, contact response or object identity), so that the assumed invariant dynamics are not actually shared?
Reviewer_ootSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ShadowDancer, an innovative framework designed to tackle the non-identifiability problem in interactive video world models by disentangling latent action dynamics from visual appearance. The authors propose learning from "shadow pairs", synthetic videos depicting identical motions but rendered with distinct characters, environments, and lighting, which forces the LAM to encode only appearance-invariant dynamics into a reusable "action asset" that controls a video diffusion model. Across various dynamics families (human motion, gameplay, and robotic manipulation…) , it achieves superior performance in action-conditioned video generation and long-horizon rollouts compared to state-of-the-art baselines.

Critical strengths

** The manuscript is well-written and easy to follow. And the accompanying supplementary videos provide examples that clearly demonstrate the model's practical capabilities and prove that the framework works effectively in practice.

** While the "shadowing" concept is an established idea previously (such as I2I translation), this work presents a highly effective and novel adaptation of this idea to the domains of controllable video generation and motion synthesis.

Critical weaknesses

** Because the identifying training signal relies heavily on synthetic rendering pipelines, any artifacts, physical inaccuracies, or rendering anomalies present in these engines can accumulate within ShadowDancer. Over large-scale training, this error propagation could potentially lead to visible, compounding distortions in the generated outputs.

** The heavy reliance on synthetic training data significantly limits the model's immediate scalability to complex, real-world human and robotic actions that are notoriously difficult or impossible to simulate accurately.

** The quantitative evaluation in Table 1 lacks comparisons to Yume1.5. Authors do not explain why a strong baseline like Yume-1.5 is omitted from the main comparison table (or I overlooked).

Questions for authors

Minor Weakness:

** In line, 1079, Fable 5 --> Table 5

Questions: See weakness

Reviewer_DMVYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ShadowDancer, a framework for learning transferable latent actions from paired videos that share identical dynamics but differ in appearance. Instead of reconstructing the source video, the proposed cross-shadow prediction objective encourages the encoder to extract dynamics that generalize across visual contexts. The learned representation, together with source-video motion assets, conditions a block-causal video diffusion model for interactive generation. Experiments across human motion, gameplay, camera control, and robotic manipulation demonstrate improvements in action transfer and long-horizon rollout.

Critical strengths
  1. Using cross-rendered pairs to bypass the appearance leakage common in self-reconstruction is neat and physically intuitive.

  2. The benchmark is quite comprehensive, spanning articulated motion, robotics, and camera pans without relying on external pose estimators.

  3. The Shadow Library pipeline is practical and easy to scale up in simulation.

Critical weaknesses
  1. Insufficient comparisons. The paper lacks comparisons with recent motion transfer methods, particularly FastVMT and Follow-Your-Motion, making it difficult to assess its advantages over existing approaches.

  2. Lack of real-robot experiments. Robotic manipulation is evaluated only in simulation. Real-world experiments would strengthen the generalization claims.

  3. Limited qualitative results. More video demonstrations are needed, especially for long-horizon generation, unseen actions, and failure cases.

  4. Practical limitations. The method relies on precisely synchronized shadow pairs and pre-recorded action assets, which may be difficult to obtain in real-world scenarios.

Questions for authors
  1. Can the authors provide quantitative comparisons with FastVMT and Follow-Your-Motion?

  2. Can the authors include real-robot experiments to validate practical applicability?

  3. Can the authors provide additional videos, failure cases, and a more detailed discussion of practical limitations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.