- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SteadyDancer, an Image-to-Video (I2V) framework for human image animation. It introduces three main components to address spatio-temporal misalignments and identity drift in existing Reference-to-Video (R2V) methods: (1) a Condition-Reconciliation Mechanism; (2) Synergistic Pose Modulation Modules (SPMM); and (3) a Staged Decoupled-Objective Training Pipeline. The authors also introduce X-Dance, a new benchmark for evaluating non-homologous cross-source animation scenarios.
Critical strengths
- This paper accurately identifies and addresses critical limitations of the prevailing R2V paradigm, particularly spatio-temporal misalignments.
- The staged training pipeline requires significantly fewer steps (14.5k) and less data (10.2 hours) than recent DiT-based models while maintaining competitive performance.
- The X-Dance benchmark addresses the critical need for evaluating models in realistic, cross-source scenarios.
- The "Pose Simulation" strategy during training is an elegant, low-cost solution that effectively simulates and mitigates real-world start-gap artifacts.
Critical weaknesses
- Stage 2 aims to preserve the teacher's prior by aligning the student's unconditional component with the teacher's. Why not directly use the frozen teacher model to produce the unconditional prediction? The paper lacks an ablation comparing the proposed Stage 2 against this training-free alternative, leaving the necessity of Stage 2 unsubstantiated.
- There is a clear mismatch between Stage 2 training and inference. Stage 2 trains the student's unconditional component, but the inference-time DC-CFG (Eq. 8) relies on negative conditions instead of a pure unconditional prediction. The paper provides no evidence showing that this unconditional distillation actually benefits the final DC-CFG sampling.
- While the Synergistic Pose Modulation Modules (SPMM) empirically improve performance, the paper lacks sufficient implementation details (e.g., network architectures, forward process) and deeper insights into exactly why and how these specific designs resolve the spatio-temporal misalignments better than standard adapters.
- The figures in the manuscripts are not well organized to show your strength.
I am open to raising my score if the authors can adequately address these concerns.
Questions for authors
- The manuscript omits several key hyperparameters necessary for reproducibility. Could you provide details on: (1) LoRA settings (rank, alpha, target modules); (2) Training specifics (resolution, frames, FPS, batch size, optimizer); and (3) Inference settings (DC-CFG guidance scales)? 2. The model is trained on a proprietary dataset of 7,338 curated clips. Do you plan to open-source this data.
- How does SteadyDancer scale to long video generation? Specifically, does the Condition-Reconciliation Mechanism successfully prevent identity drift over extended durations. Figure 10 is too small to see the details. You should exhibit a video in the supp to support your claims.
9 comments
R3-02 (W2): Relationship Between Stage 2 Distillation and Inference-Time DC-CFG
Thank you for pointing out that the relationship between Stage 2 training and inference-time guidance needs clearer explanation.
In Stage 2, the "unconditional component" is defined relative to the newly introduced pose condition. During pose-unconditional teacher distillation, neither the student nor the teacher takes pose input, while both still retain the reference image condition. This prediction is used to construct the teacher distillation target so that the student preserves the generative capability of the original Wan I2V model while learning pose control, rather than acting as a separate generation branch to be called directly during final inference. Meanwhile, task supervision under reference-image and pose conditions jointly updates the student model's parameters together with pose-unconditional teacher distillation, and final inference conditioned on the reference image and pose is carried out by the resulting trained student model.
DC-CFG, on the other hand, follows the conditional-difference formulation of CFG at inference time to adjust the sampling direction for stronger control without extra training.
Specifically, in standard text CFG with negative prompts, each denoising timestep runs two forward predictions with the same noisy latent and reference image: one with a positive prompt and one with a negative prompt. The negative prompt describes undesired content such as overexposure, blur, deformed limbs, static frames, or cluttered backgrounds. The difference between the two predictions defines a guidance direction that strengthens the positive condition while moving away from the negative condition, and the combined prediction takes the negative-prompt prediction as its base term plus the scaled conditional difference:
where controls text guidance strength. Because other conditions such as the reference image and pose remain identical across both passes, this comparison isolates the prediction difference caused by changing the text condition.
We extend the same intuition to pose control. Under the same noisy latent, one pass feeds the target driving pose while the other feeds a negative pose constructed through perturbations such as scaling and translation:
where the negative pose is an off-target pose condition constructed for contrast instead of no pose. The prediction difference between the positive and negative poses provides a guidance direction that strengthens the response to the target motion while steering away from the perturbed pose response. It aims at reducing limb blur, motion ghosting, or drift toward ambiguous poses.
Therefore, pose-unconditional distillation in Stage 2 and negative-pose guidance at inference serve distinct purposes. The former constrains the student via training regularization with the frozen teacher to retain its generative prior, whereas the latter uses conditional prediction differences from the trained student to strengthen pose control during inference sampling. In the revised manuscript, we will explicitly clarify the distinction between "pose-unconditional" and "negative-pose conditional" to make the relationship between Stage 2 training and DC-CFG sampling clear.
R3-03 (W3): Implementation Details and Design Insights of SPMM
Thank you for pointing out that the structure and function of SPMM warrant a clearer explanation. SPMM processes the pose representation before it enters the DiT, with its three modules improving the control signal across content adaptivity, temporal context, and correspondence between the generation state and pose. Instead of simply adding another condition-injection layer, our design shapes the input pose into a representation better suited to the current generation task.
SSAE performs content-adaptive feature adjustment to address spatial structural discrepancies. Because the reference subject and driving subject may differ in body shape, limb proportions, or framing, a fixed pose representation struggles to adapt fully to these variations. SSAE generates a dynamic transformation matrix and channel scaling factors from the global content of the input pose, adjusting the channel responses of pose features through two branches. Different poses can thus induce different feature transformations, providing a more adaptable representation for subsequent fusion with the reference image condition. This adjustment takes place at the feature level instead of geometrically retargeting the input skeleton.
TMCM improves the coherence of motion representations through local temporal context. Isolated errors, missing keypoints, or contradictory cues in pose estimation can make frame-by-frame control signals unstable. TMCM adopts three factorized convolutional blocks, where each block first extracts intra-frame spatial structure and then exchanges information across neighboring frames via temporal convolution. Consequently, the pose representation at any given frame draws on surrounding motion context alongside the current detection, mitigating the impact of isolated noise on the generation process. This acts as learned temporal feature modeling rather than fixed low-pass smoothing on the input poses.
FAAU uses the current denoising state to aggregate pose information via attention. Enhancing the pose representation on its own is insufficient to establish its correspondence with the active generation state. FAAU takes the denoising latent as Query and the enhanced pose features as Key and Value, selecting and aggregating relevant pose information via cross-attention followed by normalization, a feed-forward residual block, and projection to obtain the control representation. Its role is to make pose-condition processing account for the current generation state instead of applying a fixed fusion rule independent of the generation process. Here Query comes from the current noisy video latent, while coordination with the reference appearance is achieved jointly through subsequent generation with this representation and the I2V condition.
The three modules follow a hierarchical aggregation scheme where SSAE and TMCM process pose features in parallel and are added to the original pose via a residual connection, after which FAAU processes the result:
Both the enhanced pose representation and the attention output are retained as conditions and concatenated with the image condition and denoising latent before entering the DiT. Attention processing therefore does not replace the original pose information. It builds on spatial and temporal feature enhancement by adding generation-state-aware condition matching.
Compared with approaches that use only static pose encoding and condition injection, these designs respectively introduce pose-dependent feature transformations, local motion context, and generation-state-aware attention aggregation, clarifying their design motivation for addressing structural discrepancies, pose noise, and condition correspondence. Our module ablations and targeted qualitative comparisons (Fig. 12 and Appendix F, Fig. 24) further show the structural or motion failures that emerge when removing the corresponding modules, supporting their roles in the system. In the revised manuscript, we will add architectural specifications, input/output shapes, and the full forward aggregation formulation to make both the design rationale and implementation details clear.
R3-01 (W1): Role of the Frozen Teacher and Necessity of Stage 2
Thank you for the question. Our original manuscript did not explain the role of the Stage 2 teacher model clearly enough. Here, the teacher is the original Wan-2.1 I2V model, which generates videos conditioned on a reference image but lacks the pose control capability introduced in our work; its unconditional output is just a standard image-to-video result. Therefore, the teacher is not intended to replace SteadyDancer for pose-driven animation.
In fact, it provides the student with an alignment target for preserving the original I2V generation capability. In Stage 1, introducing pose conditioning enables the model to learn motion control. However, this adaptation affects the generative priors from the base model, leading to degraded visual quality and detail preservation. To solve this, we introduce the frozen teacher in Stage 2 to serve as a distillation regularizer that constrains the student to retain its original I2V generation capability while continuing to learn the pose control task. Specifically, the student continues to receive ground-truth supervision conditioned on both the reference image and driving poses. In parallel, we remove the pose condition so that both student and teacher make predictions under the same reference image condition only to compute the distillation loss. The former task supervision maintains the student's pose adherence, while the latter alignment target from the teacher bounds how far the student drifts from the original generative prior. These two losses jointly guide the student's training.
Thus, the teacher's prediction serves as a training target that constrains the student via distillation loss to retain the generative capability of the original I2V model. Because the teacher itself lacks pose control capability, it cannot directly substitute for the student in human animation conditioned jointly on a reference image and driving poses. Once Stage 2 finishes, final video generation is performed independently by the trained student model, and the teacher is not involved at inference time.
In the revised manuscript, we will clarify the teacher's role as a regularizer on generative priors and revise "unconditional" to "pose-unconditional" to make clear that the reference image condition is retained. We will also detail the computation flow of task supervision and teacher distillation so that both the training objective of Stage 2 and how the teacher is used are clear.