acceptodds
Under review as a conference paper at ICLR 2027

AnimateCanvas: Learning Implicit Motion Planning from Composable Kinematic Cues

Abstract

Professional character animation requires both natural motion and precise, versatile control. For example, creators often define the timing of a specified action, control the motion range of the character’s arm swing, or specify the route the character walks through—effectively placing various kinematic cues on a “motion canvas”. This motivates us to propose AnimateCanvas, a model that supports cue-conditioned implicit motion planning to faithfully and coherently connect all cues, dense or sparse, full or partial, into one full-body motion sequence. Specifically, AnimateCanvas represents heterogeneous kinematic cues on a shared motion canvas, where position and rotation values are specified across body joints and time. A shared flow-matching model generates motion conditioned on this canvas, with optional language and input motion; cue imputation keeps the specified canvas values fixed in both training and sampling. To learn coherent completion across different cue sets, we train with a compositional cue sampler that varies the timing of cue application, the positions or rotations specified, and how they are combined. Together, these designs enable a single generator to integrate heterogeneous kinematic cues into coherent full-body actions, giving creators fine-grained control over selected frames, joints, and position or rotation channels. We evaluate this planning ability on temporal, root, and body-part cues—alone and combined—as well as language-guided editing, and naturally extend it to sequential generation and motion repair. AnimateCanvas achieves state-of-the-art results in temporal completion, spatial control, sequential generation, language-guided editing, and motion repair, while retaining strong text-to-motion capability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion Generation, Character Animation, Digital Human

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_X1RdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper fine-tunes a large pretrained text-to-motion model to accept user cues. A user can fix any mix of poses, paths, joint positions or joint rotations at chosen frames. The model then generates a full-body motion that passes through these cues.

  • Input: A "motion canvas" over frames and 198 pose channels (root translation, 6D rotations, joint positions), with a mask of specified cells. Optional text and an optional input motion for editing or repair.
  • Output: A full-body SMPL motion with 22 joints at 30 fps.
  • Method: An MMDiT flow-matching model, initialised from HY-Motion-1.0-Lite, predicts velocities for the unspecified cells. After each sampling step, the specified cells are overwritten with their cue values, so root and rotation cues are met exactly. A compositional sampler trains the model on random combinations of frames, joints and channels. Forward-kinematic, boundary and rollout losses couple position cues to the decoded body and smooth the transitions around cues.
  • Delta: Imputation of known values and predefined constraint mixtures exist in CondMDI and KIMODO. The new parts are the shared canvas with per-axis position cues, the compositional sampler and the transition losses. The paper also uses one model for control, sequential generation, editing and repair.
  • Experimental validation: Control is evaluated on HumanML3D, sequential generation on BABEL, editing on MotionFix and PerMo, and repair on BrokenAMASS. All results report 95% confidence intervals over 20 runs. Quality metrics use a joint-position evaluator that the authors trained on their own corpus. The model reports the best FID in most settings and zero error for root and rotation cues. Three technical artists prefer it in a paired study. Matched ablations isolate training-time imputation, the compositional sampler and the transition losses.
Critical strengths

The video in the supplementary is very well done. It clearly shows that the method is amenable to a wide variety of inputs; and can generate plausible motion that satisfies a wide variety of inputs.

  • Matched ablations show that the training design, not only the prior, drives cue following. Table 4(a) keeps the initialisation, data and budget fixed. Test-time imputation on the frozen prior gives a 408 mm path error, against 30 mm for the full model. Predefined task templates, which are close to KIMODO's training, give 52 mm path error and 81.1 mm mixed-cue error, against 30 mm and 40.9 mm.
  • Exact root paths come with much better motion quality than projection. In Table 2(a), both AnimateCanvas and ProjFlow meet dense XZ paths exactly. AnimateCanvas reaches FID 0.0081 and R@3 0.9025 there, against 0.7856 and 0.3989 for ProjFlow. It shows that learned completion around imputed cues avoids the quality loss of hard projection.
  • The transition losses fix cue-boundary artefacts without hurting generation. Adding boundary, rollout and background-relative losses lowers root second differences at XZ waypoints from 13.34 to 1.62 mm (Table 4(b)). Text-to-motion FID improves at the same time, from 0.0127 to 0.0108. Usually abrupt motion at cue boundaries is a common failure of imputation methods.
  • One model covers an unusually broad set of tasks with careful evaluation. The same checkpoint handles keyframes, routes, waypoints, body-part cues, mixed cues, sequential generation, editing and repair. Every main metric has confidence intervals over 20 runs, and the supplementary material includes the source code. This breadth makes the canvas a practical interface for animation tools.
Critical weaknesses
  • Comparisons with other systems mix the method with more data and an in-domain evaluator. AnimateCanvas starts from HY-Motion-1.0-Lite and trains on about 700 hours of motion, much of it internal, whereas most baselines use public data such as HumanML3D. The evaluator is trained on the same internal corpus (Appendix B.2). Under it, the model scores above ground truth on retrieval, for example R@3 0.7134 against 0.6317 for the BABEL reference. Under the independent MotionStreamer evaluator, GoToZero has a better text-to-motion FID (3.08 against 8.28).
  • Position cues are soft, so the control claims overstate cue adherence. Root and rotation cues are exact by construction, but position cues act only through FK losses. The sparse-wrist error is 130.7 mm, the worst in Table 2(b). In Table 3, joint-axis errors reach 40.9 to 43.9 mm, whereas ProjFlow meets them exactly. Without IK, only 47.7% of position cues lie within 50 mm (Section 4.4). This matters because the abstract promises that the model "faithfully" connects all cues and gives "fine-grained spatial control".
  • The mixed-cue comparison omits the strongest unified baseline. KIMODO supports keyframes, joint positions, rotations, waypoints and paths, and Fig. 3(c) shows it on mixed cues. Table 3, however, compares mixed cues only with OmniControl, ProjFlow and CondMDI. Each other table also uses a different subset of baselines.
  • The user study cannot support strong preference claims. Three technical artists from the authors' organisation rated the pairs, one rater per case (Appendix F). The study compares only with KIMODO on four tracks and StableMotion on repair. One rater per case allows no agreement statistic.

Additional Comments

  • Fig. 3(b) omits the error of AnimateCanvas. It prints the errors of the baselines only. Table 2(b) shows that AnimateCanvas has the largest sparse-wrist error, so the figure should report it too.
  • Zero cue errors follow from imputation. Tables 1 and 2 show these zeros in bold as results. Please mark them as analytic, as Table 3 does with a star.
  • Transition smoothness on BABEL is weak. The area under the jerk curve is 3.1 times the lowest baseline (Table 1(b)). The main text should discuss this next to the retrieval gains.
  • Compute is not reported. The training hardware and time are missing, and the inference GPU is called confidential. Please state at least the GPU class.
  • The runtime comparison belongs in the main paper. Animation tools need interactive speed, and the method takes 2.47 s for 360 frames with 50 steps. Please summarise Table 20 in Section 4.
  • The PerMo pairs are constructed by the authors. Please describe their construction in the main text, because the paper reports results on them as a benchmark.
Questions for authors
  • Report the main control tables with an evaluator trained only on public data, or with the MotionStreamer evaluator.
  • Train one strong baseline, for example CondMDI-style training, from the same initialisation and corpus, and compare it.
  • Report results for a model trained only on public data, to show the effect of the internal corpus.
  • Add KIMODO to the mixed-cue comparison of Table 3.
  • Explain why the model exceeds the reference on BABEL retrieval and matching distance.
  • State how the three raters relate to the authors' team and whether they knew the method.
2 comments
AuthorsOpenReview ↗

To Reviewer X1Rd 1/2

Dear Reviewer X1Rd,

Thank you for your feedback. We address your questions below.

1. Why did we train a broader motion–text evaluator, and do the gains hold under independent evaluation?

  1. Shared joint positions avoid ambiguity from rotation reconstruction. Positions leave bone-axis twist undetermined, so evaluators that use rotations, such as MotionStreamer, can be sensitive to how rotations are recovered. Identical joint trajectories yield identical embeddings under our evaluator. MotionStreamer’s FID can also disagree with visual assessments of motion quality. We provide side-by-side examples so you can compare the generated motions directly. Anonymous T2M comparison gallery

  2. Broader training data expands coverage of actions and descriptions. We train the evaluator separately on real motion–text pairs. AnimateCanvas never uses its embeddings, and both training sets exclude benchmark test motions and derived clips.

  3. The advantage holds under independent evaluation on both tested control tasks. The retrained CondMDI uses the same corpus as AnimateCanvas.

TaskMethodOur FID ↓Our R@3 ↑MotionStreamer FID ↓MotionStreamer R@3 ↑
Sparse XZ waypointsOmniControl0.03750.812227.88190.8703
KIMODO0.48170.547998.50650.6431
AnimateCanvas0.00960.89819.70860.9393
Mixed cues with rotationsCondMDI (released)0.07390.787430.97740.8292
CondMDI (retrained)0.03040.843818.00980.8837
AnimateCanvas0.00920.90298.65590.9306

Compare scores within each evaluator, since their embedding spaces differ.

2. Why does AnimateCanvas exceed the BABEL reference on retrieval and matching distance?

Real motions can be outscored on text–motion alignment metrics. Retrieval and matching distance measure prompt alignment in a learned embedding space. Motion realism is a separate question.

BABEL’s short labels add ambiguity. We use an LLM to expand labels for generation and scoring. Expansion can add details that generators are prompted to match and recorded motions may lack, favoring generated motions on alignment scores. We clarified the caption expansion procedure in the revised paper.

3. Do the gains come from the method, or from additional data and HY-Motion initialization?

We examine training design, data access, and initialization separately. The CondMDI-style control shares our backbone, HY-Motion initialization, corpus, update count, and training/sampling imputation.

Cue samplerTransition supervisionMixed error (mm) ↓XZ WP-Δ² (mm) ↓
CondMDI-style random frame/jointGlobal reference derivatives73.35.09
OursGlobal reference derivatives51.84.11
OursOurs40.91.62

Separately, we retrained CondMDI with its original architecture and training strategy on our public-plus-internal corpus, matching the number of updates.

MethodTraining dataMixed error (mm) ↓FID ↓R@3 ↑
CondMDI (released)Public-only138.80.07390.7874
CondMDI (retrained)Public + internal96.90.03040.8438
AnimateCanvasPublic + internal40.90.00920.9029

A public-data-only AnimateCanvas experiment remains open.

The scratch run jointly learns text and cue conditioning with the same architecture, corpus, task mixture, and losses.

InitializationT2M FID ↓T2M R@3 ↑Mixed error (mm) ↓Mixed FID ↓Mixed R@3 ↑
Random0.00950.918938.60.00940.9079
HY-Motion0.01080.910340.90.00920.9029

Training took 32 days from scratch and 14 days with HY-Motion initialization, excluding pretraining.

AuthorsOpenReview ↗

To Reviewer X1Rd 2/2

  1. Do the positional-control results support the paper’s claims?**

Positional residuals arise during forward-kinematics decoding. Imputation exactly preserves supplied root translations and local rotations. Forward kinematics derives the final joint positions from these quantities, so the positions can differ from their targets. Our FK objectives encourage consistency, and optional IK reduces positional error while preserving the imputed quantities.

With refinement, AnimateCanvas achieves millimeter-level mean positional errors while largely retaining motion quality.

SettingMean error, raw → refined (mm) ↓FID, raw → refined ↓R@3, raw → refined ↑Refined Success@50 mm ↑
Dense wrists37.8 → 0.00.0062 → 0.00640.8983 → 0.8983100%
Sparse wrists130.7 → 0.20.0074 → 0.00750.9082 → 0.9033100%
Dense lower body12.2 → 0.50.0057 → 0.00590.9005 → 0.9084100%
Mixed cues40.9 → 2.80.0092 → 0.00980.9029 → 0.901799.2%

The raw mixed-cue success rate of 47.7% counts motions in which every supplied position cue is within 50 mm. IK adds 54–386 ms per motion. Foot sliding decreases and jitter increases slightly on mixed cues.

Optional refinement improves practical control. Our main comparisons use raw outputs, and the largest mixed-cue residual after IK is 89 mm. “Fine-grained control” refers to cue selection across frames, joints, and position/rotation channels. We changed the abstract to “effectively and coherently connect all cues” to clarify this scope.

5. Why is KIMODO absent from the mixed-cue quantitative comparison?

KIMODO supports mixtures of specific cue patterns, and Table 3 tests a broader range of combinations. This reflects a difference in training priorities. KIMODO emphasizes text-to-motion quality with control over specified patterns. AnimateCanvas trains for completion across arbitrary cue patterns and densities. KIMODO technical report, §4.3

We tested KIMODO under the full protocol and observed unreliable cue following. Figure 3(c) illustrates one mixed case. We quantify performance on shared tasks.

On shared tasks, AnimateCanvas improves waypoint accuracy and text alignment. Path ADE is 30 vs. 77 mm on sparse XZ waypoints. Both methods preserve nonuniform full-body keyframes exactly, where AnimateCanvas scores 0.9031 vs. 0.8560 on R@3.

6. How were the user-study raters selected and blinded?

Three in-house technical artists with no prior knowledge of our method rated 1,218 anonymous, randomized pairs across five tasks. Each pair received one rating, so inter-rater agreement cannot be estimated. We clarified the raters’ affiliation and blinding in the revised paper.

7. Does the higher BABEL AUJ indicate unnatural action transitions?

Higher AUJ indicates greater deviation from reference transition dynamics. AUJ is the area between the generated and reference average jerk curves. Differences in action timing or intensity can raise AUJ. Visible abruptness requires direct assessment.

The generation framework affects transition dynamics. FlowMDM jointly adjusts both sides of a transition. AnimateCanvas conditions each new action on the preceding action’s final five fixed frames, limiting coordination across the boundary. This experiment tests whether the same cue-conditioned model can connect successive actions.

Our anonymous gallery shows selected side-by-side examples of action execution and transitions. BABEL comparison gallery

8. Can the figure and tables distinguish measured errors from exact constraint preservation?

We added AnimateCanvas’s per-example error to Figure 3(b) using the baseline metric and marked zeros guaranteed by imputation in Tables 1 and 2.

9. Where are training compute and inference efficiency reported?

The training budget and inference measurements are in the appendix. Training took about 14 days on 64 GPUs. We added the inference time of 2.47 seconds for 360 frames at 50 Euler steps to the main text. Table 20 gives the full timing protocol and memory use. Company policy prevents us from disclosing the GPU model.

10. How were the PerMo editing pairs constructed?

Both motions in each pair come from the same actor. Style edits hold action fixed, and action edits hold style fixed. We added the pairing protocol to the main text.

Reviewer_qLrtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

(1) This paper use a unified canvas that accommodates diverse heterogeneous kinematic control cues for generating complete full‑body human motions. (2) This paper use a suite of loss functions combined with the compositional cue‑imputation mechanism enforces input constraints and guarantees the continuity and consistency of motions produced by the model.

Critical strengths

(1) A unified canvas is devised to fuse and store diverse heterogeneous kinematic control cues within a shared representation. It substantially simplifies model inputs and supports training on various input types rather than fixed task‑specific input templates. (2) With constraints imposed by the cue‑imputation mechanism and multiple auxiliary loss functions, the model can produce complete motions that smoothly bridge different constraint points.

Critical weaknesses

Although the shared canvas unifies and fuses various heterogeneous kinematic cues, this procedure introduces FK‑related fusion residuals. The paper lacks a clear account of native mitigation strategies for these residuals.

Questions for authors

(1)When some cue combinations are held‑out in training and tested afterwards with elevated FID and errors, does it mean the method cannot realize true zero‑shot arbitrary cue mixing? (2)Despite SOTA action‑recognition metrics, higher AUJ than best baseline: does it imply unnatural discontinuities still exist for multi‑action stitching?

1 comment
AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer qLrt

Dear Reviewer qLrt,

Thank you again for your thoughtful feedback. We have posted a response to your comments with additional clarifications and experimental results, and we have updated the manuscript in light of the reviewers’ feedback.

With the discussion deadline approaching, we would appreciate it if you could take a look at our response and let us know if any concerns remain. We would be glad to address any follow-up questions before the deadline. If the new evidence and clarifications address your concerns, we would be grateful if you would revisit your overall assessment in light of the revision.

Thank you for your valuable time and consideration!

Reviewer_oWKvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a flow-matching framework for controllable 3D human motion generation conditioned on composable kinematic cues, including dense/sparse joint rotations, root trajectories, and joint positions. The model employs step-wise cue imputation and zero-target flow velocities to enforce root and local rotation constraints. To enable generalizable completion across arbitraty constraint patterns, it uses a composable cue sampler coupled with mask-derived boundary smoothness objectives and truncated rollout supervision for waypoint transition dynamics. Validations performed on HumanML3D, BABEL, MotionFix, and BrokenAMASS show strong motion fidelity and broad taks performances.

Critical strengths
  • It enables heterogeneous kinematic constraints on a single cue canvas and achieves exact constraint satisfaction by using cue imputation, which is a strong point for motion generation with specific artist control.

  • The proposed mask-derived excess loss and truncated rollout supervision effectively reduce dynamic spikes near contraint boundaries.

  • The method achieves leading retrieval and FID across a wide variety of tasks and fast inference speed.

  • Ablation studies verify that joint training-and-sampling imputation and compositional cue sampler outperform channel-concatenated flow baselines and predefined template samplers.

  • The quality of the demo video is excellent.

Critical weaknesses
  • The claimed contributions in Intro section emphasizes empirical effectiveness rather than fundamental technical novelties. The flow-matching imputation and higher-order boundary penalties are mainly motion-domain specific adaptations of existing inpainting and smoothness regularization techniques.

  • The model benefits from extensive pretraining on 700 hours of proprietary data using 64 GPUs. It makes it difficult to interpret whether the benchmark gains came from data scaling against baselines or methodological innovations.

  • While local rotations and root translations are well controlled, non-root joint positions show noticeable deviations from the contraints, which may require post-processing.

Questions for authors
  • How much of the improvement across FID and retrieval is attributed to the proposed method versus the scale of the extensive dataset and training? An experiment showing benchmark results where key baselines are trained on the similar data/training budget would help understand that aspect.
Reviewer_wnULOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript proposes AnimateCanvas, a novel method that integrates heterogeneous control cues, including keyframes, joint constraints, positions, and rotations, into a unified motion canvas. This representation is then used within a flow matching model to guide motion generation and editing. The proposed approach has potential value for computer animation.

Critical strengths
  1. The paper presents a novel approach to integrating heterogeneous motion cues into a unified representation, demonstrating a strong degree of originality.
  2. The paper provides extensive experiments and detailed descriptions. It is well organized, logically structured, and easy to follow.
Critical weaknesses
  1. Some details remain unclear. For example, the cue layout in Figure 2 contains a small gray box in the i2i_2 column, but neither the caption nor the main text explains what it represents.
  2. The proposed method appears to rely primarily on MM DiT for generation and on a generation mask for cue conditioning and reconstruction. This mechanism seems closely related to that of KIMODO, raising the concern that the method may constitute an incremental extension of KIMODO through the addition of cue imputation. The authors should clarify the substantive methodological differences between the two approaches.
  3. The method incorporates multiple complementary loss terms, but the paper does not adequately explain how their weights are selected. A sensitivity analysis would help clarify how different weighting choices affect model performance.
Questions for authors
  1. Although cue imputation explicitly resets the values at cue locations to their original values, these cues still participate in the joint-attention computation within the MMDiT blocks. Overwriting their values afterward may introduce inconsistencies or additional errors, yet the paper does not appear to analyze this possibility. The authors should clarify the implications of this operation and provide supporting evidence.
  2. The model is initialized from HY-Motion-1.0-Lite and further trained, but this base model is absent from the comparisons in the main experiments (e.g. Tables 1–4). Without a direct comparison under matched conditions, it is difficult to determine whether the reported gains stem from the capabilities of the base model or from the proposed methodological contributions.
  3. The claim that the method retains strong text-to-motion capabilities requires further qualification. Table 13 shows a decline in the R@k metrics, suggesting some degradation in text-to-motion performance. The authors should explain this decline and temper the claim where appropriate.
4 comments
AuthorsOpenReview ↗

To Reviewer wnUL 1/3

Dear Reviewer wnUL,

Thank you for your careful review and constructive suggestions. Your comments have been very helpful in refining our presentation and strengthening our experimental analysis. We appreciate the time and effort you have devoted to helping us improve our work.

1. How does AnimateCanvas differ from KIMODO beyond imputation?

We agree that neither imputation nor a general mask-based interface for cue conditioning is new. Earlier work such as CondMDI already used imputation, and KIMODO supports multiple constraint types through this shared mechanism. We do not claim these ingredients as our novelty.

KIMODO emphasizes high-quality text-to-motion generation with additional cue controls; AnimateCanvas focuses on arbitrary cue conditioning while retaining strong text-to-motion capability. KIMODO produces high-quality motion and offers a flexible conditioning interface. Its curriculum first trains text-to-motion generation, then introduces predefined constraint patterns and their mixtures. Our central goal is to train a model to complete motion from arbitrary cue patterns at arbitrary densities, while preserving its ability to generate motion from text alone.

The key difference is how the model is trained to interpret and integrate the supplied cues, not simply whether its interface accepts them. KIMODO's constraint-training phase covers sparse full-body joint-position keyframes; sparse position/rotation keyframes for randomly selected hands and feet; sparse 2D root-position and heading keyframes; dense 2D root-position and heading paths; and sparse foot-contact configurations. Two patterns are combined in 25% of examples during this phase (technical report, §4.3 [1]). In our tests, we found that its flexible interface did not always translate into reliable control beyond these training patterns, for example with 10 consecutive conditioned frames or shoulder/elbow position constraints.

AnimateCanvas trains directly for variation in cue layout and density.** Our compositional sampler varies the constrained frames, joint subsets, position axes, rotations, and densities, and combines these choices within each training sequence. This requires the model to learn how changing partial constraints interact to determine the remaining motion. Text-only examples are included in the same training process to retain strong T2M capability. The distinction is therefore one of training objectives and coverage, rather than simply which conditions the interface can accept.

The comparisons below evaluate cue following and text alignment on shared tasks; the controlled sampler ablation then isolates the contribution of our training strategy.

Sparse XZ waypoints. FID, R@3, and Path ADE are reported in Table 2(a); Fail@200 is reported in Table 6(b).

MethodPath ADE (mm) ↓Fail@200 ↓FID ↓R@3 ↑
KIMODO775.7%0.48170.5479
AnimateCanvas300.0%0.00960.8981

Path ADE evaluates the entire route, including unspecified frames between waypoints. The improvement therefore reflects more accurate route completion, alongside better FID and text alignment.

Nonuniform full-body keyframes. FID and R@3 are reported in Table 1(a); Foot is reported in Table 5, under the text-conditioned nonuniform-keyframe setting.

MethodFID ↓R@3 ↑Foot ↓
KIMODO0.06940.85600.185
AnimateCanvas0.00140.90310.118

Our training explicitly targets the broader completion problem. KIMODO samples predefined constraint patterns and their mixtures, including sparse full-body and hand/foot keyframes and dense root paths (KIMODO technical report, §4.3 [1]). Our sampler varies cue locations, densities, joint subsets, position axes, rotations, and their combinations throughout the motion. Learning these interacting constraints requires more than independently learning individual control types.

The controlled sampler ablation demonstrates the importance of this design: replacing the named-task sampler—which already includes mixed-control templates—with our compositional sampler reduces mixed-cue error from 81.1 to 40.9 mm, with imputation and losses unchanged (Table 4(a)). The shared-task comparisons demonstrate practical benefits, while the ablation establishes a training contribution beyond the common imputation mechanism.

[1] Rempe et al. Kimodo: Scaling Controllable Human Motion Generation. 2026. Technical report.

Please see our next reply for the overwrite-consistency analysis and the comparison with training from scratch.

AuthorsOpenReview ↗

To To Reviewer wnUL 2/3

2. Does overwriting cues after joint attention introduce inconsistencies?

This question pinpoints one of the core challenges our work addresses. KIMODO and CondMDI use imputation, but their documented training objectives do not explicitly combine fixed-cue zero-flow targets with supervision of cue boundaries and completed sampling outputs. Our design specifically targets consistency between generated motion and the actual input cues, rather than only consistency with the model’s predictions at constrained locations.

First, boundary supervision and sampling-trajectory supervision explicitly train these connections. Boundary supervision targets reference-relative errors around specified/generated regions. Sampling-trajectory supervision uses motion obtained during sampling with imputation applied; the final ablation additionally evaluates the completed output at the solver endpoint. Table 4(b) shows:

Supervision stageXZ WP-Δ² (mm)XYZ WP-Δ² (mm)
One-step transition objective13.3413.46
+ Boundary and sampling-trajectory supervision6.186.81
+ Supervision at the completed solver output2.463.18

The last comparison keeps sampling-trajectory losses and weights unchanged, demonstrating the benefit of supervising the actual completed output.

Second, we align training and sampling around fixed input cues. Supplied cues remain fixed along the training flow path, with zero target flow velocity, so the model learns completion around unchanged input conditions. Imputation enforces preservation despite residual prediction errors. Table 4(a) evaluates this combined formulation:

FormulationPath ADE (mm) ↓Mixed error (mm) ↓
Full-state flow12261.9
Our fixed-cue training-and-sampling formulation3040.9

Together, these mechanisms directly address the overwrite-consistency problem: the model learns completion around preserved input cues, with improved control accuracy and reduced cue-local waypoint variation demonstrated by the ablations above.

3. Do the gains come from HY-Motion initialization or the proposed training?

Training from scratch was also part of our development plan. We prioritized continued training for the manuscript while pursuing a longer scratch run. That run is now complete and jointly learns text and cue conditioning without separate text-only pretraining:

InitializationT2M FID ↓T2M R@3 ↑Mixed error (mm) ↓Mixed FID ↓Mixed R@3 ↑
Random0.00950.918938.60.00940.9079
HY-Motion0.01080.910340.90.00920.9029

We use HY-Motion initialization primarily to accelerate training. Comparable scratch performance shows that our formulation learns these capabilities without HY pretraining. Both runs use the same architecture, training data, task mixture, losses, and evaluation protocol. The scratch and initialized runs took 32 and 14 days, respectively, excluding HY’s upstream pretraining.

4. Does the decrease in R@k indicate reduced text-to-motion capability?

We acknowledge a slight decrease in text–motion retrieval, but consider it a modest trade-off given the added control capabilities and the overall T2M results. Under the same joint-position evaluator, Table 13(a) reports:

ModelR@1 ↑R@2 ↑R@3 ↑FID ↓
HY-Motion0.71260.86070.91430.0157
AnimateCanvas0.70860.85620.91030.0108
GoToZero0.63340.79240.86010.0183
MotionGPT30.60320.77030.84550.0936

Relative to HY-Motion, R@1, R@2, and R@3 decrease by 0.56%, 0.52%, and 0.44%, respectively, while FID decreases by 31.21%. AnimateCanvas also remains above every evaluated generated-motion baseline other than HY-Motion in all three retrieval metrics. AnimateCanvas therefore retains strong T2M capability, with improved FID and only a small retrieval trade-off. We also provide an anonymous comparison gallery containing all 4,042 samples. We invite the reviewer to use Random sample to spot-check text alignment and motion quality side by side with HY-Motion and other baselines.

Please see our next reply for loss-weight sensitivity, and the Figure 2 clarification.

AuthorsOpenReview ↗

To To Reviewer wnUL 3/3

5. How are the loss weights selected, and how sensitive is the method to them?

Thank you for this helpful suggestion. Honestly, we did not run a systematic search when choosing the loss weights. We initially set them so that the weighted loss terms had roughly similar magnitudes, following a common practice for balancing different training objectives. Your comment encouraged us to check how much these choices affect the results.

We tested loss-weight sensitivity by changing only the loss weights during the final 50 training epochs, keeping all other settings fixed. Every variant resumes from the same checkpoint at the start of this interval. We scale one loss group at a time by 0.1× or 10×, leaving the other groups unchanged and preserving the relative weights within the selected group. All variants train for 50 epochs and are evaluated on the same test sets.

G, B, R, and E denote the geometry, boundary, rollout, and one-step background-relative excess loss groups, respectively. Each multiplier scales all loss weights within that group together.

SettingMixed error ↓Mixed FID ↓Mixed R@3 ↑XZ Δ²XYZ Δ²T2M FID ↓T2M R@3 ↑
Original weights (1×)40.90.00920.90291.621.960.01080.9103
G ×0.150.30.01030.89992.082.440.01130.9088
G ×1038.90.01310.90661.792.170.01380.9042
B ×0.145.90.00910.90293.093.360.01010.9090
B ×1042.40.01060.89711.461.750.01170.9061
R ×0.146.50.01020.89993.063.720.01130.9082
R ×1040.10.01080.89631.381.680.01200.9055
E ×0.141.70.00910.90302.182.650.01070.9108
E ×1043.50.01140.89331.201.470.01240.9033

Errors and waypoint second differences are in mm. “Original weights (1×)” reports the original training continuation over this interval. Mixed-control and T2M metrics follow their respective manuscript protocols, and WP-Δ² uses the fixed diagnostic sets in Table 17(a).

The original weights provide a good overall balance, though they are not best on every metric. G ×10 improves cue accuracy but worsens FID; B/R ×0.1 increase waypoint variation, while B/R/E ×10 reduce it at the expense of mixed FID and retrieval. These results support our weight choices for continued training, without establishing optimality or robustness from scratch. We thank the reviewer for prompting this useful check.

6. What does the gray cell in Figure 2 represent?

Thank you for carefully examining Figure 2. The gray cell between the two blue cells at i₂ represents Y (height): X and Z are specified, while Y remains free for generation. The Figure 2 caption in our latest manuscript now explicitly explains this example. We appreciate your careful reading, which helped us make the illustration clearer.

Thank you again for your thoughtful feedback and constructive suggestions, which have helped us improve both the work and its presentation. If we have misunderstood any of your comments or have not fully addressed a concern, please feel free to let us know. We would be happy to clarify further.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer wnUL

Dear Reviewer wnUL,

Thank you again for your thoughtful feedback. We have posted a response to your comments with additional clarifications and experimental results, and we have updated the manuscript in light of the reviewers’ feedback.

With the discussion deadline approaching, we would appreciate it if you could take a look at our response and let us know if any concerns remain. We would be glad to address any follow-up questions before the deadline. If the new evidence and clarifications address your concerns, we would be grateful if you would revisit your overall assessment in light of the revision.

Thank you for your valuable time and consideration!

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer qLrt: Higher AUJ on BABEL

3. Does higher AUJ mean that transitions between actions look unnatural?

A higher AUJ signals a greater discrepancy in how acceleration changes around action transitions; it does not by itself establish unnatural motion or visible artifacts. Jerk measures the rate of change of acceleration. We compute jerk from third-order differences of joint positions and average its magnitude over joints and transition windows. AUJ is the area between the generated and real-motion average jerk curves, computed by integrating their absolute difference over the transition window. $ \mathrm{AUJ}=\int \left|\mu_{\mathrm{gen}}(t)-\mu_{\mathrm{real}}(t)\right|,dt $

  1. AUJ is a dynamics diagnostic, not a direct judgment of naturalness. Natural executions of the same action sequence can differ in timing and intensity, producing different jerk profiles. A higher AUJ can therefore reflect these differences without implying visible jumps or abrupt discontinuities. We acknowledge the higher value as a discrepancy from reference dynamics, but visual artifacts need to be assessed directly.

  2. The generation framework also affects transition dynamics. FlowMDM jointly denoises the composed sequence with all action instructions available, allowing motion on both sides of a transition to be adjusted together. Our method and MotionStreamer generate motion autoregressively. In AnimateCanvas, each new action is conditioned on only the final five frames of the preceding action. These frames provide local motion context, but the model does not observe how acceleration evolves over the preceding action as a whole, and cannot revise that action. It must construct the transition from this limited context, which can increase discrepancies in the transition jerk profile. Their AUJ values are similar in our evaluation: 80,054 for AnimateCanvas and 76,289 for MotionStreamer, a 4.9% difference. Our purpose here is to demonstrate that the same cue-conditioned model can connect successive actions, not to claim that autoregression is the optimal framework for this benchmark.

  3. We invite the reviewer to judge the visible transitions and action execution directly in our anonymous comparison gallery. In our visual inspection, we found AnimateCanvas's results better overall than the compared baselines in following the requested actions while maintaining coherent transitions. This is our qualitative assessment, and we encourage the reviewer to inspect the side-by-side results independently. The gallery is a curated subset of 1,272/1,295 episodes; its selection disclosure documents the 23 exclusions based on our method's boundary displacement. Quantitative metrics are computed on the full evaluation set.

Thank you again for your time and thoughtful comments. If we have misunderstood any of your concerns or if you would like further clarification on any point, please let us know. We would be happy to elaborate.

AuthorsOpenReview ↗

First-Round Responses and Further Discussion

Dear Reviewers,

In this first round of responses, we have addressed each reviewer’s comments and would welcome any further questions or feedback. Thank you for your thoughtful comments and constructive suggestions, which have helped us improve the work.

We look forward to discussing the paper further with you. If there are any additional experiments or results you would like to see, please feel free to let us know. We would appreciate hearing your requests early so that we have time to run them, and we will do our best to provide the results as quickly as possible.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Paper Revision

Dear Reviewers,

Thank you again for your constructive feedback. We have updated the manuscript with the following revisions w.r.t each reviewer's feedback:

  1. Clarified positional-control claims in the Abstract and Section 4.4, distinguishing exact root/local-rotation preservation from learned non-root position adherence. We also moved the existing raw/IK comparison into the main paper (Table 5), reporting accuracy, success rates, quality trade-offs, and runtime. Main comparisons remain without IK. (We thank reviewers oWKv, qLrt, and X1Rd for these suggestions.)
  2. Improved method and figure clarity by explaining the gray cells and axis-level cues in Figure 2 and clarifying sampling-trajectory supervision in Section 3. (We thank reviewer wnUL for prompting these clarifications.)
  3. Marked exact-zero errors with an asterisk in Tables 1–2 and explained their origin in imputation or projection, rather than highlighting them as empirical improvements. (We thank reviewer X1Rd for this suggestion.)
  4. Clarified evaluation protocols, including same-actor PerMo editing-pair construction in Section 4.6 and Appendix D.3, and BABEL label acquisition and LLM expansion in Appendix D.2. (We thank reviewer X1Rd for raising these questions.)
  5. Added inference efficiency to Section 4.1, reporting 360 frames in 2.47 seconds with 50 Euler steps and 2.06 GB peak allocated memory at batch size 1, excluding optional IK. (We thank reviewer X1Rd for this suggestion.)
  6. Expanded the held-out-composition discussion in Section 4.4 with key numerical results, their scope, and the remaining gap to full-composition training. (We thank reviewer qLrt for raising this concern.) We also included paired qualitative examples in the main paper (Figure 4), showing editing alone versus editing combined with root-path or pose-keyframe cues under otherwise identical settings.
AuthorsOpenReview ↗

Thank You to All Reviewers

As the discussion period draws to a close, we would like to thank all the reviewers again for their careful reading and constructive feedback. We value each comment and suggestion, and your feedback has helped us see where we can clarify the paper and strengthen the work. Thank you again for the time and effort you devoted to reviewing our submission and engaging with us during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.