- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper fine-tunes a large pretrained text-to-motion model to accept user cues. A user can fix any mix of poses, paths, joint positions or joint rotations at chosen frames. The model then generates a full-body motion that passes through these cues.
- Input: A "motion canvas" over frames and 198 pose channels (root translation, 6D rotations, joint positions), with a mask of specified cells. Optional text and an optional input motion for editing or repair.
- Output: A full-body SMPL motion with 22 joints at 30 fps.
- Method: An MMDiT flow-matching model, initialised from HY-Motion-1.0-Lite, predicts velocities for the unspecified cells. After each sampling step, the specified cells are overwritten with their cue values, so root and rotation cues are met exactly. A compositional sampler trains the model on random combinations of frames, joints and channels. Forward-kinematic, boundary and rollout losses couple position cues to the decoded body and smooth the transitions around cues.
- Delta: Imputation of known values and predefined constraint mixtures exist in CondMDI and KIMODO. The new parts are the shared canvas with per-axis position cues, the compositional sampler and the transition losses. The paper also uses one model for control, sequential generation, editing and repair.
- Experimental validation: Control is evaluated on HumanML3D, sequential generation on BABEL, editing on MotionFix and PerMo, and repair on BrokenAMASS. All results report 95% confidence intervals over 20 runs. Quality metrics use a joint-position evaluator that the authors trained on their own corpus. The model reports the best FID in most settings and zero error for root and rotation cues. Three technical artists prefer it in a paired study. Matched ablations isolate training-time imputation, the compositional sampler and the transition losses.
Critical strengths
The video in the supplementary is very well done. It clearly shows that the method is amenable to a wide variety of inputs; and can generate plausible motion that satisfies a wide variety of inputs.
- Matched ablations show that the training design, not only the prior, drives cue following. Table 4(a) keeps the initialisation, data and budget fixed. Test-time imputation on the frozen prior gives a 408 mm path error, against 30 mm for the full model. Predefined task templates, which are close to KIMODO's training, give 52 mm path error and 81.1 mm mixed-cue error, against 30 mm and 40.9 mm.
- Exact root paths come with much better motion quality than projection. In Table 2(a), both AnimateCanvas and ProjFlow meet dense XZ paths exactly. AnimateCanvas reaches FID 0.0081 and R@3 0.9025 there, against 0.7856 and 0.3989 for ProjFlow. It shows that learned completion around imputed cues avoids the quality loss of hard projection.
- The transition losses fix cue-boundary artefacts without hurting generation. Adding boundary, rollout and background-relative losses lowers root second differences at XZ waypoints from 13.34 to 1.62 mm (Table 4(b)). Text-to-motion FID improves at the same time, from 0.0127 to 0.0108. Usually abrupt motion at cue boundaries is a common failure of imputation methods.
- One model covers an unusually broad set of tasks with careful evaluation. The same checkpoint handles keyframes, routes, waypoints, body-part cues, mixed cues, sequential generation, editing and repair. Every main metric has confidence intervals over 20 runs, and the supplementary material includes the source code. This breadth makes the canvas a practical interface for animation tools.
Critical weaknesses
- Comparisons with other systems mix the method with more data and an in-domain evaluator. AnimateCanvas starts from HY-Motion-1.0-Lite and trains on about 700 hours of motion, much of it internal, whereas most baselines use public data such as HumanML3D. The evaluator is trained on the same internal corpus (Appendix B.2). Under it, the model scores above ground truth on retrieval, for example R@3 0.7134 against 0.6317 for the BABEL reference. Under the independent MotionStreamer evaluator, GoToZero has a better text-to-motion FID (3.08 against 8.28).
- Position cues are soft, so the control claims overstate cue adherence. Root and rotation cues are exact by construction, but position cues act only through FK losses. The sparse-wrist error is 130.7 mm, the worst in Table 2(b). In Table 3, joint-axis errors reach 40.9 to 43.9 mm, whereas ProjFlow meets them exactly. Without IK, only 47.7% of position cues lie within 50 mm (Section 4.4). This matters because the abstract promises that the model "faithfully" connects all cues and gives "fine-grained spatial control".
- The mixed-cue comparison omits the strongest unified baseline. KIMODO supports keyframes, joint positions, rotations, waypoints and paths, and Fig. 3(c) shows it on mixed cues. Table 3, however, compares mixed cues only with OmniControl, ProjFlow and CondMDI. Each other table also uses a different subset of baselines.
- The user study cannot support strong preference claims. Three technical artists from the authors' organisation rated the pairs, one rater per case (Appendix F). The study compares only with KIMODO on four tracks and StableMotion on repair. One rater per case allows no agreement statistic.
Additional Comments
- Fig. 3(b) omits the error of AnimateCanvas. It prints the errors of the baselines only. Table 2(b) shows that AnimateCanvas has the largest sparse-wrist error, so the figure should report it too.
- Zero cue errors follow from imputation. Tables 1 and 2 show these zeros in bold as results. Please mark them as analytic, as Table 3 does with a star.
- Transition smoothness on BABEL is weak. The area under the jerk curve is 3.1 times the lowest baseline (Table 1(b)). The main text should discuss this next to the retrieval gains.
- Compute is not reported. The training hardware and time are missing, and the inference GPU is called confidential. Please state at least the GPU class.
- The runtime comparison belongs in the main paper. Animation tools need interactive speed, and the method takes 2.47 s for 360 frames with 50 steps. Please summarise Table 20 in Section 4.
- The PerMo pairs are constructed by the authors. Please describe their construction in the main text, because the paper reports results on them as a benchmark.
Questions for authors
- Report the main control tables with an evaluator trained only on public data, or with the MotionStreamer evaluator.
- Train one strong baseline, for example CondMDI-style training, from the same initialisation and corpus, and compare it.
- Report results for a model trained only on public data, to show the effect of the internal corpus.
- Add KIMODO to the mixed-cue comparison of Table 3.
- Explain why the model exceeds the reference on BABEL retrieval and matching distance.
- State how the three raters relate to the authors' team and whether they knew the method.
2 comments
To Reviewer X1Rd 2/2
- Do the positional-control results support the paper’s claims?**
Positional residuals arise during forward-kinematics decoding. Imputation exactly preserves supplied root translations and local rotations. Forward kinematics derives the final joint positions from these quantities, so the positions can differ from their targets. Our FK objectives encourage consistency, and optional IK reduces positional error while preserving the imputed quantities.
With refinement, AnimateCanvas achieves millimeter-level mean positional errors while largely retaining motion quality.
| Setting | Mean error, raw → refined (mm) ↓ | FID, raw → refined ↓ | R@3, raw → refined ↑ | Refined Success@50 mm ↑ |
|---|---|---|---|---|
| Dense wrists | 37.8 → 0.0 | 0.0062 → 0.0064 | 0.8983 → 0.8983 | 100% |
| Sparse wrists | 130.7 → 0.2 | 0.0074 → 0.0075 | 0.9082 → 0.9033 | 100% |
| Dense lower body | 12.2 → 0.5 | 0.0057 → 0.0059 | 0.9005 → 0.9084 | 100% |
| Mixed cues | 40.9 → 2.8 | 0.0092 → 0.0098 | 0.9029 → 0.9017 | 99.2% |
The raw mixed-cue success rate of 47.7% counts motions in which every supplied position cue is within 50 mm. IK adds 54–386 ms per motion. Foot sliding decreases and jitter increases slightly on mixed cues.
Optional refinement improves practical control. Our main comparisons use raw outputs, and the largest mixed-cue residual after IK is 89 mm. “Fine-grained control” refers to cue selection across frames, joints, and position/rotation channels. We changed the abstract to “effectively and coherently connect all cues” to clarify this scope.
5. Why is KIMODO absent from the mixed-cue quantitative comparison?
KIMODO supports mixtures of specific cue patterns, and Table 3 tests a broader range of combinations. This reflects a difference in training priorities. KIMODO emphasizes text-to-motion quality with control over specified patterns. AnimateCanvas trains for completion across arbitrary cue patterns and densities. KIMODO technical report, §4.3
We tested KIMODO under the full protocol and observed unreliable cue following. Figure 3(c) illustrates one mixed case. We quantify performance on shared tasks.
On shared tasks, AnimateCanvas improves waypoint accuracy and text alignment. Path ADE is 30 vs. 77 mm on sparse XZ waypoints. Both methods preserve nonuniform full-body keyframes exactly, where AnimateCanvas scores 0.9031 vs. 0.8560 on R@3.
6. How were the user-study raters selected and blinded?
Three in-house technical artists with no prior knowledge of our method rated 1,218 anonymous, randomized pairs across five tasks. Each pair received one rating, so inter-rater agreement cannot be estimated. We clarified the raters’ affiliation and blinding in the revised paper.
7. Does the higher BABEL AUJ indicate unnatural action transitions?
Higher AUJ indicates greater deviation from reference transition dynamics. AUJ is the area between the generated and reference average jerk curves. Differences in action timing or intensity can raise AUJ. Visible abruptness requires direct assessment.
The generation framework affects transition dynamics. FlowMDM jointly adjusts both sides of a transition. AnimateCanvas conditions each new action on the preceding action’s final five fixed frames, limiting coordination across the boundary. This experiment tests whether the same cue-conditioned model can connect successive actions.
Our anonymous gallery shows selected side-by-side examples of action execution and transitions. BABEL comparison gallery
8. Can the figure and tables distinguish measured errors from exact constraint preservation?
We added AnimateCanvas’s per-example error to Figure 3(b) using the baseline metric and marked zeros guaranteed by imputation in Tables 1 and 2.
9. Where are training compute and inference efficiency reported?
The training budget and inference measurements are in the appendix. Training took about 14 days on 64 GPUs. We added the inference time of 2.47 seconds for 360 frames at 50 Euler steps to the main text. Table 20 gives the full timing protocol and memory use. Company policy prevents us from disclosing the GPU model.
10. How were the PerMo editing pairs constructed?
Both motions in each pair come from the same actor. Style edits hold action fixed, and action edits hold style fixed. We added the pairing protocol to the main text.
To Reviewer X1Rd 1/2
Dear Reviewer X1Rd,
Thank you for your feedback. We address your questions below.
1. Why did we train a broader motion–text evaluator, and do the gains hold under independent evaluation?
Shared joint positions avoid ambiguity from rotation reconstruction. Positions leave bone-axis twist undetermined, so evaluators that use rotations, such as MotionStreamer, can be sensitive to how rotations are recovered. Identical joint trajectories yield identical embeddings under our evaluator. MotionStreamer’s FID can also disagree with visual assessments of motion quality. We provide side-by-side examples so you can compare the generated motions directly. Anonymous T2M comparison gallery
Broader training data expands coverage of actions and descriptions. We train the evaluator separately on real motion–text pairs. AnimateCanvas never uses its embeddings, and both training sets exclude benchmark test motions and derived clips.
The advantage holds under independent evaluation on both tested control tasks. The retrained CondMDI uses the same corpus as AnimateCanvas.
Compare scores within each evaluator, since their embedding spaces differ.
2. Why does AnimateCanvas exceed the BABEL reference on retrieval and matching distance?
Real motions can be outscored on text–motion alignment metrics. Retrieval and matching distance measure prompt alignment in a learned embedding space. Motion realism is a separate question.
BABEL’s short labels add ambiguity. We use an LLM to expand labels for generation and scoring. Expansion can add details that generators are prompted to match and recorded motions may lack, favoring generated motions on alignment scores. We clarified the caption expansion procedure in the revised paper.
3. Do the gains come from the method, or from additional data and HY-Motion initialization?
We examine training design, data access, and initialization separately. The CondMDI-style control shares our backbone, HY-Motion initialization, corpus, update count, and training/sampling imputation.
Separately, we retrained CondMDI with its original architecture and training strategy on our public-plus-internal corpus, matching the number of updates.
A public-data-only AnimateCanvas experiment remains open.
The scratch run jointly learns text and cue conditioning with the same architecture, corpus, task mixture, and losses.
Training took 32 days from scratch and 14 days with HY-Motion initialization, excluding pretraining.