- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose TimelineControl, a framework for streaming 3D human motion generation from independently timed, overlapping text instructions. Built on ActionPlan, it combines interval-aware text conditioning with a causal part-structured autoencoder and part-aware denoising to coordinate concurrent actions across body regions. The authors also construct TimelineMotion, a dataset of synthetically composed motions with overlapping instruction intervals and body-part annotations. Experiments on TimelineMotion and MTT show improved semantic alignment and temporal adherence over the evaluated streaming baselines. Ablation studies and human evaluations support the proposed design, while spatial control and humanoid robot demonstrations illustrate its potential applications.
Critical strengths
- The proposed framework supports streaming motion generation conditioned on independently timed, overlapping instructions, addressing a limitation of existing methods.
- Experiments on the TimelineMotion and MTT datasets demonstrate improved semantic alignment and temporal adherence over the evaluated streaming baselines.
Critical weaknesses
-
Several aspects of the TimelineMotion dataset construction require further clarification. (a) The dataset construction pipeline uses Qwen3-8B to extract body-part labels and motion attributes from text descriptions, without access to the corresponding motion sequences or visual evidence. It remains unclear whether text-only annotations are sufficient for reliable body-part assignment in multi-track motion composition. (b) The composition pipeline applies strict compatibility rules, excluding overlapping action pairs that involve large stance changes or full-body actions. These restrictions may limit the dataset’s coverage of complex motion composition scenarios.
-
The analysis of streaming inference efficiency is insufficient. (a) Section 5.1 reports sampling latency, but a more comprehensive analysis would include computational cost, latency variability, and a breakdown of runtime across components under different denoising settings. (b) According to Appendix B.4, uncommitted predictions are discarded and regenerated whenever a new instruction arrives. Could this regeneration process introduce noticeable delays when multiple instructions arrive in quick succession?
-
The practical applicability of the proposed interaction setting needs further discussion. Although the authors leave instruction revision, cancellation, and conflict resolution to future work, the absence of these capabilities raises questions about the current task formulation. The authors should give concrete examples of how this setting would be used in practice.
Questions for authors
- Have the authors evaluated the proposed method on challenging instructions involving rapid action changes or complex gymnastic or acrobatic movements? How does the method perform in terms of motion quality in these scenarios?
- Have the authors evaluated long-horizon motion generation over extended durations, such as tens of minutes? Can the method sustain continuous generation while maintaining consistent motion quality and temporal coherence?
4 comments
Response to reviewer drbP (2/2)
W3-a. Practical applicability
As shown in the supplementary video (8s-28s), an avatar can continue walking while receiving a brief wave command, followed by a kicking command, each with its own interval. This supports adding compatible actions during execution. This is important for real-time robot execution, which can be controlled in an online manner without stopping previous actions, as shown in the supplementary video (4:13-4:34).
W3-b. Cancellation and conflict resolution
While revision, cancellation, conflict resolution are not handled inherently by the model, our online demo can actually handle these by designing an engineering pipeline. For example, the current online demo shown in the supplementary video (3:58-4:13) already implements automatic replacement using the text classifier used for part-aware guidance. For each arriving text, we predict the corresponding body part. A new instruction can replace an earlier, temporally overlapping instruction with body-part conflicts, while retaining the other active instructions. In this example, the “kick right leg” action replaces “walk in circle” (Section 5.4, Appendix B.4). Instruction revision can similarly be implemented by updating the text of an active instruction and regenerating the uncommitted continuation. Cancellation removes the selected instruction from the active set while retaining the remaining instructions.
Q1. Challenging instructions and motion quality
HumanML3D and our proposed TimelineMotion includes some high dynamic motions such as cartwheel, handstand and jumping actions. We further select and evaluate 128 TimelineMotion test sequences, each containing at least one dynamic or acrobatic instruction. On the selected challenging instruction intervals, M2T is 0.7747, while STMC obtains M2T 0.7184, which shows the robustness of TimelineControl for challenging instructions. For motion quality, FID is 0.1677 for TimelineControl and 0.2168 for STMC on the same 128 sequences.
Some qualitative cartwheel and jumping examples are already included in the supplementary video (8s-28s).
Q2. Long-horizon motion generation
Thanks for the suggestion. We ran 20 ten-minute sequence rollouts. Across runs, M2T is 0.7116 ± 0.0161 and transition distance is 0.946 ± 0.088 cm. Transition and jitter showed no clear changes across time, which supports sustained ten-minute operation. M2T slightly decreased by 0.0260 from the first to the last minute, with minor alignment degradation. This shows the robustness of our method for long motion generation.
Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.
I thank the authors for the detailed responses and additional experiments. Most of my concerns have been addressed, and I am inclined to raise my score.
Response to reviewer drbP (1/2)
Response to reviewer drbP
We thank the reviewer for the constructive feedback. We add analyses of annotation reliability, runtime, challenging motions, and long-horizon generation, and clarify interactive instruction handling.
W1-a. Text-only body-part assignment
We agree that text alone cannot fully determine motion-specific body-part involvement. However, prior work (e.g., SINC, STMC) has shown that body parts can be derived from language as a practical approximation. For example, first work to do this, SINC (Athanasiou et al., 2023) reports 85% body part labeling accuracy with GPT-3 on 100 manual annotated action descriptions, where they include `maybe’ annotations to acknowledge the ambiguity. We follow these works and use a more modern language model to label the body parts. Appendix G also discusses limitations arising from automatically generated body-part annotations from text. To further check the annotation reliability, we also randomly select 100 TimelineMotion sequences including 300 source motion-caption pairs and check the body annotation together with visualization. The 6 part macro-F1 is 83.60%.
W1-b. Data coverage
Following the body-part compatibility principle in SINC (ICCV 2023) and STMC (CVPRW 2024), we compose actions with compatible regional requirements. We agree this is a limitation of our data pipeline and detailed in “Limitations” section that our composition pipeline with strict compatibility rules cannot capture the full diversity and complexity of naturally co-occurring actions. However, these constraints still permit independently timed, overlapping actions on different body parts, and the results on MTT provide additional evidence of the method’s effectiveness beyond our constructed dataset (Table 1).
W2-a. Computational cost and runtime breakdown
Our final model contains 284.25M denoiser parameters. On an RTX PRO 6000 Blackwell GPU at batch size one, first-output latency is 212.9 ± 4.7 ms for one active instruction at 25 steps. Steady-state latency is 16.1-16.5 ms per four-frame output across 1 to 4 active instructions.
Runtime breakdown in milliseconds with one active instruction at 25 and 10 denoising steps.
Table R2. Runtime breakdown (ms).
W2-b. Regeneration delay
A newly arriving instruction produces the first updated output after 230.3 ± 3.5 ms with one existing instruction and 234.3 ± 1.3 ms with two. These timings include regeneration of the uncommitted continuation, which only introduces a minor delay and support real-time streaming under the reported settings.