acceptodds
Under review as a conference paper at ICLR 2027

Streaming Multi-Track Timeline Control for 3D Human Motion Generation

Abstract

Text-driven human motion generation has advanced substantially, yet most methods assume instructions are available before synthesis. Interactive applications require responding to new instructions while continuing ongoing actions, such as answering a phone while walking. Existing approaches address streaming generation or simultaneous composition without explicitly combining streaming instruction arrival with independently timed, overlapping actions. We introduce streaming multi-track timeline control and propose TimelineControl to incorporate new instructions alongside ongoing actions. Interval-aware conditioning preserves instruction timing, while causal part-structured representations and part-aware denoising coordinate concurrent actions across body regions. We also construct TimelineMotion, a dataset with overlapping instruction intervals and body-part annotations. Experiments on TimelineMotion and MTT demonstrate improved semantic alignment and temporal adherence over evaluated streaming baselines, including models retrained on the same data. Ablations and human evaluations validate our design, complemented by spatial conditioning and humanoid execution demonstrations.

Primary area:
generative models
Keywords:
Human motion generation, Streaming motion generation, Multi-track timeline control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_drbPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose TimelineControl, a framework for streaming 3D human motion generation from independently timed, overlapping text instructions. Built on ActionPlan, it combines interval-aware text conditioning with a causal part-structured autoencoder and part-aware denoising to coordinate concurrent actions across body regions. The authors also construct TimelineMotion, a dataset of synthetically composed motions with overlapping instruction intervals and body-part annotations. Experiments on TimelineMotion and MTT show improved semantic alignment and temporal adherence over the evaluated streaming baselines. Ablation studies and human evaluations support the proposed design, while spatial control and humanoid robot demonstrations illustrate its potential applications.

Critical strengths
  1. The proposed framework supports streaming motion generation conditioned on independently timed, overlapping instructions, addressing a limitation of existing methods.
  2. Experiments on the TimelineMotion and MTT datasets demonstrate improved semantic alignment and temporal adherence over the evaluated streaming baselines.
Critical weaknesses
  1. Several aspects of the TimelineMotion dataset construction require further clarification. (a) The dataset construction pipeline uses Qwen3-8B to extract body-part labels and motion attributes from text descriptions, without access to the corresponding motion sequences or visual evidence. It remains unclear whether text-only annotations are sufficient for reliable body-part assignment in multi-track motion composition. (b) The composition pipeline applies strict compatibility rules, excluding overlapping action pairs that involve large stance changes or full-body actions. These restrictions may limit the dataset’s coverage of complex motion composition scenarios.

  2. The analysis of streaming inference efficiency is insufficient. (a) Section 5.1 reports sampling latency, but a more comprehensive analysis would include computational cost, latency variability, and a breakdown of runtime across components under different denoising settings. (b) According to Appendix B.4, uncommitted predictions are discarded and regenerated whenever a new instruction arrives. Could this regeneration process introduce noticeable delays when multiple instructions arrive in quick succession?

  3. The practical applicability of the proposed interaction setting needs further discussion. Although the authors leave instruction revision, cancellation, and conflict resolution to future work, the absence of these capabilities raises questions about the current task formulation. The authors should give concrete examples of how this setting would be used in practice.

Questions for authors
  1. Have the authors evaluated the proposed method on challenging instructions involving rapid action changes or complex gymnastic or acrobatic movements? How does the method perform in terms of motion quality in these scenarios?
  2. Have the authors evaluated long-horizon motion generation over extended durations, such as tens of minutes? Can the method sustain continuous generation while maintaining consistent motion quality and temporal coherence?
4 comments
AuthorsOpenReview ↗

Response to reviewer drbP (1/2)

Response to reviewer drbP

We thank the reviewer for the constructive feedback. We add analyses of annotation reliability, runtime, challenging motions, and long-horizon generation, and clarify interactive instruction handling.

W1-a. Text-only body-part assignment

We agree that text alone cannot fully determine motion-specific body-part involvement. However, prior work (e.g., SINC, STMC) has shown that body parts can be derived from language as a practical approximation. For example, first work to do this, SINC (Athanasiou et al., 2023) reports 85% body part labeling accuracy with GPT-3 on 100 manual annotated action descriptions, where they include `maybe’ annotations to acknowledge the ambiguity. We follow these works and use a more modern language model to label the body parts. Appendix G also discusses limitations arising from automatically generated body-part annotations from text. To further check the annotation reliability, we also randomly select 100 TimelineMotion sequences including 300 source motion-caption pairs and check the body annotation together with visualization. The 6 part macro-F1 is 83.60%.

W1-b. Data coverage

Following the body-part compatibility principle in SINC (ICCV 2023) and STMC (CVPRW 2024), we compose actions with compatible regional requirements. We agree this is a limitation of our data pipeline and detailed in “Limitations” section that our composition pipeline with strict compatibility rules cannot capture the full diversity and complexity of naturally co-occurring actions. However, these constraints still permit independently timed, overlapping actions on different body parts, and the results on MTT provide additional evidence of the method’s effectiveness beyond our constructed dataset (Table 1).

W2-a. Computational cost and runtime breakdown

Our final model contains 284.25M denoiser parameters. On an RTX PRO 6000 Blackwell GPU at batch size one, first-output latency is 212.9 ± 4.7 ms for one active instruction at 25 steps. Steady-state latency is 16.1-16.5 ms per four-frame output across 1 to 4 active instructions.

Runtime breakdown in milliseconds with one active instruction at 25 and 10 denoising steps.

Table R2. Runtime breakdown (ms).

Component25 steps10 steps
Text encoding12.8613.96
First output denoising191.2386.45
First output decoding5.876.26
Total first output latency212.94109.80
Subsequent-output denoising8.929.65
Subsequent-output decoding7.097.37
Total steady state latency16.1217.16

W2-b. Regeneration delay

A newly arriving instruction produces the first updated output after 230.3 ± 3.5 ms with one existing instruction and 234.3 ± 1.3 ms with two. These timings include regeneration of the uncommitted continuation, which only introduces a minor delay and support real-time streaming under the reported settings.

AuthorsOpenReview ↗

Response to reviewer drbP (2/2)

W3-a. Practical applicability

As shown in the supplementary video (8s-28s), an avatar can continue walking while receiving a brief wave command, followed by a kicking command, each with its own interval. This supports adding compatible actions during execution. This is important for real-time robot execution, which can be controlled in an online manner without stopping previous actions, as shown in the supplementary video (4:13-4:34).

W3-b. Cancellation and conflict resolution

While revision, cancellation, conflict resolution are not handled inherently by the model, our online demo can actually handle these by designing an engineering pipeline. For example, the current online demo shown in the supplementary video (3:58-4:13) already implements automatic replacement using the text classifier used for part-aware guidance. For each arriving text, we predict the corresponding body part. A new instruction can replace an earlier, temporally overlapping instruction with body-part conflicts, while retaining the other active instructions. In this example, the “kick right leg” action replaces “walk in circle” (Section 5.4, Appendix B.4). Instruction revision can similarly be implemented by updating the text of an active instruction and regenerating the uncommitted continuation. Cancellation removes the selected instruction from the active set while retaining the remaining instructions.

Q1. Challenging instructions and motion quality

HumanML3D and our proposed TimelineMotion includes some high dynamic motions such as cartwheel, handstand and jumping actions. We further select and evaluate 128 TimelineMotion test sequences, each containing at least one dynamic or acrobatic instruction. On the selected challenging instruction intervals, M2T is 0.7747, while STMC obtains M2T 0.7184, which shows the robustness of TimelineControl for challenging instructions. For motion quality, FID is 0.1677 for TimelineControl and 0.2168 for STMC on the same 128 sequences.

Some qualitative cartwheel and jumping examples are already included in the supplementary video (8s-28s).

Q2. Long-horizon motion generation

Thanks for the suggestion. We ran 20 ten-minute sequence rollouts. Across runs, M2T is 0.7116 ± 0.0161 and transition distance is 0.946 ± 0.088 cm. Transition and jitter showed no clear changes across time, which supports sustained ten-minute operation. M2T slightly decreased by 0.0260 from the first to the last minute, with minor alignment degradation. This shows the robustness of our method for long motion generation.

Dear reviewer, the authors have responded to your comment. Could you please take a look? Let us know if you have further concerns.

I thank the authors for the detailed responses and additional experiments. Most of my concerns have been addressed, and I am inclined to raise my score.

Reviewer_GzPpOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to streaming generate human motion while new instructions can arrive while earlier actions are still ongoing. It is an extension of the progessive streaming flow-mathcing backbone. The core idea is to keep each instruction's timing separate, hand it to the related body parts, so overlapping actions can coexist. TimelineControl combines these controls with a motion generator, which can preserve movement already exist. Experiments on composed action timelines show improved instruction alignment and coordinated motion.

Critical strengths

This paper addresses a useful gap between motion generation and instruction response of robotics. It propose a problem that I believe should be considered by research community. It has well-controlled empirital validation that isolates the achitectural contribution from training data and prompt concatenation. The retrained baselines, dedicated multi-instruction control, and component ablations provide convincing evidence to convince me that the proposed design contributes beyond the training data. The conclusion is also supported by human preference investigation.

Critical weaknesses

I especially care about timing and continuity claims, since better instruction alignment also comes with higher boundary displacement. The delay caused by a newly arrived instruction is unclear. The runtime results only measure startup and steady-state generation. The temporal evaluation uses broad window, so maybe cannot establish precise action onset and termination.

Questions for authors

As mentioned in weakness, I want to know about how quickly the model responds to a newly arrive instruction and how accurately the generated action follows its requested start and end times. For example, maybe report the delay from instruction arrival to the first updated motion output? Or action onset and offset errors, measured using suitable kinematic cues or annotated boundaries? With such report I will improve the review score and fully recommend this paper.

2 comments
AuthorsOpenReview ↗

Response to reviewer GzPp

Response to reviewer GzPp

We thank the reviewer for the helpful feedback. We provide latency measurements and manual annotations of action timing to clarify responsiveness and the trade-off with boundary continuity.

W1-a / Q1-a. Delay from instruction arrival

On an RTX PRO 6000 Blackwell GPU at batch size one, first-output latency is 212.9 ± 4.7 ms for one active instruction at 25 steps. Steady-state latency is 16.1-16.5 ms per four-frame output across 1 to 4 active instructions. A newly arriving instruction produces the first updated output after 230.3 ± 3.5 ms with one existing instruction and 234.3 ± 1.3 ms with two (Section 5.1, Appendix C.4). These timings support real-time streaming under the reported settings.

W1-b / Q1-b. Action onset and offset errors

Thank you for the suggestion. To examine the trade-off between action timing and boundary continuity, we manually annotate onset and offset times and missed actions in generated videos for 100 randomly sampled TimelineMotion sequences (300 actions per method). Table R1 reports the number of missed actions and the mean absolute onset/offset errors, excluding missed actions from both error averages. TimelineControl has fewer missed actions and lower onset/offset errors, while STMC and ActionPlan* have lower transition distance.

Table R1. Action execution, timing, and continuity.

MethodMissed ↓Onset (s) ↓Offset (s) ↓Trans (cm) ↓
TimelineControl39/3000.9971.0071.32
STMC93/3001.1271.1601.01
ActionPlan*130/3001.2551.2910.98

Thank you for the additional experiments. If I understand correctly, the motion output latency after new instruction is about 230 ms, while the errors in action onset/offset are around one second. It's a limitation, but I believe it is acceptable given the scope of this work, and the reduction in missed actions.

I have no further major concerns and am happy to raise my ratingto 4. Personally I look forward to seeing further applications of this work in humanoid motion planning.

Reviewer_BEQzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles streaming multi-track motion generation, where new instructions can arrive during generation and overlap with ongoing actions. The authors extend ActionPlan with interval-aware text conditioning and part-aware denoising. They also introduce TimelineMotion and evaluate the method on benchmarks, user studies, and a real humanoid robot.

Critical strengths
  1. Interesting and practical problem setting: Combining streaming instructions with overlapping actions is a meaningful problem that existing methods do not directly address. The new dataset and real-robot demonstration also make the work more convincing.

  2. Fair baseline comparisons: The authors retrain several baselines on the same data, which makes the comparisons more meaningful. I also appreciate the additional ActionPlan* experiment in Table 9, which helps separate the effect of multi-text conditioning from the proposed components.

Critical weaknesses
  1. The methodological novelty is somewhat limited: Most components are adapted from existing work, including RoTE, sigmoid attention, and part-structured representations. Table 9 shows that simply adding multi-text conditioning to ActionPlan* already improves R@3 from 0.502 to 0.566, while the full model reaches 0.603. It is unclear whether the remaining improvement justifies the added complexity.

  2. Some concerns about the evaluation: TimelineMotion is built using STMC-based motion composition, so evaluating on the same dataset may favor the proposed approach. On MTT, the FID+ is almost identical to STMC (0.478 vs. 0.476). I also find the temporal adherence results confusing. In Table 5, GT motion has much lower AP/AUC (0.647/0.728) than the generated motion in Table 1 (0.832/0.772). This makes me question what these metrics actually capture.

  3. Some claims seem stronger than the results support: The paper claims better motion quality, but the transition distance is worse than STMC and ActionPlan* (1.31 vs. 0.92). This seems important for streaming generation but is not really discussed. Also, some ablation improvements are quite small, and there are no results across multiple seeds or error bars. The user study is relatively small as well.

  4. Missing references (Minor): The related work section could be more complete. In particular, the authors may consider citing MMM (CVPR 2024), BAMM (ECCV 2024), and HGM³ (ICLR 2025) when discussing masked motion generation. MotionLCM (ECCV 2024) is also worth mentioning in the context of real-time and interactive motion generation, especially given the reported inference latency in Sec. 5.1.

Questions for authors
  1. Why is the GT temporal adherence score in Table 5 much lower than the generated results in Table 1? Are they evaluated using the same protocol?

  2. RoTE uses the midpoint and duration of each instruction, which seems to assume that the end time is known in advance. How does the method handle unknown end times? Also, how quickly does it respond to new instructions?

  3. The part-aware bias seems quite small (around ±0.21 in the attention logits). Does it actually make a meaningful difference? An attention visualization or a sweep over λ_r would help. Also, could the improvement from the part branch simply come from having more parameters?

4 comments
AuthorsOpenReview ↗

Response to reviewer BEQz (1/2)

Response to reviewer BEQz

We thank the reviewer for the feedback. We clarify our contribution and evaluation protocol and add analyses of action timing and model design.

W1-a. Methodological novelty

The novelty and contribution of our paper is primarily the new problem setting with streaming multi-track control, where instructions arrive during generation and retain independently timed, overlapping intervals. The adaptation of existing methods is non-trivial for this task as in Table 1, therefore, we construct our dataset and propose TimelineControl. While some components are adapted from other papers, the core methodological contribution lies in the denoising architecture where part and whole-body branches coordinate these concurrent actions for streaming instructions.

W1-b. Do the improvements justify the added complexity?

Please note that ActionPlan* + multi-text is our extension of ActionPlan* with separately encoded instructions and interval masks, whose adaptation is non trivial. It already includes part of our conditioning design i.e. multi-track cross attention. TimelineControl further adds our part-aware denoising mechanism. On the TimelineMotion dataset, TimelineControl reduces FID by 52.8%, from 0.036 to 0.017, and improves sequence R@3 from 0.566 to 0.603 and AP from 0.805 to 0.832 (Table 9), which is in our opinion sensible.

W2-a. STMC-based motion composition and evaluation fairness

The reviewer is right to question synthetic-data bias. In fact, this is the reason why we also retrain several baselines on our proposed dataset for a fair comparison, showing that TimelineControl has better performance regardless. To further demonstrate the robustness of our method, we further evaluate on the MTT dataset. STMC is offline generation, which has access to the complete timeline before generation, whereas TimelineControl receives instructions as they arrive. Under this streaming constraint, TimelineControl generates in an online streaming manner, the proposed method achieves comparable FID+ and better sequence alignment (sequence R@3 0.628 vs. 0.566, Table 1), which shows the robustness of TimelineControl.

W2-b / Q1. Temporal adherence results doubts

Temporal AP and AUC assess whether text-motion similarity is higher inside the requested interval than outside it, and Table 5 shows that both scores decrease under larger interval shifts. The temporal-adherence scores in Table 1 and Table 5 are not a matched comparison. Both tables use the same scoring rule, but different instruction sets. Table 5 tests sensitivity to interval shifts of up to ±2 s. To allow each instruction interval to be shifted both earlier and later by up to 2 s, Table 5 excludes instructions that begin within the first 2s or end within the last 2s of the motion. Table 1 uses the full test set. Hence, the reported values can not be compared directly. We add this explanation in Appendix C.3.

W3-a. Transition distance

Our transition distance is higher than STMC/ActionPlan*, as transition distance measures joint displacement at instruction boundaries, and responding promptly to a new action can require a larger pose change near its boundary. The original STMC paper also shows similar trade-off results between text alignment and transition distance in Table A.1. To further explain this trade-off, we manually annotated action onset and offset times in generated videos for 100 randomly sampled TimelineMotion sequences (300 actions per method). We report mean absolute onset/offset errors, excluding missed actions from both error averages (Table R1). TimelineControl has fewer missed actions and lower onset/offset errors indicating more reliable action execution and interval alignment, while STMC and ActionPlan* have lower transition distance with higher missed actions. In other words, we consider this trade-off worthwhile for interactive control, where reliably executing a command at the requested time is central to the task.

Table R1. Action execution, timing, and continuity.

MethodMissed ↓Onset (s) ↓Offset (s) ↓Trans (cm) ↓
TimelineControl39/3000.9971.0071.32
STMC93/3001.1271.1601.01
ActionPlan*130/3001.2551.2910.98

W3-b. Results uncertainty

We repeat each ablation evaluation five times and update the results in Table 4 of the revised PDF with the mean and 95% confidence interval for each metric. The full model still achieves the highest mean sequence and body-part R@3 and the lowest FID, supporting the proposed design.

W3-c User study

Our user study includes 18 participants evaluating 20 timeline sequences. The participant and sequence count are comparable to prior motion-generation studies, including SALAD (CVPR 2025) with 16 participants and 15 videos, MoScale (CVPR 2026) with 20 participants and 20 motion sequences, and StaCOM (CVPR 2026) with 12 participants 30 interaction videos.

AuthorsOpenReview ↗

Response to reviewer BEQz (2/2)

W4. Missing references

We thank the reviewer for pointers to other works. We further cite MMM, BAMM, HGM³, and MotionLCM and clarify their relationship to our setting in the updated paper.

Q2-a. Unknown end times

Following ActionPlan and MotionStreamer, in all experiments, the duration of each instruction is provided on arrival (Section 5.1). For further application with commands without an end time, we use a default 5-second interval and extend it successively (Appendix B.4). In that case, RoTE therefore still receives a finite current interval. The online demo shown in the supplementary video (3:58-4:13) provides a qualitative example with unknown end times.

Q2-b. Response to new instructions

On an RTX PRO 6000 Blackwell GPU at batch size one, first-output latency is 212.9 ± 4.7 ms for one active instruction at 25 steps. Steady-state latency is 16.1-16.5 ms per four-frame output across 1 to 4 active instructions. A newly arriving instruction produces the first updated output after 230.3 ± 3.5 ms with one existing instruction and 234.3 ± 1.3 ms with two (Section 5.1, Appendix C.4). These timings support real-time streaming under the reported settings.

Q3-a. Part-aware bias and attention visualization

The part-aware bias acts as a soft prior, and removing it lowers body-part R@3 from 0.375 to 0.360 (Table 4). We show an attention visualization with text “wave right hand” in Figure R1 (Appendix D.5), left figure without bias and right figure with bias. The bias here mainly enhances the corresponding body part where the original attention is uncertain.

Q3-b. λᵣ ablations

As suggested, we compare three values of λᵣ (0, 0.25, and 0.5) on TimelineMotion (Table R3, Appendix D.5). Among these values, 0.25 achieves the highest sequence and body-part R@3 and the lowest FID.

Table R3. Part-aware bias strength.

λᵣSequence R@3₃₂ ↑Sequence M2T ↑Body-part R@3₃₂ ↑Body-part M2T ↑FID ↓FID+ ↓Trans ↓AP ↑AUC ↑
00.5870.6500.3600.5470.0250.1941.210.8300.769
0.250.6040.6530.3750.5530.0170.1921.310.8320.772
0.50.6000.6530.3720.5510.0240.1981.280.8310.771

Q3-c. Part-branch parameters

The whole part-aware branch adds 4.334M parameters, a 1.55% increase over the same model trained with the global branch only (from 279.912M to 284.246M). Adding the branch improves mean body-part R@3 from 0.345 to 0.375 and FID decreases from 0.030 to 0.017 (Table 4). We consider these gains in body-part alignment and FID substantial relative to the parameter increase.

Thanks for the response and revisions. The latency results, λr\lambda_r sweep, attention visualizations, and parameter counts address my earlier questions. I still have a few concerns.

  1. Multi-track (W2). Could you report some TimelineMotion statistics, like tracks per timeline, overlap ratio, and types of overlapping actions? Results for overlapping vs. non-overlapping segments would also help. App. A.2 excludes locomotion conflicts, stance changes, and full-body actions, so I'm not sure how general the multi-track claim is.

  2. Foot jitter (W3). I noticed some foot jitter in the supplementary video. Transition distance alone doesn't seem enough to evaluate smoothness. Could you report foot skating and jerk metrics for TimelineControl, GT, and the baselines?

  3. Long streaming sequence. Could you show side-by-side videos of TimelineControl and ActionPlan* using the sequence in ActionPlan paper Fig. 5 (“walk forward while jumping” 0–6s → “move forward diagonally” 6–12s → “stand” 12–18s → “dancing waltz” 18–24s)? One with the original sequence, and another with “wave left hand” added at 7–10s. I'd like to see if the backbone can still generate these dynamic motions with multi-track control.

  4. Boundary discontinuities. ActionPlan* + multi-text has the same transition distance as TimelineControl (1.31), while ActionPlan* gets 0.92. Could this be due to the hard visibility mask rather than responsiveness? A comparison with a soft or ramped mask would help.

AuthorsOpenReview ↗

Follow-up response to reviewer BEQz

We thank the reviewer for the follow-up questions. We update the supplementary material with the requested videos. We further report dataset statistics, results on overlapping/non-overlapping segments, foot-skating and jerk results, and a ramped-mask comparison.

Multi-track coverage (W2).

Across 44,944 TimelineMotion sequences, each timeline contains three tracks, where 24.75% frames have two overlapping actions and 23.23% frames have 3 overlapping actions. Among overlapping action pairs, 86.01% combine locomotion, such as walking, with another compatible action, such as raising an arm. Another 13.99% combine two actions involving the arms, such as reaching with the right arm while throwing with the left. We further evaluate R@3 and M2T within overlapping or non-overlapping segments:

MethodNon-overlap R@3 ↑Non-overlap M2T ↑Overlap R@3 ↑Overlap M2T ↑
STMC0.76760.71320.48110.6120
ActionPlan*0.81350.73500.45700.6084
TimelineControl0.83840.75240.56110.6368

TimelineControl achieves higher R@3 in both groups, showing that the alignment improvement of our method also holds when multiple instructions are active.

Foot skating and jerk (W3).

We further evaluate the baselines and GT on the TimelineMotion dataset with foot skating ratio and jerk metrics. TimelineControl improves instruction alignment and reduces foot skating relative to STMC, while jerk remains an area for improvement.

MethodFoot skating ratio (%) ↓Jerk (m/s³) ↓
GT10.8937.79
STMC13.0220.97
ActionPlan*6.3851.01
TimelineControl7.9049.02

Long streaming sequence.

We provide side-by-side videos for the requested 24-second sequence and the version with an added left-hand wave in the updated supplementary materials. In the example with overlapping actions, ActionPlan* fails to generate “moving forward” actions, while our TimelineControl generates consistent motion with the given instruction.

Boundary discontinuities.

As suggested by the reviewer, we compare the original TimelineControl and the ablation with a linear ramp after instruction start and end on TimelineMotion test sequences. We agree that the mask is a factor for the boundary distance. The ramped mask reduces boundary displacement while the hard mask yields slightly better alignment and FID, offering a trade-off between instruction control and boundary continuity.

MethodSeq R@3 ↑Seq M2T ↑FID ↓FID+ ↓Trans(cm)↓AP ↑AUC ↑
TimelineControl0.6030.6530.0170.1921.310.8320.772
TimelineControl + ramped mask0.5970.6530.0190.1921.170.8300.769

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Area Chair and Reviewers,

We thank the reviewers for their constructive feedback. Below we summarize the strengths highlighted in the reviews, our responses to the main concerns, and the corresponding revisions.

Strengths noted by the reviewers

  • Practical value of streaming motion generation with overlapping instructions (all reviewers).

  • Controlled comparisons with baselines retrained on the same data (BEQz, GzPp).

  • Improved semantic alignment and temporal adherence (GzPp, drbP).

Main concerns

  1. Responsiveness and action timing (BEQz W3-a/Q2-b, GzPp W1/Q1, drbP W2). Reviewers asked whether new instructions can be incorporated quickly and whether the requested actions are actually executed at the specified times. The first updated output is available after 230-234 ms with one or two existing instructions (Section 5.1, Appendix C.4). Manual video annotation shows fewer missed actions and lower onset/offset errors, although transition distance is higher (Table R1). This makes the trade-off between action execution, timing, and boundary displacement explicit.

  2. Contribution, complexity, and evidence (BEQz W1/W3-b/W3-c/Q3). Reviewers questioned the gains beyond multi-text conditioning and whether the added complexity and architectural choices are justified. Compared with ActionPlan* + multi-text, TimelineControl reduces FID by 52.8% with 2.9% more denoiser parameters (Table 9, Appendix D.4). Ablations support the part-aware components (Tables 4 and R3), with five evaluation repeats and 95% confidence intervals reported in Table 4. The user study provides complementary perceptual evidence (Figure 7, Appendix C.5).

  3. Data reliability and coverage (BEQz W2-a, drbP W1). Reviewers questioned whether body-part labels inferred from text are reliable and whether synthetic composition restricts the data’s coverage. We clarify both limitations in Appendix G. Visual verification of 300 source motion-caption pairs yields 83.60% macro-F1 for the body-part labels (Section 4.4, Appendix A.1). On the separate MTT benchmark, TimelineControl also achieves higher text-motion alignment than offline STMC with comparable FID+ (Table 1).

  4. Interpretation of temporal metrics (BEQz W2-b/Q1). Reviewers found the lower GT scores in Table 5 apparently inconsistent with the generated-motion scores in Table 1. Table 5 uses only internal instructions for interval-shift tests, whereas Table 1 uses the full test set under the same scoring rule. Their averages therefore cannot be compared directly (Appendix C.3). Within the GT validation, scores peak at the correct intervals and decrease under larger shifts, showing sensitivity to temporal misalignment (Table 5).

  5. Interaction setting (BEQz Q2-a, drbP W3). Reviewers asked how instruction end times are obtained and how ongoing commands can be cancelled or replaced. The experiments provide durations when instructions arrive. For open-ended commands, the application extends five-second intervals during generation. Instruction revision, cancellation, and conflict replacement are handled by an external policy that updates the active instructions (Sections 5.1 and 5.4, Appendix B.4).

  6. Challenging motions and long-horizon generation (drbP Q1/Q2). Reviewers asked whether dynamic or acrobatic actions remain plausible and whether errors accumulate during long generations. On 128 challenging timelines, TimelineControl achieves higher text alignment on the selected challenging instruction intervals and lower whole-sequence FID than STMC (Appendix D.6). Across 20 ten-minute rollouts, transition distance and jitter show no clear deterioration, while text alignment declines slightly (Appendix D.7).

Changes in the revised PDF (marked in blue)

For consistency with the reviews, we keep the original table and figure numbers and use an R prefix for newly added tables and figures. The numbering will be consolidated after the rebuttal period.

  • Main paper: related work (Section 2, BEQz W4), annotation verification (Section 4.4), instruction durations and response latency (Section 5.1), action timing and continuity (Section 5.2, Table R1), and ablation confidence intervals and parameter overhead (Section 5.3, Table 4).

  • Appendices A.1, B.4, and C.3-C.4: annotation details, interactive instruction handling, temporal-adherence validation (Table 5), and runtime breakdown (Table R2).

  • Appendices D.4-D.5: multi-text conditioning, parameter analysis, and part-aware bias analysis (Table 9, Table R3, Figure R1).

  • Appendices D.6-D.7: challenging-motion and long-horizon results.

  • Appendix G: limitations of text-derived body-part annotations and motion composition.

We thank the Area Chair and reviewers for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.