acceptodds
Under review as a conference paper at ICLR 2027

TempoTrack: Temporally Grounded Multi-Track Control for Streaming Joint Audio-Video Generation

13 views

Abstract

Interactive joint audio-video generation requires users to schedule independently timed events and introduce new instructions as generation proceeds. Existing global conditioning and segment-level prompt switching offer limited support for dynamically arriving controls with asynchronous, overlapping intervals. We introduce TempoTrack, a framework for online, temporally grounded multi-track control in streaming joint audio-video generation. Its core mechanism, Hierarchical Context Routing (HCR), connects event-level instructions to blockwise generation through coarse-to-fine conditioning. An online prompt queue admits newly received instructions at block boundaries and selects events relevant to each causal window, while Temporal Attention Bias (TAB) aligns instruction intervals with audio-video query positions on a shared physical timeline through timestamp-dependent cross-attention biases. To enable few-step streaming generation, we directly compose the temporally controllable causal model with a pretrained distillation LoRA and apply self-forcing distribution matching under self-generated histories, without an additional ODE-based initialization stage. We further introduce MTBench to evaluate multi-track instruction adherence, temporal alignment, and joint audiovisual quality. Experiments demonstrate asynchronous, overlapping, and dynamically arriving controls alongside strong identity consistency and audiovisual synchronization, enabling streaming audiovisual content to be incrementally directed through independently scheduled events.

Primary area:
generative models
Keywords:
Audio-Video Joint Generation, Streaming Video Generation, Diffusion

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_oacEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TempoTrack, a framework for online, temporally grounded multi track control in streaming joint audio–video generation. The key idea, Hierarchical Context Routing (HCR), combines block-level prompt admission (a prompt queue that selects only instructions whose intervals intersect the current causal window) with a token-level Temporal Attention Bias (TAB) that introduces a timestamp dependent cross-attention bias to align instructions and audio/video latents on a shared timeline. To enable few step streaming, the authors compose a causally adapted model with a pretrained distillation LoRA and then train a separate DMD LoRA under self generated histories (self forcing). They introduce MTBench to evaluate multi track adherence and report strong identity consistency, synchronization, and controllability under overlapping and dynamically arriving controls.

Critical strengths

1,HCR’s two-tier routing (block-level admission + token-level Gaussian temporal bias) is a clean, practical way to bridge event-level timing with blockwise causal generation in a joint AV model. 2,The “sink repositioning” trick for sliding-window inference is a simple but effective identity anchor under long rollouts. 3,The multi-track adherence evaluation (Body Event, Face Success, Speech Accuracy) addresses a gap where standard metrics do not test whether controls land on the intended track and interval.

Critical weaknesses

1,Technical limitations or concerns

a, TAB is extremely close to prior temporal routing biases (e.g., Temporal Context Routing with Gaussian interval priors), raising questions about novelty at the fine-grained mechanism level; the main novelty lies more in the block-level admission and streaming composition than in the attention bias itself.

b, The few-step composition via distillation LoRA is similar in spirit to recent “decoupled” compositions; while the self forcing residual is a useful addition, the methodological delta over prior art may be incremental.

2, Experimental gaps

a, The “online” aspect (asynchronous arrivals during generation) is not directly stress tested with interactive latency/achievement metrics. MTBench appears to be a static, pre-specified timeline rather than a live, update-while-generating protocol.

b, Current evidence does not yet rule out the possibility that the primary benefits stem from offline generation with temporal annotations. The empirical scope is limited to single-person speech, facial expressions, and upper-body movements; the control of musical tracks and general audiovisual events has yet to be validated.

c, There is a lack of a fine-tuning baseline, trained on the same data, using the same backbone and training pipeline, but without the "time bias" (time-conditioning offset), as well as a direct comparison between the "Queue-only" and "Queue + TAB" configurations.

d, It is not sufficiently clarified whether the training data, LoRA capacity, and sampling budget are consistent across variants such as TIE and Hard Mask.

3,Clarity

It is not fully specified how λ in TAB is tuned across layers/heads or whether its value is shared across modalities

4,Missing related works

a, The paper does not discuss Temporal Context Routing–style approaches that use parameter-free Gaussian biases on cross-attention logits, which are very similar to TAB, nor does it position its contribution relative to such recent work beyond TIE/TS-Attn/MinT.

b, Recent streaming and distillation recipes (e.g., context-aligned AR DMD, decoupled causal+distilled LoRA composition, reward-weighted/self-forcing variants) deserve more direct comparison and discussion, especially on the few-step composition claim.

Questions for authors

1, How is the TAB strength λ selected across layers/heads and modalities? Did you study sensitivity to λ and to timestamp noise or interval mis-specification?

2, Can you report interactive, online metrics (e.g., update latency, achievement rate for instructions submitted mid-rollout, and conflict resolution under overlapping tracks) on a streaming benchmark, or release an extension of MTBench for interactive evaluation?

3, Please clarify the provenance and compatibility of the pretrained distillation LoRA: is it trained on the same backbone and data distribution? How portable is your composition across different backbones?

4, Given that AVSpeech is largely talking-head content, how were body actions and background music annotated and validated?

5 comments
AuthorsOpenReview ↗

Response to Reviewer oacE. [Part I]

W1(a) & W4(a): TCR and the contribution of TAB

Thank you for highlighting this connection. Our main contribution is the first exploration of online, temporally grounded multi-track control for streaming joint audiovisual generation, where instructions have independent, potentially overlapping target intervals and may arrive dynamically during generation. TAB serves as the temporal routing component within this framework. TCR [1] is concurrent work that also uses Gaussian temporal routing. While the two methods share this mechanism, they address different conditioning settings: TCR operates on a pre-specified script, whereas TempoTrack separates instruction arrival from the target interval and manages instruction admission and eviction during blockwise causal generation. TAB operates on the currently admitted instructions and routes their influence to both audio and video positions on a shared timeline. The distinction therefore lies in integrating temporal routing with dynamic instruction management and causal joint audiovisual generation. We clarified this relationship and acknowledged TCR as concurrent work in the revision.

W1(b) & W4(b): Few-step composition and related training approaches

We thank the reviewer for raising this concern. Although LongLive-2.0 [2] also does not use ODE initialization, it does not include a subsequent self-forcing refinement stage to further address error accumulation. TempoTrack reuses a pretrained distillation LoRA and introduces an additional self-forcing DMD training stage while freezing the existing components. Table 3 evaluates the contribution of this additional refinement within our system. We will clarify this distinction and distinguish the composition strategy from the subsequent adaptation in the revision.

W2(a) & Q2: Online arrival and overlapping controls

We thank the reviewer for raising this concern. Our original evaluation assigned each instruction group a submit_at time preceding its target intervals. We further tested live submission under four arrival conditions . We report reception-to-admission latency and instruction achievement; since users specify target intervals, content timing is evaluated against those intervals rather than requiring immediate execution.

Arrival conditionBody Event / Face Success ↑Speech Accuracy ↑Speech onset / offset MAE ↓BODY / FACE Out-of-window Activity ↓Admission latency: Median ↓
Preloaded52.63% / 66.67%75.00%1.164 / 0.69911.67% / 25.29%1.738
Mid-block56.86% / 73.68%68.75%1.085 / 0.63610.26% / 20.31%1.982
Late52.63% / 71.11%71.44%1.121 / 0.60011.28% / 32.88%2.007

Expired instructions have a 100% scheduler rejection rate. Compared with preloaded instructions, mid-block and late arrivals yield similar BODY and FACE adherence in the reported point estimates. These results support the model’s ability to execute newly arriving controls during generation.

W2(b): Online benefits and task coverage

We thank the reviewer for raising this concern. Our original evaluation scheduled instruction availability through submit_at. More directly, the added live-submission experiments withhold target instructions from the generator until they arrive during generation. Their subsequent execution provides evidence of online control beyond a fully preloaded timeline. Our experiments focus on digital-human scenarios involving speech, facial expressions, and body actions. We leave music and general audiovisual control as future directions.

W2(c): Queue-only versus Queue + TAB

We added a direct Queue-only experiment. Training data, backbone training procedure, and block-level scheduling are held fixed; only temporal attention bias is changed. The experimental results show that Queue + TAB substantially outperforms Queue-only.

MethodBODY Coverage ↑FACE Coverage ↑Speech onset MAE ↓Speech offset MAE ↓BODY Out-of-window Activity ↓FACE Out-of-window Activity ↓
Queue-only81.80%15.93%2.4022.38120.08%39.74%
Queue + TAB91.99%58.97%1.1700.6838.24%10.61%

W2(d): TIE and Hard Mask comparison settings

TIE, Hard Mask, and our method are trained through LoRA using the same training data, backbone, LoRA capacity, training procedure, and budget, with only the temporal-conditioning mechanism changed. Evaluation uses the same sampling settings. We will include the configurations in the revised appendix.

References

[1] Yichen Liu et al. “The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation.” arXiv:2609.02367, 2026.

[2] Yukang Chen et al. “LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation.” 2026.

AuthorsOpenReview ↗

Response to Reviewer oacE. [Part II]

W3 & Q1: TAB strength

TAB uses a fixed, non-learnable λ=5, shared across layers, attention heads, and the audio and video branches. It is applied to T2A and T2V cross-attention, with each branch computing the bias using the physical timestamps of its own latents.

The added ablation compares λ=2.5, 5, and 10, as well as a Flat-smooth profile. The results show a trade-off: weaker Gaussian bias improves state coverage, whereas the default λ=5 achieves the lowest Out-of-window activity on both visual tracks and the lowest speech offset MAE among the tested configurations.

ConfigurationBODY Coverage ↑FACE Coverage ↑Speech onset MAE ↓Speech offset MAE ↓BODY Out-of-window Activity ↓FACE Out-of-window Activity ↓
Gaussian λ=2.598.94%76.63%0.8970.75615.62%12.86%
Gaussian λ=1084.67%68.63%0.8110.8289.71%23.61%
Flat-smooth λ=5100.00%58.14%1.2510.96415.70%11.84%
Gaussian λ=5 (default)91.99%58.97%1.1700.6838.24%10.61%

Q3: Distillation LoRA source and compatibility

We use the official LTX-2.3 pretrained distilled LoRA, ltx-2.3-22b-distilled-lora-384.safetensors, with its matching LTX-2.3 backbone. We do not train or fine-tune this adapter on our dataset. It provides the few-step initialization, while our subsequent self-forcing DMD refinement trains an additional residual with the pretrained adapter frozen.

Q4: Body-action and background-music annotation

AVSpeech contains many clips showing the speaker’s upper body with visible gestures and movements. As described in the paper, the actions are annotated using Gemini, and we manually inspect a subset of the annotations to check their accuracy. Our body-control evaluation is limited to upper-body gestures and movements. We neither train nor evaluate background-music control and will clarify this scope in the revised manuscript.

AuthorsOpenReview ↗

Response Summary for Reviewer oacE

Thank you for your constructive feedback. Our rebuttal provides the following additions and clarifications:

  1. Positioning relative to related work. We clarified the relationship to concurrent Gaussian temporal-routing work and related distillation approaches. Our central contribution is the integration of dynamic instruction management and temporal routing into causal joint audiovisual generation. We also distinguished pretrained LoRA composition from the subsequent self-forcing refinement.
  2. Direct evidence for online control and TAB. We added live-submission experiments and a controlled Queue-only versus Queue + TAB comparison. Adding TAB reduces speech onset and offset errors, improves sustained-state coverage, and reduces out-of-window visual activity.
  3. Experimental settings and parameter analysis. We clarified that TIE, Hard Mask, and our method use the same training settings, reported bias-strength and profile ablations, and specified the source and backbone compatibility of the pretrained distillation LoRA.
  4. Task and annotation scope. We explained the body-action annotation and manual inspection procedure, and explicitly limited our empirical claims to speech, facial expressions, and upper-body movements. Background-music control is neither trained nor evaluated in this work.

We sincerely appreciate your time and valuable feedback. We have carefully responded to the raised concerns in our rebuttal and provided additional clarifications. We would be happy to further discuss any remaining questions during the discussion period.

Reviewer_4gCwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TempoTrack adds timed control to streaming audio-video generation. It starts from LTX-2.3. Face, body, and speech each have their own time range. The ranges can overlap, and a new instruction can arrive after generation has started. Each block keeps only the lines that overlap that block. A bias then makes a token at time t attend more to the nearby line. Training uses LoRA: bidirectional control, causal adaptation, then a pretrained distillation LoRA plus self-forcing. On 56 test cases, it ranks first on body events, speech timing, identity, lip sync, and speed. Face success is second. The transcript has more errors than LTX-2.3 and TaoMate.

Critical strengths

The control setup is useful. The user can schedule the face, the body, and the speech on different times, including overlap, and can add a line after the video has already started. The paper states this clearly and tests it.

Critical weaknesses

The core control method is built on splitting the prompt. Which lines enter a block, and how attention binds each line to a time, both assume that the face, the body, and the speech stay in separate lines. The paper does not show why that split is needed. The simple baseline keeps one text for the block. That text states the face, the hand, and the speech together. At 3.2s it can say "She smiles and raises her right hand." At 5.5s the smile is over and speech has started, so the same text becomes "She raises her right hand and says: ...". Only the changed sentence is edited. There is no separate face, body, or speech prompt, and no time bias in attention. The schedule still changes the forward, because this text changes at the boundary. LTX-2.3 is not this baseline: it gets one text for the whole clip. Please train a separate model on this single text. Use the same causal recipe: bidirectional first, then causal. Do not reuse the checkpoint trained on separate prompts.

If the answers below are satisfactory, I will keep the positive score.

Questions for authors
  1. Please train a comparison model with one text per block. The text states the face, the hand, and the speech together. At 3.2s it can say "She smiles and raises her right hand." At 5.5s the smile is over and speech has started, so it becomes "She raises her right hand and says: ...". Only the changed sentence is edited. Do not use separate face, body, and speech prompts, and do not add a time bias. Use the same causal recipe: bidirectional first, then causal. Do not reuse the checkpoint trained on separate prompts. This is the comparison that shows whether splitting matters.
  2. On this kind of multi-part, multi-time control, do you still do better than strong commercial models such as Seedance 2.5 and MiniMax-H3?
  3. The Fig. 3 caption says your method has the best identity preservation. Please explain why it is better than earlier methods, and which part of your method causes that.
  4. TaoMate follows facial expressions better, but keeps identity worse. What can be done to improve both together?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4gCw.

Q1: A trained unified-prompt baseline

We thank the reviewer for raising this concern. Structured prompts are used in several generation systems, providing practical support for this representation. As discussed in the Introduction, blockwise prompt switching is a straightforward control mechanism. In joint audiovisual generation, however, each block’s text must condition both audio and visual content while maintaining synchronization. Separate events also allow users to manage individual tracks without reorganizing all controls whenever one track changes.

To directly address the reviewer’s request, we trained a baseline with one unified prompt per block. The prompt jointly describes the current facial expression, body action, and speech, without separate track prompts or temporal attention bias. It starts from the same original backbone and follows bidirectional training and causal adaptation.

MethodBody Event ↑Face Success ↑Speech Accuracy ↑Speech onset MAE (s) ↓Speech offset MAE (s) ↓BODY Out-of-window Activity ↓FACE Out-of-window Activity ↓
Unified41.46%64.20%42.50%3.4153.73721.63%48.00%
Ours71.74%92.50%99.73%1.1700.6838.24%10.61%

Q2: Commercial-model comparisons

Thank you for suggesting comparisons with strong commercial models. We evaluated them using the same reference images, underlying instructions, and evaluation rules.

Commercial models provide a strong reference for generation quality. TempoTrack achieves competitive performance on control metrics while supporting newly arriving instructions during causal joint audiovisual generation. The commercial-model runs in this comparison receive the complete timed prompt before generation. These results show that TempoTrack approaches commercial-model performance while enabling online multi-track control.

Method/versionSpeech onset MAE ↓Speech offset MAE ↓Body Event ↑Face Success ↑Speech Accuracy ↑Coverage ↑Out-of-window Activity ↓
Seedance 2.50.44000.694772.41%100.00%99.42%81.18%37.04%
MiniMax-H30.51300.923055.56%95.24%97.09%64.04%30.56%
TempoTrack1.1700.68371.74%92.50%99.73%76.85%9.35%

Q3: Identity preservation

We retain the first block as an attention sink to provide a persistent reference for later generation, and reposition its coordinates to account for changes in the sliding window. The ablation in Table 4 supports the contribution of these components to identity consistency. Table 3 also shows further improvement after the final self-forcing refinement.

Q4: Improving expression adherence and identity preservation

From the visual results, TaoMate tends to produce more exaggerated facial expressions, which may contribute to identity drift as generation progresses. A possible way to improve both is to better separate expression control from identity conditioning, using a stable identity reference alongside dedicated expression guidance and jointly optimizing expression fidelity and identity consistency.

AuthorsOpenReview ↗

Response Summary for Reviewer 4gCw

Thank you for your constructive feedback. Our rebuttal includes the following additions and clarifications:

  1. A separately trained unified-prompt baseline. Starting from the same original backbone, we trained the requested baseline through bidirectional training and causal adaptation, using one unified prompt per block without separate track prompts or temporal attention bias. TempoTrack achieves higher Body Event, Face Success, and Speech Accuracy than the unified-prompt baseline, while reducing speech timing errors and out-of-window visual activity.
  2. Commercial-model comparisons. We added comparisons with Seedance 2.5 and MiniMax-H3 using the same reference images, underlying instructions, and evaluation rules. TempoTrack achieves competitive results on the reported control metrics while supporting online instruction updates.
  3. Identity preservation and expression control. We explained how the persistent first-block attention sink and sink repositioning support identity consistency, with evidence from the existing ablations. We also discussed a possible direction for jointly improving expression adherence and identity preservation through stable identity conditioning and dedicated expression guidance.

Thank you again for your time and thoughtful feedback. We would be happy to further discuss any remaining questions during the discussion period.

Reviewer_w11SOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

TempoTrack proposes a method for incorporating timed instructions that arrive during block-by-block sequential audio–video generation. Each instruction’s start and end times are externally specified. The goal is to control facial expressions, body movements, and speech over different time intervals while preserving character consistency, continuity across blocks, audio quality, and lip synchronization.

To achieve this, HCR selects the instructions available for the current block, while TAB adds a temporal bias, computed from the output timestamp and each instruction’s target interval, to the audio/video-to-text cross-attention logits. This smoothly adjusts how much the model attends to each instruction at each point in time, but does not directly specify precise action boundaries or word-level progression within an utterance.

Training first equips a bidirectional model with temporal control, then adapts it for causal generation. A pretrained distillation LoRA is subsequently incorporated to initialize a few-step student. Without a separate ODE-based initialization stage, a new DMD LoRA is trained using self-forcing while the existing components remain frozen. This stage uses the frozen Stage 1 teacher and a critic that estimates the student’s generated distribution. The paper reports an end-to-end generation throughput of approximately 15.4 FPS at 480 × 480 resolution with four sampling steps, using a configuration that keeps the models resident across four GPUs.

Critical strengths

They propose a method for generating multiple actions that change according to specified timing in a streaming setting.

They also propose an efficient training approach that initializes the model with a distilled LoRA and continues training, without requiring ODE-based initialization.

Critical weaknesses
  1. Transcript progression during streaming generation is insufficiently addressed

For joint audio-video generation of a speaking character, I consider reliable speech generation a central requirement. However, the paper does not adequately explain how the model tracks which part of the transcript has already been spoken and which part remains to be generated.

For a speech clause spanning multiple blocks, the full transcript appears to remain in the conditioning context, including words that may already have been spoken, alongside other active instructions. This raises concerns about repetition, omission, and interference between instructions, especially for long utterances. The final model’s WER of 14.48% also raises concerns about speech reliability, although the results do not establish whether transcript handling is responsible for these errors. I see the lack of a clear treatment of transcript progression as a major limitation of the proposed streaming setting.

  1. The novelty of temporal action control is not sufficiently clear

Generating different actions at specified times has already been explored in prior video generation work. The ability to schedule multiple actions over time therefore does not, by itself, establish a sufficiently distinct contribution.

The paper needs to clarify what its approach contributes beyond existing temporal control methods, particularly what is gained from supporting online instructions in joint audio-video generation. This distinction is especially important given the unresolved concerns about speech generation.

  1. TAB and the gating design seem overly heuristic

The Gaussian prior in TAB favors the midpoint of an instruction’s interval. This is not an obvious choice for actions that should persist throughout the entire interval, and the motivation for this design is not sufficiently convincing.

More broadly, the choice of temporal attention bias would benefit from stronger justification. An additive bias is a reasonable way to incorporate timing information, but it is unclear whether the same temporal profile is equally appropriate for brief events and behaviors that should persist throughout an interval. Beyond the existing comparisons, more targeted analysis of the bias shape and strength, along with evaluations of onset and offset accuracy and full-interval persistence, would help establish whether the proposed formulation adequately supports these different temporal requirements.

  1. The practical online interaction scenario is unclear

Requiring users to specify precise start and end times while a video is being generated seems cumbersome. It is unclear how a user would choose feasible target intervals as the stream advances, or how commands such as “do this now” would map to the generated timeline.

This makes the practical motivation for the online setting less convincing. The paper should explain how the intended interaction would work for a human user and clarify its practical advantages over providing a timeline of instructions in advance.


I recognize that combining online instruction updates with multi-track joint audio-video generation presents a meaningful technical challenge. Retaining the full transcript and using a simple attention bias can be reasonable design choices, and timestamped inputs may be appropriate when supplied by an external scheduler or agent. My concerns primarily concern whether the current explanations and evaluations sufficiently support these choices, particularly regarding speech continuity, sustained actions, and responsiveness to newly arriving instructions.

Questions for authors
  1. Temporal control performance of the teacher

How well does the Stage 1 teacher follow the specified temporal intervals for actions and speech? Since it defines the target distribution for DMD, could you report its temporal control performance alongside that of the final student?

  1. Stage 3 losses

Is the Stage 3 student trained only with the DMD objective, or are LPIPS regression, GAN loss, or other auxiliary losses also included?

  1. Text encoding and positional information

Are clauses encoded independently and then concatenated, as in concat(Encode(text1), Encode(text2)), or concatenated before encoding, as in Encode(text1 + separator + text2)?

How are position IDs, separators, and any RoPE handled? How is the text-conditioning cache updated when clauses are added or removed?

  1. Online conditioning and the teacher’s information access

During DMD training, does the bidirectional teacher receive the full instruction sequence, including instructions that have not yet arrived for the student? How is the teacher’s conditioning aligned with the information available to the student at each generation block?

3 comments
AuthorsOpenReview ↗

Response to Reviewer w11S. [Part I]

W1: Cross-block speech progression and reliability

Thank you for raising this concern. Retaining the full utterance does not mean that speech restarts at each block: through the causal KV cache, the student conditions on previously generated audiovisual content. Transcript progression is therefore inferred jointly from the full text and generated history. We acknowledge that this implicit mechanism is imperfect: some cases contain omissions or additional/repeated words, which contribute to WER. Nevertheless, our method achieves the highest Speech Accuracy, indicating strong reference-word adherence within the assigned intervals. Retaining the full utterance also preserves linguistic context across blocks and avoids requiring an explicit word-to-block assignment or a separate transcript-tracking module. We will clarify this design and its limitations in the revision, and discuss explicit tracking of spoken words as a potential improvement.

W2: Contribution beyond temporal action control

Thank you for asking us to clarify this distinction. The benefit of online control is that what the character should say and do next can be decided after generation has already started, instead of requiring a complete audiovisual script in advance. For example, in an interactive avatar scenario, a newly received audience question can motivate a subsequent spoken response together with accompanying expressions and gestures, each with its own target interval.

This requirement goes beyond generating predefined actions at specified times. Global conditioning assumes that the intended sequence is available beforehand. Whole-prompt switching allows content updates, but updating the text alone does not explicitly associate individual instructions with their respective audio and video times. Audio-driven animation instead conditions on a supplied audio stream, whereas our setting jointly generates the speech and visual behavior.

TempoTrack addresses this setting by combining block-level admission of newly arriving instructions with clause-level temporal routing on a shared audiovisual timeline. In our evaluation, instructions are submitted in groups, while facial, body, and speech controls retain independent, potentially overlapping intervals. The contribution is therefore the integration of online instruction availability and multi-track temporal control into causal joint audiovisual generation.

W3: Gaussian temporal bias and sustained behavior

Thank you for raising this concern. To examine this issue, we compare the default Gaussian with different bias strengths and an interval-flat profile with smooth boundary decay (Flat-smooth). In addition to event-level adherence, we evaluate state coverage for sustained BODY/FACE instructions, speech onset/offset mean absolute error, and visual activity outside the target interval. State coverage measures the proportion of the target interval during which the requested state is present; brief events are evaluated separately using event-level metrics.

The results reveal a trade-off between interval coverage and temporal selectivity. Compared with Flat-smooth, the default Gaussian achieves lower pre-target activity on both visual tracks and lower speech onset/offset MAE, while Flat-smooth provides higher BODY coverage.

ConfigurationBODY Coverage ↑FACE Coverage ↑Speech onset MAE ↓Speech offset MAE ↓BODY Out-of-window Activity ↓FACE Out-of-window Activity ↓
Gaussian λ=2.598.94%76.63%0.8970.75615.62%12.86%
Gaussian λ=1084.67%68.63%0.8110.8289.71%23.61%
Flat-smooth λ=5100.00%58.14%1.2510.96415.70%11.84%
Gaussian λ=5 (default)91.99%58.97%1.1700.6838.24%10.61%

W4: Practical interaction

TempoTrack is intended to support online interaction. Our current interface accepts structured timed instructions, enabling updates based on new inputs. In practical use, a VLM agent could generate these prompts from user requests and observations. Our current focus is on executing dynamically arriving instructions with temporal control, and we plan to explore VLM-agent integration in future work.

AuthorsOpenReview ↗

Response to Reviewer w11S. [Part II]

Q1: Temporal adherence of the Stage 1 teacher

We evaluated the Stage 1 teacher and final student using the same cases and evaluation rules.

ModelBody EventFace SuccessSpeech Accuracy
Stage 1 teacher0.78980.93670.9967
Student0.71740.92500.9973

Q2: Stage 3 losses

The Stage 3 student is trained only with the DMD objective, without LPIPS, GAN losses, or other auxiliary losses. The critic is trained separately with a flow-matching loss.

Q3: Text encoding, temporal positions, and caching

Clauses are encoded independently and then concatenated: concat(Encode(text1), Encode(text2)). Each uses Gemma’s default positional handling without cross-clause position offsets. Padding is removed, tokenizer-provided special tokens are retained, and no additional separators are inserted.

The connectors apply one-dimensional RoPE to the concatenated token sequence. This represents token order; temporal alignment instead uses TAB with clause intervals and audio/video latent timestamps, not word-level text–latent RoPE.

When clauses are added or removed, existing clause features are reused and connector outputs are recomputed for the new active set. Processed contexts are cached by active segment set and reused when that set recurs.

Q4: Text visibility during training

During DMD training, both teacher and student receive the full timed instruction sequence. The teacher therefore does not receive additional instructions hidden from the student. Training uses complete timelines and does not simulate dynamic arrivals. At inference, online scheduling determines which instructions are available to each block. There is thus a training–inference gap in instruction availability: TAB controls temporal relevance, while the online scheduler controls arrival visibility. We will clarify these two settings and their distinction in the revision.

AuthorsOpenReview ↗

Response Summary for Reviewer w11S

Thank you for your thoughtful comments. Our rebuttal provides the following clarifications and additional evidence:

  1. Cross-block speech progression and context preservation. While implicit speech progression can still involve omissions or repetitions, the model uses the causal KV cache to condition on both the full utterance and previously generated audiovisual content, achieving the highest Speech Accuracy and demonstrating strong reference-word adherence within the target intervals. Retaining the full utterance also preserves linguistic context across blocks without requiring explicit word-to-block assignments or a separate transcript-tracking module.
  2. Contribution and interaction setting. We clarified that TempoTrack combines online instruction admission with independently timed, multi-track control in joint audiovisual generation, allowing subsequent speech and visual behavior to be specified after generation has begun. We also explained the intended use of the structured timed-instruction interface.
  3. Temporal bias analysis. We compared different Gaussian bias strengths and a flat-smooth profile using sustained-state coverage, speech onset/offset errors, and out-of-window visual activity. These results characterize the trade-off between interval coverage and temporal selectivity.
  4. Text processing and instruction availability. We explained independent clause encoding and conditioning-cache updates, and clarified the distinction between complete-timeline training and queue-admitted instruction availability during online inference.

We sincerely appreciate your time and valuable feedback. We have carefully responded to the raised concerns in our rebuttal and provided additional clarifications. We would be happy to further discuss any remaining questions during the discussion period.

Reviewer_W6sfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TempoTrack introduces HCR for streaming joint audio-video generation with temporally specified controls across multiple tracks. HCR combines block-level prompt admission with a Gaussian temporal attention bias, and is trained using causal adaptation, LoRA distillation, and self-forcing DMD.

Critical strengths
  1. TempoTrack addresses an important but underexplored problem: controlling multiple audio-visual events, each defined over an explicit temporal interval, during incremental generation. Its HCR decomposition into block-level admission and token-level temporal routing is clear and intuitive, and could be useful for interactive generation.
  2. The training pipeline coherently combines causal adaptation, distillation, and self-forcing. The ablation studies further reveal meaningful trade-offs in quality control, indicating that the method is technically well developed rather than merely conceptual.
Critical weaknesses

First, how can we verify that the system truly handles dynamically arriving controls? The paper defines arrival times and block-level admission, but does not show experiments with late or mid-block instruction arrivals. Also, does the frozen bidirectional DMD teacher access future latents or prompt information during training, and how do you rule out future-information leakage into the causal student?

Second, how reliable is the MTBench evaluation? The benchmark contains only 56 synthetic cases generated with proprietary models, and the same type of model is heavily involved in judging the results. Could you provide human validation or an independent evaluator? The current metrics give credit when an event appears only once within the target interval, without checking onset, offset, persistence, or activity outside the interval. How many cases are excluded as unevaluable for each method, and can you report the full denominators, confidence intervals, and results across multiple random seeds?

Questions for authors
  1. State the arrival schedules used. Evaluate preloaded, mid-block, late, and expired arrivals, with effect latency and out-of-window errors.
  2. Clarify whether the DMD teacher sees future latents or prompts. Add a causal or future-masked ablation to rule out leakage.
  3. Add human or independent MTBench validation, exclusion counts, confidence intervals, and multi-seed results.
  4. Release baseline prompt conversions and matched throughput results, including first-block latency and real-time factor.
5 comments
AuthorsOpenReview ↗

Response to Reviewer W6sf. [Part I]

W1 & Q1: Dynamic instruction arrival and temporal adherence

Thank you for the suggestion. In our original evaluation, each instruction group was assigned a submission time preceding its target intervals to simulate dynamically arriving controls. To further test instruction arrivals during generation, we evaluated 56 cases with three random seeds under four conditions: Preloaded, Mid-block, Late, and Expired.

  • Preloaded: Submitted before generation begins.
  • Mid-block: Submitted during generation of the block preceding the target start.
  • Late: Submitted after generation of the block containing the target onset has begun.
  • Expired: Submitted after the target interval has ended.

Submission-to-content latency includes the wait until the user-specified target interval. We therefore report reception-to-admission latency for system response and onset/offset errors for content timing. We measure onset/offset on speech, whose boundaries can be located using ASR word timestamps.

All expired instructions were rejected, giving a scheduler rejection rate of 100%. Full protocols and metric definitions will be included in the revision.

Arrival conditionBody Event / Face Success ↑Speech Accuracy ↑Speech onset / offset MAE ↓BODY / FACE Out-of-window Activity ↓Reception-to-admission Median ↓
Preloaded52.63% / 66.67%75.00%1.164 / 0.69911.67% / 25.29%1.738
Mid-block56.86% / 73.68%68.75%1.085 / 0.63610.26% / 20.31%1.982
Late52.63% / 71.11%71.44%1.121 / 0.60011.28% / 32.88%2.007

Compared with preloaded instructions, mid-block and late arrivals yield similar BODY and FACE adherence in the reported point estimates. These results support the model’s ability to execute newly arriving controls during generation.

W2 & Q3: Human validation, temporal evaluation, and statistical reporting

1. Human validation.

For each metric, all three annotators independently evaluated the same 50 targets, yielding 150 human judgments per metric. Annotators followed the criteria for Body Event and Face Success and did not see the automatic judgments. Agreement was computed as the proportion of individual human judgments matching the corresponding automatic label. The agreement rates were 94.00% for Body Event and 90.67% for Face Success.

MetricUnique targetsHuman judgmentsAgreement with individual human judgments ↑
Body Event5015094%(141/150)
Face Success5015090.67%(136/150)

2. Temporal evaluation. Many actions and expressions are sustained behaviors whose exact boundaries are difficult to annotate. The original evaluation therefore checks whether the requested behavior appears within its target interval. To further assess temporal control, we report speech onset and offset MAE, measuring the mean absolute deviation of the detected speech start and end times from their target boundaries, respectively. We also measure BODY and FACE Out-of-window activity within the one-second window out of each target interval.

MethodSpeech onset MAE ↓Speech offset MAE ↓BODY Out-of-window Activity ↓FACE Out-of-window Activity ↓
OVI1.66161.585830.45%43.93%
MOVA0.92580.848321.88%55.56%
JavisDiT++--18.14%36.25%
LTX-2.30.74251.665922.20%75.00%
TaoMate-ref0.48461.505840.55%26.47%
OmniForcing1.55883.812316.00%42.17%
Ours1.1700.6838.24%10.61%

3. Statistical reporting. We report evaluation denominators and exclusion counts for each baseline in the main comparison in Appendix D, “Evaluation Coverage and Exclusion Counts.” We will additionally conduct multi-seed experiments to assess variability across random seeds.

AuthorsOpenReview ↗

Response to Reviewer W6sf. [Part II]

W1 & Q2: Teacher information access

Thank you for raising this concern. Following the training setup established in Self Forcing [1], we use a bidirectional teacher. Our Stage 1 teacher receives the complete re-noised student rollout and full timed text for sequence-level supervision. The student generates causally without accessing future latents; neither teacher nor critic participates in inference. We therefore adopt this established training setup rather than using a causal teacher.

Both models receive full timed prompts during training, while inference exposes only queue-admitted instructions. This creates a text-availability gap: TAB controls temporal relevance, not arrival visibility. We will clarify this distinction in the revision. As future work, we plan to explore a causal teacher and compare it with the bidirectional teacher under controlled conditions.

[1] Xun Huang et al. Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. NeurIPS, 2025.

Q4: Baseline prompt conversion and efficiency

We have added the baseline prompt-conversion procedures to Section C of the appendix. We describe how the original controls are converted to each model’s conditioning interface and provide examples.

The efficiency results are shown below. TaoMate requires two GPUs. DiT FPS measures denoising throughput, while end-to-end RTF is the total generation time divided by the generated audiovisual duration.

MethodGPUsDiT FPS ↑End-to-end RTF ↓First-block latency (s) ↓
OVI12.66414.54880.829
MOVA10.29090.049449.881
JavisDiT++11.45912.00961.091
LTX-2.311.32634.086170.996
TaoMate-ref220.6641.68411.585
OmniForcing119.3441.8028.534
Ours122.4033.08314.394
AuthorsOpenReview ↗

Response Summary for Reviewer W6sf

Thank you for your constructive feedback. We summarize the main additions and clarifications below:

  1. Dynamic instruction arrival. We evaluated 56 cases with three random seeds under preloaded, mid-block, late, and expired arrival conditions. The results include instruction adherence, reception-to-admission latency, and temporal errors. Body and face adherence under mid-block and late arrivals is comparable to the preloaded condition in the reported point estimates. All expired instructions were rejected.
  2. Human validation and temporal evaluation. Agreement between automatic labels and individual human judgments was 94.00% for Body Event and 90.67% for Face Success. We also added speech onset/offset errors and out-of-window visual activity to complement the event-level metrics.
  3. Teacher and student information access. We clarified the distinction between bidirectional teacher supervision during training and causal student generation at inference. We also explicitly described the instruction-availability gap between complete-timeline training and online inference.
  4. Baseline and efficiency details. We provided baseline prompt-conversion procedures and reported end-to-end RTF and first-block latency alongside denoising throughput.

Thank you again for your time and feedback. We would be happy to discuss any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair,

Thank you for your time and consideration of our submission.

To the best of our knowledge, TempoTrack is the first to study streaming joint audiovisual generation with dynamically arriving instructions, each assigned an independent, potentially overlapping target interval. This enables temporally grounded control over speech, expressions, and body movements.

Discussion progress. Reviewers W6sf, 4gCw, and oacE currently recommend weak acceptance. Following our rebuttal, oacE explicitly stated that all concerns had been addressed and that they would raise their score. W6sf thanked us for the detailed response and additional experiments and recommended incorporating them into the revised manuscript.

We summarize the additional experiments and clarifications provided in our rebuttal below:

  1. Dynamic instruction arrival experiments. We evaluated Preloaded, Mid-block, Late, and Expired arrival conditions across multiple random seeds, reporting instruction adherence, reception-to-admission latency, and temporal errors. Body and face adherence under Mid-block and Late arrivals is comparable to the preloaded condition in the reported point estimates, and all expired instructions were rejected. These results provide direct evidence of receiving and executing new instructions during generation.

  1. A trained unified-prompt baseline and Queue-only ablation. Starting from the same original backbone, we separately trained a unified-prompt baseline through bidirectional training and causal adaptation. TempoTrack outperforms this baseline in Body Event, Face Success, and Speech Accuracy, while reducing speech timing errors and out-of-window activity. In a controlled comparison between Queue-only and Queue + TAB, adding TAB reduces speech onset and offset errors, improves sustained-state coverage, and reduces out-of-window activity.

  1. Human validation and finer-grained temporal evaluation. Independent annotators evaluated Body Event and Face Success targets, showing high agreement between automatic labels and individual human judgments. We also supplemented event-level metrics with sustained-state coverage, speech onset/offset MAE, and pre-target visual activity, and provided evaluation denominators and exclusion counts for each baseline.

  1. Temporal bias analysis. We compared different Gaussian strengths and a Flat-smooth profile, revealing a trade-off between sustained-state coverage and temporal selectivity. The default configuration achieves the lowest out-of-window activity on both visual tracks and the lowest speech offset MAE among the tested configurations.

  1. Commercial-model comparisons. We compared against Seedance 2.5 and MiniMax-H3 using the same reference images, underlying instructions, and evaluation rules, demonstrating comparable performance on the reported control metrics on several reported control metrics.

  1. Clarification of contributions and implementation details. We clarified that our central contribution integrates dynamic instruction management with multi-track temporal control on a shared timeline in causal joint audiovisual generation, and explained its relationship to TCR and related distillation approaches. We provided teacher–student control comparisons and details on training losses, text encoding and caching, distillation LoRA provenance, matched comparison settings, identity preservation, prompt conversion, and efficiency metrics.

We hope these additional results and clarifications will be helpful in your assessment of our submission. Thank you again for your time and thoughtful consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.