- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TempoTrack, a framework for online, temporally grounded multi track control in streaming joint audio–video generation. The key idea, Hierarchical Context Routing (HCR), combines block-level prompt admission (a prompt queue that selects only instructions whose intervals intersect the current causal window) with a token-level Temporal Attention Bias (TAB) that introduces a timestamp dependent cross-attention bias to align instructions and audio/video latents on a shared timeline. To enable few step streaming, the authors compose a causally adapted model with a pretrained distillation LoRA and then train a separate DMD LoRA under self generated histories (self forcing). They introduce MTBench to evaluate multi track adherence and report strong identity consistency, synchronization, and controllability under overlapping and dynamically arriving controls.
Critical strengths
1,HCR’s two-tier routing (block-level admission + token-level Gaussian temporal bias) is a clean, practical way to bridge event-level timing with blockwise causal generation in a joint AV model. 2,The “sink repositioning” trick for sliding-window inference is a simple but effective identity anchor under long rollouts. 3,The multi-track adherence evaluation (Body Event, Face Success, Speech Accuracy) addresses a gap where standard metrics do not test whether controls land on the intended track and interval.
Critical weaknesses
1,Technical limitations or concerns
a, TAB is extremely close to prior temporal routing biases (e.g., Temporal Context Routing with Gaussian interval priors), raising questions about novelty at the fine-grained mechanism level; the main novelty lies more in the block-level admission and streaming composition than in the attention bias itself.
b, The few-step composition via distillation LoRA is similar in spirit to recent “decoupled” compositions; while the self forcing residual is a useful addition, the methodological delta over prior art may be incremental.
2, Experimental gaps
a, The “online” aspect (asynchronous arrivals during generation) is not directly stress tested with interactive latency/achievement metrics. MTBench appears to be a static, pre-specified timeline rather than a live, update-while-generating protocol.
b, Current evidence does not yet rule out the possibility that the primary benefits stem from offline generation with temporal annotations. The empirical scope is limited to single-person speech, facial expressions, and upper-body movements; the control of musical tracks and general audiovisual events has yet to be validated.
c, There is a lack of a fine-tuning baseline, trained on the same data, using the same backbone and training pipeline, but without the "time bias" (time-conditioning offset), as well as a direct comparison between the "Queue-only" and "Queue + TAB" configurations.
d, It is not sufficiently clarified whether the training data, LoRA capacity, and sampling budget are consistent across variants such as TIE and Hard Mask.
3,Clarity
It is not fully specified how λ in TAB is tuned across layers/heads or whether its value is shared across modalities
4,Missing related works
a, The paper does not discuss Temporal Context Routing–style approaches that use parameter-free Gaussian biases on cross-attention logits, which are very similar to TAB, nor does it position its contribution relative to such recent work beyond TIE/TS-Attn/MinT.
b, Recent streaming and distillation recipes (e.g., context-aligned AR DMD, decoupled causal+distilled LoRA composition, reward-weighted/self-forcing variants) deserve more direct comparison and discussion, especially on the few-step composition claim.
Questions for authors
1, How is the TAB strength λ selected across layers/heads and modalities? Did you study sensitivity to λ and to timestamp noise or interval mis-specification?
2, Can you report interactive, online metrics (e.g., update latency, achievement rate for instructions submitted mid-rollout, and conflict resolution under overlapping tracks) on a streaming benchmark, or release an extension of MTBench for interactive evaluation?
3, Please clarify the provenance and compatibility of the pretrained distillation LoRA: is it trained on the same backbone and data distribution? How portable is your composition across different backbones?
4, Given that AVSpeech is largely talking-head content, how were body actions and background music annotated and validated?
5 comments
Response to Reviewer oacE. [Part II]
W3 & Q1: TAB strength
TAB uses a fixed, non-learnable λ=5, shared across layers, attention heads, and the audio and video branches. It is applied to T2A and T2V cross-attention, with each branch computing the bias using the physical timestamps of its own latents.
The added ablation compares λ=2.5, 5, and 10, as well as a Flat-smooth profile. The results show a trade-off: weaker Gaussian bias improves state coverage, whereas the default λ=5 achieves the lowest Out-of-window activity on both visual tracks and the lowest speech offset MAE among the tested configurations.
| Configuration | BODY Coverage ↑ | FACE Coverage ↑ | Speech onset MAE ↓ | Speech offset MAE ↓ | BODY Out-of-window Activity ↓ | FACE Out-of-window Activity ↓ |
|---|---|---|---|---|---|---|
| Gaussian λ=2.5 | 98.94% | 76.63% | 0.897 | 0.756 | 15.62% | 12.86% |
| Gaussian λ=10 | 84.67% | 68.63% | 0.811 | 0.828 | 9.71% | 23.61% |
| Flat-smooth λ=5 | 100.00% | 58.14% | 1.251 | 0.964 | 15.70% | 11.84% |
| Gaussian λ=5 (default) | 91.99% | 58.97% | 1.170 | 0.683 | 8.24% | 10.61% |
Q3: Distillation LoRA source and compatibility
We use the official LTX-2.3 pretrained distilled LoRA, ltx-2.3-22b-distilled-lora-384.safetensors, with its matching LTX-2.3 backbone. We do not train or fine-tune this adapter on our dataset. It provides the few-step initialization, while our subsequent self-forcing DMD refinement trains an additional residual with the pretrained adapter frozen.
Q4: Body-action and background-music annotation
AVSpeech contains many clips showing the speaker’s upper body with visible gestures and movements. As described in the paper, the actions are annotated using Gemini, and we manually inspect a subset of the annotations to check their accuracy. Our body-control evaluation is limited to upper-body gestures and movements. We neither train nor evaluate background-music control and will clarify this scope in the revised manuscript.
Response Summary for Reviewer oacE
Thank you for your constructive feedback. Our rebuttal provides the following additions and clarifications:
- Positioning relative to related work. We clarified the relationship to concurrent Gaussian temporal-routing work and related distillation approaches. Our central contribution is the integration of dynamic instruction management and temporal routing into causal joint audiovisual generation. We also distinguished pretrained LoRA composition from the subsequent self-forcing refinement.
- Direct evidence for online control and TAB. We added live-submission experiments and a controlled Queue-only versus Queue + TAB comparison. Adding TAB reduces speech onset and offset errors, improves sustained-state coverage, and reduces out-of-window visual activity.
- Experimental settings and parameter analysis. We clarified that TIE, Hard Mask, and our method use the same training settings, reported bias-strength and profile ablations, and specified the source and backbone compatibility of the pretrained distillation LoRA.
- Task and annotation scope. We explained the body-action annotation and manual inspection procedure, and explicitly limited our empirical claims to speech, facial expressions, and upper-body movements. Background-music control is neither trained nor evaluated in this work.
We sincerely appreciate your time and valuable feedback. We have carefully responded to the raised concerns in our rebuttal and provided additional clarifications. We would be happy to further discuss any remaining questions during the discussion period.
Response to Reviewer oacE. [Part I]
Thank you for highlighting this connection. Our main contribution is the first exploration of online, temporally grounded multi-track control for streaming joint audiovisual generation, where instructions have independent, potentially overlapping target intervals and may arrive dynamically during generation. TAB serves as the temporal routing component within this framework. TCR [1] is concurrent work that also uses Gaussian temporal routing. While the two methods share this mechanism, they address different conditioning settings: TCR operates on a pre-specified script, whereas TempoTrack separates instruction arrival from the target interval and manages instruction admission and eviction during blockwise causal generation. TAB operates on the currently admitted instructions and routes their influence to both audio and video positions on a shared timeline. The distinction therefore lies in integrating temporal routing with dynamic instruction management and causal joint audiovisual generation. We clarified this relationship and acknowledged TCR as concurrent work in the revision.
We thank the reviewer for raising this concern. Although LongLive-2.0 [2] also does not use ODE initialization, it does not include a subsequent self-forcing refinement stage to further address error accumulation. TempoTrack reuses a pretrained distillation LoRA and introduces an additional self-forcing DMD training stage while freezing the existing components. Table 3 evaluates the contribution of this additional refinement within our system. We will clarify this distinction and distinguish the composition strategy from the subsequent adaptation in the revision.
We thank the reviewer for raising this concern. Our original evaluation assigned each instruction group a
submit_attime preceding its target intervals. We further tested live submission under four arrival conditions . We report reception-to-admission latency and instruction achievement; since users specify target intervals, content timing is evaluated against those intervals rather than requiring immediate execution.Expired instructions have a 100% scheduler rejection rate. Compared with preloaded instructions, mid-block and late arrivals yield similar BODY and FACE adherence in the reported point estimates. These results support the model’s ability to execute newly arriving controls during generation.
We thank the reviewer for raising this concern. Our original evaluation scheduled instruction availability through
submit_at. More directly, the added live-submission experiments withhold target instructions from the generator until they arrive during generation. Their subsequent execution provides evidence of online control beyond a fully preloaded timeline. Our experiments focus on digital-human scenarios involving speech, facial expressions, and body actions. We leave music and general audiovisual control as future directions.We added a direct Queue-only experiment. Training data, backbone training procedure, and block-level scheduling are held fixed; only temporal attention bias is changed. The experimental results show that Queue + TAB substantially outperforms Queue-only.
TIE, Hard Mask, and our method are trained through LoRA using the same training data, backbone, LoRA capacity, training procedure, and budget, with only the temporal-conditioning mechanism changed. Evaluation uses the same sampling settings. We will include the configurations in the revised appendix.
References
[1] Yichen Liu et al. “The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation.” arXiv:2609.02367, 2026.
[2] Yukang Chen et al. “LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation.” 2026.