- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TurboT2VA accelerates a 19B-parameter joint video-audio model through a progressive dCM$\rightarrow$sCM$\rightarrow$sCM+DMD curriculum and an architecture-aware inference stack. The joint sCM and DMD objectives use a cross-modal JVP and per-modality normalization to train a single coupled video-audio student. The inference stack combines modality-aware sparse attention, guarded W8A8, fused operations, and padded-text compaction. On the reported 200-prompt standard-resolution evaluation, the four-step student achieves 20.1$\times$ speedup. At 1024 $\times$1792 on one NVIDIA H20, the complete stack reduces generator-only latency from 318.74s to 5.83s, a 54.67$\times$ speedup. The reported Javis score and Desync also improve relative to the teacher, while some audio-fidelity metrics decline.
Critical strengths
- The scale and measured efficiency gain are consequential. Distilling the 19B joint video-audio model from 40 to 4 steps reduces standard-resolution generator latency by 20.1$\times$. The complete high-resolution stack reaches a 54.67$\times$ generator-only speedup on one H20. The paper clearly separates the distillation gain from the additional inference-stack gain.
- The joint continuous-time consistency formulation is well motivated for video-audio generation models. The cross-modal JVP retains video-to-audio and audio-to-video dependencies in the shared Transformer, while per-modality normalization addresses the differing scales of the video and audio directions in sCM and DMD.
- The progressive dCM$\rightarrow$sCM$\rightarrow$sCM+DMD curriculum receives useful empirical support.
Critical weaknesses
- The perceptual video-audio evaluation is insufficient for the claim of maintaining audio fidelity and synchronization. The submitted materials contain no playable video-audio examples or user study. Static frames, waveforms, and mel spectrograms in Fig. 3 cannot establish motion quality, audible fidelity, or temporal correspondence. This matters because Table 1's Audio score falls from 4.902 for the teacher to 4.486 for TurboT2VA, while Table 5 reports lower Enj., Comp., and CLAP. The gains in Javis and Desync are valuable, but one 200-prompt split without uncertainty estimates cannot resolve the perceptual trade-off.
- The headline 54.67× result is explicitly generator-only and excludes VAE decoding. This is a valid component measurement, but the benefit to complete video-audio generation is not measured.
- The manuscript has no explicit limitations or failure-case discussion.
Questions for authors
- Can the authors provide matched-prompt playable examples, including failures, and a blinded user study comparing TurboT2VA with the teacher and a fast baseline?
- What are the end-to-end latencies, including VAE decoding and muxing, at both resolutions?
4 comments
2. End-to-End Latency
Thank you for pointing this out. To provide a more comprehensive evaluation, we additionally measured the end-to-end latency of TurboT2VA and the 40-step teacher at both standard (512×768) and high (1024×1792) resolutions. All tests used the same output duration, frame rate, audio configuration, and batch size.
We report latency separately for each major component: text encoding, DiT generation, video VAE decoding, audio decoding, and audio–video muxing. We measure end-to-end latency directly from receipt of the raw text request until the final audio–video file has been muxed. For TurboT2VA, this measurement also includes the per-request weight transfer and cleanup overhead incurred during inference.
End-to-End Latency Comparison
| Resolution | Model | Text Encoding (s) | DiT Generation (s) | Video VAE (s) | Audio Decoding (s) | Muxing (s) | End-to-End (s) | Speedup |
|---|---|---|---|---|---|---|---|---|
| 512×768 | Teacher (40-step) | 0.569 | 157.821 | 0.894 | 0.024 | 0.429 | 160.518 | 1× |
| 512×768 | TurboT2VA (4-step) | 0.288 | 3.727 | 0.898 | 0.023 | 0.439 | 14.322 | 11.21× |
| 1024×1792 | Teacher (40-step) | 0.574 | 1159.688 | 8.538 | 0.025 | 1.496 | 1171.614 | 1× |
| 1024×1792 | TurboT2VA (4-step) | 0.285 | 18.519 | 8.417 | 0.023 | 1.293 | 36.240 | 32.33× |
Note: Latencies in this rebuttal were measured on H20 GPUs, while the original manuscript used enhanced H20z.
TurboT2VA substantially reduces both DiT generation latency and total generation time. At 512×768, it reduces end-to-end latency from 160.518 s to 14.322 s, an 11.21× speedup. At 1024×1792, it reduces end-to-end latency from 1171.614 s to 36.240 s, a 32.33× speedup. These results show that faster denoising translates into substantial end-to-end gains for joint video–audio generation.
3. Limitations and Future Directions
We include four representative bad-case examples in the supplementary materials, each comparing the 4-step student and 40-step teacher under the same prompt. These examples cover gravity-driven motion (ID 0: a ball rolling down an inclined ramp), camera trajectory control (ID 1: a camera orbiting a stationary bicycle), rebound collision (ID 2: a ping-pong ball bouncing off the floor), and text rendering (ID 3: a handwritten note displaying "MEET AT NOON."). These cases highlight the remaining challenges in physically plausible motion generation, precise camera control, collision dynamics, and accurate text rendering.
Several directions are worth exploring to address these limitations. First, our current model adopts a bidirectional generation architecture, which is suitable for offline generation but not directly optimized for low-latency real-time streaming. Future work could explore chunk-wise streaming generation with causal or limited-lookahead attention, together with cross-chunk consistency mechanisms, to enable continuous generation while preserving temporal coherence and audiovisual synchronization.
Second, improving physical realism remains an important direction, particularly for complex object interactions involving gravity and collisions. Beyond reinforcement learning with physics-aware rewards, an interesting direction is to introduce a structured scene proxy before video generation. Such a proxy could leverage physical simulation to provide explicit guidance through object trajectories, interaction states, depth, and motion information, helping the video generator produce more physically plausible results.
Third, precise camera trajectory control remains challenging. Future work could incorporate explicit camera pose trajectories, such as 6-DoF poses or spline-based keyframes, as structured conditioning signals, together with camera-aware representations and geometric consistency constraints, to achieve more accurate and controllable camera movements.
Finally, accurate and temporally consistent text rendering remains difficult. This could be addressed through text-aware supervision, OCR-based recognition and consistency objectives, or decoupled text rendering modules that explicitly generate text and integrate it into the visual scene while preserving geometric and temporal consistency.
4. Uncertainty Estimation
Thank you for suggesting that we report uncertainty estimates. We conducted a supplementary evaluation of five representative models: JavisDiT, OVI, DaVinci-MagiHuman, the 40-step LTX-2 teacher, and our four-step student. Each model was evaluated on 200 additional prompts not used in the original manuscript. We generated one sample per prompt and scored all outputs against the original captions.
For each model and metric, we performed 10,000 bootstrap resamples with replacement at the prompt level and used the 2.5th and 97.5th percentiles of the bootstrap means as the bounds of the 95% confidence interval. The means and confidence intervals below are from this supplementary evaluation and were not calculated retrospectively for the results in the original Table 1.
Supplementary Quality Evaluation with 95% Confidence Intervals
| Metric | JavisDiT | OVI | DaVinci-MagiHuman | Teacher, 40 steps | Ours, 4 steps |
|---|---|---|---|---|---|
| Perceptual Quality | |||||
| Visual ↑ | 1.253 [1.139, 1.371] | 2.300 [2.152, 2.451] | 2.528 [2.338, 2.728] | 2.102 [1.946, 2.264] | 2.302 [2.134, 2.477] |
| Motion ↑ | 0.508 [0.415, 0.602] | 0.386 [0.291, 0.487] | 0.422 [0.286, 0.566] | 0.684 [0.535, 0.841] | 0.629 [0.499, 0.768] |
| Audio ↑ | 4.232 [4.086, 4.388] | 4.079 [3.983, 4.175] | 4.335 [4.189, 4.486] | 4.552 [4.428, 4.676] | 4.653 [4.520, 4.786] |
| Text Semantic Consistency | |||||
| IB-TV ↑ | 0.321 [0.313, 0.329] | 0.346 [0.339, 0.354] | 0.335 [0.324, 0.346] | 0.349 [0.342, 0.355] | 0.345 [0.337, 0.352] |
| IB-TA ↑ | 0.174 [0.161, 0.188] | 0.153 [0.139, 0.168] | 0.166 [0.148, 0.185] | 0.186 [0.172, 0.200] | 0.185 [0.172, 0.199] |
| CLIP ↑ | 0.289 [0.284, 0.295] | 0.309 [0.303, 0.314] | 0.306 [0.299, 0.312] | 0.305 [0.300, 0.310] | 0.306 [0.301, 0.311] |
| CLAP ↑ | 0.367 [0.347, 0.387] | 0.400 [0.378, 0.422] | 0.380 [0.351, 0.408] | 0.337 [0.312, 0.361] | 0.339 [0.315, 0.362] |
| V-A Semantic Consistency | |||||
| IB-AV ↑ | 0.177 [0.164, 0.190] | 0.199 [0.183, 0.215] | 0.208 [0.187, 0.228] | 0.224 [0.207, 0.241] | 0.232 [0.217, 0.248] |
| CAVP ↑ | 0.804 [0.802, 0.806] | 0.797 [0.795, 0.799] | 0.800 [0.797, 0.802] | 0.798 [0.796, 0.800] | 0.798 [0.796, 0.800] |
| V-A Alignment | |||||
| AVH ↑ | 0.157 [0.144, 0.169] | 0.201 [0.185, 0.217] | 0.207 [0.187, 0.228] | 0.217 [0.200, 0.233] | 0.225 [0.210, 0.239] |
| Javis ↑ | 0.130 [0.118, 0.141] | 0.168 [0.154, 0.182] | 0.186 [0.167, 0.204] | 0.183 [0.169, 0.198] | 0.194 [0.180, 0.207] |
| Desync ↓ | 0.872 [0.788, 0.957] | 0.551 [0.462, 0.644] | 0.538 [0.412, 0.664] | 0.508 [0.427, 0.594] | 0.374 [0.298, 0.457] |
In this supplementary evaluation, our four-step student performs comparably overall to the 40-step teacher and outperforms all three external baselines on 7 of the 12 metrics by point estimate. Notably, it achieves the best point estimates among all five models on all three audiovisual alignment metrics (AVH, Javis, and Desync). These findings support our main conclusion: TurboT2VA reduces the number of sampling steps from 40 to 4 while retaining overall generation quality close to that of the teacher and remaining competitive with other joint video–audio generation models.
1. User Study and Supplementary Video Examples
1.1 Study Protocol
Thank you for this suggestion. We conducted a blinded, matched-prompt pairwise user study comparing our TurboT2VA output with (i) the 40-step teacher and (ii) the official 8-step distilled baseline. For each comparison, the two videos use the same prompt; model identities are hidden from participants, and A/B placement is randomized and approximately balanced. Participants evaluated visual and motion quality, audio quality, audiovisual synchronization, prompt adherence, and overall preference. Each item allowed A, B, tie, or unable to judge.
1.2 Study Scale and Response Counts
We use approximately 200 prompts as the evaluation set. Each participant completed 25 pairwise comparisons, with no repeated prompts within each participant's round. A total of 23 participants completed one round each, resulting in 575 completed pairwise comparisons: 289 against the 40-step teacher and 286 against the official 8-step distilled baseline. Each comparison covers five evaluation criteria, yielding 2,875 individual criterion-level judgments. TurboT2VA appeared on side A in 288 comparisons and on side B in 287, ensuring a nearly balanced presentation. The table reports all four response categories (win, loss, tie, and unable to judge). The win rate is calculated only among decisive A/B choices, with ties and unable-to-judge responses reported separately.
1.3 Comparison Results
Comparison with the 40-step Teacher (289 comparisons)
Comparison with the Official 8-step Distilled Baseline (286 comparisons)
Note: Win rate is calculated as Ours wins / (Ours wins + Comparator wins), excluding ties and unable-to-judge responses.
1.4 Overall Results
The results show that TurboT2VA is slightly preferred over the 40-step teacher in overall perceptual quality, with a 60.7% win rate among decisive choices, despite using only four sampling steps. This suggests that our distillation approach not only preserves the teacher's generation quality but can also yield perceptually favorable outputs in some aspects. Compared with the official 8-step distilled baseline, TurboT2VA achieves a stronger overall preference (63.9%), with a particularly notable advantage in audio quality (73.6%). The improvement in visual and motion quality is more modest (55.8%), while audiovisual synchronization receives a considerable number of tied judgments against both baselines, suggesting broadly comparable perceived synchronization quality. Overall, these results provide encouraging perceptual evidence for the effectiveness of TurboT2VA in improving the quality-efficiency trade-off for joint video-audio generation.
1.5 Objective Audio Metrics and Human Perception
Although TurboT2VA achieves slightly lower scores than the teacher on some objective audio metrics as you mentioned, including the Audio score (4.486 vs. 4.902), Enj., Comp., and CLAP, the user study provides a complementary perspective through human perceptual evaluation. Notably, TurboT2VA is preferred in audio quality in 63.1% of decisive comparisons against the teacher and 73.6% against the official 8-step baseline. These results suggest that the differences captured by automatic audio metrics do not necessarily translate directly into lower human preference. Therefore, objective metrics and human evaluations provide complementary perspectives for assessing audio quality in joint video-audio generation.
1.6 Supplementary Video Examples
We have updated the supplementary materials to include matched-prompt video-audio comparisons, with 8 samples each from the student and teacher models at both low and high resolutions (32 videos in total). These examples allow you to directly inspect the generated visual content, audio quality, and audiovisual synchronization.