acceptodds
Under review as a conference paper at ICLR 2027

TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillation

Abstract

Joint text-to-video-audio generation produces synchronized visual and acoustic content, but the long sampling trajectories and heterogeneous multimodal computation of large models make inference prohibitively expensive. We present TurboT2VA, a distillation and inference framework for accelerating a 19B-parameter joint video-audio model. Compared with discrete-time consistency, continuous-time consistency avoids finite-step discretization errors in the training target and provides more faithful supervision along the teacher trajectory, making it particularly attractive for few-step distillation. However, scaling continuous-time consistency to a 19B joint T2VA model is challenged by modality-imbalanced optimization, numerically fragile trajectory-tangent estimation, and the quality–diversity trade-off. TurboT2VA addresses these issues with per-modality normalization and a progressive dCMsCMsCM+DMD curriculum, where dCM establishes a stable initialization, sCM more faithfully learns the continuous teacher trajectory while preserving diversity, and DMD improves perceptual quality. On LTX-2, four-step distillation reduces generator latency from 50.52s to 2.51s at the standard evaluation resolution of 512768, achieving a 20.1 speedup while maintaining strong visual quality, audio fidelity, diversity, and video-audio synchronization. We further develop an architecture-aware inference stack that combines guarded W8A8 and fused operators, padded-text compaction, and modality-aware sparse attention while preserving dense cross-modal and text-conditioning paths without retraining. Under the high-resolution deployment setting at 10241792, the complete stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67 generator-only speedup over the original teacher.

Primary area:
generative models
Keywords:
Text-to-video-audio generation, consistency distillation, distribution matching, multimodal generation, inference acceleration, quantization.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3U4xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TurboT2VA accelerates a 19B-parameter joint video-audio model through a progressive dCM$\rightarrow$sCM$\rightarrow$sCM+DMD curriculum and an architecture-aware inference stack. The joint sCM and DMD objectives use a cross-modal JVP and per-modality normalization to train a single coupled video-audio student. The inference stack combines modality-aware sparse attention, guarded W8A8, fused operations, and padded-text compaction. On the reported 200-prompt standard-resolution evaluation, the four-step student achieves 20.1$\times$ speedup. At 1024 $\times$1792 on one NVIDIA H20, the complete stack reduces generator-only latency from 318.74s to 5.83s, a 54.67$\times$ speedup. The reported Javis score and Desync also improve relative to the teacher, while some audio-fidelity metrics decline.

Critical strengths
  1. The scale and measured efficiency gain are consequential. Distilling the 19B joint video-audio model from 40 to 4 steps reduces standard-resolution generator latency by 20.1$\times$. The complete high-resolution stack reaches a 54.67$\times$ generator-only speedup on one H20. The paper clearly separates the distillation gain from the additional inference-stack gain.
  2. The joint continuous-time consistency formulation is well motivated for video-audio generation models. The cross-modal JVP retains video-to-audio and audio-to-video dependencies in the shared Transformer, while per-modality normalization addresses the differing scales of the video and audio directions in sCM and DMD.
  3. The progressive dCM$\rightarrow$sCM$\rightarrow$sCM+DMD curriculum receives useful empirical support.
Critical weaknesses
  1. The perceptual video-audio evaluation is insufficient for the claim of maintaining audio fidelity and synchronization. The submitted materials contain no playable video-audio examples or user study. Static frames, waveforms, and mel spectrograms in Fig. 3 cannot establish motion quality, audible fidelity, or temporal correspondence. This matters because Table 1's Audio score falls from 4.902 for the teacher to 4.486 for TurboT2VA, while Table 5 reports lower Enj., Comp., and CLAP. The gains in Javis and Desync are valuable, but one 200-prompt split without uncertainty estimates cannot resolve the perceptual trade-off.
  2. The headline 54.67× result is explicitly generator-only and excludes VAE decoding. This is a valid component measurement, but the benefit to complete video-audio generation is not measured.
  3. The manuscript has no explicit limitations or failure-case discussion.
Questions for authors
  1. Can the authors provide matched-prompt playable examples, including failures, and a blinded user study comparing TurboT2VA with the teacher and a fast baseline?
  2. What are the end-to-end latencies, including VAE decoding and muxing, at both resolutions?
4 comments
AuthorsOpenReview ↗

1. User Study and Supplementary Video Examples

1.1 Study Protocol

Thank you for this suggestion. We conducted a blinded, matched-prompt pairwise user study comparing our TurboT2VA output with (i) the 40-step teacher and (ii) the official 8-step distilled baseline. For each comparison, the two videos use the same prompt; model identities are hidden from participants, and A/B placement is randomized and approximately balanced. Participants evaluated visual and motion quality, audio quality, audiovisual synchronization, prompt adherence, and overall preference. Each item allowed A, B, tie, or unable to judge.

1.2 Study Scale and Response Counts

We use approximately 200 prompts as the evaluation set. Each participant completed 25 pairwise comparisons, with no repeated prompts within each participant's round. A total of 23 participants completed one round each, resulting in 575 completed pairwise comparisons: 289 against the 40-step teacher and 286 against the official 8-step distilled baseline. Each comparison covers five evaluation criteria, yielding 2,875 individual criterion-level judgments. TurboT2VA appeared on side A in 288 comparisons and on side B in 287, ensuring a nearly balanced presentation. The table reports all four response categories (win, loss, tie, and unable to judge). The win rate is calculated only among decisive A/B choices, with ties and unable-to-judge responses reported separately.

1.3 Comparison Results

Comparison with the 40-step Teacher (289 comparisons)

CriterionOurs winsTeacher winsTieUnable to judgeWin rate
Visual and motion quality1298965659.2%
Audio quality1186995763.1%
Audiovisual synchronization88471361865.2%
Prompt adherence11971841562.6%
Overall preference1459445560.7%

Comparison with the Official 8-step Distilled Baseline (286 comparisons)

CriterionOurs winsBaseline winsTieUnable to judgeWin rate
Visual and motion quality13010349455.8%
Audio quality1595767373.6%
Audiovisual synchronization101501241166.9%
Prompt adherence1227580961.9%
Overall preference1528643563.9%

Note: Win rate is calculated as Ours wins / (Ours wins + Comparator wins), excluding ties and unable-to-judge responses.

1.4 Overall Results

The results show that TurboT2VA is slightly preferred over the 40-step teacher in overall perceptual quality, with a 60.7% win rate among decisive choices, despite using only four sampling steps. This suggests that our distillation approach not only preserves the teacher's generation quality but can also yield perceptually favorable outputs in some aspects. Compared with the official 8-step distilled baseline, TurboT2VA achieves a stronger overall preference (63.9%), with a particularly notable advantage in audio quality (73.6%). The improvement in visual and motion quality is more modest (55.8%), while audiovisual synchronization receives a considerable number of tied judgments against both baselines, suggesting broadly comparable perceived synchronization quality. Overall, these results provide encouraging perceptual evidence for the effectiveness of TurboT2VA in improving the quality-efficiency trade-off for joint video-audio generation.

1.5 Objective Audio Metrics and Human Perception

Although TurboT2VA achieves slightly lower scores than the teacher on some objective audio metrics as you mentioned, including the Audio score (4.486 vs. 4.902), Enj., Comp., and CLAP, the user study provides a complementary perspective through human perceptual evaluation. Notably, TurboT2VA is preferred in audio quality in 63.1% of decisive comparisons against the teacher and 73.6% against the official 8-step baseline. These results suggest that the differences captured by automatic audio metrics do not necessarily translate directly into lower human preference. Therefore, objective metrics and human evaluations provide complementary perspectives for assessing audio quality in joint video-audio generation.

1.6 Supplementary Video Examples

We have updated the supplementary materials to include matched-prompt video-audio comparisons, with 8 samples each from the student and teacher models at both low and high resolutions (32 videos in total). These examples allow you to directly inspect the generated visual content, audio quality, and audiovisual synchronization.

AuthorsOpenReview ↗

2. End-to-End Latency

Thank you for pointing this out. To provide a more comprehensive evaluation, we additionally measured the end-to-end latency of TurboT2VA and the 40-step teacher at both standard (512×768) and high (1024×1792) resolutions. All tests used the same output duration, frame rate, audio configuration, and batch size.

We report latency separately for each major component: text encoding, DiT generation, video VAE decoding, audio decoding, and audio–video muxing. We measure end-to-end latency directly from receipt of the raw text request until the final audio–video file has been muxed. For TurboT2VA, this measurement also includes the per-request weight transfer and cleanup overhead incurred during inference.

End-to-End Latency Comparison

ResolutionModelText Encoding (s)DiT Generation (s)Video VAE (s)Audio Decoding (s)Muxing (s)End-to-End (s)Speedup
512×768Teacher (40-step)0.569157.8210.8940.0240.429160.5181×
512×768TurboT2VA (4-step)0.2883.7270.8980.0230.43914.32211.21×
1024×1792Teacher (40-step)0.5741159.6888.5380.0251.4961171.6141×
1024×1792TurboT2VA (4-step)0.28518.5198.4170.0231.29336.24032.33×

Note: Latencies in this rebuttal were measured on H20 GPUs, while the original manuscript used enhanced H20z.

TurboT2VA substantially reduces both DiT generation latency and total generation time. At 512×768, it reduces end-to-end latency from 160.518 s to 14.322 s, an 11.21× speedup. At 1024×1792, it reduces end-to-end latency from 1171.614 s to 36.240 s, a 32.33× speedup. These results show that faster denoising translates into substantial end-to-end gains for joint video–audio generation.

AuthorsOpenReview ↗

3. Limitations and Future Directions

We include four representative bad-case examples in the supplementary materials, each comparing the 4-step student and 40-step teacher under the same prompt. These examples cover gravity-driven motion (ID 0: a ball rolling down an inclined ramp), camera trajectory control (ID 1: a camera orbiting a stationary bicycle), rebound collision (ID 2: a ping-pong ball bouncing off the floor), and text rendering (ID 3: a handwritten note displaying "MEET AT NOON."). These cases highlight the remaining challenges in physically plausible motion generation, precise camera control, collision dynamics, and accurate text rendering.

Several directions are worth exploring to address these limitations. First, our current model adopts a bidirectional generation architecture, which is suitable for offline generation but not directly optimized for low-latency real-time streaming. Future work could explore chunk-wise streaming generation with causal or limited-lookahead attention, together with cross-chunk consistency mechanisms, to enable continuous generation while preserving temporal coherence and audiovisual synchronization.

Second, improving physical realism remains an important direction, particularly for complex object interactions involving gravity and collisions. Beyond reinforcement learning with physics-aware rewards, an interesting direction is to introduce a structured scene proxy before video generation. Such a proxy could leverage physical simulation to provide explicit guidance through object trajectories, interaction states, depth, and motion information, helping the video generator produce more physically plausible results.

Third, precise camera trajectory control remains challenging. Future work could incorporate explicit camera pose trajectories, such as 6-DoF poses or spline-based keyframes, as structured conditioning signals, together with camera-aware representations and geometric consistency constraints, to achieve more accurate and controllable camera movements.

Finally, accurate and temporally consistent text rendering remains difficult. This could be addressed through text-aware supervision, OCR-based recognition and consistency objectives, or decoupled text rendering modules that explicitly generate text and integrate it into the visual scene while preserving geometric and temporal consistency.

AuthorsOpenReview ↗

4. Uncertainty Estimation

Thank you for suggesting that we report uncertainty estimates. We conducted a supplementary evaluation of five representative models: JavisDiT, OVI, DaVinci-MagiHuman, the 40-step LTX-2 teacher, and our four-step student. Each model was evaluated on 200 additional prompts not used in the original manuscript. We generated one sample per prompt and scored all outputs against the original captions.

For each model and metric, we performed 10,000 bootstrap resamples with replacement at the prompt level and used the 2.5th and 97.5th percentiles of the bootstrap means as the bounds of the 95% confidence interval. The means and confidence intervals below are from this supplementary evaluation and were not calculated retrospectively for the results in the original Table 1.

Supplementary Quality Evaluation with 95% Confidence Intervals

MetricJavisDiTOVIDaVinci-MagiHumanTeacher, 40 stepsOurs, 4 steps
Perceptual Quality
Visual ↑1.253 [1.139, 1.371]2.300 [2.152, 2.451]2.528 [2.338, 2.728]2.102 [1.946, 2.264]2.302 [2.134, 2.477]
Motion ↑0.508 [0.415, 0.602]0.386 [0.291, 0.487]0.422 [0.286, 0.566]0.684 [0.535, 0.841]0.629 [0.499, 0.768]
Audio ↑4.232 [4.086, 4.388]4.079 [3.983, 4.175]4.335 [4.189, 4.486]4.552 [4.428, 4.676]4.653 [4.520, 4.786]
Text Semantic Consistency
IB-TV ↑0.321 [0.313, 0.329]0.346 [0.339, 0.354]0.335 [0.324, 0.346]0.349 [0.342, 0.355]0.345 [0.337, 0.352]
IB-TA ↑0.174 [0.161, 0.188]0.153 [0.139, 0.168]0.166 [0.148, 0.185]0.186 [0.172, 0.200]0.185 [0.172, 0.199]
CLIP ↑0.289 [0.284, 0.295]0.309 [0.303, 0.314]0.306 [0.299, 0.312]0.305 [0.300, 0.310]0.306 [0.301, 0.311]
CLAP ↑0.367 [0.347, 0.387]0.400 [0.378, 0.422]0.380 [0.351, 0.408]0.337 [0.312, 0.361]0.339 [0.315, 0.362]
V-A Semantic Consistency
IB-AV ↑0.177 [0.164, 0.190]0.199 [0.183, 0.215]0.208 [0.187, 0.228]0.224 [0.207, 0.241]0.232 [0.217, 0.248]
CAVP ↑0.804 [0.802, 0.806]0.797 [0.795, 0.799]0.800 [0.797, 0.802]0.798 [0.796, 0.800]0.798 [0.796, 0.800]
V-A Alignment
AVH ↑0.157 [0.144, 0.169]0.201 [0.185, 0.217]0.207 [0.187, 0.228]0.217 [0.200, 0.233]0.225 [0.210, 0.239]
Javis ↑0.130 [0.118, 0.141]0.168 [0.154, 0.182]0.186 [0.167, 0.204]0.183 [0.169, 0.198]0.194 [0.180, 0.207]
Desync ↓0.872 [0.788, 0.957]0.551 [0.462, 0.644]0.538 [0.412, 0.664]0.508 [0.427, 0.594]0.374 [0.298, 0.457]

In this supplementary evaluation, our four-step student performs comparably overall to the 40-step teacher and outperforms all three external baselines on 7 of the 12 metrics by point estimate. Notably, it achieves the best point estimates among all five models on all three audiovisual alignment metrics (AVH, Javis, and Desync). These findings support our main conclusion: TurboT2VA reduces the number of sampling steps from 40 to 4 while retaining overall generation quality close to that of the teacher and remaining competitive with other joint video–audio generation models.

Reviewer_31VgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a training curriculum for distilling a large (19B) joint text-to-audio-video model. The distillation consists of three steps: dCM warmup, sCM refinement, and joint sCM+DMD optimization. The authors claim that this setup maintains a good stability, quality, and diversity tradeoff. The results show that the distilled model is comparable to the base model in terms of quality and multimodal alignment, while being much faster to run. The authors also show that the proposed staged training objective performs better than training with a single objective directly throughout.

Critical strengths

This paper presents a clear, practical, and useful approach for distilling a large joint text-to-AV model. This is useful because direct distillation with existing approaches often proves to be suboptimal in practice in multimodal generation. The training pipeline is also simple and easy to reproduce, so overall this is a neat technique. The results that the authors showed are promising, compared to the baseline teacher model.

Critical weaknesses

One of the issues of distillation, and one of the motivations for doing sCM+DMD joint training, is the diversity of the generations. However, the comparisons on diversity metrics remain very limited in this paper, except in Table 2. For a complete picture, this paper should include diversity metrics whenever quality metrics are being compared, such as in Table 1 and Figure 5. Otherwise, readers would not be able to make a complete assessment.

Secondly, the ablations on the three-stage training pipeline are insufficient. The authors ablated the proposed pipeline versus 1) direct training with any one of the objectives and 2) continuing with only DMD/sCM from the sCM baseline. None of these supports the use of a sCM refinement stage, and only weakly supports the use of dCM initialization. Two experiments that could be useful are dCM->sCM+DMD, and sCM->sCM+DMD.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

1. Diversity Evaluation across Representative Models

Thank you for pointing out that, outside Table 2, comparisons on diversity metrics are limited. We agree that, for a complete picture, diversity metrics should be reported alongside quality metrics. To address your concern about Table 1, we add video diversity, audio diversity, and their arithmetic mean (V-A Avg.) for five representative models: JavisDiT, OVI, DaVinci-MagiHuman, the 40-step LTX-2 teacher, and our four-step student. The inference times and quality scores are unchanged from the original Table 1. We address your point about Figure 5 in the training ablation response below, where we report diversity alongside quality for the different training configurations.

Following the diversity evaluation protocol in the manuscript, we generate eight samples per original prompt using different random seeds. For each prompt, we compute the mean pairwise cosine distance between ImageBind embeddings separately for video and audio, then average the scores across prompts. We evaluate our four-step student on 200 original prompts and each of the other four models on 50 original prompts. Diversity is measured on a separate batch of generated samples from the one used for quality scoring.

Extended Table 1

MetricJavisDiTOVIDaVinci-MagiHumanTeacher (40 steps)Ours (4 steps)
Time (s) ↓49.7376.796.7050.522.51
Perceptual Quality
Visual ↑1.4882.1450.8541.9682.389
Motion ↑0.5910.2940.3820.7830.849
Audio ↑4.4055.0245.2454.9024.486
Text Consistency
IB-TV ↑0.2500.2440.2270.2680.218
IB-TA ↑0.1450.1320.1240.1390.145
CLIP ↑0.2970.3070.2840.2910.292
CLAP ↑0.4130.4070.3620.3400.347
V-A Consistency
IB-AV ↑0.2120.2090.2010.1990.229
CAVP ↑0.7980.7950.8000.7960.803
V-A Alignment
AVH ↑0.1830.2060.1810.1950.224
Javis ↑0.1520.1750.1450.1650.196
Desync ↓0.8480.6380.7300.6170.388
Diversity
Video Diversity ↑0.2130.2690.2270.2570.216
Audio Diversity ↑0.3790.4820.4910.4340.448
V-A Avg. ↑0.2960.3760.3590.3460.332

TurboT2VA reduces the number of sampling steps from 40 to 4 while maintaining overall generation quality and average audiovisual diversity close to those of the teacher and remaining competitive with other joint video–audio generation models.

AuthorsOpenReview ↗

2. Quality–Diversity Trade-off across Training Strategies

As you pointed out, reporting quality alone in Figure 5 does not give a complete picture of the trade-off across training configurations. We therefore supplement Figure 5 with diversity results alongside quality scores for the configurations with usable checkpoints. We also evaluate both ablations you suggested: dCM → sCM+DMD (Warmup-DCM) skips sCM refinement, while sCM → sCM+DMD (Warmup-SCM) starts sCM training without dCM initialization.

Failed Warmup-SCM: Training without dCM Initialization

On visual inspection, samples from Warmup-SCM exhibited noticeable visual quality degradation around step 1,000. To quantify this deterioration, we report the VBench imaging_quality score at 200-step intervals. As shown below, the score drops sharply from 0.6151 at step 800 to 0.1536 at step 1,000, and remains substantially lower in subsequent checkpoints. The run ultimately did not yield a usable final checkpoint.

For qualitative inspection, we also provide an anonymized visualization (view visualization), showing three frames sampled from the generated video at every 200 training steps. This allows direct visual comparison of generation quality throughout the training process.

Training step2004006008001,0001,200...
Warmup-SCM (VBench imaging_quality) ↑0.71680.59270.64210.61510.15360.1090...

These results indicate that starting sCM training directly from the LTX-2 base model without dCM initialization was unstable.

Quality–Diversity Trade-off across Training Configurations

To complement the Javis curves in Figure 5, we report diversity alongside quality for the five configurations with usable checkpoints: the full curriculum (Ours), direct sCM+DMD training from initialization (Direct), continuation from the same dCM → sCM checkpoint with either sCM alone (Final-SCM) or DMD alone (Final-DMD), and joint training that skips sCM refinement (Warmup-DCM). We list Warmup-SCM as Failed in the table to document the outcome of this ablation, but do not include it in the scatter plot (view scatter plot), which shows Javis Score on the vertical axis and V-A Avg. on the horizontal axis.

Training ConfigurationJavis Score ↑Video Diversity ↑Audio Diversity ↑V-A Avg. ↑
Ours0.19350.21600.44760.3318
Direct0.16340.13310.41050.2718
Final-SCM0.13900.31640.54980.4331
Final-DMD0.18270.24890.37090.3099
Warmup-DCM0.16870.20870.52370.3662
Warmup-SCMFailed———

Reporting diversity alongside quality reveals differences that the quality curves alone do not show. Among the five configurations with usable checkpoints, the full three-stage pipeline achieves the highest Javis Score (0.1935) and outperforms both Direct and Final-DMD on Javis Score and V-A Avg. Compared with Warmup-DCM, which skips sCM refinement, the full pipeline achieves a higher Javis Score (0.1935 vs. 0.1687) but a lower V-A Avg. (0.3318 vs. 0.3662). Final-SCM achieves the highest V-A Avg. (0.4331) but has a lower Javis Score (0.1390) than the full pipeline. These comparisons show how the full pipeline balances generation quality and diversity rather than maximizing either metric alone.

Thank you for the response. My concerns have been addressed, and I now tend to recommend acceptance. I highly recommend that the authors include these new results in the paper.

Thank you for your valuable review. We are glad that our response addressed your concerns, and we will incorporate the new results into the revised manuscript.

Reviewer_JcVROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TurboT2VA, a few-step T2VA model based on the pretrained LTX-2. The core idea is to port rCM into joint T2VA flow-matching models by naturally extending it to a multimodal formulation with modality-specific normalization. To improve the quality–diversity trade-off, the paper uses a three-stage curriculum: dCM warmup, sCM refinement, and joint sCM+DMD optimization. Moreover, the paper introduces inference acceleration techniques (e.g., sparse attention, guarded W8A8, kernel fusion) for further speedup. Experiments show that the proposed method achieves up to a 54.67× speedup over the 40-step teacher while maintaining comparable quantitative results.

Overall, the reviewer acknowledges the substantial speedups and their practical value. However, the methodological contribution beyond rCM appears limited, and the current experiments do not sufficiently isolate the benefits of the proposed training stages. Therefore, the reviewer currently leans toward rejection.

Critical strengths

This paper offers practical techniques for building a few-step generator for LTX-2. The reported speedups are substantial and address the high computational cost of joint T2VA generation. The resulting training and inference recipe could be valuable to the community.

Critical weaknesses
  • The main concern is novelty. Eq. (4) follows directly from applying the chain rule to the joint video-audio state, and the modality-specific normalizations appear to be straightforward adaptations of the normalization mechanisms used in sCM and DMD to balance optimization signals across modalities. The proposed curriculum is conceptually related to prior work. CM, iCT [1], and ECT [2] progressively reduce the timestep separation, and sCM introduces a tangent-coefficient warmup. Although they do not propose the exact dCM→sCM→sCM+DMD schedule, progressively strengthening consistency training has already been explored. The paper should therefore clarify what additional methodological contribution is provided by the particular stage sequence. Regarding inference acceleration, LightX2V [3] and the official LTX-2 implementation [4] already support reduced-precision execution for LTX-2, and LightX2V also includes fused operators. The incremental benefits of the proposed inference stack over these implementations remain unclear.
  • Experiments are limited. The paper claims that the proposed method is a unified acceleration framework for large-scale joint T2VA generation, but it is tested only with LTX-2. As in rCM, the authors should evaluate the proposed method's effectiveness across multiple base models (e.g., JavisDiT, OVI, and DaVinci-MagiHuman) to demonstrate generalizability. The paper also uses fixed transition points between the three training stages but does not report sensitivity to these choices or independently isolate the contribution of each warmup stage. Although the paper provides component-wise latency ablations, it does not directly compare its inference stack with existing optimized LTX-2 implementations under matched conditions. It therefore remains unclear how much additional benefit the proposed stack provides over readily available alternatives.

[1] Improved Techniques for Training Consistency Models, ICLR 2024
[2] Consistency Models made Easy, ICLR 2025
[3] https://github.com/ModelTC/LightX2V
[4] https://github.com/Lightricks/LTX-2

Questions for authors
  • Can the authors clarify the positioning of the proposed method against existing work in both CM training and inference acceleration?
  • Have the authors tested the proposed method with other backbones and confirmed its generalizability?
  • What additional benefits does the proposed inference stack provide over existing optimized LTX-2 implementations?
4 comments
AuthorsOpenReview ↗

1. Overall Novelty of TurboT2VA and Overview of Additional Experiments

Thank you for raising this important question about the methodological novelty of TurboT2VA. Our primary contribution is not a new consistency objective or a standalone optimization operator, but an effective and scalable training paradigm for few-step distillation of large-scale joint text-to-video-audio (T2VA) foundation models. Scaling consistency training remains challenging. As you noted, CM, iCT, and ECT explore progressively narrowing the timestep separation used for consistency supervision, while sCM introduces a warmup for the tangent coefficient. These works informed our design, and we have updated the discussion and added the relevant citations in Section 2.2 of the revised manuscript (highlighted in blue). The reported experiments focus primarily on image generation and use smaller models: approximately 0.30B parameters for iCT (0.59B for iCT-deep), up to approximately 1.1B for ECT, and up to 1.5B for sCM. In contrast, we study few-step distillation on two large-scale joint video–audio backbones, LTX-2 (19B) and OVI (15B), where the substantially larger model scales and the coupling between two heterogeneous modalities make stable optimization more challenging. To address these challenges, we explicitly change the training objective across stages to balance generation quality and sample diversity in the student. This is the main distinction from the prior distillation methods discussed above, and the additional ablations in Reply 2 further support this staged design. The additional experiments on OVI in Reply 3 provide evidence that the distillation paradigm generalizes to a different joint video–audio backbone. For inference acceleration, rather than applying existing techniques unchanged, we tailor them to the distinct computation paths of the joint video–audio model to improve inference speed while maintaining competitive generation quality. Finally, Reply 4 presents comparisons with existing optimized LTX-2 implementations.

AuthorsOpenReview ↗

2. Ablation of the Progressive Training Curriculum

To address your concern that the original ablations did not independently assess the contribution of each warmup stage, we compare the full dCM → sCM → sCM+DMD curriculum with two variants. Both retain the same final joint objective but omit one of the first two stages: dCM → sCM+DMD (Warmup-DCM) skips sCM refinement, whereas sCM → sCM+DMD (Warmup-SCM) starts sCM training without dCM initialization.

Failed Warmup-SCM: Training without dCM Initialization

On visual inspection, samples from Warmup-SCM exhibited noticeable visual quality degradation around step 1,000. To quantify this deterioration, we report the VBench imaging_quality score at 200-step intervals. As shown below, the score drops sharply from 0.6151 at step 800 to 0.1536 at step 1,000, and remains substantially lower in subsequent checkpoints. The run ultimately did not yield a usable final checkpoint.

For qualitative inspection, we also provide an anonymized visualization (https://anonymous.4open.science/r/rebuttal-figure-BF35/fig-r2.png), showing three frames sampled from the generated video at every 200 training steps. This allows direct visual comparison of generation quality throughout the training process.

Training step2004006008001,0001,200...
Warmup-SCM (VBench imaging_quality) ↑0.71680.59270.64210.61510.15360.1090...

These results indicate that starting sCM training directly from the LTX-2 base model without dCM initialization was unstable.

Quality–Diversity Trade-off across Training Configurations

To complement the Javis curves in Figure 5, we report diversity alongside quality for the five configurations with usable checkpoints: the full curriculum (Ours), direct sCM+DMD training from initialization (Direct), continuation from the same dCM → sCM checkpoint with either sCM alone (Final-SCM) or DMD alone (Final-DMD), and joint training that skips sCM refinement (Warmup-DCM). We list Warmup-SCM as Failed in the table to document the outcome of this ablation, but do not include it in the scatter plot (https://anonymous.4open.science/r/rebuttal-figure-BF35/fig-r1.png), which shows Javis Score on the vertical axis and V-A Avg. on the horizontal axis.

Training ConfigurationJavis Score ↑Video Diversity ↑Audio Diversity ↑V-A Avg. ↑
Ours0.19350.21600.44760.3318
Direct0.16340.13310.41050.2718
Final-SCM0.13900.31640.54980.4331
Final-DMD0.18270.24890.37090.3099
Warmup-DCM0.16870.20870.52370.3662
Warmup-SCMFailed———

Reporting diversity alongside quality reveals differences that the quality curves alone do not show. Among the five configurations with usable checkpoints, the full three-stage pipeline achieves the highest Javis Score (0.1935) and outperforms both Direct and Final-DMD on Javis Score and V-A Avg. Compared with Warmup-DCM, which skips sCM refinement, the full pipeline achieves a higher Javis Score (0.1935 vs. 0.1687) but a lower V-A Avg. (0.3318 vs. 0.3662). Final-SCM achieves the highest V-A Avg. (0.4331) but has a lower Javis Score (0.1390) than the full pipeline. These comparisons show how the full pipeline balances generation quality and diversity rather than maximizing either metric alone.

AuthorsOpenReview ↗

3. Generalization to Another T2VA Backbone

Experimental Setup

To further investigate the generalizability of our approach beyond LTX-2, we extend our experiments to OVI, another large-scale joint text-to-audio-video backbone. We evaluate four configurations under the same OVI backbone: the original OVI teacher, DMD-only distillation, dCM warmup followed by sCM-only refinement (sCM-only), and our complete staged training paradigm (dCM warmup → sCM refinement → joint sCM+DMD optimization).

We report the same evaluation metrics as in the original Table 1, covering inference time, perceptual quality, semantic consistency, and audiovisual alignment. We further include video diversity, audio diversity, and their average (V-A Avg.) to provide a comprehensive assessment of the quality-diversity trade-off.

Extended Table

MetricOVI TeacherDMD-onlysCM-onlyOurs
Inference Time (s) ↓76.7913.9113.9113.91
Perceptual Quality
Visual ↑2.88712.57462.30753.5343
Motion ↑0.60670.80150.86081.1940
Audio ↑4.26395.40793.98844.6710
Text Semantic Consistency
IB-TV ↑0.33920.34040.34000.3520
IB-TA ↑0.14320.12100.08990.1208
CLIP ↑0.30970.29760.30190.3167
CLAP ↑0.42050.33750.29520.3810
V-A Semantic Consistency
IB-AV ↑0.21720.19330.12490.2112
CAVP ↑0.80050.78760.80470.7994
V-A Alignment
AVH ↑0.21260.17580.10460.1879
Javis ↑0.17380.14820.08710.1430
Desync ↓0.30000.58130.77250.3050
Diversity
Video Diversity ↑0.28300.10450.24100.2235
Audio Diversity ↑0.48190.38170.57050.6161
V-A Avg. ↑0.38240.24310.40570.4198

Our experiments on OVI further demonstrate the generalizability of the proposed training paradigm beyond LTX-2. Compared with the original OVI teacher, our method reduces inference time from 76.79 s to 13.91 s, achieving a 5.52× speedup while maintaining competitive overall generation quality and diversity. Compared with DMD-only and sCM-only, our staged approach achieves a more favorable quality–diversity trade-off: it maintains competitive generation quality while achieving higher average diversity (0.4198 vs. 0.2431 and 0.4057, respectively). These results demonstrate that our progressive training strategy effectively balances generation quality, diversity, and inference efficiency, and can be transferred to different joint T2VA backbones.

AuthorsOpenReview ↗

4. Comparison with Existing Optimized LTX-2 Implementations

Thank you for raising this important point. Regarding inference acceleration, both LightX2V and the official LTX-2 implementation already provide inference optimizations for LTX-2. The official implementation supports optional FP8 low-precision execution and FlashAttention 3 (FA3), while LightX2V further adopts FP8 per-tensor quantization, SPAS + SageAttention2 sparse attention, and Triton fused modulation operators. Therefore, our method does not simply introduce low-precision computation, sparse attention, or operator fusion. Instead, we design a different combination of optimizations tailored to the computational characteristics of joint video–audio models.

Specifically, our inference optimization stack consists of three components: (1) TileLang W8A8: quantized linear-layer kernels with dynamic INT8 activations and static INT8 weights, reducing the computational and memory access overhead of matrix multiplication; (2) FastNorm: fuses normalization and modulation operations to reduce operator invocations and intermediate tensor read/write overhead; and (3) SageSLA: introduces sparse linear attention for video and audio self-attention, using a nominal block retention ratio of 0.3 to reduce attention computation overhead. Unlike LightX2V's FP8 + SPAS + SageAttention2 pipeline, we combine W8A8 quantization with SageSLA and selectively optimize self-attention across different modalities. Compared with the optional low-precision and FA3 paths provided by the official implementation, we further integrate quantized linear layers, fused normalization and modulation, and specialized optimizations for audiovisual attention computation. These optimizations target linear layers, normalization and modulation operations, and audiovisual self-attention, respectively, forming a complete inference optimization stack tailored to the computational structure of joint video–audio generation.

To quantify the incremental benefits of our inference optimization stack, we further conduct direct latency comparisons with optimized LTX-2 implementations from LightX2V and the official repository, covering both standard- and high-resolution settings.

Latency and Quality Comparison

Configuration512×768 (s) ↓1024×1792 (s) ↓Visual ↑Motion ↑Audio ↑Javis ↑CLIP ↑CLAP ↑CAVP ↑AVH ↑Desync ↓IB-TV ↑IB-TA ↑IB-AV ↑
Official teacher, BF16, CFG305.652264.832.30420.71724.60590.23210.31230.37410.80280.27080.32690.008410.027250.28530
Official 8-step distilled15.86115.392.11490.72494.45540.14090.31550.24500.79340.17480.45180.013440.028480.18613
LightX2V5.3924.452.23880.81344.61760.19670.31690.35450.80000.23170.43550.011260.022800.23638
Ours student, 4 steps3.7318.522.43990.88774.73970.20660.31500.34390.80000.24180.30560.007440.033940.24727

Note: Latencies in this rebuttal were measured on H20 GPUs, while the original manuscript used enhanced H20z, thus slower.

To further isolate the contribution of our inference stack, we compare our four-step student with and without the full optimization stack. Under the same model and sampling steps, our stack achieves additional 2.16× and 3.15× speedups at 512×768 and 1024×1792, respectively. Compared with the already optimized LightX2V, TurboT2VA remains 1.45× and 1.32× faster, while achieving 4.25× and 6.23× speedups over the official eight-step distilled implementation.

In terms of generation quality, our method achieves better results on several perceptual quality and audiovisual alignment metrics while remaining competitive on other metrics. Together, these experiments demonstrate that TurboT2VA is not merely benefiting from fewer sampling steps: its architecture-aware inference stack, combined with the four-step student model, delivers further practical acceleration over existing optimized LTX-2 implementations while maintaining competitive multimodal generation quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.