- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a parallel adapter composition approach for converting a pretrained bidirectional joint audio-video diffusion model into a few-step streaming generator. The authors use an off-the-shelf four-step distillation LoRA and train only a causal LoRA on the original frozen, undistilled MiniMax-H3 backbone using clean-context teacher forcing and block-causal attention. At inference, the two adapters are added to the backbone, avoiding a sequential pipeline in which causal flow-matching may weaken the existing few-step capability. Experiments show that this composition generally provides better visual quality and long-horizon stability than the chained baselines while achieving approximately 26 FPS, although its lip-sync performance and generality remain limited.
Critical strengths
-
The proposed method is simple and practical. It trains only a causal LoRA on the original frozen backbone and combines it with an off-the-shelf few-step distillation LoRA, avoiding complex joint or sequential distillation. The analysis of chained pipelines is also insightful, particularly the observation that applying a standard flow-matching objective to an already distilled model may weaken its few-step capability.
-
The experiments show that the proposed approach outperforms Chained-DA on most reported metrics while maintaining strong visual quality and relatively stable long-horizon generation. These results demonstrate the practical effectiveness of the composition strategy.
Critical weaknesses
-
Limited empirical scope. The authors appropriately acknowledge that all training and evaluation data are restricted to talking-head and digital-human footage. Nevertheless, this remains a substantial limitation because the paper presents the method as a general approach to real-time joint audio-video generation. The current experiments establish its effectiveness only for speech-centric content, where the main challenges and metrics are dominated by lip-speech synchronization and identity preservation. It remains unclear whether the method generalizes to broader audio-visual content with non-speech sounds, diverse events, or more complex cross-modal interactions.
-
The technical contribution is relatively incremental, as it mainly combines existing techniques for few-step distillation, causal generation, and LoRA merging. Moreover, the near-orthogonality of the two LoRA updates under the Euclidean or Frobenius metric provides only limited evidence of functional non-interference. Such orthogonality is not generally preserved after subsequent non-orthogonal transformations and nonlinear operations. The more relevant quantity is the Hessian-weighted cross-term, which the paper does not directly measure.
Questions for authors
-
Scope of the training data and evaluation. Could the authors clarify why both the adaptation data and all evaluation sets are restricted to talking-head or digital-human footage? Since the paper presents the method as a general approach to real-time joint audio-video generation, could it be evaluated on broader benchmarks such as VABench, T2AV-Compass, or AVGen-Bench? VABench appears particularly relevant because it includes an image-to-audio-video setting. If the other benchmarks are incompatible with the first-frame-conditioned streaming protocol, could the authors explain the specific incompatibilities?
-
Alignment between the claims and the empirical scope. The paper appropriately acknowledges the talking-head-only setting as a limitation. However, the title and several claims are phrased in terms of general joint audio-video generation, while the current evidence supports only speech-centric digital-human generation. If broader evaluation is not feasible, would the authors consider narrowing the title and claims to better reflect the demonstrated scope? Alternatively, what evidence supports the expectation that the method will transfer to non-speech audio-visual content?
-
Justification for linear LoRA composition. Could the authors provide a clearer explanation of why the independently obtained causal and distillation LoRAs can be linearly merged while preserving both capabilities? The paper motivates this using small update norms, different functional axes, and near-orthogonal write directions under the Frobenius metric. However, Frobenius orthogonality alone does not guarantee functional non-interference in a nonlinear network. Appendix E identifies the Hessian-weighted cross-term as the relevant second-order interaction, but this quantity is not directly measured. Could the authors provide stronger evidence, such as an estimate of this cross-term, merge-coefficient sweeps, or functional interference measurements? More generally, under what conditions should this composition be expected to succeed or fail?
2 comments
3. Frobenius orthogonality is a necessary condition for orthogonality under the Hessian, and we do not treat it as a sufficient one. The signed alignment of the two updates is not distinguishable from a depth-shuffled null ( versus , ). Their inner product in weight space is close to 0, so the write directions do not cancel. This does not guarantee that the network outputs do not interfere.
During the discussion period we added a further experiment. On 10 VidChatBench clips and three timesteps, let be the velocity predicted by the base model, the velocity with only the few-step LoRA, and the velocity with only the causal LoRA. The angle between and is for video and for audio on the prefix chunk, and and on the next chunk. The two individual changes in velocity are nearly perpendicular, and they are not opposite.
The functional evidence comes from a merge-coefficient sweep added during the discussion period. On 100 VidChatBench videos we set the few-step coefficient and the causal coefficient each to 0.5, 1.0, and 1.5, nine settings in all, with the same generation setup as the paper. Each cell below is Sync-C, IQA, and script error, in that order. Rows are the few-step coefficient and columns are the causal coefficient.
| Few-step / Causal | 0.5 | 1.0 | 1.5 |
|---|---|---|---|
| 0.5 | 7.401 / 0.869 / 0.135 | 7.356 / 0.818 / 0.066 | 6.055 / 0.721 / 0.074 |
| 1.0 | 7.743 / 0.888 / 0.138 | 7.830 / 0.867 / 0.054 | 7.118 / 0.828 / 0.074 |
| 1.5 | 7.363 / 0.889 / 0.140 | 7.852 / 0.886 / 0.062 | 7.542 / 0.865 / 0.060 |
The two coefficients do different jobs. The causal coefficient determines whether the spoken line is correct. At 0.5 the script error is about 0.14, and image quality is highest. At 1.0 the script error falls to about 0.05. The few-step coefficient determines whether 4-step sampling holds.
Raising the causal coefficient further to 1.5 hurts lip sync and image quality, and the hurt is larger when the few-step coefficient is smaller. At few-step coefficients 0.5, 1.0, and 1.5, Sync-C drops by 1.30, 0.71, and 0.31, and IQA drops by 0.097, 0.039, and 0.021. With the few-step coefficient at 1.5 and the causal coefficient at 1.0, Sync-C and IQA remain 7.852 and 0.886; what drops is speaker similarity, from 0.774 to 0.744.
Thank you for reading the paper carefully and for these questions. We address them in turn.
1. We chose digital humans as the experimental setting for three reasons.
(1) Digital humans are a widely used setting for streaming video generation, so a model that can generate in real time transfers directly to this kind of downstream task.
(2) Digital humans demand close audio–video alignment, especially between lip motion and speech. Consistency of the face and of the voice also measures whether long-form generation stays stable, so the task is a useful test of streaming quality.
(3) We did not have the time or the compute to cover a broader range of video generation, so we first checked that the method works on digital humans, and can extend it to more general video later.
Our training data are entirely digital humans, so the evaluations in the main text concentrate on human-centered tests. The model is conditioned on a first frame. T2AV-Compass and AVGen-Bench provide text only, with no first image; a first frame can be obtained by text-to-image generation and then passed to the model. VABench’s image-to-audio-video subset has 521 clips, each with a first image, covering all 14 categories and mostly non-speech. During the discussion period we evaluated this full subset. Both the autoregressive and the non-autoregressive arms use the same 4-step setting as the paper. The first two result columns in the table below are this run. The Sora2, Veo3, and Wan2.5 columns are copied from Tables 2 and 3 of the VABench paper, on the same 521 prompts and under each system’s own default settings; we did not re-run them.
The autoregressive model is worse on text–audio alignment and on DeSync; the other absolute gaps are small. These 521 clips are mostly non-speech audio–video, whereas the training set is entirely digital humans. The autoregressive adapter stays close to the non-autoregressive model on this distribution, so AR-LoRA extends to more general non-speech audio–video generation. On some metrics, it exceeds the published systems in the table: Alignment is higher than all three, and audio aesthetics is higher than Sora2 and Wan2.5. It generalizes reasonably strongly. The paper’s demo page includes some VABench samples.
VABench image-to-audio-video subset (521 clips), 4-step setting:
2. As shown in our response to Q1, the training set is entirely digital humans, but we additionally evaluated all 521 mostly non-speech clips of VABench’s image-to-audio-video subset under the same 4-step setting, where the autoregressive model stays close to the non-autoregressive one. This supports transfer beyond speech-centric digital humans. The revision states this scope explicitly: the main text covers digital humans, together with the non-speech test added during the discussion. The numbers are in our response to Q1.