acceptodds
Under review as a conference paper at ICLR 2027

REAL-TIME JOINT AUDIO–VIDEO GENERATION BY PARALLEL ADAPTER COMPOSITION

Abstract

Deploying a joint audio–video diffusion transformer for real-time, interactive generation normally requires two essential modifications: block-autoregressive attention, so frames can be emitted before the whole clip is finished, and few-step sampling, so each block is cheap. Conventionally, the streaming video literature obtains both capabilities from a chained pipeline. It first distills a bidirectional teacher into a causal student, then into a few-step one, or proceeds in reverse order. Each stage of such a chain fine-tunes the weights the previous one produced, so a later objective can undo an earlier capability. Following the idea of model merging, we show that on a packed audio–video backbone the two capabilities can be acquired in parallel. A causal adapter is trained against the frozen backbone, and an off-the-shelf few-step adapter provides the few-step capability. As the two edit different functional axes, we predict, and then verify, that their weight-update directions are near-orthogonal, without any explicit orthogonality constraint during training. Orthogonal updates should combine without interfering, so parallel composition is a direct sum. The two adapters are simply added at inference, with no joint training, yielding few-step, streaming audio–video whose image quality tracks the bidirectional teacher. Compared to the chained baselines, the composed model matches or beats them on most metrics, making parallel composition a practical approach. The resulting streaming system generates joint audio–video in real time,  fps at without quantization, and sustains  s of continuous generation with stable image quality. Demo Page: https://pac-demo-2027.github.io/demo/

Primary area:
generative models
Keywords:
Joint audio-video generation, Real-time streaming generation, Model merging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jatqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a parallel adapter composition approach for converting a pretrained bidirectional joint audio-video diffusion model into a few-step streaming generator. The authors use an off-the-shelf four-step distillation LoRA and train only a causal LoRA on the original frozen, undistilled MiniMax-H3 backbone using clean-context teacher forcing and block-causal attention. At inference, the two adapters are added to the backbone, avoiding a sequential pipeline in which causal flow-matching may weaken the existing few-step capability. Experiments show that this composition generally provides better visual quality and long-horizon stability than the chained baselines while achieving approximately 26 FPS, although its lip-sync performance and generality remain limited.

Critical strengths
  1. The proposed method is simple and practical. It trains only a causal LoRA on the original frozen backbone and combines it with an off-the-shelf few-step distillation LoRA, avoiding complex joint or sequential distillation. The analysis of chained pipelines is also insightful, particularly the observation that applying a standard flow-matching objective to an already distilled model may weaken its few-step capability.

  2. The experiments show that the proposed approach outperforms Chained-DA on most reported metrics while maintaining strong visual quality and relatively stable long-horizon generation. These results demonstrate the practical effectiveness of the composition strategy.

Critical weaknesses
  1. Limited empirical scope. The authors appropriately acknowledge that all training and evaluation data are restricted to talking-head and digital-human footage. Nevertheless, this remains a substantial limitation because the paper presents the method as a general approach to real-time joint audio-video generation. The current experiments establish its effectiveness only for speech-centric content, where the main challenges and metrics are dominated by lip-speech synchronization and identity preservation. It remains unclear whether the method generalizes to broader audio-visual content with non-speech sounds, diverse events, or more complex cross-modal interactions.

  2. The technical contribution is relatively incremental, as it mainly combines existing techniques for few-step distillation, causal generation, and LoRA merging. Moreover, the near-orthogonality of the two LoRA updates under the Euclidean or Frobenius metric provides only limited evidence of functional non-interference. Such orthogonality is not generally preserved after subsequent non-orthogonal transformations and nonlinear operations. The more relevant quantity is the Hessian-weighted cross-term, which the paper does not directly measure.

Questions for authors
  1. Scope of the training data and evaluation. Could the authors clarify why both the adaptation data and all evaluation sets are restricted to talking-head or digital-human footage? Since the paper presents the method as a general approach to real-time joint audio-video generation, could it be evaluated on broader benchmarks such as VABench, T2AV-Compass, or AVGen-Bench? VABench appears particularly relevant because it includes an image-to-audio-video setting. If the other benchmarks are incompatible with the first-frame-conditioned streaming protocol, could the authors explain the specific incompatibilities?

  2. Alignment between the claims and the empirical scope. The paper appropriately acknowledges the talking-head-only setting as a limitation. However, the title and several claims are phrased in terms of general joint audio-video generation, while the current evidence supports only speech-centric digital-human generation. If broader evaluation is not feasible, would the authors consider narrowing the title and claims to better reflect the demonstrated scope? Alternatively, what evidence supports the expectation that the method will transfer to non-speech audio-visual content?

  3. Justification for linear LoRA composition. Could the authors provide a clearer explanation of why the independently obtained causal and distillation LoRAs can be linearly merged while preserving both capabilities? The paper motivates this using small update norms, different functional axes, and near-orthogonal write directions under the Frobenius metric. However, Frobenius orthogonality alone does not guarantee functional non-interference in a nonlinear network. Appendix E identifies the Hessian-weighted cross-term as the relevant second-order interaction, but this quantity is not directly measured. Could the authors provide stronger evidence, such as an estimate of this cross-term, merge-coefficient sweeps, or functional interference measurements? More generally, under what conditions should this composition be expected to succeed or fail?

2 comments

Thank you for reading the paper carefully and for these questions. We address them in turn.

1. We chose digital humans as the experimental setting for three reasons.

(1) Digital humans are a widely used setting for streaming video generation, so a model that can generate in real time transfers directly to this kind of downstream task.

(2) Digital humans demand close audio–video alignment, especially between lip motion and speech. Consistency of the face and of the voice also measures whether long-form generation stays stable, so the task is a useful test of streaming quality.

(3) We did not have the time or the compute to cover a broader range of video generation, so we first checked that the method works on digital humans, and can extend it to more general video later.

Our training data are entirely digital humans, so the evaluations in the main text concentrate on human-centered tests. The model is conditioned on a first frame. T2AV-Compass and AVGen-Bench provide text only, with no first image; a first frame can be obtained by text-to-image generation and then passed to the model. VABench’s image-to-audio-video subset has 521 clips, each with a first image, covering all 14 categories and mostly non-speech. During the discussion period we evaluated this full subset. Both the autoregressive and the non-autoregressive arms use the same 4-step setting as the paper. The first two result columns in the table below are this run. The Sora2, Veo3, and Wan2.5 columns are copied from Tables 2 and 3 of the VABench paper, on the same 521 prompts and under each system’s own default settings; we did not re-run them.

The autoregressive model is worse on text–audio alignment and on DeSync; the other absolute gaps are small. These 521 clips are mostly non-speech audio–video, whereas the training set is entirely digital humans. The autoregressive adapter stays close to the non-autoregressive model on this distribution, so AR-LoRA extends to more general non-speech audio–video generation. On some metrics, it exceeds the published systems in the table: Alignment is higher than all three, and audio aesthetics is higher than Sora2 and Wan2.5. It generalizes reasonably strongly. The paper’s demo page includes some VABench samples.

VABench image-to-audio-video subset (521 clips), 4-step setting:

MetricAutoregressiveNon-autoregressiveSora2Veo3Wan2.5
Audio Aes3.4723.4672.9743.5743.455
T–V Align0.22330.22540.21880.23340.2374
T–A Align0.32770.35690.40450.41300.2865
A–V Align0.21770.2350.26230.32150.2112
DeSync ↓0.66930.59920.91710.61360.3539
Alignment4.9194.9214.8854.9064.812
Expressiveness4.4884.5224.3904.6314.495
Visual Realism4.94.9234.9644.9214.766
Audio Realism4.434.4414.5974.6604.395
Audio QA0.70750.73920.82870.85840.8084
Visual QA0.73380.74930.76110.79820.7889
Artistry3.9853.9813.9313.9833.929

2. As shown in our response to Q1, the training set is entirely digital humans, but we additionally evaluated all 521 mostly non-speech clips of VABench’s image-to-audio-video subset under the same 4-step setting, where the autoregressive model stays close to the non-autoregressive one. This supports transfer beyond speech-centric digital humans. The revision states this scope explicitly: the main text covers digital humans, together with the non-speech test added during the discussion. The numbers are in our response to Q1.

3. Frobenius orthogonality is a necessary condition for orthogonality under the Hessian, and we do not treat it as a sufficient one. The signed alignment of the two updates is not distinguishable from a depth-shuffled null (+0.0006+0.0006 versus +0.0000+0.0000, p=0.26p = 0.26). Their inner product in weight space is close to 0, so the write directions do not cancel. This does not guarantee that the network outputs do not interfere.

During the discussion period we added a further experiment. On 10 VidChatBench clips and three timesteps, let v00v_{00} be the velocity predicted by the base model, v10v_{10} the velocity with only the few-step LoRA, and v01v_{01} the velocity with only the causal LoRA. The angle between v10−v00v_{10}-v_{00} and v01−v00v_{01}-v_{00} is 105∘±8∘105^\circ\pm8^\circ for video and 95∘±6∘95^\circ\pm6^\circ for audio on the prefix chunk, and 108∘±6∘108^\circ\pm6^\circ and 97∘±8∘97^\circ\pm8^\circ on the next chunk. The two individual changes in velocity are nearly perpendicular, and they are not opposite.

The functional evidence comes from a merge-coefficient sweep added during the discussion period. On 100 VidChatBench videos we set the few-step coefficient and the causal coefficient each to 0.5, 1.0, and 1.5, nine settings in all, with the same generation setup as the paper. Each cell below is Sync-C, IQA, and script error, in that order. Rows are the few-step coefficient and columns are the causal coefficient.

Few-step / Causal0.51.01.5
0.57.401 / 0.869 / 0.1357.356 / 0.818 / 0.0666.055 / 0.721 / 0.074
1.07.743 / 0.888 / 0.1387.830 / 0.867 / 0.0547.118 / 0.828 / 0.074
1.57.363 / 0.889 / 0.1407.852 / 0.886 / 0.0627.542 / 0.865 / 0.060

The two coefficients do different jobs. The causal coefficient determines whether the spoken line is correct. At 0.5 the script error is about 0.14, and image quality is highest. At 1.0 the script error falls to about 0.05. The few-step coefficient determines whether 4-step sampling holds.

Raising the causal coefficient further to 1.5 hurts lip sync and image quality, and the hurt is larger when the few-step coefficient is smaller. At few-step coefficients 0.5, 1.0, and 1.5, Sync-C drops by 1.30, 0.71, and 0.31, and IQA drops by 0.097, 0.039, and 0.021. With the few-step coefficient at 1.5 and the causal coefficient at 1.0, Sync-C and IQA remain 7.852 and 0.886; what drops is speaker similarity, from 0.774 to 0.744.

Reviewer_Nzp9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • In this paper, the authors propose Parallel Adapter Composition that makes pretrained bi-directional joint audio-video generation to a few-step autoregressive generator. Central idea of this method is disentangling the LoRA trainings for causal and a few-step distillation, which are typically done in sequential manner, e.g., causal -> few-step or few-step -> causal. In this paper, specifically, the authors use a frozen MiniMax-H3 backbone to train a causal LoRA via clean-context teacher forcing, and add it to an existing 4-step distillation LoRA at inference time to achieve Parallel Adapter Composition.

  • The authors also analyze why Parallel Adapter Composition works. They attribute its effectiveness to the near-orthogonality of the two LoRA weight updates (causal LoRA and few-step distillation LoRA), arguing that adding both updates to the original MiniMax-H3 weights allows their respective capabilities to be combined with minimal interference.

  • The authors compare Parallel Adapter Composition against both sequential training orders on a held-out set of 211 clips and test sets drawn from AVSpeech, HDTF, and VidChatBench. They also report streaming generation at approximately 26 fps at a resolution of 480 × 832, with a time to first frame (TTFF) of 4.2–4.3 seconds on six GPUs without quantization.

Critical strengths
  • Parallel Adapter Composition has practical value because it reuses an existing few-step adapter and requires training only the causal adapter, reducing the training requirements. The finding that straightforward addition of the LoRA updates is effective provides a useful approach to building streaming joint audio–video generation models at lower computational cost.

  • The comparison of multiple training orders on an identical backbone is a notable strength. It controls for confounding variables across different foundation models while comparing the teacher model, whole-segment generation, parallel synthesis, and sequential training. Evaluations on public data and longer-than-training generation lengths are also provided, clearly illustrating both the benefits of the proposed approach and the trade-offs between speech intelligibility and speaker similarity.

Critical weaknesses
  • The discussion of closely related prior work is incomplete. LongLive-2.0 (Appendix H) [1] already demonstrates that independently trained causal LoRA and few-step distillation LoRA components can be combined for streaming video generation. Although I acknowledge the extension to joint audio–video generation as a distinction, I do not see a clearly established, substantial difference from this prior work. Furthermore, other techniques used to enable streaming generation, such as first-frame anchoring and sliding-window attention, have already been explored in LongLive [2].

  • The evaluation is insufficient to support the central claims of real-time audio-visual generation and quality preservation. While the reported 26 fps supports average throughput, the paper omits the output intervals of decoded video and audio, latency variance (jitter), the sequential execution strategy for the audio/video decoders. Responsiveness to interactive prompt changes is also left unevaluated, making it impossible to verify the method's effectiveness for interactive applications. Furthermore, the evaluation relies on automatic metrics from the talking-head domain without perceptual validation through human evaluation. Specifically, the WER degrades to 0.127 with the proposed method compared to 0.047 for whole-segment generation, and speaker similarity drops by approximately 41% from its initial level during 30-second generation. Given these degradations, it is difficult to conclude that overall audio-visual quality is maintained based solely on the preservation of image quality.

[1] Chen, Yukang, et al. "LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation." arXiv preprint arXiv:2605.18739 (2026).

[2] Yang, Shuai, et al. "Longlive: Real-time interactive long video generation." arXiv preprint arXiv:2509.22622 (2025).

Questions for authors
  • Relative to LongLive-2.0’s contribution, what specific and substantial novelty does this work offer? For example, please highlight any modality-specific designs or insights developed to tackle the unique challenges of audio-visual multimodal data.
  • Regarding of streaming generation, could you clarify whether the video/audio decoders are causal, and detail how the full streaming pipeline from text input to RGB frame and audio waveform decodings implemented? Furthermore, to "real-time" in the title, please provide frame-level latency breakdowns (e.g., per-frame mean, median, P95, and P99).
  • While the introduction motivates streaming generation via interactive applications, does the method actually support text prompt updates in the middle (like interactive generation), user audio/video inputs, or speech interruptions? If so, I suggest providing the corresponding details.
4 comments

Thank you for reading the paper carefully and for these questions. We address them in turn.

1. Our method is close to LongLive-2.0, and the revised version cites it in Section 2. LongLive-2.0, however, is video-only; the shared ingredient is an autoregressive stage followed by a few-step stage that is stacked onto the AR model. Unlike LongLive-2.0, which trains its AR model with full-parameter fine-tuning and stacks a few-step AR onto that fully fine-tuned backbone, we instead realize both stages as two independently trained lightweight AR adapters that can be freely stacked without any full fine-tuning. Our contribution is what this stacking does on a packed audio-video backbone, where both modalities must survive the same weight update.

Audio and video do not share a time grid: the video VAE compresses 17 frames into 5 latents, while audio runs at 40 Hz against 24 fps, so a 34-frame chunk maps to 56.67 audio latents and never lands exactly on a boundary. Accumulating audio positions chunk by chunk would let the rounding error grow and drift the two streams apart; we instead recompute each chunk's audio position from its absolute frame number, keeping the error within a third of a latent and constant. The two modalities also have separate noise schedules, yet each chunk uses one shared progress index into both, so video and audio stay at the same denoising stage within each chunk.

The two updates edit different functional axes: context routing and the denoising trajectory. We study the geometry of the two updates and find them near-orthogonal without any explicit constraint, a condition for a safe sum. LongLive-2.0 provides no such analysis.

2. Video is decoded by LightVAE’s lightweight decoder. The decoder is non-causal. Its minimum unit is 5 visual latents, which yields 17 frames at a time, so the latencies below are reported per chunk rather than per frame. Training and inference both use 10 latents per chunk: once the DiT finishes a chunk, decoding produces about 34 frames, about 1.42 seconds at 24 fps. The text encoder, the DiT, and the VAE decoders are deployed separately. Each chunk is handed to the decoder as it leaves the DiT, while the DiT denoises the next chunk.

We have added a real-time interactive video to the paper’s demo page. The DiT uses W8A8 quantization and fused operators in the latest demo page. These latency numbers are measured on the deployed demo stack, which uses more GPUs and a lightweight VAE in addition to W8A8; they are therefore not directly comparable to the 1.6–1.7 s first packet and ≈26 fps reported for the six-GPU unquantized setup in Appendix I.

When a new line arrives, the first packet has three stages: text encoding, denoising of the first chunk, and decoding of the first chunk. Over 20 rounds, text encoding in the first packet takes 0.103 s (a single measurement without the cache), the DiT takes 0.390±0.0010.390\pm0.001 s, and decoding takes 0.305±0.0060.305\pm0.006 s (video 0.244 s, audio 0.061 s). Muxing takes 0.024±0.0010.024\pm0.001 s, and the remaining 0.064±0.0040.064\pm0.004 s is preparation before denoising and handing the latents to the decoder. The KV-cache commit happens after the first chunk has been handed to the decoder, so it is not on the first-packet path. Taken together, the first-packet latency over the 20 rounds is 0.886±0.0060.886\pm0.006 s.

In steady state, later chunks are the full 34 frames (1.42 s of video). Over the 20 rounds, the median interval between two successive chunks is 0.571 s, shorter than 1.42 s. This is the direct generation speed measured on our log: the DiT renders at about 66.8 fps, and steady emission after decoding is about 59.5 fps. Both are faster than playback at 24 fps.

In actual playback, latency is also affected by variation in the network. In the paper’s demo page, the avatar’s replies are smooth there, and the network latency is about 500 ms to 1 s.

3. Yes for text updates and interruptions; the user’s live video is not a condition. A session alternates between listening and speaking rounds, with audio and video generated chunk by chunk by our audio–video model.

In a speaking round, user speech is turned into a reply line by an omni model, whose text KV cache carries the earlier dialogue. That line becomes the current prompt. The user’s live video is not fed to our model: its conditions are only the first frame and the current text. Text is not written into the audio–video KV cache; each chunk re-encodes the current text, while generated frames remain cached. Round switches therefore change only the text, and the video continues without a jump.

A listening round has no line to speak. It is used while the user is speaking and after an interruption: if the user speaks again during playback, unplayed chunks are discarded, cached chunks are kept, and the next chunk uses the listening prompt. Round switching is scheduling outside the model; quantitative evaluation remains on offline clips.

Thank you for the clarifications and the additional VABench evaluation. However, my main concerns remain:

  • Significance of the contribution is still limited. LongLive-2.0 v2 (Appendix H) already trains a standalone DMD LoRA independently of AR training and combines them at inference. PAC’s use of two LoRAs and an existing bidirectional adapter for joint AV generation is useful, but the substantial benefit beyond this prior principle remains insufficiently established. The weight-space orthogonality analysis does not establish functional noninterference, particularly since Chained-DA exhibits similar geometry. Moreover, Chained-DA applies plain flow matching to distilled weights, unlike OmniForcing’s ODE regression and self-forcing; its failure therefore does not establish a general advantage over strong sequential approaches.

  • Validation of the quality trade-offs. The VABench results strengthen the evidence for transfer beyond talking heads. However, the rebuttal does not resolve the concerns about speech errors and long-term speaker consistency: WER remains 0.127 versus 0.047 for Whole-seg, and speaker similarity declines substantially during longer generation (Tables 1–2). These results indicate trade-offs whose perceptual importance remains untested. Representative blinded comparisons against Whole-seg and Chained-AD would materially strengthen the assessment of PAC’s practical value.

Thank you for raising these key concerns. We address each point below and have tempered the affected claims in the revised manuscript where warranted.

Response 1: Our differences from LongLive-2.0 lie not in the shared principle of "independently trained LoRAs, combined at inference," but in three points:

First, the two capabilities can be independently enabled, disabled, or replaced — this is a fact already verified in Table 1, not a hypothetical: WHOLE-SEG is Δd alone (4-step, bidirectional) and DECOUPLED-50 is Δc alone (50-step, causal); each capability stands on its own, and their sum stands as well. In contrast, LongLive-2.0's autoregressive capability comes from a full-parameter fine-tune that cannot be swapped out separately.

Second, PAC removes the causal distillation stage entirely. When training its few-step LoRA, LongLive-2.0 still applies an AR mask to the teacher and runs a three-model causal DMD (generator, real-score teacher, fake-score critic) alongside full-parameter AR training**. PAC directly reuses a community purely-bidirectional few-step LoRA (which has never seen a causal mask or a KV cache); the entire training involves a single LoRA, a single plain flow-matching objective, and a single stage — no critic, no ODE rollout, no multi-model online distillation.

Third, the settings differ: LongLive-2.0 is video-only; extending it to joint audio–video would require training an audio-video Turbo LoRA under an AR mask, and would have to confront modality-specific difficulties such as misaligned video/audio temporal grids, a shared progress index, and the packed sequence. PAC resolves these directly on the audio–video backbone and provides functional/geometric evidence for when direct addition is safe (Appendix E).

Our LoRA-swap experiment shows that within LongLive-2.0's own video-only setting, its causal DMD stage is redundant for itself — this reduces the engineering required by LongLive-2.0; PAC's contribution is precisely to show that under the harder joint audio–video setting, that same causal distillation can likewise be omitted wholesale, and that the safety of doing so is verifiable.

Setup: ref = reproduction of the official LongLive-2.0 repo; exp1a = replacing its AR Turbo LoRA with an NAR Turbo LoRA. Tables 4/5 below refer to the video-only VBench numbers self-reported in the LongLive-2.0 paper, which are different from Table 4 (VABench transfer) in our paper. Results: the 5s Total score is nearly unchanged (84.29 → 84.34). On VBench-Long, all metrics except dynamic change by ≤ 1.6 (subject 94.99 → 93.45, smoothness 98.82 → 99.12, imaging 67.01 → 68.14, etc.); for dynamic, exp1a (60.03) is close to the value reported in the LongLive-2.0 paper (60.62).

Full VBench (5s, 946 prompts × 5 seeds, official formula)

QualitySemanticTotal
LongLive-2.0 Table 486.6778.6385.06
Ours · ref85.3480.0784.29
Ours · exp1a85.4979.7584.34
Note: the LongLive-2.0 paper reports using prompt augmentation, which we did not use; this accounts for part of the gap to their Table 4.

VBench-Long (60s, clip-level protocol, 50 prompts × 2 seeds)

subjectbackgroundsmoothnessdynamicaestheticimaging
LongLive-2.0 Table 597.4897.0098.8660.6253.6865.51
Ours · ref94.9995.0898.8266.7853.7467.01
Ours · exp1a93.4594.7199.1260.0353.9468.14
Note: our computation of subject/background consistency differs from the paper's, which may explain the discrepancy between our reproduction and their reported values.

Response 2: We agree, and the revised §4.3 states this as its opening conclusion: "neither chain fails for geometric reasons." Geometry serves one purpose only — verifying that directly adding two individually valid LoRAs is safe — and there it is conclusive: write interference exceeds the null by only 26% while read interference is 5.7× the null (overlapping reads, no overwriting; route success is decided by the objective, §4.3). For functional noninterference, Appendix E adds two output-space tests: velocity-shift angles of 105°±8° (video) / 95°±6° (audio) — nearly perpendicular, not opposing — and the merge sweep (Table 5): at unit weights Sync-C/IQA stay stable (7.830/0.867) while script error drops 0.138→0.054, degrading only at αc = 1.5.

Response 3: On the positioning of Chained-DA and its relation to strong sequential approaches. We fully agree: Chained-DA's failure does not imply that parallel composition is superior to strong multi-stage sequential approaches such as OmniForcing, which use ODE regression and self-forcing; we have revised the wording in Section 2 and Section 3.1 to avoid this misunderstanding. Chained-DA serves strictly as a controlled comparison at matched training budget, showing that without ODE trajectories and self-forcing, flow-matching causalization can only be applied to an undistilled backbone.

Response 4: We performed a per-word edit-distance decomposition on the English portion of the held-out set. Substitutions — actually misread words — barely increase: 0.029 vs 0.020 for Whole-seg; 89% of the gap (+0.071 of +0.080) comes from deletions + insertions (0.042 + 0.056 vs 0.011 + 0.016). The cause is the evaluation protocol: held-out clips are forced to use the same fixed total frame count as the bidirectional teacher, while a block-causal model cannot see the total length — slightly slower speech drops the final words at truncation (counted as deletions), and slightly faster speech produces extra words in the trailing silence (counted as insertions). This explanation has been added to §4.2. The per-clip median WER is 0.062.

It should be noted that the 0.127-vs-0.047 gap is a common cost of causalization, not specific to PAC: all four 4-step causal systems fall in 0.126–0.136 (DECOUPLED 0.127, on par with the best chain; its Chinese CER of 0.051 is the lowest of all systems).

Response 5: Within the 5–10s interactive-turn range covered by training, DECOUPLED's voice similarity is 0.666 (close to the 0.688 of the same-budget Chained-DA). The drop to 0.521 (−41.0%) occurs only after the N = 7 (~10s) sliding window has evicted the opening audio from the KV cache. The revised manuscript repositions the 30s protocol as an extrapolation stress test at 3–6× the training length (§4.2, Table 2) and expands the speaker-drift discussion in §5 with its architectural cause (audio lacks a persistent first-frame-style anchor) and candidate remedies.

Response 6: We did not conduct a blinded human study, and we have stated this candidly in the revised §5: "blinded human preference studies against WHOLE-SEG and CHAINED-AD remain needed to weigh the perceptual salience of these metric trade-offs." All conclusions in the paper are based on automatic metrics (Sync-C, CSIM, IQA, WER/CER) and contain no claims of perceptual superiority.

Reviewer_VqkPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts a pretrained joint audio–video generator to four-step streaming generation. An existing distillation LoRA is combined with a causal LoRA trained independently on the original backbone using clean-context teacher forcing. The authors evaluate adapter composition against sequential training, analyze weight-space overlap, and test generation quality, 30-second stability, and inference speed.

Critical strengths

S1: Reuse of independently acquired adapters. The method combines an existing four-step adapter with one newly trained causal adapter, avoiding additional distillation on the causal backbone. This is a useful practical result for reusing pretrained components.

S2: Good preservation of visual quality. In Table 1, image-quality scores are close to the offline four-step model (0.914 versus 0.915) and higher than both sequential baselines (0.854 and 0.877). Public-benchmark results also support the visual-quality benefit, although synchronization and speech quality show trade-offs.

Critical weaknesses

W1: The benefit of independent training is not fully isolated. The composition strategy is simple and provides convincing improvements in visual consistency over the evaluated CHAINED-AD baseline. However, it remains unclear how much of this benefit comes from independent adaptation rather than differences in training objectives and optimization. CHAINED-DA is particularly difficult to interpret because its objective can undo distillation (Section 3.1). A comparison with matched training budgets and compatible sequential objectives would strengthen the claim that parallel adaptation itself is advantageous.

Long-horizon visual quality remains a concern. In the shown decoupled sample, noticeable visual artifacts appear later. Although the method improves visual consistency over CHAINED-AD, Figure 3 measures similarity to the first frame and does not directly assess artifact severity.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.