acceptodds
Withdrawn from ICLR 2027

Injected but Not Accumulated: Measuring and Stabilizing Synchronization in Training-Free Streaming Audio-Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Joint audio-video diffusion models generate video with synchronized sound, but only within native windows of a few seconds. We study their training-free streaming extension: a causal rolling window whose memory and per-second cost do not grow with duration. Concurrent work trains streaming generators on the premise that extension makes audio-visual synchronization drift, and a concurrent benchmark reports endpoint drift of mixed sign across trained systems without resolving its mechanism. We present a measurement protocol that tracks synchronization over generation time with a calibrated offset estimator whose sensitivity is audited on every compared system, sample-accurate seam diagnostics, content-validity gates, and a stratified prompt suite with a negative control. The measurement overturns the premise. Naive streaming injects a conditioning-anchor error of −120 ms at 91% of window seams, as a closed-form analysis of the conditioning arithmetic predicts to within 2.5 ms across ten operating points, yet the errors do not accumulate: 4.3 s of injected error over two minutes leaves a median realized offset of 0.23 s, and offset curves stay flat for five minutes, because each window's joint denoising re-synchronizes audio to video. The same protocol yields three training-free stabilizers, modality-aligned conditioning arithmetic, audio-lead conditioning, and cross-window fixed noise, whose combination removes the seam fingerprint and improves synchronization over both naive streaming and an offline joint-denoising baseline on a 200-prompt benchmark (0.253 vs. 0.285 and 0.437 s expected offset), at matched instrument sensitivity and no measured visual-quality cost. On a second backbone with a different fusion design and conditioning path, naive continuation does degrade synchronization after the first window, and the fixed-noise stabilizer recovers part of the loss, with prompt-dependent magnitude. Cross-window subject identity remains open: our system re-rolls identity at 72% of seams, where the offline baseline, at full-duration latency, almost never does. We release the protocol, the prompt suite, and code.

Primary area:
generative models
Keywords:
audio-video generation, streaming generation, long-form video generation, audio-visual synchronization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jH7TOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether audio-video synchronization errors accumulate when a joint audio-video generation model is extended to streaming generation using a sliding-window approach. The authors find that naive extension does not lead to significant accumulation of synchronization errors, although it introduces boundary artifacts and other issues. To address these problems, they propose a training-free method to stabilize streaming generation and improve generation quality.

Critical strengths

The authors provide detailed implementation descriptions and release the code.

Critical weaknesses
  1. The paper reads more like an experimental report than a well-structured research paper. The method section contains excessive implementation details, making it difficult to follow the main idea and understand the key contributions.
  2. The evaluation is not sufficiently convincing. The experiments mainly focus on a single 2.1B joint audio-video generation model (JavisDiT++). Moreover, several methodological choices rely heavily on the specific architecture and implementation of JavisDiT++, such as the audio-video token rates and alignment mechanisms. Therefore, it remains unclear whether the findings generalize to other joint audio-video generation models.
  3. The paper does not appear to be fully prepared for submission. For example, Line 468 states, "These interim results do not constitute the completed pre-registered test." This raises concerns about whether the experimental evaluation has been fully completed.
Questions for authors

Can the authors better distinguish the key methodological contributions from model-specific implementation details?

Reviewer_hD2bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • This paper studies synchronization when a pretrained short-window joint audio-video generator is extended through training-free rolling-window generation. Using JavisDiT++ as the primary backbone, each new window is conditioned on the tail of the previous output. The authors analyze local timing mismatches introduced by the conditioning intervals and report no sustained increase in estimated audio-video offsets in the evaluated continuations, including a small set of five-minute examples.

  • The paper also combines three inference-time interventions: modality-aligned conditioning, audio-lead conditioning, and shared initial noise across windows. Evaluation includes offset-estimator (Synchformer) sensitivity audits, boundary and identity diagnostics, quality metrics, and a held-out comparison on 200 prompts. The combined method improves the reported offset estimates in the primary comparison, while its benefits at longer durations and on a second backbone are less consistent. The human evaluation remains preliminary.

Critical strengths
  • The main experiments match the backbone, sampling settings, prompts, and seeds, and separate development prompts from a held-out 200-prompt evaluation. The paper also discloses smaller effects on held-out data, a nonsignificant synchronization difference at 60 seconds, and worse continuity on the second backbone. This transparency strengthens the value of the empirical observations within the evaluated setting.
Critical weaknesses
  • The definitions and validation of the measurements are insufficient to support the central interpretation concerning cumulative drift. The paper needs to distinguish conditioning-interval placement errors, cross-correlation lags between audio anchors, and estimated audio-video offsets. Their definitions and relationships are scattered across the main text, figures, and appendix, making the central argument difficult to verify. The offset estimator is also reported to return toward zero outside its operating range. The paper does not establish that it remains sensitive in the later portions of the 120–300-second continuations, or that the complete protocol reliably detects known cumulative drift. The five-minute evaluation covers only four periodic prompts. Given the frequent identity discontinuities, the measurements do not clearly distinguish sustained synchronization of continuing events from local synchronization of content that is repeatedly regenerated at each window.

  • The incremental contribution of the proposed interventions and their perceptual benefit remain insufficiently established. The mean absolute offset estimate improves from 0.285 to 0.253 seconds on the held-out 200-prompt benchmark, but the held-out 60-second comparison does not establish a significant difference. The additional benefit over shared noise alone is also unclear. In the reported exploratory 60-second contrasts, adding shared noise to naive streaming improves the estimate by approximately 0.051 seconds, compared with approximately 0.058 seconds for the complete method. A direct held-out comparison is needed to establish the value of the full combination. The human evaluation is incomplete, with only two raters satisfying the prespecified strict inclusion criteria. The gain-corrected effect also lacks a sufficiently explicit derivation and uncertainty estimate. Consequently, independent evidence that the reported metric improvements translate into perceptually better synchronization and viewing quality remains limited.

Questions for authors
  1. Validation of the incremental contribution: Can you provide a direct held-out comparison between the complete method and naive streaming with shared noise alone? This comparison is needed to isolate the additional effects of anchor correction and audio lead on synchronization, continuity, and quality.

  2. Measurement and perceptual validity: Please provide the formula, calibration data, and confidence interval for the gain-corrected effect. Does the completed human study support improvements in synchronization and overall quality under the prespecified exclusion criteria and an analysis accounting for within-rater dependence?

  3. Validation of the central conclusion: Can the complete measurement protocol detect a known, progressively increasing drift, including offsets beyond ±2 seconds? Also, which signals are cross-correlated to obtain the “end-to-end offset” reported in Section 4.1?

Reviewer_Vf6rOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether causal, constant-memory rolling-window generation in a joint audio-video diffusion transformer leads to accumulated audio-video synchronization errors over long-form generation. It distinguishes predictable synchronization offsets introduced at window boundaries from cumulative temporal drift, and finds that no evidence of sustained drift over the evaluated durations. The authors also introduce a measurement protocol incorporating estimator calibration, sensitivity analysis, and content-validity checks to ensure reliable evaluation. Furthermore, they explore several sampling interventions, whose combination yields modest improvements in synchronization accuracy and generation quality compared to naive streaming.

Critical strengths
  • The paper studies an interesting and important task. Distinguishing local boundary errors from cumulative synchronization drift provides a more practical understanding of failure modes in long-form audio-video generation.
  • Estimator calibration, per-system sensitivity audits, negative controls, and content-validity checks reveal some findings, which might be useful for the community.
  • In their experiments, the matched prompts and seeds, component ablations, and a held-out 200-prompt evaluation strengthen the comparison with naive streaming. The authors also transparently report limitations and unfavorable results.
Critical weaknesses
  • The proposed interventions primarily combine conditioning-index corrections, a temporal lead parameter, and previously explored shared-noise sampling. The paper's contribution lies more in its empirical analysis and evaluation protocol than in methodological novelty for generation.
  • The generalizability of the proposed stabilizers remains unclear. The main improvement is modest and demonstrated on one backbone at 30 seconds, while the held-out 60-second synchronization comparison is not statistically significant. The second backbone uses image-only continuation without an audio anchor, so it does not validate the complete method. Its fixed-noise results are also prompt-dependent and show increased identity discontinuities. A comparable evaluation on another backbone with both audio and video conditioning would strengthen the evidence.
  • Including human evaluation is appreciated. However, it only involves two participants passing the strict screening criteria. The current results cannot establish robust perceptual benefits or whether the remaining fixed seam displacement is noticeable.
Questions for authors
  • Can the no-drift finding be validated on a larger and more diverse long-duration test set, using independent event-level annotations or another synchronization measure that remains reliable beyond the current estimator’s range?
  • How much benefit does the full method provide over simply adding shared noise to naive streaming on the held-out benchmark? This comparison would clarify the practical contribution of the anchor correction and audio lead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.