acceptodds
Withdrawn from ICLR 2027

StoryWeaver: Towards Consistent and Synchronized Multi-Shot Audio-Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating multi-shot videos with synchronized audio remains a significant challenge. While single-shot models perform well, extending them to multi-shot story generation is hindered by consistency degradation across shots. Current approaches typically rely on decoupled paradigms that generate segments independently, inherently limiting long-range coherence and causing identities, voices, and backgrounds to drift over time. Furthermore, existing continuous story frameworks often lack synchronized audio synthesis. To address these limitations, we introduce StoryWeaver, a framework for consistent multi-shot audio-video generation. Instead of decoupled steps, it adopts a joint generation paradigm conditioned on text and an initial prompt to improve global coherence. To model audio-visual interactions across extended contexts, we propose the Multi-Modal Sink Shot (MSS) mechanism, which couples windowed audio-visual cross-attention for local synchronization with sink-mediated inter-shot self-attention for cross-shot consistency. To extend generation beyond a single denoising window, we introduce Audio-Visual Latent Stitching (AVLS), a train-test matched continuation scheme that operates entirely in the joint audio-video latent space, reducing identity and acoustic drift compared to decode-reencode approaches. Experiments on both video-only and audio-visual benchmarks, supported by audio-specific ablations, adjacent-shot transition metrics, and human preference studies, show that StoryWeaver achieves improved visual-acoustic consistency, character preservation, and temporal coherence over existing methods.

Primary area:
generative models
Keywords:
multi-shot story generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UH73OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StoryWeaver, a multi-shot audio-video generation framework built upon MOVA, aiming to improve cross-shot identity preservation, acoustic consistency, and audio-visual synchronization. The proposed Multi-Modal Sink Shot (MSS) mechanism combines windowed audio-visual cross-attention with sink-mediated inter-shot self-attention, while Audio-Visual Latent Stitching (AVLS) enables long-horizon continuation in the joint audio-video latent space. The model is trained on 350K multi-shot examples and evaluated on ST-Bench and the newly constructed WeaverBench, with component ablations, audio quality evaluations, long-horizon generation experiments, and human preference studies. The results demonstrate improvements in cross-shot consistency, identity preservation, and speaker consistency.

Critical strengths
  1. Task-driven architectural design for multi-shot audio-video generation. The proposed Multi-Modal Sink Shot (MSS) mechanism combines windowed audio-visual cross-attention for local synchronization with sink-mediated inter-shot self-attention for cross-shot consistency. This design enables the model to maintain character identity and acoustic consistency across shots while respecting shot boundaries and reducing the computational cost of dense global attention.
  2. Controlled ablation studies and comprehensive empirical evaluation. The paper provides controlled ablation studies of MSS and Audio-Visual Latent Stitching (AVLS) under matched backbone and training settings, demonstrating their contributions to cross-shot consistency and generation quality. Additional audio-specific evaluations, continuation strategy comparisons, long-horizon generation experiments, and human preference studies further support the effectiveness of the proposed framework.
Critical weaknesses
  1. Limited comparison with strong audiovisual baselines. The main audiovisual evaluation compares only against StoryDiffusion + MOVA and IC-LoRA + MOVA. Although Appendix G includes LTX-2, it reports only two metrics and shows a modest synchronization improvement. Together with the 48.56% lip-sync preference against MOVA, this leaves the system’s advantages over strong contemporary audiovisual generators insufficiently established.

  2. Insufficient qualitative comparisons. The supplementary videos showcase StoryWeaver without readily accessible, prompt-matched baseline outputs. This limits direct assessment of the claimed improvements in voice consistency and cross-shot transitions, which are difficult to judge from static figures and aggregate metrics alone.

My main concern is that comparisons with strong contemporary audiovisual generation systems are limited, making the method’s practical competitiveness difficult to assess. The small number of supplementary videos and lack of baseline videos further limit direct qualitative assessment. While the controlled MOVA experiments support the proposed components, evidence on other backbones remains preliminary, leaving it unclear whether these gains transfer reliably to stronger base models.

Questions for authors
  1. Could the authors clarify the baseline selection and the feasibility of comparison with stronger recent audiovisual generators, such as recent LTX variants or Seedance?
  2. Could the authors specify the LTX-2 checkpoint and adaptation budget, and report audio quality and speaker similarity to better assess generalization beyond MOVA?
  3. Given the modest synchronization gains, are the main benefits primarily cross-shot consistency? Could uncertainty estimates help clarify this distinction?
  4. Could the authors provide an anonymous comparison page with full, prompt-matched audiovisual outputs from StoryWeaver and the main baselines, including representative good examples and failures?
Reviewer_iSACOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StoryWeaver addresses multi-shot audiovisual generation with consistent character appearance, voice, and synchronized audio. Built on MOVA, the framework introduces two components. Multi-Modal Sink Shot (MSS) combines shot-local audio–visual cross-attention with per-shot, modality-specific sink tokens that mediate cross-shot communication. These sink states persist across transformer blocks and denoising steps within each generation window. Audio-Visual Latent Stitching (AVLS) extends generation across windows by caching overlapping audio and video latents, re-noising and clamping them during sampling, and training with a corresponding masked continuation objective. The authors curate 350K multi-shot training examples and introduce WeaverBench, comprising 20 synthetic reference characters in 10 situations. Evaluation includes visual storytelling on ST-Bench, audiovisual generation on WeaverBench, component and continuation ablations, human preferences, generation up to 180 seconds, and preliminary transfer to LTX-2. The results indicate improved visual identity and voice consistency relative to the evaluated baselines.

Critical strengths

A practically relevant task supported by complementary evaluations. The paper addresses visual identity, acoustic continuity, and synchronization jointly rather than evaluating visual storytelling alone. On WeaverBench, StoryWeaver improves FaceSIM from 0.82 to 0.87 and DeSync from 0.552 to 0.446 relative to StoryDiffusion + MOVA. The human study provides complementary evidence, with particularly strong preferences for voice consistency and transition naturalness. These results support the practical usefulness of the complete system.

Meaningful controls and continuation analysis. Appendix K explicitly states that the Table 4 variants share initialization, data, training steps, and inference settings, strengthening attribution beyond a comparison with an unadapted backbone. Table 8 additionally separates decode–reencode continuation, clean latent prefixes, noisy prefixes, and the complete AVLS procedure. The long-horizon and preliminary LTX-2 experiments further examine the scope and limitations of the method.

Critical weaknesses

1.The training of persistent sink states is not adequately defined, undermining the claimed training–inference consistency. Section 3.3 and Algorithms 1–2 specify that sinks retain their hidden states across denoising steps. In contrast, the flow-matching objectives describe training at a randomly sampled noise level, without explaining how the preceding sink-state trajectory is constructed. A single forward pass initialized from learned sinks would expose training to a different state distribution from inference, where sinks accumulate information over many denoising evaluations.

2.The evidence does not sufficiently isolate the audio-specific contribution of MSS. Tables 4–5 evaluate the main components and MSS submodules using exclusively visual ST-Bench metrics. They therefore cannot directly establish that audio sinks preserve speaker identity or that windowed attention suppresses acoustic leakage. Table 7 reports audio metrics for complete systems, and Table 8 provides useful continuation comparisons, but neither isolates the audio sinks or windowing mechanism. Moreover, the global cache explicitly contains both audio and video summaries from all shots, so local cross-attention windows alone do not guarantee that unrelated acoustic content cannot propagate through the cache. The paper promises adjacent-shot transition metrics, but no dedicated objective boundary/leakage metric is clearly defined or reported.

Questions for authors

1.During a randomly sampled training timestep, how are the incoming sink states obtained? Are earlier denoising steps unrolled, simulated, or omitted? Please specify gradient truncation/detachment and compare persistent sinks against sinks reset at every denoising step under matched training conditions.

2.Can you report SpkSim, synchronization, and boundary leakage results for variants without audio sinks, without video sinks, and without temporal windowing? How does the global sink cache preserve timbre while avoiding transmission of shot-specific dialogue or sound events?

3.Equations 10–11 appear to construct training context from ground-truth adjacent windows, whereas inference uses self-generated context. Does training include generated or perturbed prefixes? Please distinguish matching the masking/noising interface from eliminating the remaining context-distribution mismatch.

  1. Is AVLS enabled in every Table 5 row? Please also specify the FAD embedding and reference dataset, speaker segmentation/matching protocol, and uncertainty estimates. For the human study, each baseline comparison appears to contain 600 binary judgments; how are percentages such as 71.47% aggregated from those judgments?
Reviewer_rjscOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents StoryWeaver, a framework for consistent multi-shot audio-video generation. The main goal is to maintain character identity, audio continuity, and audiovisual synchronization across different shots.

The method consists of two components. Multi-Modal Sink Shot (MSS) combines local audio-video cross-attention with sink-mediated inter-shot communication to capture both local synchronization and cross-shot consistency. Audio-Visual Latent Stitching (AVLS) connects consecutive generation windows in the joint audio-video latent space to reduce error accumulation during long-form generation.

The authors build on MOVA and construct a multi-shot training dataset containing 350K examples. Experiments are conducted on ST-Bench and the proposed WeaverBench, with evaluations of visual consistency, identity preservation, audio-video synchronization, and audio quality. The paper also includes ablation studies, long-horizon generation experiments, and human preference evaluations.

Critical strengths
  1. The proposed MSS and AVLS designs are reasonable. MSS separates local audiovisual synchronization from cross-shot information exchange, while AVLS connects consecutive generation windows directly in the joint audio-video latent space. Both components target specific problems and are relatively easy to understand. In particular, handling the audio and video context together during continuation seems useful for maintaining audiovisual consistency.

  2. The paper includes a relatively broad set of experiments. Besides the main comparisons on ST-Bench and WeaverBench, the authors provide component ablations, comparisons between continuation strategies, longer generation experiments at 120 and 180 seconds, and a transfer experiment on another backbone. These experiments provide some evidence for the effectiveness of the proposed components, although I still have concerns about several aspects of the evaluation.

Critical weaknesses
  1. The paper lacks systematic qualitative comparisons, especially on WeaverBench. This is one of my main concerns. The authors introduce WeaverBench and report improvements in identity preservation, audiovisual synchronization, and video quality in Table 3. However, I could not find corresponding side-by-side qualitative comparisons on this benchmark in the paper.

Figures 1 and 5 mainly show results generated by StoryWeaver itself, without comparisons against other methods under the same prompts. For multi-shot audiovisual generation, I think qualitative comparisons are particularly important. Metrics such as FaceSIM, DeSync, and Sync-C cannot fully capture whether characters remain visually consistent, whether transitions between shots look natural, or whether voices and background audio remain stable.

  1. The training mechanism for persistent sink tokens is not sufficiently clear. The authors mention that the MSS sink tokens persist and are updated across denoising steps, rather than being reinitialized at every step. However, standard flow-matching training typically samples a single random timestep, so I am not sure how the model learns this cross-step state accumulation during training.

  2. The train-test matching of AVLS and the ablation settings need clarification. AVLS uses ground-truth history latents during training but generated latents during inference. Therefore, although the stitching format is matched, the distribution gap may still remain, especially for longer sequences.

Additionally, the Full MSS results in Table 5 are identical to the complete StoryWeaver results in Table 4, rather than MOVA + MSS. It is unclear whether AVLS is enabled in Table 5, making it difficult to isolate the contribution of MSS.

Questions for authors
  • Qualitative comparisons: Could the authors provide side-by-side qualitative comparisons on WeaverBench under the same prompts, especially for character reappearance, cross-shot dialogue, and audio continuity? The current figures mainly show results from StoryWeaver, making it difficult to assess the perceptual differences between methods.

  • Persistent sink training: How are sink tokens trained to maintain their states across denoising steps? Does training involve multiple denoising steps, or are the sink tokens only updated within a single sampled timestep? Have the authors compared this design with reinitializing sink tokens at each denoising step?

  • AVLS train-test mismatch: During training, AVLS uses ground-truth history latents, while inference relies on previously generated latents. How do the authors account for this remaining distribution mismatch? Have they considered using self-generated histories during training or evaluation?

  • MSS ablation settings: Why are the Full MSS results in Table 5 identical to the complete StoryWeaver results in Table 4? Is AVLS enabled for all configurations in Table 5? Clarifying this would help readers understand the contribution of each MSS sub-module.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.