acceptodds
Under review as a conference paper at ICLR 2027

SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing

Abstract

Visual and acoustic events in the physical world are inherently coupled, yet existing video editing methods typically adopt decoupled pipelines, lacking bidirectional modality interaction. This results in two key limitations: (i) audio-visual desynchronization and (ii) contextual conflicts between generated audio and preserved content. To address these, we propose **SpongeBob**, an end-to-end audio-visual joint editing framework featuring bidirectional cross-modal interaction. For synchronization, a Sync-Aware Editing Mechanism aligns visual edits with sound events via bidirectional attention, temporal alignment, and spatial constraints. For contextual consistency, a Context-Aware Module leverages acoustic and visual context attention to prevent semantic clashes. Additionally, we introduce Sync-Preserving Training and Guidance (SPTG) to enhance alignment without degrading quality. Due to the scarcity of paired data, we construct a scalable data pipeline and a large-scale subject-level dataset. We also propose SpongeBob-Bench for systematic evaluation. Experiments show SpongeBob significantly outperforms existing baselines, improving Sync-C by 19.0% and Ctx-F1 by 9.5%.

Primary area:
generative models
Keywords:
Instruction-Guided Video Edit, Video Edit, Audio-Visual Edit

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5FCYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpongeBob, a framework for subject-level joint audio-visual editing. Given a video, a mask of the target subject, a reference image, and new visual and audio descriptions, the model regenerates the masked visual region together with the subject's sound, while preserving the unedited frames and the background audio.

The central idea is to cast editing as self-supervised inpainting. During training, the target subject is masked out in the video and its sound is separated with SAM-Audio, and the model learns to reconstruct both from their original descriptions. At inference, editing is achieved simply by providing a different description, so no paired (pre-edit, post-edit, instruction) data is needed for training.

SpongeBob is a dual-stream DiT built on Wan2.2-TI2V-5B with bidirectional cross-modal attention between the two modalities, and introduces three designs on top of it:

  1. Sync-Aware Editing Mechanism: the condition video, target video, and audio share a unified temporal RoPE alignment, and mask-guided asymmetric routing restricts audio-to-video injection to the masked region.
  2. Context-Aware Module: the target audio attends, through zero-initialized cross-attention layers, to the masked video and to the background audio, respectively.
  3. Sync-Preserving Training and Guidance (SPTG): four training modes are mixed during training, and a two-stage guidance scheme is applied at inference (context CFG followed by sync CFG).

The authors build a 400K-sample dataset with an automated pipeline and introduce SpongeBob-Bench with 700 test samples. The proposed method outperforms the compared methods on most evaluation metrics.

Critical strengths
  1. Bidirectional joint editing of both modalities. In AVI-Edit, audio serves only as a one-way condition for video editing. In contrast, SpongeBob edits the two modalities together with bidirectional interaction, so that the edited visual motion and the generated sound can constrain each other. The paper also identifies a concrete and practically relevant problem, namely that newly generated sound may conflict with the preserved original audio, and introduces a dedicated module to address it.

  2. Broad experimental evaluation. The experiments include per-subset results, an external benchmark, component ablations, a user study, and supplementary analyses of separation leakage, background-audio preservation, long-video editing, and inference cost.

  3. Valuable dataset and benchmark. The 400K subject-level audio-visual samples, with separated target audio, background audio, and per-frame masks, are a useful resource for this task. The release of the benchmark should help drive follow-up research in this area.

Critical weaknesses
  1. The first claimed contribution overlaps with existing editing paradigms. The paper lists "recasting audio-visual editing as mask-based self-supervised inpainting" as its first contribution. However, training an editing model through mask conditioning and reconstruction is already the paradigm adopted by related work such as AVI-Edit, as well as by mask-based video editing methods such as VACE. The authors should clarify what fundamentally distinguishes their formulation from these existing approaches.

  2. Close relation to recent joint audio-video generation architectures. The architecture of SpongeBob is closely related to recent joint audio-video generation models. The dual-stream DiT with bidirectional cross-attention, the audio branch that reuses the video DiT structure, and the scaled temporal RoPE alignment are all quite similar to Ovi and LTX-2; indeed, the InstructAV2AV baseline itself is initialized from Ovi. In my view, the strength of this work lies in joint editing across the two modalities and in the context-aware design, and the contribution statement and the related-work section should be revised accordingly.

  3. Limited technical contribution of the data pipeline. The data pipeline is a combination of existing tools (Gemini, Grounded-SAM-2, SAM-Audio) and is not particularly new by today's standards. That said, the resulting dataset itself is a contribution.

Questions for authors
  1. Drastic changes to the masked region. Does SpongeBob-Bench include cases where the content of the masked region is changed drastically, e.g., replacing an animal with a human, or a human with a car? Since the model is trained on a reconstruction task, does its performance degrade when the target description differs substantially from the original content of the region? Results on such cases would help clarify the model's editing capability.

  2. Ablation of each auxiliary training mode. The paper introduces multi-task alignment training to mitigate correspondence drift. To justify each additional mode (audio-driven, video-driven, and context-null), please provide ablations that remove each mode individually while keeping the other key settings (e.g., training steps and the remaining sampling probabilities) as fixed as possible.

Reviewer_9FGFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to edit a person or object in a video together with its sound, while preserving the other people, visual content, and background audio. Training is formulated as reconstructing a masked subject, which avoids the need for a large collection of paired before-and-after edits. The model generates video and target audio jointly, using aligned temporal positions, restrictions on the editing region, and attention to the remaining visual and acoustic context. SPTG combines several training tasks and strengthens context conditioning early in generation, followed by audio-video synchronization later. The paper uses approximately 400K training samples and reports improvements in quality, synchronization, and context consistency on the 700-sample SpongeBob-Bench. The appendices also include an external benchmark, human evaluation, and robustness studies.

Critical strengths
  1. The paper addresses a practical problem: editing the target person or object is only part of the task, since the new sound also needs to fit the other speakers and background audio. The method brings the visual edit, target sound, and preserved scene into one framework, while constructing training examples from ordinary videos to reduce reliance on paired edits. This is a useful direction for audio-visual editing.

  2. The experiments provide several kinds of support for the complete system. In Table 1, Sync-C increases from 3.78 for the strongest baseline to 4.50, and Ctx-F1 increases from 0.74 for the best baseline on that metric to 0.81. Beyond the main benchmark and component ablations, Appendices F-I cover external evaluation, human judgments, non-speech synchronization, source-separation errors, and longer videos. The evaluation is reasonably broad.

Critical weaknesses
  1. I am still unsure how much SPTG adds beyond existing synchronization methods. Its three basic training tasks and synchronization guidance using muted audio or static video as comparison conditions closely resemble Harmony (CVPR 2026), Sections 3.2.2 and 3.4. The context-related additions here are well motivated, but the current manuscript does not discuss this relationship or clearly separate the effects of the new parts. There is also a compute difference: Table 5 compares guidance strategies on the same trained model, while Appendix B.3 uses 140 model forward passes for the full method versus 100 for standard CFG. This makes it harder to tell how much of the improvement comes from the guidance design and how much may be related to the additional computation.

  2. The reported metrics give useful evidence about output quality and synchronization, but I would like a more direct check of whether the requested edits are carried out correctly. For example, replacing a dog with a white cat requires checking the requested object and appearance, as well as whether the result looks and sounds natural and synchronized. Appendix J acknowledges that training mainly reconstructs the original subject and that cross-category replacement relies on generalization. However, Appendices C.1-C.3 do not give the proportions and separate success rates of within-category and cross-category edits, or directly report adherence to reference appearance or identity and the requested speech text. Ctx-F1 is also described rather briefly in Appendix C.3: the reference intervals for conflict rate and target-audio coverage, and the way target and non-target speakers are distinguished, are not fully clear. These details matter when replacement utterances have different lengths or overlapping speech is appropriate for the scene. The human study in Appendix G supports an overall improvement, but these specific questions remain.

Reference: Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy, CVPR 2026. https://arxiv.org/html/2511.21579

Questions for authors
  1. [Clarifying the contribution] Could you explain the similarities and differences between SPTG and Harmony, Sections 3.2.2 and 3.4? What improvements come from the added context training, context guidance, and the schedule that handles context before strengthening synchronization? If there are relevant comparisons already available, please point to them.

  2. [Checking edit success] How many of the 700 test examples involve within-category changes versus cross-category replacements (Appendices C.1 and J)? Could you report, on the relevant subsets of the existing test set, whether the requested object or appearance is produced and whether the reference identity is followed? For speech edits, an error rate against the requested transcript, such as WER or CER, would also help establish whether the intended edit was completed.

  3. [Explaining Ctx-F1] What is the denominator of the conflict rate in Appendix C.3? Is target-audio coverage measured against the original speaker's active intervals or some other reference intervals? How are target and non-target speakers distinguished after generation? For example, how is a two-second utterance replaced by a four-second one evaluated, or a scene where two speakers are allowed to overlap? Please provide the calculation rules and thresholds so the metric can be understood and reproduced.

  4. [Separating design from computation] Appendix B.3 uses 140 forward passes for the full method and 100 for standard CFG. Does SPTG retain an advantage under the same computation budget? Also, were the muted-audio and static white-video conditions in Equations (9)-(10) seen during training? Results with alternative comparison conditions would help clarify whether the guidance mainly improves accurate audio-video correspondence or more generally increases motion or sound activity.

1 comment

Thank you for the detailed response. The equal-budget comparison, the schedule-reversal study, and the shifted-anchor experiment address my concerns about computation and the guidance mechanism. However, the response confirms substantial overlap with Harmony, and most of the synchronization gain comes from the inherited SyncCFG. The new adherence metrics (Gemini judgment, ArcFace/DINOv2, WER) are reported without baselines, so they don’t show better edit correctness. The benchmark also largely excludes distant cross-category edits. I therefore maintain my original rating.

Reviewer_8mF2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SpongeBob, a framework for subject-level audio and video editing that avoids the need for paired editing examples by formulating training as joint inpainting. The model uses a dual-stream DiT with temporal alignment, mask-guided interaction, and conditioning on preserved visual and acoustic content. Multi-task training and two-stage guidance are further introduced to improve generation. The authors construct 400K training samples and a 700-sample benchmark. Experiments show improvements over several baselines, supported by ablations, external evaluations, robustness tests, and a human study.

Critical strengths
  • Practical data construction. Formulating editing as conditional reconstruction reduces the need for paired source–target examples. The pipeline combines segmentation, source separation, and quality filtering to construct 400K training samples, with manual checks providing additional support for data quality. This is a useful approach to addressing the lack of suitable editing data.

  • Effective use of preserved context. Conditioning generation on both the unedited video and the remaining background audio is a sensible way to maintain consistency with the original scene. The ablation results show improvements in context consistency, synchronization, and perceptual quality, supporting this design choice.

  • Effective inference guidance. The authors compare different guidance strategies using the same trained model, making the results relatively easy to interpret. Combining context and synchronization guidance gives the best overall performance. The external evaluation and human study also provide additional support for the system's effectiveness.

Critical weaknesses
  • Limited evidence for context consistency. In Table 4, visual context accounts for most of the Ctx-F1 improvement (0.75 to 0.80), while acoustic context alone lowers it to 0.72. Combining both reaches 0.81. Although acoustic context helps other metrics, its role in preventing conflicts is less clear. Moreover, Ctx-F1 focuses on temporal overlap with non-target speakers and target audio coverage, rather than broader acoustic compatibility such as loudness or reverberation. Reporting conflict and coverage separately would help clarify whether the improvement mainly comes from reducing excessive silence.

  • Unclear synchronization mechanism of SPTG. During training, the audio-driven and video-driven branches use real clean modalities, while inference uses silent audio or repeated white frames as negative anchors (Eqs. 9–10). This may introduce a training–inference mismatch. The negative anchors also change semantic content and potentially noise levels, making it difficult to attribute the guidance effect specifically to temporal alignment. Table 5 shows that SPTG improves performance, but does not fully support the proposed synchronization interpretation. Using temporally shifted negative anchors with matched content and noise levels would provide a more direct test.

  • Incomplete evaluation of editing success. For speech editing with a specified transcript, lip synchronization and perceptual quality do not indicate whether the generated speech contains the correct words. Reporting WER or CER against the target transcript would help assess content fidelity. More broadly, Appendix J acknowledges limitations in cross-category editing, but the evaluation does not clearly break down performance across different editing operations. A more detailed analysis of supported edits and failure cases would help clarify the method's scope.

Questions for authors
  • Bidirectional interaction: What evidence shows that bidirectional interaction is responsible for the improvements over cascaded pipelines? Have the authors compared unidirectional or sequential variants under matched backbone, data, conditioning, and computational settings? The current comparisons involve several differences beyond interaction direction.

  • Editing capabilities: Which editing operations are covered by SpongeBob-Bench, and which can be reliably learned through reconstruction training? In particular, could the authors clarify performance and failure cases for identity or appearance changes, transcript edits, and cross-category replacements?

Comments on the paper

1 comment

Thank you for the detailed response. The clarifications help me better understand how SpongeBob differs from VACE/AVI-Edit and from Ovi/LTX-2, and the revised contribution statement and the per-mode ablation address my questions. The new cross-category experiment is also useful, and I suggest reporting it in the paper together with the recommendation to use bounding-box masks for large shape changes. After considering the response and the other reviews, I have decided to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.