- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SpongeBob, a framework for subject-level joint audio-visual editing. Given a video, a mask of the target subject, a reference image, and new visual and audio descriptions, the model regenerates the masked visual region together with the subject's sound, while preserving the unedited frames and the background audio.
The central idea is to cast editing as self-supervised inpainting. During training, the target subject is masked out in the video and its sound is separated with SAM-Audio, and the model learns to reconstruct both from their original descriptions. At inference, editing is achieved simply by providing a different description, so no paired (pre-edit, post-edit, instruction) data is needed for training.
SpongeBob is a dual-stream DiT built on Wan2.2-TI2V-5B with bidirectional cross-modal attention between the two modalities, and introduces three designs on top of it:
- Sync-Aware Editing Mechanism: the condition video, target video, and audio share a unified temporal RoPE alignment, and mask-guided asymmetric routing restricts audio-to-video injection to the masked region.
- Context-Aware Module: the target audio attends, through zero-initialized cross-attention layers, to the masked video and to the background audio, respectively.
- Sync-Preserving Training and Guidance (SPTG): four training modes are mixed during training, and a two-stage guidance scheme is applied at inference (context CFG followed by sync CFG).
The authors build a 400K-sample dataset with an automated pipeline and introduce SpongeBob-Bench with 700 test samples. The proposed method outperforms the compared methods on most evaluation metrics.
Critical strengths
-
Bidirectional joint editing of both modalities. In AVI-Edit, audio serves only as a one-way condition for video editing. In contrast, SpongeBob edits the two modalities together with bidirectional interaction, so that the edited visual motion and the generated sound can constrain each other. The paper also identifies a concrete and practically relevant problem, namely that newly generated sound may conflict with the preserved original audio, and introduces a dedicated module to address it.
-
Broad experimental evaluation. The experiments include per-subset results, an external benchmark, component ablations, a user study, and supplementary analyses of separation leakage, background-audio preservation, long-video editing, and inference cost.
-
Valuable dataset and benchmark. The 400K subject-level audio-visual samples, with separated target audio, background audio, and per-frame masks, are a useful resource for this task. The release of the benchmark should help drive follow-up research in this area.
Critical weaknesses
-
The first claimed contribution overlaps with existing editing paradigms. The paper lists "recasting audio-visual editing as mask-based self-supervised inpainting" as its first contribution. However, training an editing model through mask conditioning and reconstruction is already the paradigm adopted by related work such as AVI-Edit, as well as by mask-based video editing methods such as VACE. The authors should clarify what fundamentally distinguishes their formulation from these existing approaches.
-
Close relation to recent joint audio-video generation architectures. The architecture of SpongeBob is closely related to recent joint audio-video generation models. The dual-stream DiT with bidirectional cross-attention, the audio branch that reuses the video DiT structure, and the scaled temporal RoPE alignment are all quite similar to Ovi and LTX-2; indeed, the InstructAV2AV baseline itself is initialized from Ovi. In my view, the strength of this work lies in joint editing across the two modalities and in the context-aware design, and the contribution statement and the related-work section should be revised accordingly.
-
Limited technical contribution of the data pipeline. The data pipeline is a combination of existing tools (Gemini, Grounded-SAM-2, SAM-Audio) and is not particularly new by today's standards. That said, the resulting dataset itself is a contribution.
Questions for authors
-
Drastic changes to the masked region. Does SpongeBob-Bench include cases where the content of the masked region is changed drastically, e.g., replacing an animal with a human, or a human with a car? Since the model is trained on a reconstruction task, does its performance degrade when the target description differs substantially from the original content of the region? Results on such cases would help clarify the model's editing capability.
-
Ablation of each auxiliary training mode. The paper introduces multi-task alignment training to mitigate correspondence drift. To justify each additional mode (audio-driven, video-driven, and context-null), please provide ablations that remove each mode individually while keeping the other key settings (e.g., training steps and the remaining sampling probabilities) as fixed as possible.
Thank you for the detailed response. The equal-budget comparison, the schedule-reversal study, and the shifted-anchor experiment address my concerns about computation and the guidance mechanism. However, the response confirms substantial overlap with Harmony, and most of the synchronization gain comes from the inherited SyncCFG. The new adherence metrics (Gemini judgment, ArcFace/DINOv2, WER) are reported without baselines, so they don’t show better edit correctness. The benchmark also largely excludes distant cross-category edits. I therefore maintain my original rating.