- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TV-AudioRemover for selective sound removal after visual object editing. The framework combines text-visual guidance with a flow-matching architecture. It introduces task tokens, modality-specific global guidance, and multi-task hard-mixture curriculum learning. The authors construct a million-scale training corpus and a 77-sample evaluation benchmark. Experiments show promising improvements in sound removal, audio preservation, and audio-visual consistency.
Critical strengths
-
The paper addresses the mismatch between audio and video after visual objects are removed. The proposed framework uses the edited video to guide the removal of the corresponding sounds.
-
The authors construct a large-scale training dataset through multimodal annotation, source separation, and quality filtering.
-
Table 1 shows that the proposed method achieves higher TSSR, PSF, and IB-AV scores than SAM Audio and MMAudioSep.
Critical weaknesses
-
The paper does not clearly distinguish its contribution from prior work, and several relevant baselines are missing. OmniSep [1] already supports positive and negative multimodal queries. SAM Audio [2] provides residual audio for target removal. SAVE [3] directly studies object-level audio-visual removal. The paper lacks comparisons with OmniSep and SAVE, and Section 4.1 does not consistently evaluate SAM Audio through direct residual-based removal.
-
The evidence for the benefits of visual guidance is limited. Table 4 shows identical SAJ scores (3.46) for text-only and text-visual inputs. TSSR improves only from 0.63 to 0.64, and PSF improves from 1.07 to 1.08. The main improvements concern audio-visual consistency.
-
The ablation study does not isolate the effects of several key components. Table 4 lacks independent ablations for task tokens, generalized instructions, global guidance, and synchronization features.
References:
[1] Cheng et al. OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup. ICLR, 2025.
[2] Shi et al. SAM Audio: Segment Anything in Audio. arXiv:2512.18099, 2025.
[3] Xu et al. Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removal. arXiv:2512.12875, 2025.
Questions for authors
Q1. What is the main technical distinction from OmniSep and SAM Audio? Can the authors evaluate SAM Audio using direct residual-based removal? Can the authors also explain the absence of OmniSep and SAVE from the comparisons?
Q2. Effectiveness of visual guidance. Why does visual conditioning provide only marginal improvements in TSSR and PSF in Table 4?
Q3. Can the authors provide separate ablations for task tokens, instruction modeling, global guidance, and synchronization features?
Q4. Metric consistency and evaluation protocol. Why do TSSR and PSF differ by approximately tenfold between Tables 1 and 4? What scaling is applied to IB-AV? How are AST categories mapped to fine-grained target sounds? What is the effective sample count for Table 2?
7 comments
A2 (For W2 and Q2)
A2: (For W2 and Q2)
Thank you for your attention to the visual branch. Limited gains of visual conditioning on certain editing metrics do not imply weak efficacy. This is a known difficulty in GT-free audio editing evaluation: current metrics are insensitive to fine-grained source discrimination, though they work for coarse sound categories. In a male–female dialogue example targeting female voice removal, text-only conditioning often fails to fully suppress the female voice even when speakers talk at separate times, as it cannot reliably link the editing target to the desired source. Visual conditioning provides agent and temporal activity cues to separate sources, yet such benefits may evade existing metrics. In our example, text-only incomplete removal yields SAJ = 3.4, while successful audio-visual removal yields SAJ = 3.3, showing SAJ is limited by audio-text matching and description granularity. TSSR and similar metrics are constrained by sound detector accuracy; both underestimate fine-grained source-selection mistakes. Audio-visual synchrony metrics offer complementary clues: residual sound after the target leaves the frame becomes asynchronous with visible speakers. This supports improved post-editing cross-modal consistency but cannot serve as standalone causal ablation evidence for the visual branch. We therefore evaluate jointly over semantic editing, target fidelity and audio-visual synchrony rather than relying on one metric. Human subjective scoring remains the most reliable measure, with listening procedures and statistics provided under “Subjective Evaluation Statistics” in the supplement.
Second, we agree that audio–visual consistency alone cannot confirm whether the visual branch helps locate the intended sound source. We therefore introduce four visual-condition control settings: correctly matched videos, semantically mismatched videos, time-shuffled original videos, and original videos with a 2 s visual shift. Mixed audio, prompt, parameters, inference budget, latent sampling and initial noise are fixed to isolate visual effects. Semantic mismatch tests reliance on visual agents; shuffling and shifting test temporal alignment between visual vocalization and audio events. All outputs are referenced against the same correct video to eliminate reference bias. Results are analyzed using source removal/preservation, synchrony and GT fidelity metrics. For instance, to keep on-screen speech and remove dog barking: matched frames show the speaker; mismatch shows barking dogs; shuffle uses 1 s chunks; shift uses a 2 s visual circular delay. All groups share the same text instruction.
| Visual Condition | SAJ Overall ↑ | TSSR ↑ | PSF ↑ | DeSync ↓ | MR-STFT↓ | REC↑ |
|---|---|---|---|---|---|---|
| Correct Edited Video | 4.48 | 9.14 | 12.98 | 0.00 | 0.92 | 0.99 |
| Semantically Mismatched Video | 1.48 | -7.21 | 4.30 | 2.00 | 5.72 | 0.24 |
| Temporally Shuffled Video | 1.04 | 8.98 | 5.06 | 2.00 | 4.69 | 0.15 |
| Temporally Displaced Video | 1.47 | 8.96 | 11.18 | 1.80 | 4.38 | 0.02 |
The contrast results on this sample demonstrate that semantic mismatch weakens source removal and preservation performance, and temporal perturbation hurts target sound fidelity and synchrony. These findings support that the visual branch contributes to both source selection and temporal guidance, instead of only boosting holistic audio–video compatibility. As future work, we plan to replicate these visual-condition control experiments on a larger, category-diverse test set.
A3 (For W3 and Q3)
A3: (For W3 and Q3)
We thank the reviewer very much for the reminder. Task embeddings, instruction diversification, modality-specific global guidance, and hard-mixture fine-tuning are exactly the designs we introduced incrementally during model training and iteration. The original manuscript mainly presented the overall performance of the final model and did not fully show the role of these designs during iteration. Your suggestion helps us state each design's contribution more clearly. To this end, the revision will add four controls evaluated on the same test set: without task embeddings, without instruction diversification, without modality-specific global guidance, and fine-tuning with generic random-mixture data under the same number of training steps.
| Method | SAJ Overall ↑ | TSSR ↑ | PSF ↑ | Instr. Comp._a ↑ | Fidelity_a ↑ | IB-AV ↑ | DeSync ↓ |
|---|---|---|---|---|---|---|---|
| w/o Task Embeddings | 3.22 | 2.88 | 9.84 | 4.16 | 3.95 | 25.75 | 0.68 |
| w/o Instruction Diversification | 3.29 | 3.57 | 9.99 | 4.34 | 4.13 | 25.89 | 0.75 |
| w/o Modality-Specific Global Guidance | 3.50 | 5.97 | 10.05 | 4.66 | 3.94 | 24.24 | 0.74 |
| Random-Mixture Fine-Tuning | 3.24 | 3.64 | 10.57 | 4.28 | 3.90 | 25.47 | 0.71 |
| full | 3.46 | 6.35 | 10.79 | 4.79 | 3.96 | 27.32 | 0.60 |
The comparison of these iterative versions shows that, without task embeddings, the model is more prone to removal failure and over-removal. Task embeddings help specify the editing task, while instruction diversification strengthens the model's adaptability to different instruction phrasings, letting it perform the intended edit more stably. Modality-specific global guidance helps make effective use of visual and text conditions, improving the synchronization and consistency between the edited sound and the video frames. Furthermore, under the same number of training steps, compared with generic random-mixture data, hard-mixture fine-tuning further strengthens the model's ability to distinguish confusable sources, improving the suppression of the sound to remove and the fidelity of the preserved sound.
A1 (For W1 and Q1)
Thanks for the valuable comments. Our responses are as follows:
A1: (For W1 and Q1)
OmniSep is an ImageBind-based, negative-query-capable mask-discriminative separation model that supports only static-image queries, lacks video temporal interaction, and struggles to localize sources from consecutive frames; SAM-Audio adopts a flow-matching generative paradigm and, with its self-developed PE-AV encoder, can leverage temporal information for interactive source extraction. Since SAVE releases neither code nor weights, a reproducible fair comparison is infeasible; OmniSep supports only static images as visual queries, and its negative-text query mechanism is similar to but weaker than MMAudioSep's—so neither is included in the original main comparison. For reference, we additionally report OmniSep on AV-Remove-Bench, using the edited-video first frame together with negative text (for test data composed of two foreground sources, the source to keep is also fed to the positive query).
For SAM Audio: for multi-sounding-object cases, we feed SAM-Audio the visual mask of the source to keep (the intermediate result of the visual-removal pipeline), the original audio to be edited, and the text instruction for extracting the target source to keep, and directly take the model's extraction output as the sound-removal result. We also tested a residual scheme: feeding the visual mask of the source to remove, the audio to be edited, and the text instruction for extracting that removal target, and taking the model's residual audio as the sound-removal result. However, this scheme has a clear drawback: the residual audio usually still carries residual signal of the removal target and fails to preserve the target source to keep intact. For foreground-target-plus-background cases, since the audio to keep cannot be abstracted as a single, well-defined extraction target, we uniformly adopt the residual paradigm: feeding the visual mask of the source to remove, the audio to be edited, and the corresponding text instruction, and taking the residual audio as the sound-removal output. Below we report results under the all-residual setting.
We adapted and tuned SAM-Audio for the sound-removal task across different scenarios. Combined with the results above, its flexibility and final performance on this task are both weaker than those of a model designed specifically for sound removal.