acceptodds
Under review as a conference paper at ICLR 2027

TV-AudioRemover: Joint Text‑Visual Guided Sound Removal with Multi‑Task Hard‑Mixture Curriculum

Abstract

Visual object removal can eliminate a target from video frames, yet its acoustic trace persists in the soundtrack, causing obvious audio-visual inconsistency. Existing video inpainting models operate solely on pixels, while audio editing models, especially for the sound removal task, are typically driven by text and therefore rely on limited single-modal control, which is less effective than multimodal guidance that provides stronger semantic grounding and temporal synchronization cues. In this paper, we present Text‑Visual Guided Sound Removal (TV-AudioRemover), a target sound removal framework that leverages natural-language instructions and optionally visually edited videos to suppress the sound corresponding to the removed visual object from the original audio mixture. To acquire high-quality training data, we devise a pipeline to construct a million-scale dataset of single-object audio-visual aligned samples, from which we synthesize mixture-target pairs customized for model training. To effectively leverage visual context and follow instruction intent, we augment the model architecture with task tokens, generalizable instruction modeling, and modality-specific global guidance. We further adopt multi-task training to strengthen task-role comprehension, and employ a hard-mixture curriculum that leverages semantically similar acoustic mixtures during fine-tuning to enhance fine-grained source discrimination. To support evaluation, we present AV-Remove-Bench, a comprehensive audio-visual object removal benchmark, along with dedicated objective metrics and an MLLM-based evaluation protocol. Experiments demonstrate that our method achieves state-of-the-art performance on both subjective and objective metrics.

Primary area:
generative models
Keywords:
Audio Edit, Sound Removal, Generative Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p3msOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TV-AudioRemover for selective sound removal after visual object editing. The framework combines text-visual guidance with a flow-matching architecture. It introduces task tokens, modality-specific global guidance, and multi-task hard-mixture curriculum learning. The authors construct a million-scale training corpus and a 77-sample evaluation benchmark. Experiments show promising improvements in sound removal, audio preservation, and audio-visual consistency.

Critical strengths
  1. The paper addresses the mismatch between audio and video after visual objects are removed. The proposed framework uses the edited video to guide the removal of the corresponding sounds.

  2. The authors construct a large-scale training dataset through multimodal annotation, source separation, and quality filtering.

  3. Table 1 shows that the proposed method achieves higher TSSR, PSF, and IB-AV scores than SAM Audio and MMAudioSep.

Critical weaknesses
  1. The paper does not clearly distinguish its contribution from prior work, and several relevant baselines are missing. OmniSep [1] already supports positive and negative multimodal queries. SAM Audio [2] provides residual audio for target removal. SAVE [3] directly studies object-level audio-visual removal. The paper lacks comparisons with OmniSep and SAVE, and Section 4.1 does not consistently evaluate SAM Audio through direct residual-based removal.

  2. The evidence for the benefits of visual guidance is limited. Table 4 shows identical SAJ scores (3.46) for text-only and text-visual inputs. TSSR improves only from 0.63 to 0.64, and PSF improves from 1.07 to 1.08. The main improvements concern audio-visual consistency.

  3. The ablation study does not isolate the effects of several key components. Table 4 lacks independent ablations for task tokens, generalized instructions, global guidance, and synchronization features.

References:

[1] Cheng et al. OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup. ICLR, 2025.

[2] Shi et al. SAM Audio: Segment Anything in Audio. arXiv:2512.18099, 2025.

[3] Xu et al. Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removal. arXiv:2512.12875, 2025.

Questions for authors

Q1. What is the main technical distinction from OmniSep and SAM Audio? Can the authors evaluate SAM Audio using direct residual-based removal? Can the authors also explain the absence of OmniSep and SAVE from the comparisons?

Q2. Effectiveness of visual guidance. Why does visual conditioning provide only marginal improvements in TSSR and PSF in Table 4?

Q3. Can the authors provide separate ablations for task tokens, instruction modeling, global guidance, and synchronization features?

Q4. Metric consistency and evaluation protocol. Why do TSSR and PSF differ by approximately tenfold between Tables 1 and 4? What scaling is applied to IB-AV? How are AST categories mapped to fine-grained target sounds? What is the effective sample count for Table 2?

7 comments
AuthorsOpenReview ↗

A1 (For W1 and Q1)

Thanks for the valuable comments. Our responses are as follows:

A1: (For W1 and Q1)

OmniSep is an ImageBind-based, negative-query-capable mask-discriminative separation model that supports only static-image queries, lacks video temporal interaction, and struggles to localize sources from consecutive frames; SAM-Audio adopts a flow-matching generative paradigm and, with its self-developed PE-AV encoder, can leverage temporal information for interactive source extraction. Since SAVE releases neither code nor weights, a reproducible fair comparison is infeasible; OmniSep supports only static images as visual queries, and its negative-text query mechanism is similar to but weaker than MMAudioSep's—so neither is included in the original main comparison. For reference, we additionally report OmniSep on AV-Remove-Bench, using the edited-video first frame together with negative text (for test data composed of two foreground sources, the source to keep is also fed to the positive query).

GroupMethodIS ↑SAJ Overall ↑TSSR ↑PSF ↑Instr. Comp._a ↑Fidelity_a ↑IB-AV ↑DeSync ↓ParamsTimes (s)
T-AVAVI-Edit3.162.242.609.583.562.5719.220.777.24B255.89
T-AVInstructAV2AV3.452.60-2.889.862.653.0414.810.7615.63B299.55
T-AZETA3.272.955.7110.543.693.2218.710.751.5B86.98
T-AAudio-Omni2.152.204.969.014.391.688.231.133.05B6.18
T-AUNISON3.632.77-2.399.994.273.4418.000.78731.71M5.82
IT-AOmniSep3.552.160.5910.133.753.0821.670.83116M0.85
VT-ASAM Audio3.232.423.5410.073.703.9017.890.933.72B1.94
VT-AMMAudioSep3.482.672.9810.153.413.9423.490.971.03B2.03
VT-A (Ours)TV-AudioRemover3.553.466.3510.794.793.9627.320.601.04B2.07

For SAM Audio: for multi-sounding-object cases, we feed SAM-Audio the visual mask of the source to keep (the intermediate result of the visual-removal pipeline), the original audio to be edited, and the text instruction for extracting the target source to keep, and directly take the model's extraction output as the sound-removal result. We also tested a residual scheme: feeding the visual mask of the source to remove, the audio to be edited, and the text instruction for extracting that removal target, and taking the model's residual audio as the sound-removal result. However, this scheme has a clear drawback: the residual audio usually still carries residual signal of the removal target and fails to preserve the target source to keep intact. For foreground-target-plus-background cases, since the audio to keep cannot be abstracted as a single, well-defined extraction target, we uniformly adopt the residual paradigm: feeding the visual mask of the source to remove, the audio to be edited, and the corresponding text instruction, and taking the residual audio as the sound-removal output. Below we report results under the all-residual setting.

MethodSAJ Overall ↑TSSR ↑PSF ↑Instr. Comp._a ↑Fidelity_a ↑IB-AV ↑DeSync ↓
SAM Audio (full residual)2.373.579.753.693.1015.470.96
SAM Audio (partial residual)2.423.5410.073.703.9017.890.93
TV-AudioRemover3.466.3510.794.793.9627.320.60

We adapted and tuned SAM-Audio for the sound-removal task across different scenarios. Combined with the results above, its flexibility and final performance on this task are both weaker than those of a model designed specifically for sound removal.

AuthorsOpenReview ↗

A2 (For W2 and Q2)

A2: (For W2 and Q2)

Thank you for your attention to the visual branch. Limited gains of visual conditioning on certain editing metrics do not imply weak efficacy. This is a known difficulty in GT-free audio editing evaluation: current metrics are insensitive to fine-grained source discrimination, though they work for coarse sound categories. In a male–female dialogue example targeting female voice removal, text-only conditioning often fails to fully suppress the female voice even when speakers talk at separate times, as it cannot reliably link the editing target to the desired source. Visual conditioning provides agent and temporal activity cues to separate sources, yet such benefits may evade existing metrics. In our example, text-only incomplete removal yields SAJ = 3.4, while successful audio-visual removal yields SAJ = 3.3, showing SAJ is limited by audio-text matching and description granularity. TSSR and similar metrics are constrained by sound detector accuracy; both underestimate fine-grained source-selection mistakes. Audio-visual synchrony metrics offer complementary clues: residual sound after the target leaves the frame becomes asynchronous with visible speakers. This supports improved post-editing cross-modal consistency but cannot serve as standalone causal ablation evidence for the visual branch. We therefore evaluate jointly over semantic editing, target fidelity and audio-visual synchrony rather than relying on one metric. Human subjective scoring remains the most reliable measure, with listening procedures and statistics provided under “Subjective Evaluation Statistics” in the supplement.

Second, we agree that audio–visual consistency alone cannot confirm whether the visual branch helps locate the intended sound source. We therefore introduce four visual-condition control settings: correctly matched videos, semantically mismatched videos, time-shuffled original videos, and original videos with a 2 s visual shift. Mixed audio, prompt, parameters, inference budget, latent sampling and initial noise are fixed to isolate visual effects. Semantic mismatch tests reliance on visual agents; shuffling and shifting test temporal alignment between visual vocalization and audio events. All outputs are referenced against the same correct video to eliminate reference bias. Results are analyzed using source removal/preservation, synchrony and GT fidelity metrics. For instance, to keep on-screen speech and remove dog barking: matched frames show the speaker; mismatch shows barking dogs; shuffle uses 1 s chunks; shift uses a 2 s visual circular delay. All groups share the same text instruction.

Visual ConditionSAJ Overall ↑TSSR ↑PSF ↑DeSync ↓MR-STFT↓REC↑
Correct Edited Video4.489.1412.980.000.920.99
Semantically Mismatched Video1.48-7.214.302.005.720.24
Temporally Shuffled Video1.048.985.062.004.690.15
Temporally Displaced Video1.478.9611.181.804.380.02

The contrast results on this sample demonstrate that semantic mismatch weakens source removal and preservation performance, and temporal perturbation hurts target sound fidelity and synchrony. These findings support that the visual branch contributes to both source selection and temporal guidance, instead of only boosting holistic audio–video compatibility. As future work, we plan to replicate these visual-condition control experiments on a larger, category-diverse test set.

AuthorsOpenReview ↗

A3 (For W3 and Q3)

A3: (For W3 and Q3)

We thank the reviewer very much for the reminder. Task embeddings, instruction diversification, modality-specific global guidance, and hard-mixture fine-tuning are exactly the designs we introduced incrementally during model training and iteration. The original manuscript mainly presented the overall performance of the final model and did not fully show the role of these designs during iteration. Your suggestion helps us state each design's contribution more clearly. To this end, the revision will add four controls evaluated on the same test set: without task embeddings, without instruction diversification, without modality-specific global guidance, and fine-tuning with generic random-mixture data under the same number of training steps.

MethodSAJ Overall ↑TSSR ↑PSF ↑Instr. Comp._a ↑Fidelity_a ↑IB-AV ↑DeSync ↓
w/o Task Embeddings3.222.889.844.163.9525.750.68
w/o Instruction Diversification3.293.579.994.344.1325.890.75
w/o Modality-Specific Global Guidance3.505.9710.054.663.9424.240.74
Random-Mixture Fine-Tuning3.243.6410.574.283.9025.470.71
full3.466.3510.794.793.9627.320.60

The comparison of these iterative versions shows that, without task embeddings, the model is more prone to removal failure and over-removal. Task embeddings help specify the editing task, while instruction diversification strengthens the model's adaptability to different instruction phrasings, letting it perform the intended edit more stably. Modality-specific global guidance helps make effective use of visual and text conditions, improving the synchronization and consistency between the edited sound and the video frames. Furthermore, under the same number of training steps, compared with generic random-mixture data, hard-mixture fine-tuning further strengthens the model's ability to distinguish confusable sources, improving the suppression of the sound to remove and the fidelity of the preserved sound.

Reviewer_C7JyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to remove, from the soundtrack, the sound of an object that has been erased from the video. The authors propose TV-AudioRemover, an MM-DiT flow-matching model that takes the original audio mixture, a text instruction, and an optional object-removed video, and outputs the audio without the target sound. They build about 1M single-source clips using Qwen3-Omni tagging, SAM Audio separation, and quality filtering, then mix these clips online so that a single mixture supervises three tasks: extract, delete, and both. Training uses task tokens, modality-specific global conditions, and an easy-to-hard mixture curriculum. The paper also introduces AV-Remove-Bench (77 clips) along with two new metrics (TSSR, PSF) and a Gemini-based judge. On AV-Remove-Bench and the MMAE removal subset, the proposed model outperforms text-guided editors, extraction models, and joint audio-visual editors, including in human evaluation.

Critical strengths
  1. The task is useful in practice. Video object removal works quite well now, but the removed object can still be heard. This paper fills that gap. The design is also flexible: it works with any video remover, and it can take text only, video only, or both.

  2. The data and training setup are solid. One mixture gives labels for all three tasks, and the hard-mixture stage focuses on similar-sounding sources. The results are strong, also in human evaluation.

Critical weaknesses

Overall I think this is a useful work. Below are my main concerns.

  1. The paper says SAM Audio is an extraction model "not tailored for target removal". But SAM Audio already outputs a residual track, which is basically the removal result, and the authors use it this way for the foreground-plus-background cases (Section 4.1). It is also not clear what visual input SAM Audio gets, since Section 4.2 only mentions the SVOR video for TV-AudioRemover and MMAudioSep. SAM Audio can take a visual mask, and the video removal step already has the object mask. So I would like to see SAM Audio run with the original video plus this mask, taking the residual as output.

  2. I worry about how much the audio result depends on the video removal quality. In training, the visual input is the original video of the kept source, so the model never sees edited videos. In testing, only SVOR is used, which is much better than the other removers in Appendix H. What if a weaker remover is used, or the object is not fully removed from the video? Results with other removers, or a few failure cases, would help.

  3. A simpler way to do this might be reference-based generation: give the edited video and the original audio to a model like Seedance 2.5 or MiniMax H3 as references, and let it regenerate the audio. Appendix F compares with these models, but the setup is not clear. The example prompt just asks them to edit video and audio together. What exactly was given as input, and did the authors try reference-based generation?

  4. The paper only promises part of the inference code and part of the benchmark, with no model weights. I do not see much technical novelty here, but the model itself could be a useful tool for the community. So whether it is open-sourced will affect my final score. Will the full model be released?

Questions for authors

Please see Weaknesses.

One suggestion: the paper does not say how the timestep t is sampled in training. For removal, which source to keep is probably decided in the early, high-noise steps. Putting more training weight on these steps (small t in this paper, for example with a shifted timestep distribution) might be a cheap way to improve target selection.

3 comments
AuthorsOpenReview ↗

A1 (For W1), A2 (For W2), and A3 (For W3)

Thanks for the valuable comments. Our responses are as follows:

A1: (For W1)

For multi-sounding-object cases, we first explore the residual paradigm: we feed SAM-Audio the visual mask (the intermediate result of the visual-removal pipeline) of the source to remove, the original audio to be edited, and the text instruction for extracting this unwanted source, then take the residual audio as the sound-removal result. However, this scheme has a clear drawback: the residual audio usually still carries residual signal of the removal target, and it is hard to preserve the target source to keep intact. To mitigate this issue, we switch to an extraction-based paradigm. We feed SAM-Audio the visual mask of the source to keep, the original audio to be edited, and the text instruction for extracting the target source to keep, and directly take the model's extraction output as the final sound-removal result. For foreground-target-plus-background cases, since the audio to keep cannot be abstracted as a single, well-defined extraction target, we uniformly adopt the residual paradigm: feeding the visual mask of the source to remove, the audio to be edited, and the corresponding text instruction, and taking the residual audio as the sound-removal output. Below we report results under the all-residual setting.

MethodSAJ Overall ↑TSSR ↑PSF ↑Instr. Comp._a ↑Fidelity_a ↑IB-AV ↑DeSync ↓
SAM Audio (full residual)2.373.579.753.693.1015.470.96
SAM Audio (partial residual)2.423.5410.073.703.9017.890.93
TV-AudioRemover3.466.3510.794.793.9627.320.60

We adapted and tuned SAM-Audio for the sound-removal task across different scenarios. Combined with the results above, its flexibility and final performance on this task are both weaker than those of a model designed specifically for sound removal.

A2: (For W2)

The table below shows results when the outputs of different visual-removal models are used as the visual-condition input to our model. Among them, some visual-removal models produce samples where the target object is not fully removed from the video.

Upstream visual removersSAJ Overall ↑TSSR ↑PSF ↑IB-AV ↑DeSync ↓
ROSE3.296.2810.7126.070.77
EffectErase3.225.5410.6824.210.89
UnderEraser3.235.8110.6626.120.75
SVOR3.466.3510.7927.320.60

Regarding the concern about "how much the audio result depends on the quality of video removal," this precisely reflects that the visual information provides effective guidance for our model. When video removal is poor, we can first assess the quality of the video removal, and for unqualified videos we can directly choose text-only guided removal. This is exactly the advantage of our model's support for multimodal instructions, giving it robustness under different input conditions.

A3: (For W3)

The original comparison experiment in Appendix F uses a joint audio-visual editing setting: the original pre-edit video and original audio are input together with an editing instruction expanded and refined by a large model, aiming to fully exploit the native audio-visual temporal consistency of models such as Seedance 2.5 and MiniMax H3. Inspired by your comment, we add the reference-based generation experiment you mentioned: the edited video and the original audio are provided as reference inputs, constraining the model to edit only the audio. Below we give the comparison under this setting. We will add this new set of results to Appendix F and clearly label the two input paradigms for readers to distinguish. As can be seen, under the audio-only editing setting the performance of both models drops, a phenomenon worth further investigation. Such models, natively designed for joint audio-visual generation / co-editing, are trained on data where "the video frames change and the audio evolves synchronously"; their strength lies in jointly aligning frame changes + audio changes, rather than independently editing the audio while keeping the frames fixed.

MethodSAJ Overall ↑TSSR ↑PSF ↑Instr. Comp._a ↑Fidelity_a ↑IB-AV ↑DeSync ↓
Seedance 2.5 (Joint Edit)2.933.6310.594.364.1130.660.49
Seedance 2.5 (Audio-only Edit)2.23-4.6410.014.203.5528.170.74
MiniMax H3 (Joint Edit)2.250.769.973.414.3023.520.64
MiniMax H3 (Audio-only Edit)2.18-0.949.593.204.2524.721.01
TV-AudioRemover3.466.3510.794.793.9627.320.60
AuthorsOpenReview ↗

A4 (For W4) and A5 (For the suggestion)

A4: (For W4)

We plan to fully open-source this work. In addition to the benchmark and the complete model, we are also developing a lightweight open-source version with fewer parameters and faster inference, to further facilitate use by the open-source community and to become a genuinely useful and usable tool.

A5: (For the suggestion)

We thank the reviewer for the suggestion. In the revision we will add the training-timestep sampling scheme: we currently use a logit-normal distribution, i.e., we first sample z∼N(0,1)z \sim \mathcal{N}(0,1) and then set t=sigmoid(z)t = \mathrm{sigmoid}(z), without applying any extra timestep-dependent loss weighting. In the Flow Matching path adopted in this paper, xt=(1−t) x0+t x1x_t = (1-t)\,x_0 + t\,x_1, a smaller tt corresponds to the high-noise stage.

To preliminarily examine this suggestion, we performed a generation-trajectory analysis on a "keep barking, remove ocean waves" sample. The RMS-envelope correlation between the decoded original latent and the GT rises from 0.0400.040 at t=0t=0 to 0.9790.979 at the final output; at t=0.04t=0.04, the envelope correlation of the first-order endpoint prediction already reaches 0.9770.977, and the AST barking-category score reaches 0.8000.800. This indicates that the early velocity prediction for this sample already contains the semantics and rhythm of the source to keep, consistent with the reviewer's judgment that the early stage may play an important role in target selection. However, a single-sample first-order endpoint prediction provides only a preliminary observation; it cannot prove that this mechanism is general, nor that increasing early-stage training weight improves performance.

We acknowledge and will adopt this suggestion as a key direction for future work. Specifically, we will compare different timestep-sampling distributions via controlled ablations. For example, shifting the mean of the logit-normal distribution from 00 to −0.5-0.5 raises the sampling proportion of the high-noise region t<0.2t<0.2 from 8.28%8.28\% to 18.77%18.77\%, without increasing the per-update compute. In the future we will combine stage-conditioned ablations and jointly evaluate sound-removal effectiveness and the spectral, envelope, and energy fidelity of the preserved sound, to verify the effectiveness and potential trade-offs of this strategy.

We will keep adding relevant experiments and updating the paper PDF accordingly. If the reviewers have any further questions, we are happy to continue the discussion in the thread.

AuthorsOpenReview ↗

Following up on our response

Thank you again for your detailed review. We know this is a busy time, so we just wanted to check in briefly. Did our response address your concerns? If anything still seems unclear or unconvincing, please let us know. We would really like to hear your thoughts and are happy to discuss further.

Reviewer_MJq1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TV-AudioRemover studies selective sound removal after a visual object has been erased from a video. Given the original audio mixture, a natural-language instruction, and optionally the edited video, the model generates a soundtrack that suppresses the designated source while preserving unrelated sounds. The method uses a latent flow-matching multimodal diffusion Transformer with mixture-audio conditioning. Task embeddings distinguish extraction, deletion, and combined preservation-removal instructions; modality-specific global guidance separates visual conditioning from textual editing intent. Training combines these tasks and proceeds from semantically distinct mixtures to harder combinations containing acoustically or semantically similar sources.

Critical strengths
  1. The task formulation closely matches a concrete editing need. A visual eraser can leave an acoustically inconsistent scene, and preserving the existing soundtrack's untargeted components is a more demanding and useful objective than muting or regenerating everything. Treating the edited video as a positive description of what should remain is a sensible design choice, especially when text expresses a negative operation. The model's optional visual input also permits use when only an audio-removal instruction is available.

  2. The training procedure is grounded in the ambiguity between extraction and deletion. Explicit task embeddings, complementary extraction supervision, and instructions that name both retained and removed sources give the model a plausible basis for distinguishing opposing operations. Hard mixtures of similar voices, instruments, and environmental sounds are relevant to failures that simple category-disjoint mixtures would not expose. The data appendix describes annotation fields, filtering thresholds, instruction templates, and hard-example groups in enough detail to make the construction strategy assessable.

Critical weaknesses
  1. The claimed importance of joint text-visual guidance exceeds the evidence currently supplied for audio removal itself. Table 4 gives identical SAJ scores for text-only and joint conditioning, with TSSR changing only from 0.63 to 0.64 and PSF from 1.07 to 1.08. The improvement is clearer in audiovisual alignment. This distinction matters: the results support complementary visual grounding, but do not yet establish that vision materially resolves source ambiguity. A test deliberately containing indistinguishable textual categories would be more diagnostic.
  2. AV-Remove-Bench is also small relative to the breadth of the claims: it contains 42 selected public clips, 27 generated clips, and eight recordings. Aggregate scores lack uncertainty estimates and may conceal strong differences across these sources. TSSR and PSF measure AST category-confidence changes rather than isolated source energy or waveform fidelity. A preserved category can retain high confidence despite altered timbre or missing events, while a target's confidence can change through masking. PSF is evaluated only in the subset with two describable sounding objects, further restricting its coverage. Human listening is valuable, but inter-rater reliability and paired significance are not reported.
Questions for authors
  1. Can you evaluate cases where text alone is genuinely ambiguous, such as two visible objects of the same sound category with different temporal activity? Comparing correct, shuffled, and temporally displaced edited videos would establish whether the visual branch identifies the intended source rather than mainly improving global audiovisual compatibility.
  2. Please reconcile the TSSR and PSF scales in Tables 1 and 4. Are the ablation values divided by ten, and are the sample subsets and aggregation rules identical? Also report how undefined TSSR cases are handled and how many benchmark clips contribute to each preservation score.
  3. How well do AST-based TSSR and PSF track human assessments of suppression and waveform preservation at the individual-clip level? A controlled mixture test with known clean stems could measure target leakage, background distortion, and source identity directly, providing a useful check on category-confidence metrics without requiring references for the natural benchmark.
  4. Could you report separate ablations for task embeddings, instruction diversification, and modality-specific global guidance, with the training data and optimization budget held fixed? For the curriculum, please distinguish the effects of additional fine-tuning steps, the smaller fine-tuning corpus, and the use of hard rather than randomly sampled mixtures.
4 comments
AuthorsOpenReview ↗

A1 (For W1 and Q1) and A2 (For Q2)

Thanks for the valuable comments. Our responses are as follows:

A1: (For W1 and Q1)

Thank you for your attention to the visual branch. Limited gains of visual conditioning on certain editing metrics do not imply weak efficacy. This is a known difficulty in GT-free audio editing evaluation: current metrics are insensitive to fine-grained source discrimination, though they work for coarse sound categories. In a male–female dialogue example targeting female voice removal, text-only conditioning often fails to fully suppress the female voice even when speakers talk at separate times, as it cannot reliably link the editing target to the desired source. Visual conditioning provides agent and temporal activity cues to separate sources, yet such benefits may evade existing metrics. In our example, text-only incomplete removal yields SAJ = 3.4, while successful audio-visual removal yields SAJ = 3.3, showing SAJ is limited by audio-text matching and description granularity. TSSR and similar metrics are constrained by sound detector accuracy; both underestimate fine-grained source-selection mistakes. Audio-visual synchrony metrics offer complementary clues: residual sound after the target leaves the frame becomes asynchronous with visible speakers. This supports improved post-editing cross-modal consistency but cannot serve as standalone causal ablation evidence for the visual branch. We therefore evaluate jointly over semantic editing, target fidelity and audio-visual synchrony rather than relying on one metric. Human subjective scoring remains the most reliable measure, with listening procedures and statistics provided under “Subjective Evaluation Statistics” in the supplement.

Second, we agree that audio–visual consistency alone cannot confirm whether the visual branch helps locate the intended sound source. We therefore introduce four visual-condition control settings: correctly matched videos, semantically mismatched videos, time-shuffled original videos, and original videos with a 2 s visual shift. The correct visual condition achieves the best results across all six reported metrics in the tested example, demonstrating that visual conditions provide guidance for our model. Mixed audio, prompt, parameters, inference budget, latent sampling and initial noise are fixed to isolate visual effects. Semantic mismatch tests reliance on visual agents; shuffling and shifting test temporal alignment between visual vocalization and audio events. All outputs are referenced against the same correct video to eliminate reference bias. Results are analyzed using source removal/preservation, synchrony and GT fidelity metrics. For instance, to keep on-screen speech and remove dog barking: matched frames show the speaker; mismatch shows barking dogs; shuffle uses 1 s chunks; shift uses a 2 s visual circular delay. All groups share the same text instruction.

Visual ConditionSAJ Overall ↑TSSR ↑PSF ↑DeSync ↓MR-STFT↓REC↑
Correct Edited Video4.489.1412.980.000.920.99
Semantically Mismatched Video1.48-7.214.302.005.720.24
Temporally Shuffled Video1.048.985.062.004.690.15
Temporally Displaced Video1.478.9611.181.804.380.02

The contrast results on this sample demonstrate that semantic mismatch weakens source removal and preservation performance, and temporal perturbation hurts target sound fidelity and synchrony. These findings support that the visual branch contributes to both source selection and temporal guidance, instead of only boosting holistic audio–video compatibility. As future work, we plan to replicate these visual-condition control experiments on a larger, category-diverse test set.

A2: (For Q2)

We thank the reviewer for the careful check. Table 4 contains a typo: the TSSR and PSF values shown there were not multiplied by 10. In addition, all compared methods use exactly the same test-sample subset and metric-aggregation rules, ensuring a fair comparison; we will correct the table values in the revision.

Of the 77 test samples, 60 are eligible for PSF evaluation; the rest do not meet the metric's evaluation conditions: in foreground-removal cases the background may contain multiple sound components, so the single-category assumption no longer holds, and these samples are therefore excluded from the PSF statistics. This filtering does not undermine the aggregation logic of the final PSF mean.

AuthorsOpenReview ↗

A3 (For W2 and Q3)

A3: (For W2 and Q3)

We thank the reviewer for the suggestion. Following the evaluation procedure you proposed, we add the corresponding experiments. To address the problem that existing audio-visual editing test sets lack post-edit ground-truth audio, we build VTAR-GT1K, containing 1,000 test samples with ground-truth references. Using the audio-visual clip of a single salient sounding subject as the post-edit reference, we randomly pair it with extra audio whose label and secondary category both differ; after aligning sampling rate, channels, and duration, and taking the original video's audio track as reference, we mix with the SNR sampled independently from U(-3, 6) dB, while keeping the video frames unchanged. The video frames, the mixture audio, and the removal instruction pointing to the extra sound together form the model input, and the original audio track serves as the ground truth, used to evaluate the model's ability to remove the sound not present in the frame while preserving the sound corresponding to the frame. The table below lists the newly added metrics. For generative models, even when the output preserves the correct sound semantically and perceptually, it may regenerate phase, rhythm, and detail; Therefore, SDR/SIR serve only as auxiliary waveform reference metrics and are not the primary ranking criteria for generative models.

MetricEvaluation
MR-STFT↓Multi-resolution STFT distance
LMD↓Log-Mel spectral distance
REC↑RMS envelope correlation coefficient
AED↓Absolute energy deviation
SDR↑Signal-to-distortion ratio
SIR↑Signal-to-interference ratio

The table below presents results on VTAR-GT1K. For AVI-Edit, only its audio branch is used. OmniSep takes the first frame as visual input; SAM Audio, MMAudioSep and TV-AudioRemover use full videos. In this benchmark, the target sound to preserve is uniquely defined by an object label. SAM Audio generates a visual mask for this labeled sound source to extract it from the mixture. Since the source to remove is absent from frames, we avoid the “extract-and-residual” pipeline. MMAudioSep and OmniSep use positive/negative text labels for retained/removed sounds; TV-AudioRemover uses removal prompts. All models receive mixed audio and model-specific conditions. Visual inputs only contain the source to keep, simulating the case where the unwanted object has left the frame while its sound remains.

GroupMethodSAJ Overall ↑TSSR ↑PSF ↑IB-AV ↑DeSync ↓MR-STFT↓LMD↓REC↑AED↓SDR↑SIR↑
T-AAVI-Edit1.95-28.588.6412.741.024.8013.860.188.37-12.164.69
T-AZETA3.288.1210.6625.180.803.1310.500.644.06-12.747.30
T-AAudio-Omni2.06-29.268.818.450.894.2214.170.356.43-15.922.18
T-AUNISON2.747.2110.3321.750.852.818.970.794.82-12.119.28
IT-AOmniSep3.016.8810.8420.830.673.0311.820.82‾\underline{0.82}1.56‾\underline{1.56}5.5813.70
VT-ASAM Audio3.00-10.7610.2321.770.902.9013.540.542.23-5.974.63
VT-AMMAudioSep3.29‾\underline{3.29}8.20‾\underline{8.20}11.1528.45‾\underline{28.45}0.56‾\underline{0.56}2.25‾\underline{2.25}8.46‾\underline{8.46}0.743.70-12.4711.74
VT-A (Ours)TV-AudioRemover3.318.7910.91‾\underline{10.91}30.320.511.484.800.891.36−11.03‾\underline{-11.03}12.54‾\underline{12.54}

TV-AudioRemover shows strong overall performance on VTAR-GT1K. It attains the best overall SAJ and TSSR for semantic editing and source suppression, along with the lowest MR-STFT, LMD, AED and highest REC, yielding spectra, energy and loudness dynamics closer to GT. It outperforms MMAudioSep and SAM Audio across these four acoustic metrics. It also achieves the highest IB-AV and lowest DeSync for superior audio–visual semantic alignment and temporal synchrony. While PSF, SDR and SIR are second-best, the results demonstrate its balanced performance in source removal, preserved-source fidelity and post-editing audio–visual consistency.

Moreover, detection-based TSSR and PSF reliably capture suppression of unwanted sound and semantic retention of target sound. For most methods, these category-level metrics correlate with spectral and envelope metrics, validating their reasonableness. Discrepancies arise from different focuses: TSSR/PSF measure removal and preservation at the sound-category level, while MR-STFT, LMD, REC and AED characterize acoustic structure and energy matching against GT. Hence category metrics effectively assist semantic-quality assessment and, combined with signal-level metrics, enable holistic evaluation for generative audio-editing models.

AuthorsOpenReview ↗

A4 (For Q4)

A4: (For Q4)

We thank the reviewer very much for the reminder. Task embeddings, instruction diversification, modality-specific global guidance, and hard-mixture fine-tuning are exactly the designs we introduced incrementally during model training and iteration. The original manuscript mainly presented the overall performance of the final model and did not fully show the role of these designs during iteration. Your suggestion helps us state each design's contribution more clearly. To this end, the revision will add four controls evaluated on the same test set: without task embeddings, without instruction diversification, without modality-specific global guidance, and fine-tuning with generic random-mixture data under the same number of training steps.

MethodSAJ Overall ↑TSSR ↑PSF ↑Instr. Comp._a ↑Fidelity_a ↑IB-AV ↑DeSync ↓
w/o Task Embeddings3.222.889.844.163.9525.750.68
w/o Instruction Diversification3.293.579.994.344.1325.890.75
w/o Modality-Specific Global Guidance3.505.9710.054.663.9424.240.74
Random-Mixture Fine-Tuning3.243.6410.574.283.9025.470.71
full3.466.3510.794.793.9627.320.60

The full model achieves the best TSSR, PSF, instruction compliance, IB-AV, and DeSync among these variants. The comparison of these iterative versions shows that, without task embeddings, the model is more prone to removal failure and over-removal. Task embeddings help specify the editing task, while instruction diversification strengthens the model's adaptability to different instruction phrasings, letting it perform the intended edit more stably. Modality-specific global guidance helps make effective use of visual and text conditions, improving the synchronization and consistency between the edited sound and the video frames. Furthermore, under the same number of training steps, compared with generic random-mixture data, hard-mixture fine-tuning further strengthens the model's ability to distinguish confusable sources, improving the suppression of the sound to remove and the fidelity of the preserved sound.

We will keep adding relevant experiments and updating the paper PDF accordingly. If the reviewers have any further questions, we are happy to continue the discussion in the thread.

AuthorsOpenReview ↗

Following up on our response

Thank you again for your detailed review. We know this is a busy time, so we just wanted to check in briefly. Did our response address your concerns? If anything still seems unclear or unconvincing, please let us know. We would really like to hear your thoughts and are happy to discuss further.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary for the Area Chair (1/2)

We thank the Area Chair and all three reviewers for their thoughtful assessment and constructive suggestions. TV-AudioRemover addresses a challenging and practically important gap in audio–visual editing: an object can be convincingly erased from a video, yet its sound may remain, undermining the realism and perceptual coherence of the entire edit. Resolving this mismatch requires more than suppressing a sound category: the model must identify the intended source, preserve other overlapping sounds, and maintain their alignment with the edited video. Our method is specifically designed for this task, combining text and edited-video guidance with task-aware training and hard-mixture fine-tuning. The original results demonstrate its strong overall performance in removal effectiveness, preserved-sound fidelity, and audio–visual consistency. Importantly, TV-AudioRemover also ranks among the leading methods for text-guided sound removal. Its capabilities therefore extend beyond removing sounds associated with visually erased objects: text instructions enable effective removal of specified sounds while preserving the remaining audio, and visual guidance provides additional source-selection and temporal-alignment cues when video is available. This strong standalone audio-editing capability makes our model useful both for audio-only applications and for coherent audio–visual editing.

Following the reviewers’ suggestions, our rebuttal extends this evidence along a coherent progression—from reference-based fidelity and baseline comparability to visual guidance and component contributions. The additional results reinforce, rather than overturn, the original conclusions and establish advantages across a broader set of evaluation criteria.

1. The new ground-truth evaluation extends our lead from semantic editing to acoustic fidelity (MJq1 W2/Q2/Q3; p3ms Q4).

We added VTAR-GT1K, a 1,000-sample benchmark with ground-truth audio for the sound to preserve, together with four reference-based acoustic metrics. TV-AudioRemover ranks first among the evaluated methods on eight of the eleven reported metrics: SAJ, TSSR, IB-AV, DeSync, MR-STFT, LMD, REC, and AED; it ranks second on PSF, SDR, and SIR. Thus, its strong semantic-editing performance is accompanied by superior agreement with the reference audio in spectral structure, temporal envelope, and energy—not merely favorable category-level scores. For example, compared with MMAudioSep, MR-STFT improves from 2.25 to 1.48 and REC from 0.74 to 0.89. This additional evidence strengthens the original conclusion that our model effectively removes unwanted sounds while preserving the remaining audio.

The new metrics complement, rather than replace, the original protocol: TSSR/PSF assess category-level removal and preservation, while reference-based measures assess acoustic similarity to the original unmixed audio that should remain after editing. SDR/SIR are auxiliary measures because generative outputs may differ in phase despite preserving the intended sound.

We also clarified shared sample subsets and aggregation rules, PSF eligibility (60/77), Table 2’s sample count (50), AST coverage, and IB-AV scaling, and will correct the Table 4 scaling typo. These clarifications do not change the comparative conclusions.

2. Additional baseline settings confirm that our advantage is not an artifact of unfavorable baseline configurations (C7Jy W1/W3; p3ms W1/Q1).

We additionally evaluated SAM Audio using the reviewer-requested fully residual setting and commercial models using reference-based, audio-only editing. TV-AudioRemover remains ahead of both SAM Audio settings on all seven reported metrics and ahead of the commercial models under both editing paradigms on SAJ, TSSR, PSF, and instruction compliance. Importantly, the original configurations were chosen to make effective use of each baseline’s capabilities, not to weaken them.

For SAM Audio, we had already identified incomplete target removal and damage to retained sounds in the residual approach. The original comparison therefore used direct extraction when the retained source was well defined, and residual processing when it was not. The added fully residual results confirm that this scenario-adapted configuration provides a better overall trade-off, particularly for preservation and audio–visual consistency. For Seedance 2.5 and MiniMax H3, the original joint-editing setup was deliberately designed to exploit their native joint audio–visual capabilities. The requested audio-only setup yields lower SAJ and TSSR for both models, rather than revealing an underestimated baseline. These results strengthen the case for a model specifically designed for selective sound removal. We also report OmniSep as a reference baseline and explain the current reproducibility barrier for SAVE, whose code and weights are unavailable.

AuthorsOpenReview ↗

Summary for the Area Chair (2/2)

3. Controlled visual conditions further support the effectiveness of our multimodal guidance (MJq1 W1/Q1; p3ms W2/Q2; C7Jy W2).

We added correct, semantically mismatched, temporally shuffled, and temporally shifted visual conditions while fixing the audio, instruction, model parameters, and inference randomness. The correct visual condition achieves the best results across all six reported metrics in the tested example, demonstrating that visual conditions provide guidance for our model: semantic mismatch weakens source removal and preservation, while temporal perturbations degrade reference-audio fidelity and synchrony. These results provide additional, more targeted evidence that the visual branch contributes to both source selection and temporal guidance, extending the original evidence of improved audio–visual consistency. The controlled example is preliminary; broader, category-diverse validation remains future work.

We also evaluate outputs generated by four upstream visual removal models, showing the compatibility of our approach with various visual editing workflows and confirming the effectiveness of visual conditioning for our model. The support for text-only guidance additionally provides an alternative input mode when edited visual conditions are unreliable.

Category-level metrics remain useful for their intended purpose, but may not fully resolve distinctions between similar sources, such as two speakers. Combining them with reference-based fidelity, synchrony, and the original human evaluation gives a more complete account of our model’s benefits.

4. Component ablations confirm the full model’s strongest overall balance across the core task objectives (MJq1 Q4; p3ms W3/Q3).

We added controlled ablations of task embeddings, instruction diversification, and modality-specific global guidance, and replaced hard-mixture fine-tuning with random-mixture fine-tuning under the same training-step budget. The full model achieves the best TSSR, PSF, instruction compliance, IB-AV, and DeSync among these variants.

The results make the complementary contributions of our existing design choices explicit: task-aware and diversified instructions improve editing reliability; multimodal global guidance improves cross-modal alignment; and hard-mixture training improves suppression and preservation of confusable sources. Although individual variants lead on SAJ or fidelity alone, none matches the full model’s balance across these five core measures. These experiments therefore reinforce the effectiveness of the complete method rather than introducing a different model or changing the original contribution.

Overall, the rebuttal broadens the evidence for TV-AudioRemover’s advantages: stronger reference-based fidelity, sustained superiority on core editing measures under additional baseline settings, and clearer support for its multimodal and training design. We are grateful to the reviewers for helping us articulate and substantiate these strengths more comprehensively. We will incorporate the new results and protocol clarifications into the revision, and release the benchmark and full model, with a smaller and faster version under development. Our goal is to turn this research into a genuinely useful, accessible community tool for multimodal guiaded audio object removal.

AuthorsOpenReview ↗

Invitation to Further Reviewer Discussion

We would also appreciate the Area Chair’s help, if possible, in encouraging further engagement from the reviewers during the discussion period. We hope to discuss their concerns in depth in light of the clarifications and additional results presented in our rebuttal, and would welcome any follow-up questions or remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.