acceptodds
Under review as a conference paper at ICLR 2027

Knowing Whom to Erase: Instruction-Routed Audio-Visual Subject Removal for Human-Centric Speech and Music Videos

Abstract

Audio-visual subject removal erases the person specified by an instruction and the sound they produce, while preserving other people and their sounds. Existing audio-visual editors learn text- or mask-conditioned edits from paired examples, but matching individual edits does not establish reliable target switching within the same recording. In multi-person scenarios, existing editors struggle to jointly remove the requested person and preserve competing sources. Single-target supervision can contribute to this failure by permitting an instruction-agnostic shortcut: when every instruction for an input refers to the same target, a fixed edit can fit the training data without learning whom to remove. To address this problem, we propose Audio-Visual Instruction-Routed Erasure (AV-IRE), a joint audio-video editor combining complementary-target supervision with explicit target routing. Source recordings and their teacher-edited counterparts are composed to supervise different target-specific outputs for the same input. The target is conveyed to the generator through a spatial routing map and modality-specific codes, while its text prompt remains target-neutral. Route-dependent output changes are supervised by a shared-state route-intervention loss. We also construct 54,705 automatically filtered, teacher-generated removal pairs from real recordings for task adaptation. AV-IRE outperforms strong audio-visual editing baselines in selective multi-person removal. Compared with Task-Adapted LTX-2.5, blind-rated joint success rises from 11.7% to 41.7% on natural two-musician recordings. On new recordings, stricter bidirectional success, requiring both target requests per input to succeed, rises from 3.3% to 26.7%.

Primary area:
generative models
Keywords:
audio-visual editing, instruction-guided video editing, subject removal, diffusion transformer, instruction following, counterfactual supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RRmKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AV-IRE, a framework for targeted audio-visual subject removal using textual instruction. The goal is to remove a specified subject along with their associated sound while preserving the rest of the scene unchanged. The main idea is to use complementary-target supervision, where the same video is paired with different removal instructions and corresponding outputs. The method is based on LTX-2.5 and introduces a spatial selector and modality-specific routing codes to condition the joint audio-video generator. The paper also introduces a teacher-generated dataset of 54,705 removal pairs, along with AV-SwapTarget, a benchmark designed to evaluate whether a model can correctly switch between removal targets in the same video. On the Natural-2 benchmark, AV-IRE improves blind-rated joint success from 11.7% to 41.7% compared to a task-adapted LTX-2.5 baseline. A follow-up evaluation on new recordings also shows an improvement in bidirectional success from 3.3% to 26.7%.

Critical strengths
  1. Relevant problem formulation and evaluation.

The paper addresses an interesting problem of instruction=based audio-video editing. Removing one target successfully does not necessarily mean that a model can correctly identify and remove different targets from the same video. I think the proposed evaluation is useful to address this issue, since requiring both removal instructions to succeed provides a stronger measure of selective editing. The complementary-target supervision is also a reasonable way to address this issue during training.

  1. Promising results and careful experimental validation.

The improvement in joint success on Natural-2 is substantial, going from 11.7% to 41.7%. I also appreciate the effort of the authors in validating their results, using independent raters, preregistered analyses and a separate evaluation on new recordings, providing additional evidence that the model can generalize its target-selection behavior beyond the original evaluation set. Another positive aspect is that the authors discuss negative results and limitations openly, including cases where the proposed method performs worse than the baseline; this makes their results more credible. 

Critical weaknesses
  1. Limited statistical evidence and relatively low absolute performance.

Although the improvements on Natural-2 are encouraging, the primary bidirectional success result does not reach statistical significance (0/30 vs 5/30, p = 0.0625). The fresh evaluation provides stronger evidence, but it is still based on only 30 recordings and a single reviewer. Furthermore, Natural-2 was also used during model development, making the fresh evaluation particularly important for validating the results. Even in this additional evaluation, bidirectional success reaches only 26.7%. Overall, I find the results promising, but the relatively small evaluation sets and low absolute success rates make it difficult to assess how reliable the method would be in more general scenarios.

  1. Mixed results on acoustic removal and generalization.

While AV-IRE substantially improves joint success on Natural-2, acoustic target removal remains almost unchanged compared with the baseline (61.7% vs 60.0%). The improvements therefore seem to come mainly from better visual editing and preservation of non-target sources. The results on other benchmarks are also less convincing. In particular, AV-IRE performs considerably worse than the task-adapted baseline on InsAVE (63.3% vs 95.9% joint success) and achieves lower acoustic removal on Natural-3. These results raise some concerns about the generalization of the method, especially given that acoustic target removal is an important part of the proposed task.

  1. Minor weakness: presentation and readability.

Although the motivation is clearly presented, the technical description is quite dense and requires substantial effort to follow. The experimental section could also be more straightforward. In particular, the different evaluation rounds, the distinction between primary and secondary results, and the statistical analyses are not always easy to keep track of. Considering how important the experimental validation is to the paper, I think a more concise and structured presentation of the results would make the contribution easier to assess.

Questions for authors
  1. The fresh evaluation shows promising improvements, but it is based on only 30 recordings. Do the authors consider this sufficient to draw conclusions about the reliability of the method, especially given the relatively low absolute bidirectional success rate?
  2. Acoustic target removal on Natural-2 is almost unchanged compared with the baseline, despite the substantial improvement in joint success. Do the authors believe that the main advantage of AV-IRE is better preservation of non-target sources rather than more effective target removal?
  3. AV-IRE performs considerably worse than the task-adapted baseline on InsAVE and shows weaker acoustic removal on Natural-3. Could the authors explain what they believe causes these differences in performance?
  4. The paper includes several evaluation rounds and follow-up experiments. Could the authors clarify which results they consider the strongest evidence for the effectiveness of the proposed approach, and how the different evaluations should be interpreted together?
1 comment
AuthorsOpenReview ↗

Title: Following up on our response

Thank you again for your detailed review. We know this is a busy time, so we just wanted to check in briefly. Did our response address your concerns? If anything still seems unclear or unconvincing, please let us know. We would really like to hear your thoughts and are happy to discuss further.

Reviewer_6pUqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes AV-IRE, a framework for removing a specified person and their associated sound from audio-video recordings while preserving other people and sounds. The main idea is to train on complementary targets, where the same input has different edited outputs depending on the requested person. The method also introduces spatial routing and modality-specific conditioning within an LTX-2.5 backbone.

The authors evaluate the method on natural multi-musician recordings, synthetic composites, and existing single-speaker benchmarks. The results show substantial improvements over a task-adapted LTX-2.5 baseline on selective multi-person removal, particularly in preserving non-target sources.

Critical strengths

I think the paper addresses an interesting problem that is not fully captured by existing audio-visual editing evaluations. Successfully removing one person does not necessarily mean the model can remove a different person from the same recording. Evaluating both instructions on the same input is a useful way to test whether the editing behavior actually depends on the requested target.

The complementary-target training strategy is also reasonable. By constructing two different removal targets for the same input, the authors directly address the possibility that a model learns an input-dependent edit without properly following the instruction.

The evaluation is relatively careful. I appreciate the use of blind human ratings, a second reviewer, and an additional evaluation set collected after model freezing. The authors also report cases where the method performs worse than the baseline, rather than presenting only favorable results.

Critical weaknesses

My main concern is whether the proposed routing mechanism actually improves audio-visual grounding, or whether most of the gains come from better visual localization and preservation. In Table 1, acoustic target removal barely changes compared with Task-Adapted LTX-2.5 (36/60 versus 37/60), while the much larger improvements are in visual removal and preservation of other sounds. More importantly, the audio-only intervention in Appendix E.3 shows identical audio success (32/60) when the pooled visual feature is replaced by a spatial mean or by the other performer's feature. Although this does not rule out cross-modal information passing through other parts of the backbone, it leaves the contribution of the explicit visual-to-audio routing mechanism unclear.

I also find the ablation study insufficient to establish which components are responsible for the final improvement. Table 4 combines several changes in the final configuration, including the target-neutral prompt, spatial map, and crossed route interventions. Since these are introduced together, it is difficult to determine whether the improvement comes from one component or their combination. Each ablation configuration is also trained only once, and the authors acknowledge execution variability. A more controlled ablation would make the methodological contribution more convincing.

Another limitation is the relatively narrow evaluation setting. The main experiments involve two musicians with different instrument categories. The authors explicitly exclude selecting between performers of the same category. However, distinguishing two different instruments is substantially easier than selecting one of two people who are speaking or playing the same instrument. It is unclear whether the proposed approach would work in these more ambiguous situations.

The generalization results are also mixed. On Natural-3, AV-IRE has lower target-removal performance than Task-Adapted LTX-2.5 (38/81 versus 49/81). On the InsAVE single-speaker benchmark, its joint success is also considerably lower (31/49 versus 47/49). This suggests that the improvements in multi-person preservation may come with a loss of target-removal capability in other settings. I would like the authors to discuss this tradeoff more directly.

Finally, the primary bidirectional-success comparison on Natural-2 does not reach statistical significance (p = 0.0625). The fresh-set follow-up is encouraging, but it still contains only 30 recordings, and five inputs had incorrect target labels. The authors provide sensitivity analyses, which is helpful, but a larger evaluation would make the claimed improvement more convincing.

Questions for authors
  1. Given the null result from the audio-only visual-feature intervention, what evidence supports the claim that the proposed audio routing learns a meaningful association between visual performers and their sounds?

  2. Can the authors separately ablate the target-neutral prompt, spatial routing map, and crossed route interventions? The current cumulative ablation does not isolate their individual contributions.

  3. How does the method perform when two performers belong to the same category, such as two speakers or two guitarists? I understand that this is outside the current scope, but it would help establish the limitations of instruction-based target selection.

  4. Why does the method underperform the task-adapted baseline on InsAVE and show lower removal rates on Natural-3? Is this caused by the preservation-oriented training objective or by limitations in target localization and audio conditioning?

1 comment
AuthorsOpenReview ↗

Title: Following up on our response

Thank you again for your detailed review. We know this is a busy time, so we just wanted to check in briefly. Did our response address your concerns? If anything still seems unclear or unconvincing, please let us know. We would really like to hear your thoughts and are happy to discuss further.

Reviewer_oP5FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies erasing/removal of a person and their sound under a multi-person audio-visual recorded setting with instruction guided removal. This is done under a complex scenario where under different instructions, different objects/entities should be removed from a same recording. It argues that single-target easily route to an instruction-agnostic shortcut and proposes AV-IRE.

Critical strengths
  1. The problem is very well motivated and clearly deliverd.This is also a very practical problem case that has not been well addressed in current literature.

  2. With supplementary audio-video samples, the results are very well demonstrated.

Critical weaknesses
  1. It seems from the qualitative evaluation, that most improvements are focused on preservation, visual removal than a thorough acoustic removal (Table 1), This seems somewhat contrary to the task.

  2. Continued from above, the method seems to substantially fail to do "both" (PP). While ther in substantial increase from a task-adapted LTX-2.5 implemented, the paper lacks analysis on this failure case.

  3. From supplemental content, it seems many videos that are "automatic evaluation only" are not very accurately judged. For instance, in Kiwi-Edit to CoherentAVEdit cascade for Example 01 (remove the banjo...) - the automatic evaluation says the exact MP4V original is retained but noticeably the banjo instrument has been removed from the video. While this may be somewhat minor, I believe this suggests a gap in preciseness of the automatic evaluation and scores. This is reucrring in all Kiwi-Edit evaluations (instrument is removed the evaluation says exact MP4V is being retained). Especially since Composite-N and Natural-3 are only automatically evaluated.

  4. I could not refer to any seed variance / statistical significance over multiple removal runs for the experimental results.

  5. Despite the paper title being " INSTRUCTION-ROUTED AUDIO-VISUAL SUBJECT REMOVAL FOR HUMAN-CENTRIC SPEECH AND MUSIC VIDEOS", the paper lacks support in speech domain. The qualitative samples do now show any speech samples. The table and evaluation in multi-source environment is only in musical settings. There is only one speaker setting that is single-target transfer.

  6. Continued from above, in the single-target speaker setting - speech-specific evaluation metircs (WER, speaker verification) were not tested against. In speech removing setting, the inteligibility of the audio is what makes a retaining audio meaningful .

Questions for authors
  1. Since target removal is very small (table 1 36-> 37), could the authors confirm whether a small improvement in acoustic forgetting is something to be addressed or is simply because it is already a substantially good score across baselines?

  2. Is there a specific analysis or diagnosis on why "both / bi-directional" modes are difficult to be erased? Does this hint overfitting to removing specific targets per video? The oracle-map (App E) seems to improve visual criteria but leaves PP with subtle change. Clarification on remaining failures would help strengthen the soundness of the paper.

  3. Could the authors elaborate on the evaluation system? It seems the evaluation says "exact MP4V is being retained" when it is not. While it is noticeable that the removal quality is poor on these baseline videos, it still is not same as "exact MP4v is being retained."

  4. Could the authors provide statistical significance and scores on multiple seed runs for removal? Rather than holding a single train run fixed, I believe multiple runs would help validate the soundness of the work.

  5. Could you provide a multi-speaker speech evaluation, or a justifiable reason for this not being provided?

  6. Also, if a removal of speech under multi-speaker setting is possible - could you provide WER , speaker similarity metric (original audio vs retained audio from removed video)? I believe these results on speech will substantially affect my assessment of the work.

  • below are question more related to discussion rather than assessment ;
  • Would this method be applicable in possibly editing rather than removal? E.g., chaning the trumpet to another instrument like trumbone. In useful scenarios - a person could just film themselves holding a stick and edit it to be a piccollo.

  • How are two performers of the similar instrument class handled? (trumpet and trumbone) Were there any cases observed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.