acceptodds
Under review as a conference paper at ICLR 2027

AVIO: Learning to Add and Remove Sounding Objects in Audiovisual Scenes

Abstract

Adding or removing a sounding object requires coordinated changes to visual content and sound while preserving the surrounding scene. Yet paired supervision for localized non-speech audiovisual editing remains limited, as visual and acoustic edits must target the same object and isolate its sound from overlapping sources. To address this gap, we introduce AVIOBench, a dataset comprising 37.9 hours of paired audiovisual examples spanning 1, 878 target-object names. AVIOBench links the visual presence and acoustic contribution of each target object through a shared identity and visual mask. Our automated pipeline uses visual grounding and cross-modal consistency to select target-sound removal candidates, then jointly refines the audiovisual pairs to improve perceptual quality and cross-modal consistency. Building on this dataset, we propose AVIO, which adapts a pretrained text-to-audiovisual generation model through source conditioned feature modulation to jointly learn object addition and removal. A reference-frame curriculum gradually reduces reference conditioning during training, enabling one model to perform instruction-only editing with optional visual guidance. Quantitative and qualitative evaluations demonstrate effective audiovisual object removal and addition, with optional reference guidance providing appearance and placement control for addition. Video examples are available on our project page.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Editing, Video Editing, Audiovisual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Rmu9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AVIOBench, a paired dataset for audiovisual object addition and removal, and AVIO, a unified model for editing visual objects and their associated sounds. AVIO adapts a pretrained text-to-audiovisual diffusion transformer through source-conditioned feature modulation and a reference-guided training curriculum. The paper evaluates the model on AVIOBench using quantitative metrics and qualitative comparisons.

Critical strengths
  1. Audiovisual editing is an important and practically relevant task. Object addition and removal are fundamental operations that require coordinated changes across visual and acoustic content while preserving the surrounding scene.
  2. The approach is straightforward and clearly presented. Adapting a pretrained audiovisual DiT with source conditioning and progressively dropping reference-frame guidance provides a practical architecture for unified addition and removal.
Critical weaknesses

Major Weakness:

  1. The methodological novelty appears limited. The dataset construction pipeline primarily combines existing segmentation, audio separation, video inpainting, and audiovisual refinement models. The model also builds on established components, including source-conditioned modulation and reference-guided training. Closely related work such as InstructAV2AV already adapts audiovisual generation backbones using paired editing data, while the manuscript explicitly draws inspiration from Ditto for its reference-guided curriculum. Although integrating these components may be useful, the paper does not sufficiently establish what new technical insight or capability distinguishes AVIO from existing approaches. A stronger justification of the dataset contribution would also require more rigorous quality validation.

  2. The experimental comparison does not isolate the benefits of the proposed method from those of dataset-specific training. AVIO is trained on AVIOBench, whereas the baselines use released checkpoints without adaptation to this dataset. Consequently, the reported gains may partly reflect differences in training distribution rather than the proposed architecture. I suggest that the authors: (1) fine-tune suitable baselines on the same training split, or provide controlled comparisons using the same backbone and training data; and (2) evaluate generalization on external data, such as held-out examples from JAVEdit-100k and relevant subsets of JAVEditBench.

  3. The qualitative examples raise concerns about the quality of the constructed dataset and the resulting editing model. Some examples on the demo page exhibit conspicuous artifacts after object removal. In the “flute” example, the region around the person’s hands becomes noticeably blurred after the instrument is removed. In the “train engine” example, removing the locomotive leaves a large, blurry artifact in its place. These examples suggest that object removal can substantially degrade the surrounding visual content, rather than produce a clean and plausible scene. Their inclusion on the demo page also raises questions about the effectiveness of the quality-control process.

Minor Weakness:

  1. The scope of instruction following is limited. AVIO is trained and evaluated using fixed templates—“a video with {object}” and “a video without {object}”—rather than diverse, free-form editing instructions. Its ability to handle more expressive requests therefore remains unclear. Compared with broader instruction-guided frameworks such as JAVEdit and InstructAV2AV, the demonstrated capabilities are restricted to category-level object addition and removal.
  2. The applicability of the proposed design across backbones remains unexplored. All experiments use JavisDiT++, leaving it unclear whether the benefits of source-conditioned modulation and the reference-guided curriculum transfer to other audiovisual generation backbones, such as LTX-2,Minimax-h3. A discussion of the backbone choice, and ideally an additional experiment with another backbone, would help clarify the generality of the approach.
Questions for authors
  1. I would encourage the authors to include comparisons with suitable baselines fine-tuned on the same AVIOBench training split, to distinguish the benefits of the proposed architecture from those of dataset-specific training.
  2. Additional evaluation on external data, such as held-out examples from JAVEdit-100k or relevant subsets of JAVEditBench, would help establish the generalization of the approach.
  3. A clearer justification for choosing JavisDiT++, together with evidence from another audiovisual backbone such as LTX-2, would help substantiate the generality of the proposed design.

Substantive evidence addressing these points would provide a basis for reconsidering my assessment.

5 comments
AuthorsOpenReview ↗

Baseline fine-tuned on AVIOBench, Zero-shot results on JAVEdit-100k, Component ablation, Backbone choice

We thank the reviewer for recognizing that audiovisual object addition and removal is an important, practically relevant task requiring coordinated visual and acoustic changes while preserving the scene, and that our approach is clearly presented and gives a practical architecture for unified addition and removal.

[Weakness 1] Each component is necessary. Paired tests on 200 randomly sampled test clips, scored by a Gemini judge that compares each output with its source and is validated against blinded human raters (exact McNemar):

change to the modeleffectp
source-conditioned modulation → cross-attentionbackground preserved: −42.5 pp (addition), −25.0 pp (removal)< 0.001
no reference frameobject added visually: 18.0 % → 0.5 %< 0.001

What AVIO adds over prior work is one model that removes or adds an object together with its sound, with or without a reference. InstructAV2AV [1] releases a separate checkpoint per task. On the design itself, Reviewer r1hr highlights three points: the unified design is economical, reference conditioning resolves the ambiguity in the appearance and placement of the added object, and reference access is controlled across methods.

[Weakness 2 and Question 1] We fine-tuned InstructAV2AV on the same AVIOBench training split at comparable compute. 200 randomly sampled test clips:

Rem. oursRem. fine-tunedAdd. oursAdd. fine-tuned
video: PSNR ↑20.0421.1518.7917.44
audio: KL ↓1.491.591.411.79
audio: LSD ↓12.8417.3111.6416.73
human core success(11 clips) %14.020.040.945.5

Core success: the object and its sound are both edited, the rest is preserved, and there is no major artifact (blinded human study, 11 questionnaires). Performance is comparable: human core success does not differ significantly, ours is better on the audio metrics on both tasks and on addition video, and the fine-tuned baseline is better on removal video. Raters find that ours preserves other sounds better in removal (86 % vs 70 %, significant). The fine-tuned baseline is two specialist models, one per task; ours is one set of weights for removal and for addition, with or without a reference. One checkpoint is preferable: each training pair supervises both directions, so an object and its sound are learned once and serve both tasks, and only one model has to be trained, stored and run.

[Weakness 3] We asked blinded raters to audit the pseudo-targets. Percentage of ratings that are not Pass:

pseudo-targetsremovaladdition
visual edit fails21.81.9
background damaged27.31.9
sound edit fails72.77.4
major artifact32.77.4

Addition targets are reliable. For removal, the visual checks pass in 73–78 % of ratings; such failures are most common for hand-held instruments and large foreground objects. The main limit is sound removal, which is a limit of current audio separation: our pipeline is on par with SAM-Audio [3] and well above three audio editors (demo page). We will release per-clip quality labels for the test set, so that users can filter such cases, and report results on the verified subset.

[Weakness 4] In the revised version we will constrain the scope to editing from an object name. We also ran the model, without retraining, on phrasings it never saw in training (an instruction, a descriptive sentence, a paraphrase; 40 clips). They behave close to the training template: for example, the sound is added in 35.0–42.5 % of clips against 35.0 %.Examples are under "Other prompts" on our demo page.

[Question 2] We tested zero-shot performance on JAVEdit-100k [2]: the model, without any retraining, is run on its source clips, and on real videos from an external pool whose YouTube ids are in no AVIOBench split. The model generalizes to some extent: removal works on compact objects. Examples are under "Zero-shot performance" on our demo page.

[Question 3] Our design is not tied to JavisDiT++. Source-conditioned modulation acts through the normalization layers and the reference frame through the input sequence. Larger audiovisual diffusion transformers such as LTX-2 share both, so the method can be applied to them. Because of resource constraints we tested it on JavisDiT++, a lightweight joint audio-video backbone (our model has 2.26 B parameters and trains in 359 H100-hours).

The ablation shows that the mechanism itself is useful: it supports inputs with and without a reference and both tasks in one backbone. We will state in the paper that the results are demonstrated on JavisDiT++ and that applying the design to larger backbones is the next step.

Demo page: https://goatlazy.github.io/ICLR2027/

[1] Zheng et al., InstructAV2AV, arXiv:2605.18467. [2] Chen et al., JAVEdit, arXiv:2606.03168. [3] Shi et al., SAM Audio, arXiv:2512.18099.

Reviewer_Rmu9OpenReview ↗

Response to Authors

Thank you for the rebuttal and the additional experiments. After reading the response and going through the demo page again, my concerns are not resolved.

I still think the novelty is limited. Both the data pipeline and the model are mostly a combination of existing tools (segmentation, inpainting, source separation, AdaLN conditioning, and a Ditto-style reference curriculum). Using one checkpoint for both addition and removal is convenient, but I do not see it as a new technical insight. The new same-data baseline also does not help the case: the fine-tuned InstructAV2AV actually has higher human core success on both removal and addition.

My bigger concern is the visual quality. On the demo page, "A4 Add white-tailed eagle" is heavily distorted and blurry. In my view it is clearly worse than the other methods shown in the same comparison. "A5 Add helicopter" has the same problem: the added helicopter is warped and deformed, while the other methods do not show this kind of distortion. Both examples are listed under your "Best results." If I ranked the methods on these examples, I would put AVIO behind the others. This also makes me doubt the reliability of the human study, which is based on only 11 questionnaires, because what I see on the demo page does not match the reported results.

Overall, I do not think this work meets the bar for ICLR, and I will maintain my score.

We thank the reviewer for reading our response and going through the demo page again. We answer each point in two comments (1/2).

1. Novelty. We agree that the building blocks (segmentation, inpainting, source separation, AdaLN conditioning, a Ditto-style reference schedule) exist. What is new is the audiovisual setting and what we learn about it. The edited object is a sounding object, and its sound matters as much as its appearance: an edit that removes the object from the picture but leaves its sound, or adds it without a sound, has failed.

Why audio is hard, and the audio of a sounding object harder. In an image, the removed object has a mask and all pixels outside it are kept by construction. A sound has no mask: it is mixed with all other sounds at every moment. Editing the sound of a sounding object is harder still, because the target is defined by the video and not by the audio. The model has to decide which part of the mixture belongs to the visible object, remove or add only that part, and keep it in time with what the object does on screen. The raters confirm this: in our pseudo-targets the object is visually removed in 78 % of ratings, its sound in only 27 %.

What the work contributes. Joint audiovisual object editing is a new and difficult task, and the strict joint criterion leaves clear room for improvement for all current editors. Our work paves the way for progress on it in four places, none of which is given by the tools themselves:

  • Dataset. AVIOBench provides paired removal and addition of a non-speech sounding object together with its sound, with pseudo-targets audited by raters. Its audio pipeline keeps the other sounds better than single-pass SAM-Audio [1] (84.5 % vs 72.5 %, p = 0.002). The data also improve other models: fine-tuning InstructAV2AV [2] on it raises its visual removal rate from 24.0 % to 47.2 %.
  • Evaluation. Core success, scored by a judge validated against blinded raters, makes the task measurable. It replaces per-modality scores that degenerate outputs could pass.
  • Audiovisual model with diverse inputs. One set of weights edits the video and the sound together in both directions, with a reference frame or from text alone. Without the reference the visual addition falls from 18.0 % to 0.5 %, while the sound is still added (29.5 % vs 33.0 %): the reference tells the model what the object looks like and where it is, and the object name is enough to generate its sound.
  • Conditioning that transfers to other backbones. Replacing source-conditioned modulation with cross-attention, the common choice, lowers background preservation by 42.5 pp on addition and 25.0 pp on removal (p < 0.001). It acts only through the normalization layers, so it is not tied to JavisDiT++ and can be applied to larger backbones such as LTX-2.

2. Same-data baseline. The human differences are −6.0 pp (95 % CI [−17.7, +7.7]) on removal and −4.5 pp [−34.1, +27.3] on addition. Neither is significant, so the two are on par, not the baseline ahead. On the sound, which is half of the task, ours is ahead:

  • Fine-tuned InstructAV2AV, 200 clips: KL 1.49 vs 1.59 (removal) and 1.41 vs 1.79 (addition); LSD 12.84 vs 17.31 and 11.64 vs 16.73. Raters find the other sounds preserved in 86 % of our removal ratings vs 70 % (+16 pp [+2, +32]).
  • Released InstructAV2AV, full test set, addition: core success 5.5 % vs 2.4 % (+3.1 pp [+1.7, +4.4]); the sound is added in 37.3 % of clips vs 22.2 %.

[1] Shi et al., SAM Audio: Segment Anything in Audio, arXiv:2512.18099. [2] Zheng et al., InstructAV2AV: Instruction-Guided Audio-Video Joint Editing, arXiv:2605.18467.

Reviewer_hmxLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces AVIOBench, a 37.9-hour dataset of object-present/object-absent audiovisual pairs constructed through segmentation, video inpainting, audio separation, and joint generative refinement. AVIO adapts JavisDiT++ with source-conditioned feature modulation and a reference-frame curriculum to add and remove sounding objects.

Critical strengths
  1. Coordinating edits to a visual object and its sound is closer to real creative workflows.
  2. The data pipeline explicitly considers cross-modal target identity. A shared object identity and visual mask guide audio and video processing. Multiple separation candidates and consistency filtering are reasonable design choices.
  3. The paper describes source modulation, initialization, conditioning dropout, and flow matching. Table 2 shows substantial reconstruction improvements for AdaLN conditioning over source cross-attention.
Critical weaknesses
  • My major concern is the quality of the pseudo-ground-truth targets and the reliability of the automatic judge. In Table 1(c), even the pseudo-ground-truth Oracle/Oracle achieves only 55.40% VES and 11.10% AES for removal, compared with 32.70% and 11.30% for AVIO. However, Appendix G does not clearly explain whether these low scores are due to incomplete removal or errors from the automatic judge. As a result, it is difficult to assess how reliable the pseudo-targets and evaluation results actually are. The construction filters also do not guarantee complete removal or preservation of the background content. I think a human evaluation of removal quality, artifacts, and content preservation, along with validation of the Gemini-based judge, is essential to address this concern.

  • Another concern is that the current evaluation does not clearly show whether the editing is actually successful. As shown in Appendix A.2, some degraded AVIO-XAttn outputs can still pass the removal checks, and AVC can pass even when there are no events to verify. This raises questions about whether the reported VES, AES, and AVC scores really reflect successful editing while preserving the original scene, especially since the judge does not compare the edited output with the source. I think a joint evaluation that considers all these aspects on the same clips would be more convincing. Also, while the 200-clip subset used in Table 2 explains the differences from Table 1, uncertainty estimates based on paired comparisons would help determine whether the small improvements in the ablation study are meaningful.

  • The addition setting is not well evaluated under realistic conditions. The current experiments use inpainted or refined source videos and reference frames derived from the target videos. It is therefore unclear how well the method would work on real, unedited videos or with imperfect user-provided references. While using reference frames is reasonable and consistent with the baselines, it does not show whether the main model can perform addition using only text instructions. The separately trained AVIO-NoRef model does not fully address this question. I think additional experiments on real videos, imperfect references, and the main model without reference frames would make the evaluation more convincing. Also, the instruction-following claims seem a bit too strong given that only two object-name templates are evaluated.

Questions for authors
  1. Can a stratified human audit establish pseudo-target quality and calibrate Gemini's judgments, including removal completeness and preservation of unrelated content?
  2. What fraction of clips jointly satisfies visual editing, acoustic editing, content preservation, and verifiable consistency? Can paired uncertainty estimates be supplied for the 200-clip ablation subset?
  3. Do gains persist against a baseline trained on the same AVIOBench data? Are baseline prompts compatible with each method's documented input conventions?
4 comments
AuthorsOpenReview ↗

Human Evaluation and New Experiments

We thank the reviewer for the constructive review and for recognizing that audiovisual object addition and removal is an important, practically relevant task requiring coordinated visual and acoustic changes while preserving the scene, and that our unified design is practical and clearly presented.

[Weakness 1 and Question 1] We ran a blinded human study and collected 11 questionnaires. Percentage of ratings that are not Pass:

pseudo-targetsremovaladdition
sound edit fails72.77.4
visual edit fails21.81.9
background damaged27.31.9
other sounds damaged25.51.9
major artifact32.77.4
core success16.483.3

The raters confirm that the judge's scores in Table 1(c) are reasonable: like the judge (VES 55.4 %, AES 11.1 %), they find the object visually removed far more often than its sound (78 % vs 27 %). The low AES thus reflects the limit of current audio separation, not a judge error. Our pipeline is already at the level of the best available method: on 200 randomly sampled test clips our removal audio succeeds on 16.0 %, on par with SAM-Audio [1] (17.0 %) while keeping other sounds better (84.5 % vs 72.5 %), and well above CoherentAVEdit [2], ZEUS [3] and AUDIT [4] (1.0–4.5 %). Addition pseudo-targets are reliable (83 % core success). We will release a per-clip "verified" flag and report removal on that subset.

Judge validation. We further updated the judge to the newly released Gemini model and let it compare the edited output with the source, answering the same questions as the raters. On 86 rated outputs it agrees with the rater majority on 80–90 % of removal outputs per question (κ = 0.52–0.80), and on core success on 93 % (removal) and 70 % (addition, where it is stricter). The revised appendix will include those new results.

[Weakness 2 and Question 2] Joint success. Core success requires, on one clip, the edit in both modalities, preservation of unrelated visual and audio content, and no major artifact. The judge, updated to the new Gemini model, first decides whether the clip has a checkable audio-visual event, so a clip without one cannot pass consistency. Full test set (1,425 clips):

% of clipscore successcore success and verified consistency
removal, ours0.90.0
removal, pseudo-target4.80.7
addition, ours5.55.0
addition, pseudo-target43.641.4

AVIO-XAttn outputs now receive core success 0 %.

Paired tests on those 200 clips (updated judge, exact McNemar vs the main model): without the reference, visual addition drops by 17.5 pp; with cross-attention conditioning, background preservation drops by 42.5 pp (addition) and 25.0 pp (removal); all p < 0.001. We will add these tests to Table 2.

[Weakness 3] Further tests, with examples for each on our demo page:

  • Main model without a reference: the sound is still added from text alone (29.5 % vs 33.0 % with the reference, not significant). The visual addition needs the reference, as text does not fix appearance and placement.
  • Imperfect references: a frame taken from a later moment works almost as well as the exact first frame. Heavily damaged frames fail.
  • Real, unedited videos: the model generalizes to some extent. Removal works on compact objects (dog, rooster, train); addition works in some cases with a first frame from an independent image editor (no target video).

Thank you for pointing this out. In the revised version we will constrain the scope to first-frame-conditioned addition from an object name.

[Question 3] We fine-tuned InstructAV2AV [5] on AVIOBench at comparable compute. 200 test clips:

Removal. oursRemoval. fine-tunedAdd. oursAdd. fine-tuned
human core success(11 clips) %14.020.040.945.5
video: PSNR ↑20.0421.1518.7917.44
audio: KL ↓1.491.591.411.79
audio: LSD ↓12.8417.3111.6416.73

Performance is comparable: human core success does not differ significantly. Ours is better on the audio metrics on both tasks and on addition video; the fine-tuned baseline is better on removal video. Raters find that ours preserves other sounds better in removal (86 % vs 70 %, significant). The advantage of our model is that one set of weights handles both removal and addition, with or without a reference frame, while the baseline needs a separately fine-tuned model for each task.

Prompts. Yes. Each baseline receives the object name in its own input format: InstructAV2AV, an instruction model, receives "Add / Remove the {object}."; CoherentAVEdit and AVI-Edit take a text description and receive "a video with / without {object}", the same as our model.

Demo page: https://goatlazy.github.io/ICLR2027/

[1] SAM Audio, arXiv:2512.18099. [2] CoherentAVEdit, arXiv:2512.07209. [3] ZEUS, ICML 2024. [4] AUDIT, NeurIPS 2023. [5] InstructAV2AV, arXiv:2605.18467.

Reviewer_hmxLOpenReview ↗

Comment

Thank the authors for the additional experiments. However, i still have some concerns on the paper.

  • Could you clarify the relationship between the “11 questionnaires,” “11 clips,” and “86 rated outputs,” including the number of unique clips per task and raters per clip? Please also provide the core-success confusion matrices between the updated judge and human majority votes. This would help interpret the substantially different success rates reported for the human-rated subset and the full test set, particularly for addition.

  • How will the proposed “verified” subset be constructed—through human annotation or automatic filtering—and how many removal examples qualify? Please evaluate AVIO and the fine-tuned InstructAV2AV baseline on the same subset, selected solely by pseudo-target quality and independently of either model’s outputs, with confidence intervals for core success. This would help determine whether the low success rates primarily reflect unreliable targets or limitations of the editing models.

  • For the paper presentation, the figure quality is not good. Several plots and diagrams appear rasterized and become blurry when enlarged. The audio waveform visualizations also lack a consistent presentation across examples and methods.

I also remain concerned about the methodological novelty. Constructing object-addition supervision by reversing object-removal pairs has clear precedents in image editing, including EraseDraw (ECCV 2024) and Paint by Inpaint (CVPR 2025). Extending this idea to audiovisual scenes introduces meaningful challenges in sound attribution, content preservation, and temporal consistency. However, the current evidence does not clearly establish a substantial technical advance in addressing these challenges. The updated curriculum comparison shows no significant advantage, and the same-data baseline achieves comparable human-rated core success. While the unified checkpoint is practically useful, the overall methodological contribution appears incremental.

Overall, thanks for additional experiments. I can raise my score but still tend to weak reject current version.

AuthorsOpenReview ↗

Details of Human Study

We thank the reviewer for reading the response and for raising the score. Our answer is in two comments (1/2).

1. Human study: questionnaires, clips and rated outputs. Thank you for pointing this out. The two numbers coincide but mean different things: raters returned 11 questionnaires, and the system comparison uses 11 clips per task(addition/removal). The structure is:

  • There are two questionnaire versions. Each has 43 questions. One question shows the source, the instruction and one anonymised output, and asks the six checks (object edited, sound edited, background preserved, other sounds preserved, visual and audio artifacts).
  • The 43 questions are 10 pseudo-targets (5 removal, 5 addition) and 33 system outputs (11 clips × 3 systems: ours, fine-tuned InstructAV2AV, and our video paired with the audio of a fine-tuned audio editor; its results are on the demo page, §7).
  • 2 × 43 = 86 rated outputs from 42 unique clips: 21 per task (10 for the pseudo-target audit, 11 for the system comparison, no overlap).
  • Raters returned 11 questionnaires, so every output has 4 to 6 raters (391 ratings in total).

Core success, rater majority against the judge (6 tied addition outputs left out):

judge passjudge fail
removal, raters pass32
removal, raters fail137
addition, raters pass129
addition, raters fail214

The two sets of rates are not expected to match, for two simple reasons. First, the judge is stricter than the raters: on the same addition outputs it passes 38 % where the raters pass 57 %, and it rarely passes an output that they fail (2 of 16). The reason is that the judge follows a more detailed rubric. Raters and judge see the same reference frame, and raters decide by overall impression whether the object was added. The judge passes the visual check only if the object matches the reference, fits the scene (scale, lighting, occlusion) and stays the same object for the whole clip. As a result, raters see the object added in 92 % of outputs and the judge in 67 %. Its rates on the full test set are therefore conservative. Second, the study has 11 clips per task(addition/removal), so one clip changes a rate by 9 points.

We received 11 questionnaires, some of which were partially completed. Across both versions, we collected 391 output ratings, with 4–6 raters per output.

2. Verified subset. It is built by automatic filtering: the validated judge is applied to the pseudo-target only, and no model output is used. On removal, 261 of 1,425 test clips pass the sound check (we call these verified) and 69 pass all checks. We ran the judge on the fine-tuned InstructAV2AV for this comparison; its outputs exist for the 200-clip test sample, which contains 41 verified clips.

removal, % [95 % CI]clipscore success: ourscore success: fine-tuned Instructav2avsound removed: ourssound removed: fine-tuned Instructav2av
200-clip sample, not verified1580.0 [0.0, 2.4]1.9 [0.6, 5.4]10.1 [6.3, 15.8]13.9 [9.4, 20.2]
200-clip sample, verified412.4 [0.4, 12.6]2.4 [0.4, 12.6]29.3 [17.6, 44.5]22.0 [12.0, 36.7]
full test, not verified1,1640.7 [0.3, 1.4]–11.5 [9.8, 13.5]–
full test, verified2611.9 [0.8, 4.4]–38.7 [33.0, 44.7]–
full test, pseudo-target passes all checks697.2 [3.1, 15.9]–46.4 [35.1, 58.0]–

This answers the reviewer's question in two parts.

  • Unreliable targets explain the low sound removal. On verified clips our model removes the sound about three times as often as on the others (38.7 % vs 11.5 % on the full test set; 29.3 % vs 10.1 % on the 200-clip sample). The fine-tuned Instructav2av baseline also improves, but less (22.0 % vs 13.9 %). This agrees with our earlier answer that sound removal is limited by audio separation.
  • The rest is a limit of current editing models. Core success rises with target quality (0.7 % → 1.9 % → 7.2 %) but stays low for both models even on verified clips, because it requires all six checks on one clip. The two models are equal on core success there (difference 0.0 pp [−7.3, +7.3]).

In short, when the targets are reliable, our single model matches the two task-specific fine-tuned Instructav2av models on core success and gains the most on the sound. It removes the sound in 29.3 % of verified clips against 22.0 % for the fine-tuned baseline (41 clips, not significant), and in 38.7 % against 20.7 % for the released InstructAV2AV on all 261 verified clips (p < 0.001).

(2/2, continued from the previous comment.)

3. Figures. We will fix them in the final version: all plots and diagrams will be vector graphics, and the audio visualizations will use one format across examples and methods.

4. Novelty. We agree that building addition pairs by reversing removal pairs follows EraseDraw [1] and Paint by Inpaint [2], and we will cite both. Those works edit an image. Our setting is audiovisual: the edited object is a sounding object, and its sound matters as much as its appearance. An edit that removes the object from the picture but leaves its sound, or adds it without a sound, has failed. This joint requirement has no counterpart in image editing, and the audio side is where most of the difficulty lies.

Why audio is hard, and the audio of a sounding object harder. In an image, the removed object has a mask and all pixels outside it are kept by construction. A sound has no mask: it is mixed with all other sounds at every moment. Editing the sound of a sounding object is harder still, because the target is defined by the video and not by the audio. The model has to decide which part of the mixture belongs to the visible object, remove or add only that part, and keep it in time with what the object does on screen. The raters confirm this: in our pseudo-targets the object is visually removed in 78 % of ratings, its sound in only 27 %.

What we gain on audio.

  • Data: our pseudo-target audio keeps the other sounds better than single-pass SAM-Audio (84.5 % vs 72.5 %, p = 0.002) and succeeds far more often than three audio editors (16.0 % vs 1.0–4.5 %).
  • Model, against the baseline trained on the same data: our audio is closer to the target on both tasks (KL 1.49 vs 1.59 and 1.41 vs 1.79; LSD 12.8 vs 17.3 and 11.6 vs 16.7), and raters find the other sounds better preserved in removal (86 % vs 70 %, significant).
  • Against the released InstructAV2AV on the 261 verified clips, our model removes the sound in 38.7 % vs 20.7 % of clips (p < 0.001).

More diverse inputs with one set of weights. The image precedents take one image and a text instruction, and InstructAV2AV uses a separate checkpoint for each edit type. Our single model accepts:

  • both directions, removal and addition;
  • text with a reference frame, or text alone (the sound is still added in 29.5 % of clips, against 33.0 % with the reference);
  • a reference frame from a later moment, which works almost as well as the exact first frame.

What the work contributes. Joint audiovisual object editing is a new and difficult task, and the strict joint criterion leaves clear room for improvement for all current editors. Our work paves the way for progress on it in four places:

  • Dataset. AVIOBench provides paired removal and addition of a sounding object together with its sound, with pseudo-targets audited by raters. It also improves other models: fine-tuning InstructAV2AV on it raises its visual removal rate from 24.0 % to 47.2 % (200 clips).
  • Evaluation. Core success, scored by a judge validated against blinded raters, makes the task measurable. It replaces per-modality scores that degenerate outputs could pass, and it shows how far all current editors are from the goal.
  • Audiovisual model. One model edits the video and the sound together in both directions. Against the same-data baseline it is closer to the target audio on both tasks and preserves the other sounds better, as listed above.
  • Conditioning that transfers to other backbones. Source-conditioned modulation acts only through the normalization layers, which every audiovisual diffusion transformer has, so it is not tied to JavisDiT++ and can be applied to larger backbones such as LTX-2.

[1] Canberk et al., EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Images, ECCV 2024, arXiv:2409.00522. [2] Wasserman et al., Paint by Inpaint: Learning to Add Image Objects by Removing Them First, CVPR 2025, arXiv:2404.18212.

Reviewer_r1hrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AVIOBench and AVIO for coordinated addition and removal of sounding objects in audiovisual scenes. AVIOBench contains 37.9 hours of paired examples spanning 1,878 target names. Its automated pipeline identifies a primary sounding object, obtains its mask, removes it visually through inpainting, and constructs an object-absent audio track using source separation. Multiple separation candidates are ranked through audiovisual correspondence, and a joint audiovisual denoising stage refines the edited streams. Swapping the original and edited clips produces supervision for both removal and addition, yielding 24,444 unique training pairs, 1,389 validation pairs, and 1,425 test pairs.

Critical strengths
  1. The model is deliberately economical in its added machinery. Source-aligned feature modulation introduces about 7.23 million parameters into a 2.26-billion-parameter editor and retains the backbone's joint temporal representation. A single checkpoint handles both directions of editing, which avoids assembling separate addition and removal systems. The reference mechanism also addresses a real ambiguity: a category-level instruction cannot uniquely determine a new object's appearance or initial position. Supplying the same edited first frame to competing video-generating methods for addition is an appropriate attempt to control this advantage.
  2. The experimental presentation is commendably explicit about privileged information. Oracle audio and video supplied to cross-modal baselines are marked, and the paper does not present those systems as operating under identical inputs. The appendix also confronts evaluation failures rather than hiding them: degraded outputs can pass absence checks, and clips with no checkable event can receive positive consistency judgments. These observations are valuable in themselves. Together with reconstruction and perceptual measures, they provide a more honest account of the gap between apparently successful target deletion and an edit that actually preserves a coherent surrounding scene.
Critical weaknesses
  1. The consistency metric has a related degeneracy: clips without a checkable audiovisual event receive a positive judgment. The source-attention ablation illustrates that severe degradation can increase removal and consistency scores. Existing measurements therefore do not cleanly separate successful editing from destruction of evidence. Preservation-aware human judgments, joint success rates, and explicit treatment of unscorable clips are needed. The addition model also has worse DeSync than the compared joint editor, so improved reconstruction should not be conflated with universally better audiovisual coordination.
  2. Several design claims remain only partly isolated. The reference ablation compares a model using a target first frame against one trained and tested without it; this cannot identify the benefit of the curriculum separately from the availability of privileged visual information. The dataset construction and learned model are likewise not disentangled through matched-data baseline fine-tuning. Finally, the split protocol does not establish separation by original YouTube identity or target category, and the predominance of three broad source groups leaves generalization to rare or unseen sounding objects insufficiently characterized.
Questions for authors
  1. Could you manually audit a stratified sample of the removal pseudo-targets and model outputs, separating incomplete sound removal, visual remnants, background damage, and judge mistakes? Given the reference AES of 11.10%, this evidence is necessary to determine whether the main bottleneck lies in the dataset, the evaluator, or the editor.
  2. Can you report a joint editing measure requiring target removal or addition in both modalities while preserving unrelated content? Please distinguish genuinely synchronized clips from those with no checkable event, and evaluate agreement with blinded human raters. This would address the degeneracy demonstrated by the degraded source-attention outputs.
  3. How does progressive reference dropout compare with fixed-probability dropout and ordinary mixed reference/no-reference training under the same budget? Please also test the full model without a reference at inference and with an imperfect user-edited frame, since the current addition comparison uses a first frame from the desired target video.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.