- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AVIOBench, a paired dataset for audiovisual object addition and removal, and AVIO, a unified model for editing visual objects and their associated sounds. AVIO adapts a pretrained text-to-audiovisual diffusion transformer through source-conditioned feature modulation and a reference-guided training curriculum. The paper evaluates the model on AVIOBench using quantitative metrics and qualitative comparisons.
Critical strengths
- Audiovisual editing is an important and practically relevant task. Object addition and removal are fundamental operations that require coordinated changes across visual and acoustic content while preserving the surrounding scene.
- The approach is straightforward and clearly presented. Adapting a pretrained audiovisual DiT with source conditioning and progressively dropping reference-frame guidance provides a practical architecture for unified addition and removal.
Critical weaknesses
Major Weakness:
-
The methodological novelty appears limited. The dataset construction pipeline primarily combines existing segmentation, audio separation, video inpainting, and audiovisual refinement models. The model also builds on established components, including source-conditioned modulation and reference-guided training. Closely related work such as InstructAV2AV already adapts audiovisual generation backbones using paired editing data, while the manuscript explicitly draws inspiration from Ditto for its reference-guided curriculum. Although integrating these components may be useful, the paper does not sufficiently establish what new technical insight or capability distinguishes AVIO from existing approaches. A stronger justification of the dataset contribution would also require more rigorous quality validation.
-
The experimental comparison does not isolate the benefits of the proposed method from those of dataset-specific training. AVIO is trained on AVIOBench, whereas the baselines use released checkpoints without adaptation to this dataset. Consequently, the reported gains may partly reflect differences in training distribution rather than the proposed architecture. I suggest that the authors: (1) fine-tune suitable baselines on the same training split, or provide controlled comparisons using the same backbone and training data; and (2) evaluate generalization on external data, such as held-out examples from JAVEdit-100k and relevant subsets of JAVEditBench.
-
The qualitative examples raise concerns about the quality of the constructed dataset and the resulting editing model. Some examples on the demo page exhibit conspicuous artifacts after object removal. In the “flute” example, the region around the person’s hands becomes noticeably blurred after the instrument is removed. In the “train engine” example, removing the locomotive leaves a large, blurry artifact in its place. These examples suggest that object removal can substantially degrade the surrounding visual content, rather than produce a clean and plausible scene. Their inclusion on the demo page also raises questions about the effectiveness of the quality-control process.
Minor Weakness:
- The scope of instruction following is limited. AVIO is trained and evaluated using fixed templates—“a video with {object}” and “a video without {object}”—rather than diverse, free-form editing instructions. Its ability to handle more expressive requests therefore remains unclear. Compared with broader instruction-guided frameworks such as JAVEdit and InstructAV2AV, the demonstrated capabilities are restricted to category-level object addition and removal.
- The applicability of the proposed design across backbones remains unexplored. All experiments use JavisDiT++, leaving it unclear whether the benefits of source-conditioned modulation and the reference-guided curriculum transfer to other audiovisual generation backbones, such as LTX-2,Minimax-h3. A discussion of the backbone choice, and ideally an additional experiment with another backbone, would help clarify the generality of the approach.
Questions for authors
- I would encourage the authors to include comparisons with suitable baselines fine-tuned on the same AVIOBench training split, to distinguish the benefits of the proposed architecture from those of dataset-specific training.
- Additional evaluation on external data, such as held-out examples from JAVEdit-100k or relevant subsets of JAVEditBench, would help establish the generalization of the approach.
- A clearer justification for choosing JavisDiT++, together with evidence from another audiovisual backbone such as LTX-2, would help substantiate the generality of the proposed design.
Substantive evidence addressing these points would provide a basis for reconsidering my assessment.
5 comments
Response to Authors
Thank you for the rebuttal and the additional experiments. After reading the response and going through the demo page again, my concerns are not resolved.
I still think the novelty is limited. Both the data pipeline and the model are mostly a combination of existing tools (segmentation, inpainting, source separation, AdaLN conditioning, and a Ditto-style reference curriculum). Using one checkpoint for both addition and removal is convenient, but I do not see it as a new technical insight. The new same-data baseline also does not help the case: the fine-tuned InstructAV2AV actually has higher human core success on both removal and addition.
My bigger concern is the visual quality. On the demo page, "A4 Add white-tailed eagle" is heavily distorted and blurry. In my view it is clearly worse than the other methods shown in the same comparison. "A5 Add helicopter" has the same problem: the added helicopter is warped and deformed, while the other methods do not show this kind of distortion. Both examples are listed under your "Best results." If I ranked the methods on these examples, I would put AVIO behind the others. This also makes me doubt the reliability of the human study, which is based on only 11 questionnaires, because what I see on the demo page does not match the reported results.
Overall, I do not think this work meets the bar for ICLR, and I will maintain my score.
We thank the reviewer for reading our response and going through the demo page again. We answer each point in two comments (1/2).
1. Novelty. We agree that the building blocks (segmentation, inpainting, source separation, AdaLN conditioning, a Ditto-style reference schedule) exist. What is new is the audiovisual setting and what we learn about it. The edited object is a sounding object, and its sound matters as much as its appearance: an edit that removes the object from the picture but leaves its sound, or adds it without a sound, has failed.
Why audio is hard, and the audio of a sounding object harder. In an image, the removed object has a mask and all pixels outside it are kept by construction. A sound has no mask: it is mixed with all other sounds at every moment. Editing the sound of a sounding object is harder still, because the target is defined by the video and not by the audio. The model has to decide which part of the mixture belongs to the visible object, remove or add only that part, and keep it in time with what the object does on screen. The raters confirm this: in our pseudo-targets the object is visually removed in 78 % of ratings, its sound in only 27 %.
What the work contributes. Joint audiovisual object editing is a new and difficult task, and the strict joint criterion leaves clear room for improvement for all current editors. Our work paves the way for progress on it in four places, none of which is given by the tools themselves:
- Dataset. AVIOBench provides paired removal and addition of a non-speech sounding object together with its sound, with pseudo-targets audited by raters. Its audio pipeline keeps the other sounds better than single-pass SAM-Audio [1] (84.5 % vs 72.5 %, p = 0.002). The data also improve other models: fine-tuning InstructAV2AV [2] on it raises its visual removal rate from 24.0 % to 47.2 %.
- Evaluation. Core success, scored by a judge validated against blinded raters, makes the task measurable. It replaces per-modality scores that degenerate outputs could pass.
- Audiovisual model with diverse inputs. One set of weights edits the video and the sound together in both directions, with a reference frame or from text alone. Without the reference the visual addition falls from 18.0 % to 0.5 %, while the sound is still added (29.5 % vs 33.0 %): the reference tells the model what the object looks like and where it is, and the object name is enough to generate its sound.
- Conditioning that transfers to other backbones. Replacing source-conditioned modulation with cross-attention, the common choice, lowers background preservation by 42.5 pp on addition and 25.0 pp on removal (p < 0.001). It acts only through the normalization layers, so it is not tied to JavisDiT++ and can be applied to larger backbones such as LTX-2.
2. Same-data baseline. The human differences are −6.0 pp (95 % CI [−17.7, +7.7]) on removal and −4.5 pp [−34.1, +27.3] on addition. Neither is significant, so the two are on par, not the baseline ahead. On the sound, which is half of the task, ours is ahead:
- Fine-tuned InstructAV2AV, 200 clips: KL 1.49 vs 1.59 (removal) and 1.41 vs 1.79 (addition); LSD 12.84 vs 17.31 and 11.64 vs 16.73. Raters find the other sounds preserved in 86 % of our removal ratings vs 70 % (+16 pp [+2, +32]).
- Released InstructAV2AV, full test set, addition: core success 5.5 % vs 2.4 % (+3.1 pp [+1.7, +4.4]); the sound is added in 37.3 % of clips vs 22.2 %.
[1] Shi et al., SAM Audio: Segment Anything in Audio, arXiv:2512.18099. [2] Zheng et al., InstructAV2AV: Instruction-Guided Audio-Video Joint Editing, arXiv:2605.18467.
Baseline fine-tuned on AVIOBench, Zero-shot results on JAVEdit-100k, Component ablation, Backbone choice
We thank the reviewer for recognizing that audiovisual object addition and removal is an important, practically relevant task requiring coordinated visual and acoustic changes while preserving the scene, and that our approach is clearly presented and gives a practical architecture for unified addition and removal.
[Weakness 1] Each component is necessary. Paired tests on 200 randomly sampled test clips, scored by a Gemini judge that compares each output with its source and is validated against blinded human raters (exact McNemar):
What AVIO adds over prior work is one model that removes or adds an object together with its sound, with or without a reference. InstructAV2AV [1] releases a separate checkpoint per task. On the design itself, Reviewer r1hr highlights three points: the unified design is economical, reference conditioning resolves the ambiguity in the appearance and placement of the added object, and reference access is controlled across methods.
[Weakness 2 and Question 1] We fine-tuned InstructAV2AV on the same AVIOBench training split at comparable compute. 200 randomly sampled test clips:
Core success: the object and its sound are both edited, the rest is preserved, and there is no major artifact (blinded human study, 11 questionnaires). Performance is comparable: human core success does not differ significantly, ours is better on the audio metrics on both tasks and on addition video, and the fine-tuned baseline is better on removal video. Raters find that ours preserves other sounds better in removal (86 % vs 70 %, significant). The fine-tuned baseline is two specialist models, one per task; ours is one set of weights for removal and for addition, with or without a reference. One checkpoint is preferable: each training pair supervises both directions, so an object and its sound are learned once and serve both tasks, and only one model has to be trained, stored and run.
[Weakness 3] We asked blinded raters to audit the pseudo-targets. Percentage of ratings that are not Pass:
Addition targets are reliable. For removal, the visual checks pass in 73–78 % of ratings; such failures are most common for hand-held instruments and large foreground objects. The main limit is sound removal, which is a limit of current audio separation: our pipeline is on par with SAM-Audio [3] and well above three audio editors (demo page). We will release per-clip quality labels for the test set, so that users can filter such cases, and report results on the verified subset.
[Weakness 4] In the revised version we will constrain the scope to editing from an object name. We also ran the model, without retraining, on phrasings it never saw in training (an instruction, a descriptive sentence, a paraphrase; 40 clips). They behave close to the training template: for example, the sound is added in 35.0–42.5 % of clips against 35.0 %.Examples are under "Other prompts" on our demo page.
[Question 2] We tested zero-shot performance on JAVEdit-100k [2]: the model, without any retraining, is run on its source clips, and on real videos from an external pool whose YouTube ids are in no AVIOBench split. The model generalizes to some extent: removal works on compact objects. Examples are under "Zero-shot performance" on our demo page.
[Question 3] Our design is not tied to JavisDiT++. Source-conditioned modulation acts through the normalization layers and the reference frame through the input sequence. Larger audiovisual diffusion transformers such as LTX-2 share both, so the method can be applied to them. Because of resource constraints we tested it on JavisDiT++, a lightweight joint audio-video backbone (our model has 2.26 B parameters and trains in 359 H100-hours).
The ablation shows that the mechanism itself is useful: it supports inputs with and without a reference and both tasks in one backbone. We will state in the paper that the results are demonstrated on JavisDiT++ and that applying the design to larger backbones is the next step.
Demo page: https://goatlazy.github.io/ICLR2027/
[1] Zheng et al., InstructAV2AV, arXiv:2605.18467. [2] Chen et al., JAVEdit, arXiv:2606.03168. [3] Shi et al., SAM Audio, arXiv:2512.18099.