acceptodds
Withdrawn from ICLR 2027

BEYOND TARGET MATCHING: MEASURING AND MITIGATINGNON-TARGETLEAKAGE INSELECTIVE VIDEO-TO-AUDIOGENERATION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selective video-to-audio (V2A) generation aims to synthesize sounds only for user-specified sources while suppressing other visible sound sources in the scene. Generated audio may contain the requested sound along with sounds from other visible sources. We call this non-target leakage and evaluate source selectivity through both target retention and competing-source suppression. To measure this behavior, we introduce VGG-SourcePair, a paired evaluation benchmark for selective generation in dual-source scenes. Given the same silent video containing two sound sources, we alternately designate each source as the target and jointly evaluate target preservation and non-target leakage, providing a more direct measure of source selection ability. We further propose FocusV2A, which uses the target prompt to select local visual evidence relevant to the requested source before audio generation. A FocusSelector estimates the target relevance of visual features and uses it to modulate spatial attention, yielding a target-focused visual condition. Training proceeds in two stages: the first learns target-aware visual selection, and the second adapts the generator to the resulting visual condition. At inference time, only a silent video and a target prompt are required. Experiments show that FocusV2A achieves stronger source selection performance on VGG-SourcePair while maintaining competitive audio quality and audiovisual synchronization on VGG-MonoAudio.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video-to-Audio Generation, Selective Sound Synthesis, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Pe5COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies selective video-to-audio (V2A) generation, where audio should be generated only for a user-specified source among multiple visible sources. It argues that generating the target is not sufficient if competing visible sources leak into the output.

The paper addresses three research questions:

-R1: How can source selectivity be evaluated so that both target retention and non-target leakage are measured?

-R2: Can leakage be reduced by selecting target-relevant local visual evidence before the visual condition reaches the generator?

-R3: How can a pretrained generator be adapted to such a target-focused visual condition without degrading quality and synchronization?

The corresponding proposed methods are:

-M1: VGG-SourcePair, a paired benchmark of two-source composites. Each source is requested in turn and scored against both component recordings, with dedicated metrics for target retention and non-target leakage.

-M2: Focus Selector, which competitively allocates each local visual token between the target and its complement and biases the frozen spatial aggregation attention of the visual encoder. It is trained to match unmixed-component teacher features (Stage 1).

-M3: Stage-2 adaptation of MMAudio to the selected visual condition, with the selector frozen.

FocusV2A outperforms the video-conditioned baselines on VGG-SourcePair source-selection metrics with competitive audio quality, and it improves results on VGG-MonoAudio. The evaluation also includes a human study, component ablations, and a matched-vs-opposite selection analysis.

Critical strengths

-R1/M1: The paper proposes metrics that explicitly penalize the presence of the non-target source (Leak@Dur, TempSuccess), in addition to target correspondence. This aspect is under-emphasized in the paper, but it is the most reusable part of M1. Controlled mixtures validate the metric behavior (Fig. 6), and detector calibration is disjoint from the test set.

-R2/M2: The Focus Selector reweights frozen attention multiplicatively, so pretrained features are preserved. Removing it raises FAD from 24.75 to 30.10 (Table 3). The matched-vs-opposite comparison, with checkpoint, text, and noise fixed, shows that the selection path causally changes the generated content.

Critical weaknesses

The main limitation of the paper is limited novelty relative to prior selective V2A work (SelVA and VGG-MonoAudio).

-R1/M1: The distinction between VGG-SourcePair and the existing VGG-MonoAudio is only briefly stated (Sec. 2.2) and is not sufficiently justified. Both target the evaluation of selective V2A generation, and both concatenate two videos. The novelty therefore lies mainly in the metric suite that explicitly measures counterpart leakage (non-target reference, Leak@Dur, TempSuccess), not in the dataset, and the paper does not clearly stress this. Moreover, these metrics are not applied to VGG-MonoAudio. It is unclear why a new benchmark is needed rather than an extension of the existing one.

-R2/M2: The distinction between FocusV2A and SelVA is not clear either. Both adopt a two-stage scheme: (1) fine-tuning the video feature extractor with a teacher–student scheme on mixed videos, and (2) fine-tuning MMAudio. The main distinction is the Focus Selector, which substitutes for SelVA’s text-guided cross-attention. However:

  • The paper provides no analysis or intuition on why FocusV2A performs better than SelVA.

  • Competition itself contributes marginally (Table 3, w/o Competition: RefSep 0.3400 vs. 0.3441, TempSuccess 0.7573 vs. 0.7635, nearly unchanged FAD).

  • The comparison with SelVA is not controlled. The variant without any visual selection (w/o Focus Selector) already outperforms SelVA on source-selection metrics by a large margin (RefSep 0.3324 vs. 0.2177; TempSuccess 0.7764 vs. 0.6130). This gap cannot be attributed to the Focus Selector. It may stem from SelVA’s selection mechanism itself (e.g., if modifying the visual features degrades the condition, whereas FocusV2A keeps them frozen) or from differences in training recipe, and the two cannot be separated without a controlled comparison.

A fair controlled comparison between the Focus Selector and cross-attention, under the same data, schedule, and generator adaptation, is required to support the claimed contribution.

In addition, the evaluation does not verify whether the generated target is grounded in the target’s visual events:

  • DeSync on VGG-SourcePair is measured between the generated audio and the mixed video (Appendix B). This is ill-posed, because the mixed video contains visible events from both the target and the counterpart.
  • Target@Dur and TempSuccess consider only the presence of the target class in each window. They do not penalize sound events that are off-sync with the video, so temporally over-filled outputs are rewarded.
  • The human study has no item on audio-visual synchronization alongside the target/non-target identity assessment.
Questions for authors
  1. Can you provide a controlled comparison that replaces the Focus Selector with SelVA’s cross-attention (with supplementary tokens) under the same training recipe?
  2. Please specify the Focus Selector’s decoder:
  • Do the target and complement banks share one decoder and attend to each other?
  • Do the queries attend to all spatiotemporal tokens at once (~37k for an 8 s clip), or per segment or temporal position?
  • How does the prompt representation condition the query banks?
  • Are the logits ℓᵗ and ℓᶜ computed with the projected or the raw visual features?
  1. In w/o Focus Selector, is the CLIP image condition also empty?
  2. Can you report RefSep, Leak@Dur, and TempSuccess on VGG-MonoAudio? Its target references and counterpart metadata appear sufficient to compute them.
  3. Please report DeSync against the target-only view and the non-target view separately, under matched and opposite selection.
  4. The Table 3 differences are within about 1 pp. Please report confidence intervals (the bootstrap procedure is described in Sec. 5.1 but not reported in Tables 1 and 3) or results over multiple seeds.
  5. The human study uses 12 author-selected videos and reports no significance tests. Can you add confidence intervals or paired tests?

Suggestions (not affecting my score)

  • The Focus Selector design builds on ideas that should be cited: competitive allocation across slots (Slot Attention, Locatello et al., NeurIPS 2020), learned-query decoders producing per-token masks (Mask2Former, Cheng et al., CVPR 2022), and additive attention-logit biases, of which the log-bias is a soft form of Mask2Former’s masked attention.
Reviewer_WgfmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies selective video-to-audio (V2A) generation, where the model should generate sound only for a user-specified source in a video with several visible sound sources. The authors argue that evaluation should check not only whether the target is generated, but also whether the other visible source leaks into the output (non-target leakage). To measure this, they build VGG-SourcePair, which places two VGGSound test clips side by side and requests each source in turn. Each output is evaluated against both component recordings, using CLAP-based margins and window-level PANN detections. They also propose FocusV2A, built on MMAudio. A Focus Selector uses the text prompt to predict target/complement assignments over local Synchformer tokens, and these assignments are added as a log-bias to the frozen spatial attention. The training process has two stages, where the selector is first trained to match the features of the unmixed component videos, and then part of the generator is adapted. FocusV2A outperforms SelVA and general V2A models on VGG-SourcePair and is competitive on VGG-MonoAudio. The paper also includes ablations, an attention analysis, and a small user study.

Critical strengths
  • The problem framing is useful. Measuring target retention and non-target leakage separately, with paired requests where the two sources swap roles, is a clear improvement over only checking correspondence to the target (as in VGG-MonoAudio). The benchmark construction is carefully documented, and the controlled-mixture analysis in Figure 6 is a good sanity check that the metrics react to changes in the competing source.

  • The method is simple and lightweight. It only biases the frozen spatial attention, keeps the pretrained visual features unchanged, and trains a small number of parameters. It improves over SelVA consistently on both benchmarks and in the user study, and the attention analysis shows that the selection does shift toward the requested half of the frame.

Critical weaknesses
  • My main concern is that VGG-SourcePair may mostly measure whether a model follows the text label, rather than whether it selects the right visual source. The benchmark places two unrelated clips side by side, so the sources never overlap or interact, and selection basically reduces to choosing the left or right half. The prompts are event labels, and the two clips always have different labels. As a result, the text-only AudioLDM2 control, which cannot see the competing source at all, is already close to FocusV2A on most selection metrics (TempSuccess 0.741 vs. 0.764, Leak@Dur 0.194 vs. 0.184, Target@Dur 0.921 vs. 0.942). In addition, the detector already reports about 15.6% leakage on the clean target recording alone (Figure 6), so the best methods are close to the detector's floor. Since the benchmark is one of the two main contributions, I think it needs cases where text alone is not enough, for example natural scenes with co-occurring sources, or two sources of the same category.

  • My second concern is that the evidence that the Focus Selector drives the improvement is weak. In Table 3, adapting the generator without any visual selection already reaches a RefSep of 0.332 (vs. 0.344), and its TempSuccess is even higher (0.776 vs. 0.764). In Table 9, pointing the selection at the wrong source only increases Leak@Dur by about 3 pp (21.6% vs. 18.4%), so the output mostly follows the text even when the visual evidence is wrong. This suggests that most of the gain over the baselines comes from fine-tuning MMAudio on mixed two-source data with text targets, which the off-the-shelf V2A baselines never saw. That makes the comparison in Table 1 less informative. In addition, the VGG-MonoAudio baseline numbers are copied from SelVA, while FocusV2A is run with the official evaluator. The large DeSync drop on the intra-class split (0.639 →\rightarrow 0.346) is not explained either, given that the backbone is the same.

Questions for authors
  • What scores do the reference recordings themselves get on VGG-SourcePair, i.e., if the target component recording is used as the output? This would give the ceiling of TextSep/RefSep and the floor of Leak@Dur, and make the gaps between methods easier to interpret.

  • Was the "w/o Focus Selector" variant trained with exactly the same data and budget as the full model? Given how close the two are, what does the selector add in practice?

  • For VGG-MonoAudio, are the evaluation settings identical to those used in SelVA for the copied baselines? Why does DeSync improve so much on the intra-class split with the same MMAudio backbone?

  • About 17% of FocusV2A outputs are flagged as low-energy. How do the leakage metrics change if these outputs are analyzed separately?

Reviewer_V8bBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies non-target source leakage in selective video-to-audio generation: a model generates sounds from other visible sources in addition to the source requested by the user. The authors introduce VGG-SourcePair, which horizontally concatenates two component videos and requests each source in turn. The benchmark contains 256 dual-source videos and 512 directed generation tasks, jointly evaluating target retention and non-target suppression.

The proposed FocusV2A uses a text-conditioned Focus Selector to competitively assign local visual features to the target and its complement, converting the selection weights into spatial attention biases. Training has two stages: learning visual selection using teacher features from component videos, then freezing the selector and adapting parts of the MMAudio generator. Evaluations on VGG-SourcePair and VGG-MonoAudio are supplemented by ablations, selector-prompt interventions, and human evaluation.

Critical strengths
  1. The problem is well defined and the paired evaluation is valuable. Evaluating only the match between generated audio and the target can overlook contamination by other sources. The paper jointly evaluates target retention and non-target suppression and swaps the target and competing sources for the same video, providing a useful controlled diagnostic for a more complete assessment of selective generation.
  2. The overall system shows substantial improvements supported by several forms of validation. On VGG-SourcePair, compared with the closest selective-generation baseline SelVA, FocusV2A improves RefSep from 0.2177 to 0.3441, reduces Leak@Dur from 24.49% to 18.39%, and increases TempSuccess from 61.30% to 76.35%. Results on VGG-MonoAudio and human evaluation offer additional support. Table 9 changes the selector prompt while fixing the video, the generator's native text condition, and the noise, demonstrating that the selection pathway affects the final generated content.
Critical weaknesses
  1. The rationale and necessity of key internal Focus Selector components are insufficiently validated. The overall motivation for prompt-guided visual selection is clear, and Table 3 includes ablations of the selector, generator adaptation, and competitive normalization. However, the paper does not sufficiently explain or test why two query banks are needed, why prompt projections with opposite signs are added to the branches, or which problems the separation loss and cross-request constraints address individually. I suggest comparison with a simpler single-branch text-conditioned selector and separate ablations of the additional prompt projections and training losses. Some gains from competitive normalization are small: RefSep increases by only 0.0041 and TempSuccess by 0.62 percentage points. Paired confidence intervals or repeated runs are needed to establish whether these choices offer stable benefits. These tests directly concern the necessity and independent contribution of the core architecture.
  2. The representativeness of VGG-SourcePair and the quality of its reference data need stronger validation. Horizontally concatenating videos of two different events is useful for controlled evaluation, but the sources have a clear spatial boundary. The benchmark lacks shared backgrounds, occlusion, spatial overlap, and complex source interference found in natural scenes, and closely matches the method's binary selection structure. The range of real scenes to which its advantages transfer therefore remains uncertain. The original component soundtracks may also contain background sounds beyond the labeled event, potentially affecting reference-audio comparisons and leakage assessment. Although Table 2 provides results on an existing benchmark, I still recommend a human audit of source presence and reference quality in the new benchmark, results for its existing difficulty strata, and a small evaluation on natural multi-source scenes to establish the scope of both the benchmark and the method.
  3. The reliability of the core metrics and their characterization of target-event timing remain incomplete. Target@Dur, Leak@Dur, and TempSuccess depend on a PANN prototype detector and its thresholds. Relatively few examples per class are used for prototype construction and threshold calibration, and calibration windows inherit clip-level labels that may not accurately indicate whether an event occurs within each window. The controlled-mixture experiments and human evaluations are valuable, but they do not directly establish window-level detection accuracy on generated audio. A human window-level audit, false-positive and false-negative analysis, and threshold-sensitivity evaluation would strengthen the measurement claims. In addition, Target@Dur and TempSuccess do not use the target event's true activity intervals, so continuous target sound may score higher than correctly intermittent generation. DeSync on VGG-SourcePair is computed against the entire composite video and does not specifically distinguish synchronization with the requested source. The metrics therefore support target retention and non-target suppression, but provide less complete validation of generating the target sound at the correct time. This limits the interpretation of relationships among detector coverage, actual source duration, and fidelity to target activity.
  4. The extent to which generation uses the target video information still lacks direct validation. Table 9 shows that the selector prompt affects the result while video, native generator text, and noise are held fixed, but it does not directly quantify the contribution of the correct video content. In Table 1, text-only AudioLDM2 achieves 74.08% TempSuccess, close to 76.35% for FocusV2A. Although its audio quality and synchronization are substantially worse, this suggests that high selectivity scores do not necessarily establish effective use of visual information. I recommend comparing correct and mismatched videos with the same model, text, and noise, and extending the comparison between matched selection and zero attention bias to the full test set. This would more directly establish that the gains arise from effective selection of target visual content.
Questions for authors

Why does generator adaptation worsen DeSync? In Table 3, adding generator adaptation to the version that trains only the Focus Selector improves FAD from 28.30 to 24.75, but increases DeSync from 0.5879 to 0.6422, where lower is better. Could the authors explain this change and indicate whether it occurs consistently across repeated experiments or mainly reflects generation or evaluation variability? This question concerns the local effect of the adaptation stage; the full model still improves synchronization over MMAudio in Tables 1 and 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.