- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies selective video-to-audio (V2A) generation, where audio should be generated only for a user-specified source among multiple visible sources. It argues that generating the target is not sufficient if competing visible sources leak into the output.
The paper addresses three research questions:
-R1: How can source selectivity be evaluated so that both target retention and non-target leakage are measured?
-R2: Can leakage be reduced by selecting target-relevant local visual evidence before the visual condition reaches the generator?
-R3: How can a pretrained generator be adapted to such a target-focused visual condition without degrading quality and synchronization?
The corresponding proposed methods are:
-M1: VGG-SourcePair, a paired benchmark of two-source composites. Each source is requested in turn and scored against both component recordings, with dedicated metrics for target retention and non-target leakage.
-M2: Focus Selector, which competitively allocates each local visual token between the target and its complement and biases the frozen spatial aggregation attention of the visual encoder. It is trained to match unmixed-component teacher features (Stage 1).
-M3: Stage-2 adaptation of MMAudio to the selected visual condition, with the selector frozen.
FocusV2A outperforms the video-conditioned baselines on VGG-SourcePair source-selection metrics with competitive audio quality, and it improves results on VGG-MonoAudio. The evaluation also includes a human study, component ablations, and a matched-vs-opposite selection analysis.
Critical strengths
-R1/M1: The paper proposes metrics that explicitly penalize the presence of the non-target source (Leak@Dur, TempSuccess), in addition to target correspondence. This aspect is under-emphasized in the paper, but it is the most reusable part of M1. Controlled mixtures validate the metric behavior (Fig. 6), and detector calibration is disjoint from the test set.
-R2/M2: The Focus Selector reweights frozen attention multiplicatively, so pretrained features are preserved. Removing it raises FAD from 24.75 to 30.10 (Table 3). The matched-vs-opposite comparison, with checkpoint, text, and noise fixed, shows that the selection path causally changes the generated content.
Critical weaknesses
The main limitation of the paper is limited novelty relative to prior selective V2A work (SelVA and VGG-MonoAudio).
-R1/M1: The distinction between VGG-SourcePair and the existing VGG-MonoAudio is only briefly stated (Sec. 2.2) and is not sufficiently justified. Both target the evaluation of selective V2A generation, and both concatenate two videos. The novelty therefore lies mainly in the metric suite that explicitly measures counterpart leakage (non-target reference, Leak@Dur, TempSuccess), not in the dataset, and the paper does not clearly stress this. Moreover, these metrics are not applied to VGG-MonoAudio. It is unclear why a new benchmark is needed rather than an extension of the existing one.
-R2/M2: The distinction between FocusV2A and SelVA is not clear either. Both adopt a two-stage scheme: (1) fine-tuning the video feature extractor with a teacher–student scheme on mixed videos, and (2) fine-tuning MMAudio. The main distinction is the Focus Selector, which substitutes for SelVA’s text-guided cross-attention. However:
-
The paper provides no analysis or intuition on why FocusV2A performs better than SelVA.
-
Competition itself contributes marginally (Table 3, w/o Competition: RefSep 0.3400 vs. 0.3441, TempSuccess 0.7573 vs. 0.7635, nearly unchanged FAD).
-
The comparison with SelVA is not controlled. The variant without any visual selection (w/o Focus Selector) already outperforms SelVA on source-selection metrics by a large margin (RefSep 0.3324 vs. 0.2177; TempSuccess 0.7764 vs. 0.6130). This gap cannot be attributed to the Focus Selector. It may stem from SelVA’s selection mechanism itself (e.g., if modifying the visual features degrades the condition, whereas FocusV2A keeps them frozen) or from differences in training recipe, and the two cannot be separated without a controlled comparison.
A fair controlled comparison between the Focus Selector and cross-attention, under the same data, schedule, and generator adaptation, is required to support the claimed contribution.
In addition, the evaluation does not verify whether the generated target is grounded in the target’s visual events:
- DeSync on VGG-SourcePair is measured between the generated audio and the mixed video (Appendix B). This is ill-posed, because the mixed video contains visible events from both the target and the counterpart.
- Target@Dur and TempSuccess consider only the presence of the target class in each window. They do not penalize sound events that are off-sync with the video, so temporally over-filled outputs are rewarded.
- The human study has no item on audio-visual synchronization alongside the target/non-target identity assessment.
Questions for authors
- Can you provide a controlled comparison that replaces the Focus Selector with SelVA’s cross-attention (with supplementary tokens) under the same training recipe?
- Please specify the Focus Selector’s decoder:
- Do the target and complement banks share one decoder and attend to each other?
- Do the queries attend to all spatiotemporal tokens at once (~37k for an 8 s clip), or per segment or temporal position?
- How does the prompt representation condition the query banks?
- Are the logits ℓᵗ and ℓᶜ computed with the projected or the raw visual features?
- In w/o Focus Selector, is the CLIP image condition also empty?
- Can you report RefSep, Leak@Dur, and TempSuccess on VGG-MonoAudio? Its target references and counterpart metadata appear sufficient to compute them.
- Please report DeSync against the target-only view and the non-target view separately, under matched and opposite selection.
- The Table 3 differences are within about 1 pp. Please report confidence intervals (the bootstrap procedure is described in Sec. 5.1 but not reported in Tables 1 and 3) or results over multiple seeds.
- The human study uses 12 author-selected videos and reports no significance tests. Can you add confidence intervals or paired tests?
Suggestions (not affecting my score)
- The Focus Selector design builds on ideas that should be cited: competitive allocation across slots (Slot Attention, Locatello et al., NeurIPS 2020), learned-query decoders producing per-token masks (Mask2Former, Cheng et al., CVPR 2022), and additive attention-logit biases, of which the log-bias is a soft form of Mask2Former’s masked attention.