acceptodds
Under review as a conference paper at ICLR 2027

SelFoley: Turning Audio-Conditioned Video-to-Audio Models into Text-Guided Source Selectors

Abstract

Text-guided selective video-to-audio (V2A) asks a model to generate only a requested sound from a silent video that contains several plausible sources. Existing systems learn source selection and soundtrack generation together from large, automatically constructed data. We take a different route. Audio-conditioned V2A (AC-V2A) models normally transfer the content and acoustic character of a reference sound to a new video. In the open-weight ControlFoley model, we find that an isolated reference does more: it suppresses other visible events, and a controlled pathway intervention localizes this choice to an internal vector that represents what sound the reference contains. We call this the source-content condition. The Clean Source Isolation Operator (CSIO) is a 1.31M-parameter adapter that predicts this condition from target text while the generator remains frozen. CSIO alone improves selection but weakens audiovisual timing; SelFoley combines CSIO with early visual-synchronization guidance to recover alignment. We train the adapter on StemFoleyDB-3.3K, a dataset of 3,357 clips and 5,383 human-audited source stems. Its training split contains 3,287 clips (about 9.0 clip-hours); 3,274 clips with complete frozen features supply 300 adapter updates. Relative to SelVA's reported 179k-video, roughly 500-hour task corpus, SelFoley uses 98.2% fewer task-training clips and 96.8% fewer trainable parameters. On a fixed 50-clip test set, SelFoley reaches 78.76% target selection versus 51.33% for SelVA, while its Synchformer error is 0.384 versus 0.413. Eighteen listeners also rate non-target suppression at 3.30 versus 1.98 on a five-point scale. Separately trained CSIO adapters improve target selection by 7.4–23.6 points within three open-weight AC-V2A backbones, showing that generator-specific native reference-content interfaces can be reused instead of relearning separation end to end. Dataset examples and model outputs are available at https://sel-foley.github.io/.

Primary area:
generative models
Keywords:
video-to-audio generation, selective audio generation, audio-conditioned video-to-audio, source selection, frozen generator adaptation, Foley, multi-source audio dataset

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vtwUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles text-guided selective V2A: given a silent video and a text description, generate only the requested source and suppress other visible sources. The authors notice that ControlFoley's semantic reference pathway, originally for timbre transfer, actually also encodes source selection. They train a small 1.31M adapter (CSIO) to map text into the same reference-content space so the frozen backbone can be reused, with a time-dependent CFG term at sampling to fix the sync side-effect.

They also release StemFoleyDB-3.3K, a human-audited multi-source dataset with per-source clean stems. Experiments cover their own test split, SelVA's benchmark, two additional AC-V2A backbones, and an 18-listener study.

Critical strengths

S1:Clever core idea. Instead of training a selective V2A model from scratch, the authors identify an existing source-selection capability inside a frozen AC-V2A backbone and reuse it through a tiny adapter. The angle of locating a hidden semantic interface via controlled intervention and then tapping into it with textual inversion is quite novel.

S2:Release StemFoleyDB-3.3K, a human-audited multi-source dataset with per-source clean stems where target and competing sources share the same visual context. This kind of resource is rare and useful beyond this method.

Critical weaknesses

W1. The method is not really general. It only works on V2A models where the reference audio pathway dominates content/timbre. Most mainstream V2A models (MMAudio, HunyuanVideo-Foley, ThinkSound etc) don't satisfy this, authors only found 3 AC-V2A backbones that work. And Table 5 shows the gain differs by 3x across the three backbones (ControlFoley +23.60 vs AC-Foley +7.37 vs Video-Foley +8.26), which means the method really depends on how strong the reference pathway is in that specific backbone. So the "1.31M vs 2.82B, 9h vs 500h" efficiency story is kind of misleading, it only counts the adapter cost and hides the sunk cost of having to pretrain a 2.82B ControlFoley first. If the method is not general, training such a backbone is itself expensive, so the real total cost isn't lower than from-scratch.

W2. Generalization is limited in several ways. (a) CSIO input is M2D-CLAP (M2D ViT + GTE-base) but ControlFoley internally uses LAION-CLAP (HTSAT + RoBERTa, confirmed by the code). Two totally different CLAPs, no shared pretraining, different audio and text backbones. With only ~5200 training pairs, I don't believe the model can really align these two independent CLAP spaces well, even if it works on the benchmark it's probably just local alignment on the training distribution. Also a more fundamental concern: using CLAP as the text encoder for caption control is inherently weaker than T5-base style encoders used in mainstream T2A. SelFoley has to use CLAP to tap into the reference pathway, which basically trades off text control precision for source selection ability, not sure this trade-off is worth it. (b) All 3287 training captions are short Foley terms like "dog barking", with only 1.31M trainable params, hard to say if it generalizes to longer / different-style captions or novel classes. Would be good to add two tests: longer captions in different styles, and unseen classes like ESC-50 / FSD50K. (c) The data pipeline is bottlenecked by SAM-Audio separation quality + human audit, so training set can't scale up and hard-to-separate sources (overlapping same-class, heavy reverb, music, physically coupled sounds) are systematically missing.

W3. Test set is too small and in-distribution. Only 50 clips / 113 sources, and the captions come from the same 15 annotators as training, so there's no real OOD evaluation. The 113 sources also cluster in 50 clips so the effective sample size is even smaller. Authors themselves admit in limitations that "small differences may reflect separator or clip-level uncertainty", which makes the generalization claims hard to support with current numbers.

Minor issues:

  1. The writing could use some work. There are quite a few places that feel like AI-polished prose, with overly defensive hedging and some unnecessarily complex phrasing. It makes the paper less fluid to read than it should be. Would suggest another human pass to make it more natural and readable.

  2. No dedicated related work section, which makes it hard to place the paper in the landscape. The paper is also not very self-contained — to really follow it you need prior familiarity with ControlFoley and its internal structure, otherwise most of the mechanism discussion in §2 is hard to parse.

Questions for authors

Please see weaknesses.

Reviewer_CyVEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SelFoley, a method for text-guided selective video-to-audio generation. The key idea is that pretrained audio-conditioned V2A models already have some ability to select sound sources through their reference-audio conditioning. Based on this observation, the authors introduce a lightweight adapter (CSIO) that maps text to the model's internal source-content representation, without updating the generator. They also introduce a human-audited dataset and a synchronization guidance strategy. Experiments show improved source selection over existing baselines, with additional analysis across three V2A backbones.

Critical strengths
  1. Interesting observation and analysis. I find the discovery that the semantic reference pathway can control source selection interesting. The controlled interventions and representation analysis provide useful evidence beyond simply showing performance improvements.

  2. Promising results with limited training. The method achieves substantial improvements in source selection while keeping the generator frozen. The human evaluation and experiments on multiple backbones further support the practical value of this approach.

Critical weaknesses
  1. Limited methodological novelty. The CSIO adapter mainly learns a mapping from text embeddings to an existing conditioning space, which is relatively straightforward. I think the main novelty lies more in the observation than in the proposed method itself.

  2. Some experimental claims could be better supported. The comparison with SelVA involves different pretrained backbones, making it difficult to isolate the benefits of the proposed approach. Also, although the automatic synchronization metric improves, human evaluations still favor SelVA in synchronization. This suggests that the timing issue is not fully resolved.

Questions for authors
  1. Could the authors compare CSIO with a similarly sized direct text-conditioning adapter on the same ControlFoley backbone? This would help clarify how much improvement comes specifically from using the reference-content interface.

  2. In the experiments on AC-Foley and Video-Foley, simple regression performs slightly better than the full CSIO objective. Could the authors explain why the additional losses are less effective on these backbones?

  3. How do the authors interpret the difference between the improved DeSync score and the lower human synchronization rating compared with SelVA? Does this indicate a limitation in target-specific temporal alignment?

Reviewer_NUVyOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies text-guided selective video-to-audio generation, where a model should synthesize a requested sound while suppressing competing sources visible in the same scene. The authors show through controlled interventions that ControlFoley’s semantic reference-audio pathway influences source selection. They introduce CSIO, a lightweight adapter that predicts this native reference-content condition from text while keeping the generator frozen. The complete SelFoley system additionally strengthens early visual-synchronization guidance. The paper also introduces StemFoleyDB-3.3K, a dataset of human-audited source stems. Experiments demonstrate improved source selection and perceived non-target suppression, while revealing remaining limitations in target-specific synchronization.

Critical strengths

1, The semantic–timbre intervention provides convincing evidence that source selection is controlled primarily through ControlFoley’s semantic reference pathway. Interpolation, competing-source, and gain-control experiments further strengthen this interpretation beyond a simple clean-versus-mixed reference comparison.

2, CSIO adapts a frozen generator using only 1.31M trainable parameters and approximately 3.3K training clips. On the proposed benchmark, SelFoley improves TargetSel from 51.33% for SelVA to 78.76%. These results demonstrate the practical value of learning an existing conditioning interface.

3, StemFoleyDB provides human-audited targets and competing sources from the same scene. This supports both source-discriminative training and evaluation under a practically relevant setting that differs from synthetic side-by-side video mixtures.

4, The paper includes alternative embedding evaluators, controlled listening experiments, and separately trained adapters for three backbones. The independent listening study supports improved non-target suppression, with MOS increasing from 1.98 to 3.30 relative to SelVA.

Critical weaknesses

1, Although SelFoley improves aggregate DeSync over SelVA, listeners rate its synchronization lower: 2.94 versus 3.40. The proposed guidance therefore improves the automatic metric without establishing superior perceptual or target-specific timing. This remains a substantive limitation for selective video-to-audio generation.

2, In Table 23, simple target-condition regression outperforms CSIO under M2D TargetSel on AC-Foley (62.83% versus 61.36%) and Video-Foley (82.60% versus 81.71%). These results support the broader interface-learning strategy more strongly than the general necessity of CSIO’s additional source-selection losses.

3, Fine-tuning SelVA on StemFoleyDB raises TargetSel to 76.11%, substantially narrowing the selection gap, although synchronization deteriorates. On VGG-MonoAudio’s harder intra-class split, SelFoley does not improve source Top-1 over SelVA (72.02% versus 72.41%). The results suggest a useful selection–synchronization trade-off rather than uniformly superior performance.

4, TargetSel measures relative embedding similarity rather than residual interference. Its perceptual validation uses screened interventions, and the test references are separated, human-audited stems rather than independently recorded ground truth. Moreover, Appendix D reports collecting but excluding naturalness ratings. Reporting these alongside suppression would help determine whether improved selectivity comes at an audible quality cost.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.