- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tackles text-guided selective V2A: given a silent video and a text description, generate only the requested source and suppress other visible sources. The authors notice that ControlFoley's semantic reference pathway, originally for timbre transfer, actually also encodes source selection. They train a small 1.31M adapter (CSIO) to map text into the same reference-content space so the frozen backbone can be reused, with a time-dependent CFG term at sampling to fix the sync side-effect.
They also release StemFoleyDB-3.3K, a human-audited multi-source dataset with per-source clean stems. Experiments cover their own test split, SelVA's benchmark, two additional AC-V2A backbones, and an 18-listener study.
Critical strengths
S1:Clever core idea. Instead of training a selective V2A model from scratch, the authors identify an existing source-selection capability inside a frozen AC-V2A backbone and reuse it through a tiny adapter. The angle of locating a hidden semantic interface via controlled intervention and then tapping into it with textual inversion is quite novel.
S2:Release StemFoleyDB-3.3K, a human-audited multi-source dataset with per-source clean stems where target and competing sources share the same visual context. This kind of resource is rare and useful beyond this method.
Critical weaknesses
W1. The method is not really general. It only works on V2A models where the reference audio pathway dominates content/timbre. Most mainstream V2A models (MMAudio, HunyuanVideo-Foley, ThinkSound etc) don't satisfy this, authors only found 3 AC-V2A backbones that work. And Table 5 shows the gain differs by 3x across the three backbones (ControlFoley +23.60 vs AC-Foley +7.37 vs Video-Foley +8.26), which means the method really depends on how strong the reference pathway is in that specific backbone. So the "1.31M vs 2.82B, 9h vs 500h" efficiency story is kind of misleading, it only counts the adapter cost and hides the sunk cost of having to pretrain a 2.82B ControlFoley first. If the method is not general, training such a backbone is itself expensive, so the real total cost isn't lower than from-scratch.
W2. Generalization is limited in several ways. (a) CSIO input is M2D-CLAP (M2D ViT + GTE-base) but ControlFoley internally uses LAION-CLAP (HTSAT + RoBERTa, confirmed by the code). Two totally different CLAPs, no shared pretraining, different audio and text backbones. With only ~5200 training pairs, I don't believe the model can really align these two independent CLAP spaces well, even if it works on the benchmark it's probably just local alignment on the training distribution. Also a more fundamental concern: using CLAP as the text encoder for caption control is inherently weaker than T5-base style encoders used in mainstream T2A. SelFoley has to use CLAP to tap into the reference pathway, which basically trades off text control precision for source selection ability, not sure this trade-off is worth it. (b) All 3287 training captions are short Foley terms like "dog barking", with only 1.31M trainable params, hard to say if it generalizes to longer / different-style captions or novel classes. Would be good to add two tests: longer captions in different styles, and unseen classes like ESC-50 / FSD50K. (c) The data pipeline is bottlenecked by SAM-Audio separation quality + human audit, so training set can't scale up and hard-to-separate sources (overlapping same-class, heavy reverb, music, physically coupled sounds) are systematically missing.
W3. Test set is too small and in-distribution. Only 50 clips / 113 sources, and the captions come from the same 15 annotators as training, so there's no real OOD evaluation. The 113 sources also cluster in 50 clips so the effective sample size is even smaller. Authors themselves admit in limitations that "small differences may reflect separator or clip-level uncertainty", which makes the generalization claims hard to support with current numbers.
Minor issues:
-
The writing could use some work. There are quite a few places that feel like AI-polished prose, with overly defensive hedging and some unnecessarily complex phrasing. It makes the paper less fluid to read than it should be. Would suggest another human pass to make it more natural and readable.
-
No dedicated related work section, which makes it hard to place the paper in the landscape. The paper is also not very self-contained — to really follow it you need prior familiarity with ControlFoley and its internal structure, otherwise most of the mechanism discussion in §2 is hard to parse.
Questions for authors
Please see weaknesses.