- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TAP-Foley, an inference-time method for counterfactual video-to-audio generation. It aims to replace the audible source identity according to a target text prompt while preserving video-aligned timing. The method uses a target-only prediction as an anchor and adaptively corrects the joint video–text prediction along a source–target direction, without retraining the backbone. Experiments on MMAudio and Omni2Sound use counterfactual evaluation sets constructed from VGGSound and Greatest Hits, supplemented by a human study.
Critical strengths
-
The paper investigates an important and practically relevant task: counterfactual video Foley generation, where visual timing must be preserved while overriding the visually implied sound identity. The proposed guidance approach operates entirely at inference time on frozen backbones without requiring retraining or architecture modifications, making it modular and convenient to deploy across compatible VT2A diffusion and flow-matching models.
-
The target-anchored projection offers a clear closed-form geometric formulation to mitigate cross-modal conflicts while retaining video conditioning throughout all denoising steps, rather than abruptly dropping it mid-trajectory. Furthermore, the evaluation framework simultaneously examines target realization, source suppression, and temporal synchronization, providing a structured perspective to analyze the inherent trade-offs between competing modalities.
Critical weaknesses
-
Comparisons with reference-audio control methods are missing. AC-Foley, FoleyGenEx, and ControlFoley’s reference-audio mode offer relevant alternatives for controlling sound identity. Although avoiding reference audio is a practical advantage, the paper does not establish the performance trade-offs in target fidelity, naturalness, and synchronization when a suitable reference is available.
-
Evidence for generality and superiority over CounterFlow remains limited. Only two backbones are evaluated, and TAP does not show a consistent advantage across metrics on MMAudio. On Omni2Sound, CounterFlow’s guidance weights are transferred through proportional scaling without independent validation-set tuning, leaving its best achievable tradeoff insufficiently explored.
-
The evaluation relies exclusively on author-constructed counterfactual protocols, without results on CounterFlow’s established VGGSound-Sparse Clean protocol. Quantitative audits of the MLLM-assisted filtering and target selection are also missing. Selecting temporally compatible source–target pairs further limits the evidence for robustness to prompts outside these selection criteria.
-
A direct baseline that removes visual semantics while retaining synchronization conditioning is missing. MMAudio supports masking CLIP features while preserving synchronization features, and AC-Foley separately examines semantic video and synchronization conditioning. FoleyGenEx also explicitly supports text semantics with video synchronization. Without these comparisons, the benefit of TAP’s additional inference computation over straightforward condition masking remains unclear.
-
The cost of intervention and dependence on the backbone’s text prior are insufficiently characterized. Synchronization deteriorates relative to default CFG, while source-referenced FAD cannot establish whether perceptual audio quality is preserved. Moreover, TAP derives its correction direction and anchor from text-only predictions, leaving robustness to weak or ambiguous target-text conditioning unvalidated.
Questions for authors
-
Could the authors compare TAP with AC-Foley, FoleyGenEx, and ControlFoley’s reference-audio mode, explicitly accounting for their additional inputs, to clarify the trade-offs in target fidelity, naturalness, and synchronization?
-
Do TAP’s gains persist after validation-set tuning of CounterFlow’s guidance weights and transition time on each backbone? What evidence supports generalization beyond the two tested backbones?
-
Could the authors report results on the VGGSound-Sparse Clean protocol and provide quantitative audits of the curation pipeline? How does TAP behave when source and target sounds have incompatible temporal envelopes?
-
How does TAP compare with MMAudio and AC-Foley using target text and synchronization conditioning while disabling semantic video conditioning and reference audio, and with FoleyGenEx’s corresponding text-and-synchronization mode, under comparable inference budgets?
-
How robust is TAP to target prompts poorly supported by the backbone’s text prior? Could the authors evaluate perceptual audio quality independently of source-referenced FAD?
Thank you for the thoughtful responses. The additional ablations and runtime comparisons are helpful and address several of my concerns. I would appreciate some further clarification on the following two points.
For W1, I understand that the paper does not claim a formal timing guarantee. My concern is that the minimum-norm property concerns distance in velocity space, while the semantic information carried by the correction remains unclear. Both dST and its orthogonal complement may contain timing information, so preserving the latter does not necessarily preserve the desired temporal structure. The higher DeSync than Default CFG in Table R1 is consistent with this concern, although it does not establish the underlying cause.
For W2, I agree that departing from the Default CFG trajectory is necessary for sound replacement. By “error,” I meant unintended changes to timing or other attributes that may accompany the identity correction. Since subsequent directions and weights depend on the updated latent, these changes could carry forward or compound during sampling. That said, the reported results and provided examples are encouraging and support the method’s practical effectiveness. My remaining question concerns how these unintended changes behave across steps, which comparable Audiobox PQ does not directly address. Given the positive empirical results, a brief further discussion of this possibility and the scope of the per-step analysis would help clarify this point.