acceptodds
Under review as a conference paper at ICLR 2027

TAP-Foley: Inference-Time Target-Anchored Projection for Counterfactual Video Foley Generation

Abstract

Counterfactual Video Foley Generation replaces the source sound identity that a silent video implies with the target sound identity that a text prompt describes, while preserving the timing of visible events. Video-text-to-audio (VT2A) mod- els struggle with this task because video conditioning carries not only the timing but also the source identity, which conflicts with the target prompt. Existing training-based methods suppress the source identity by adding a visual adapter or by training on video–text conflicts, yet it remains partially audible. Inference-time guidance methods suppress it more fully, by negative source prompting with a fixed weight or by dropping video conditioning at an intermediate step, yet both degrade temporal alignment. We propose TAP-Foley, an inference-time guidance method that adaptively suppresses the source identity while retaining video conditioning throughout generation. At each step, it measures how strongly the video steers the prediction toward the source identity beyond what the target prompt alone would produce. It removes this excess along a single source–target axis with a scaled closed-form projection onto a target-anchored constraint, so the conflict between the two modalities is mitigated without dropping either. To reliably evaluate sound identity replacement and temporal alignment, we introduce VGGSound-Counterfactual, a benchmark for Counterfactual Video Foley Generation. Comprehensive experiments show that TAP-Foley achieves a more favorable trade-off between sound identity replacement and temporal alignment than the evaluated configurations of prior methods. At matched replacement, it improves temporal alignment by 20% over the strongest prior inference-time guidance method. Code, benchmark, and demos are available at https://anonymous129108.github.io/TAP-Foley/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
counterfactual video Foley generation, video-to-audio generation, inference-time control, flow matching, multimodal generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dgrXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TAP-Foley, an inference-time method for counterfactual video-to-audio generation. It aims to replace the audible source identity according to a target text prompt while preserving video-aligned timing. The method uses a target-only prediction as an anchor and adaptively corrects the joint video–text prediction along a source–target direction, without retraining the backbone. Experiments on MMAudio and Omni2Sound use counterfactual evaluation sets constructed from VGGSound and Greatest Hits, supplemented by a human study.

Critical strengths
  1. The paper investigates an important and practically relevant task: counterfactual video Foley generation, where visual timing must be preserved while overriding the visually implied sound identity. The proposed guidance approach operates entirely at inference time on frozen backbones without requiring retraining or architecture modifications, making it modular and convenient to deploy across compatible VT2A diffusion and flow-matching models.

  2. The target-anchored projection offers a clear closed-form geometric formulation to mitigate cross-modal conflicts while retaining video conditioning throughout all denoising steps, rather than abruptly dropping it mid-trajectory. Furthermore, the evaluation framework simultaneously examines target realization, source suppression, and temporal synchronization, providing a structured perspective to analyze the inherent trade-offs between competing modalities.

Critical weaknesses
  1. Comparisons with reference-audio control methods are missing. AC-Foley, FoleyGenEx, and ControlFoley’s reference-audio mode offer relevant alternatives for controlling sound identity. Although avoiding reference audio is a practical advantage, the paper does not establish the performance trade-offs in target fidelity, naturalness, and synchronization when a suitable reference is available.

  2. Evidence for generality and superiority over CounterFlow remains limited. Only two backbones are evaluated, and TAP does not show a consistent advantage across metrics on MMAudio. On Omni2Sound, CounterFlow’s guidance weights are transferred through proportional scaling without independent validation-set tuning, leaving its best achievable tradeoff insufficiently explored.

  3. The evaluation relies exclusively on author-constructed counterfactual protocols, without results on CounterFlow’s established VGGSound-Sparse Clean protocol. Quantitative audits of the MLLM-assisted filtering and target selection are also missing. Selecting temporally compatible source–target pairs further limits the evidence for robustness to prompts outside these selection criteria.

  4. A direct baseline that removes visual semantics while retaining synchronization conditioning is missing. MMAudio supports masking CLIP features while preserving synchronization features, and AC-Foley separately examines semantic video and synchronization conditioning. FoleyGenEx also explicitly supports text semantics with video synchronization. Without these comparisons, the benefit of TAP’s additional inference computation over straightforward condition masking remains unclear.

  5. The cost of intervention and dependence on the backbone’s text prior are insufficiently characterized. Synchronization deteriorates relative to default CFG, while source-referenced FAD cannot establish whether perceptual audio quality is preserved. Moreover, TAP derives its correction direction and anchor from text-only predictions, leaving robustness to weak or ambiguous target-text conditioning unvalidated.

Questions for authors
  1. Could the authors compare TAP with AC-Foley, FoleyGenEx, and ControlFoley’s reference-audio mode, explicitly accounting for their additional inputs, to clarify the trade-offs in target fidelity, naturalness, and synchronization?

  2. Do TAP’s gains persist after validation-set tuning of CounterFlow’s guidance weights and transition time on each backbone? What evidence supports generalization beyond the two tested backbones?

  3. Could the authors report results on the VGGSound-Sparse Clean protocol and provide quantitative audits of the curation pipeline? How does TAP behave when source and target sounds have incompatible temporal envelopes?

  4. How does TAP compare with MMAudio and AC-Foley using target text and synchronization conditioning while disabling semantic video conditioning and reference audio, and with FoleyGenEx’s corresponding text-and-synchronization mode, under comparable inference budgets?

  5. How robust is TAP to target prompts poorly supported by the backbone’s text prior? Could the authors evaluate perceptual audio quality independently of source-referenced FAD?

Reviewer_KmssOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAP-Foley for counterfactual video Foley generation. It combines a text-only source-target contrast direction with a target-anchored projection to suppress source identity while retaining video conditioning throughout generation. The paper also introduces VGGSound-Counterfactual and evaluates replacement-alignment trade-offs on two frozen backbones.

Critical strengths
  1. The method is simple, requires no retraining, and has a clear geometric formulation. Ablations support the roles of the contrast direction, target anchor, and adaptive correction weight.

  2. Experiments demonstrate favorable replacement–alignment trade-offs on both backbones, supported by comparable-compute guidance comparisons, human evaluation, and onset-level testing on Greatest Hits.

  3. The benchmark addresses relevant evaluation confounds through source verification and temporally compatible target selection, combining automated screening with human review.

Critical weaknesses
  1. Incremental methodological novelty. The contrast direction and projection mechanism build closely on existing guidance approaches. The main distinction is the target-conditioned anchor. Although useful and empirically supported, the methodological advance is relatively narrow.

  2. The benefit of adaptivity is not fully isolated. Tables 4 and 9 compare adaptive correction against a single mean-matched constant weight. The constant achieves weaker source suppression but slightly better temporal alignment, indicating another trade-off. A fixed-weight sweep with comparisons at matched replacement would better establish the advantage of adaptive correction.

  3. Evaluation coverage remains limited. The benchmark focuses on clean, single-source clips and temporally compatible prompts, with substantial concentration of targets within source families. Family-stratified results would clarify whether gains generalize across replacement types. The listening test covers only 20 triplets, and its confidence intervals treat these triplets as fixed, limiting generalization to broader video content.

Questions for authors

Stronger fixed-weight comparisons and broader perceptual evaluation would further strengthen the paper.

Reviewer_rUTPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to change the sound identity implied by a video while keeping the timing of its visible events. TAP-Foley uses a frozen video-text-to-audio model. At each sampling step, it forms a source–target direction from two text-only predictions and corrects the video-text prediction along this direction. A target-only prediction sets the boundary, and a projection determines the correction weight. The method needs no training. The paper introduces VGGSound-Counterfactual with 11,890 triplets and evaluates MMAudio and Omni2Sound through automatic metrics, a listening study, and an onset evaluation on Greatest Hits.

Critical strengths
  1. The method is simple and conceptually elegant. The target-only anchor gives the correction a clear reference. Using the current projection to set the weight is a reasonable way to adjust the correction for each sample and step.
  2. TAP-Foley gives useful gains over the base models without training. Table 2 shows much stronger sound replacement on both backbones, and Table 7 supports the joint improvement in identity and onset alignment. The examples in the anonymous repository also show convincing sound changes.
  3. The anonymous repository provides code, backbone adapters, and benchmark metadata.
Critical weaknesses
  1. The theory does not establish that the correction changes only sound identity. The source–target direction can also contain timing information. Leaving components perpendicular to this direction unchanged gives no guarantee that timing is preserved, as Appendix E.1 notes.
  2. The theoretical analysis concerns individual corrections. In Algorithm 1, each corrected latent is used to compute the next direction, so errors can carry into later steps. The paper does not analyze whether these errors grow over the full sampling process. A deterministic solver does not prevent this type of error.
  3. The benefit of adaptive weights is not clear, especially for timing. Table 4 tests only one constant weight per backbone. The constant has better DeSync than TAP-Foley: 0.408 versus 0.427 on MMAudio and 0.503 versus 0.510 on Omni2Sound, though its identity scores are lower. A constant-weight sweep at similar identity scores is needed to show the value of adaptation.
  4. The default setting still trails CounterFlow on some identity metrics. In Table 2(b), MMAudio reaches ΔFLAM 0.384 and classification accuracy 0.885, versus 0.475 and 0.916 for CounterFlow. Table 8 reaches similar ΔFLAM by increasing the correction scale, with worse timing than the default TAP-Foley setting. These results show a trade-off and limit the gain on identity alone.
Questions for authors
  1. Runtime clarification: Appendix C.2 reports the number of function evaluations. What is the actual generation time per clip for TAP-Foley, native Default CFG, and CounterFlow on the same hardware and clip length? How much extra time does TAP-Foley require on each backbone?
  2. Additional validation: Table 3 uses 20 triplets and 17 participants. Could the listening study cover more videos and sound categories to check whether the identity and timing gains hold more broadly?
3 comments

Thank you for the thoughtful responses. The additional ablations and runtime comparisons are helpful and address several of my concerns. I would appreciate some further clarification on the following two points.

For W1, I understand that the paper does not claim a formal timing guarantee. My concern is that the minimum-norm property concerns distance in velocity space, while the semantic information carried by the correction remains unclear. Both dSTd_{ST} and its orthogonal complement may contain timing information, so preserving the latter does not necessarily preserve the desired temporal structure. The higher DeSync than Default CFG in Table R1 is consistent with this concern, although it does not establish the underlying cause.

For W2, I agree that departing from the Default CFG trajectory is necessary for sound replacement. By “error,” I meant unintended changes to timing or other attributes that may accompany the identity correction. Since subsequent directions and weights depend on the updated latent, these changes could carry forward or compound during sampling. That said, the reported results and provided examples are encouraging and support the method’s practical effectiveness. My remaining question concerns how these unintended changes behave across steps, which comparable Audiobox PQ does not directly address. Given the positive empirical results, a brief further discussion of this possibility and the scope of the per-step analysis would help clarify this point.

AuthorsOpenReview ↗

Response to Reviewer rUTP's follow-up

We thank the reviewer for the follow-up.

W1. Concern: what does the correction remove? d_ST and its orthogonal complement may both contain timing information, and DeSync exceeds Default CFG's. Indeed, preserving the orthogonal complement does not necessarily preserve the desired temporal structure; but TAP-Foley keeps timing better than CounterFlow and negative-source CFG at matched ΔFLAM. App. E.1 already states that the projection "does not certify" that the orthogonal complement carries the timing. The correction removes "the steering toward the source identity that the video adds beyond what the target prompt alone produces" (Sec. 2.2; the refined rule goes a margin δ past this point, GR1); under idealized predictions, it moves the latent along the gradient of the target-versus-source log-density ratio (App. B.1). Yet d_ST may contain timing information, so the correction can also change timing; this can contribute to the DeSync gap to Default CFG and its growth with α and δ (Sec. 4.1; Figure R1(b, d)). Even so, TAP-Foley has lower DeSync than CounterFlow at matched ΔFLAM within our sweeps (Figure 2(b), Table 8; 17–24% lower with the refined rule, GR1), and higher ΔFLAM and lower DeSync than negative-source CFG on both backbones (Sec. 4.1; Table R1). We will clarify in the paper that d_ST, like its orthogonal complement, may contain timing information.

W2. Concern: unintended timing changes that accompany the identity correction may carry forward or compound across steps; what does the per-step analysis cover? Such changes can carry forward, but the refined rule's γ has not raised the timing cost over constant weights. Prop. 1 is a one-step result in velocity space and does not bound such changes across steps. A step's change enters the next latent, and the next weight γ and direction d_ST depend on that latent. Figure R1(b, d) in GR1 compares the refined rule's γ, measured at every step, with constant weights on the same d_ST. At seven of the eight points, including δ = 0.3, the refined rule has lower DeSync at matched ΔFLAM and higher ΔFLAM at matched DeSync (Holm p < 0.05). So measuring γ from the updated latent has not raised the timing cost over constant weights. Since both sides of Figure R1 use the same d_ST, we have not verified whether such changes compound through d_ST. TAP-Foley and CounterFlow share this limitation, as both correct the velocity along d_ST (Sec. 2.3; Table 1). Still, the step-wise correction does not leave the final audio with worse temporal alignment or audio quality than the other guidance methods that replace the sound: on each backbone, the refined rule's final audio is better in both (Table R1(b); onset timing also in Table R2(b)). We will add this scope and discussion to Sec. 2.2 and App. E.1. If the reviewer has other attributes in mind, we would be glad to discuss them.

We appreciate the careful reading.

Thank you for the clarification. This addresses my remaining questions about the scope of the analysis. I appreciate the explicit acknowledgment that dSTd_{ST} may carry timing information and that Proposition 1 does not bound unintended changes across sampling steps.

The constant-weight comparisons provide useful empirical support for the adaptive rule, although they do not establish whether timing changes compound through dSTd_{ST}. Given the encouraging results and examples, I am satisfied with treating this as a remaining limitation, with the proposed discussion added to the paper.

Taken together with the additional ablations and runtime comparisons, these clarifications have improved my assessment, and I will raise my recommendation to weak accept. Please incorporate the stated clarifications into Sec. 2.2 and App. E.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.