acceptodds
Under review as a conference paper at ICLR 2027

Generating Visual Sounds The Hard Way

Abstract

We can often infer what an event should sound like simply by watching it. To generate such sounds from video, a model must identify which visual details matter acoustically and use them to guide synthesis. Yet many current video-to-audio models rely on captions for these cues and face a trade-off: removing captions degrades semantic and physical accuracy, while using them can worsen event timing. We introduce TacitV2A, which learns to guide audio generation from visual evidence without captions at inference. Captions serve only as a training signal: caption-guided self-distillation adapts its visual reasoner through a shared frozen audio generator, matching video-conditioned audio-velocity predictions to their stronger caption-conditioned counterparts. This alone lets the reasoner internalize much of the guidance that captions provide. We further refine the audio generator with reinforcement learning to better match paired real recordings. Across FoleyBench, VGGSounder, and FlatSounds, quantitative and human evaluations show that TacitV2A is competitive with caption-conditioned models. On FlatSounds, it outperforms all external baselines in overall physical correctness and approaches its caption-conditioned teacher, matching its event coverage with lower timing error. Interactive results are available in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video-to-Audio Generation, Audio-Visual Learning, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5D5ROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes TacitV2A, a caption-free video-to-audio generation framework. Instead of using captions at inference, the method uses a caption-conditioned teacher during training to supervise a video-conditioned visual reasoner through a shared frozen audio generator. Specifically, the student matches the teacher's audio-velocity predictions while also being grounded by flow matching to paired real audio. A second-stage reinforcement learning procedure further adapts the audio generator using audio-audio similarity with paired recordings. The method is evaluated on FoleyBench, VGGSounder, and FlatSounds, showing competitive performance against caption-conditioned systems, particularly in physical correctness and temporal synchronization.

Critical strengths
  1. The motivation is clear and practically meaningful. The paper identifies an interesting trade-off in existing V2A systems: captions improve semantic and physical correctness but can hurt synchronization. TacitV2A provides a natural way to exploit captions as privileged training information while removing their dependence at inference.

  2. The experiments are broad and reasonably comprehensive. The authors evaluate on three benchmarks and cover distributional quality, semantic alignment, physical correctness, event coverage, and synchronization, together with a relatively substantial pairwise human study.

  3. The ablation supports the usefulness of distillation. Flow matching alone is clearly weaker than combining flow matching and teacher distillation, while distillation alone is also inferior to the combined objective, providing evidence that the two supervision sources are complementary.

Critical weaknesses
  1. The proposed self-distillation mechanism is only validated on Cosmos3-Nano. Since it relies on a particular shared reasoner-generator architecture and layer-wise conditioning interface, it is unclear how broadly the method generalizes to other V2A architectures. Experiments on more backbones would strengthen the claim of general applicability, which may be future works.

  2. The RL stage uses audio similarity from LAION-CLAP as its sole reward, while LAION-CLAP is also used for evaluation. The shared representation space may favor the proposed model and makes it difficult to separate genuine perceptual improvement from reward-specific optimization.

  3. The quality of the caption supervision is an important dependency that is not sufficiently characterized. Training captions are automatically generated from video-audio pairs and further expanded into structured descriptions, so hallucinated sound sources, materials, or temporal events may propagate into the student through distillation.

Questions for authors

Please see weaknesses. If the authors adequately address the above concerns, I would consider raising my score.

Reviewer_9L93OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TacitV2A for video-to-audio generation without per-sample textual descriptions. Through text-guided self-distillation, the method transfers guidance from a text-conditioned teacher to a visual reasoner by matching their audio velocity predictions through the same frozen generator. It then freezes the reasoner and optimizes the generator using an audio–audio similarity reward. Experiments on FoleyBench, VGGSounder, and FlatSounds demonstrate improvements in caption-free generation, physical attributes, and event timing, supported by human evaluation.

Critical strengths
  1. The problem is meaningful.
  2. The experiments and evaluations are fairly comprehensive.
  3. The paper is clearly written.
Critical weaknesses

See Questions.

Questions for authors
  1. RL uses CLAP similarity between generated and reference audio as its reward, while the physical consistency score on FlatSounds incorporates weighting based on CLAP and event coverage. Could the improvement in this score partly reflect higher CLAP similarity induced by reward optimization, rather than a genuine improvement in physical consistency? Reporting unweighted physical consistency results would help clarify this issue.
  2. Does distillation improve the model’s ability to use visual cues? For example, when sound-relevant visual cues occupy a small region or are partially occluded, does distillation improve generation quality under these weaker visual conditions?
  3. Could richer reasoner descriptions also introduce more hallucinations? Appendix D.4 shows that distillation improves coverage of the reference descriptions, but does it also increase unsupported or incorrect details?
Reviewer_AsVfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TacitV2A, a caption-free video-to-audio approach built on Cosmos3-Nano. A caption-conditioned teacher and a video-conditioned student share a frozen audio generator. In the first stage, the student reasoner is adapted through LoRA by matching the teacher's audio-velocity predictions, complemented by flow matching to paired real recordings. In the second stage, the learned reasoner is frozen and generator LoRA adapters are optimized using DiffusionNFT with an audio-audio CLAP similarity reward. Inference uses only video and a shared system prompt, without per-sample captions or intermediate caption decoding. Experiments on FoleyBench, VGGSounder, and FlatSounds, together with ablations and a pairwise listening study, indicate that self-distillation recovers much of the caption-conditioned teacher's performance and that subsequent generator refinement improves several metrics.

Critical strengths
  1. A well-motivated conditioning-transfer method with informative controls. Matching the generator’s audio-velocity predictions is a reasonable way to transfer text guidance to visual inputs, since text and visual tokens do not directly correspond to each other. The generator is frozen during self-distillation, so the improvement comes from adapting the conditioning pathway. Table 5 also shows that using both losses works better than using either loss alone.

  2. Complementary automatic and human evidence supports practical relevance. The evaluation spans distributional quality, semantic alignment, physical-property trends, and event-level synchronization across three benchmarks. It also includes a 50-clip listening study with 139 participants and 5,400 pairwise evaluations, with randomized presentation and clip-level uncertainty estimation.

Critical weaknesses
  1. The distinct value of the “hard way” formulation needs clearer justification The paper learns to generate audio from video without producing captions. However, an automatic caption-then-generate pipeline also needs only video as input. It is therefore unclear what practical benefit comes from removing the caption step. Appendix D.3 gives a useful comparison, but it mainly measures physical correctness, and each method is better on some properties. The paper should explain the necessity of choosing a " hard way " more clearly and provide evidence for this.
  2. The contribution of distillation versus RL remains unclear On VGGSounder, distillation alone achieves an FD_PANN of 22.84, substantially worse than caption-free MMAudio (5.10). It reaches 4.72 only after RL. Table 12 shows that direct RL on the zero-shot model is ineffective, but this does not establish that the proposed distillation is necessary. A comparison between flow-matching-only reasoner training followed by RL and the proposed distillation followed by the same RL, under matched training budgets, would better establish its added value.
  3. Hyperparameter selection and evaluation on FoleyBench. Appendix B.2 states that the loss weights were selected from the FoleyBench ablation in Table 5, while FoleyBench is also a main evaluation benchmark. Were the samples used for weight selection separate from those used in Table 1? If the same evaluation samples were used for both purposes, the reported results may be biased by test-set tuning.
Questions for authors
  1. Do you have direct evidence that captions reduce attention to relevant visual events, or should this explanation be treated as a hypothesis?
  2. Under matched training budgets and identical RL settings, how does flow-matching-only reasoner training followed by RL compare with the proposed approach?
  3. Were the FoleyBench samples used for loss-weight selection separate from those used for evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.