- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes TacitV2A, a caption-free video-to-audio generation framework. Instead of using captions at inference, the method uses a caption-conditioned teacher during training to supervise a video-conditioned visual reasoner through a shared frozen audio generator. Specifically, the student matches the teacher's audio-velocity predictions while also being grounded by flow matching to paired real audio. A second-stage reinforcement learning procedure further adapts the audio generator using audio-audio similarity with paired recordings. The method is evaluated on FoleyBench, VGGSounder, and FlatSounds, showing competitive performance against caption-conditioned systems, particularly in physical correctness and temporal synchronization.
Critical strengths
-
The motivation is clear and practically meaningful. The paper identifies an interesting trade-off in existing V2A systems: captions improve semantic and physical correctness but can hurt synchronization. TacitV2A provides a natural way to exploit captions as privileged training information while removing their dependence at inference.
-
The experiments are broad and reasonably comprehensive. The authors evaluate on three benchmarks and cover distributional quality, semantic alignment, physical correctness, event coverage, and synchronization, together with a relatively substantial pairwise human study.
-
The ablation supports the usefulness of distillation. Flow matching alone is clearly weaker than combining flow matching and teacher distillation, while distillation alone is also inferior to the combined objective, providing evidence that the two supervision sources are complementary.
Critical weaknesses
-
The proposed self-distillation mechanism is only validated on Cosmos3-Nano. Since it relies on a particular shared reasoner-generator architecture and layer-wise conditioning interface, it is unclear how broadly the method generalizes to other V2A architectures. Experiments on more backbones would strengthen the claim of general applicability, which may be future works.
-
The RL stage uses audio similarity from LAION-CLAP as its sole reward, while LAION-CLAP is also used for evaluation. The shared representation space may favor the proposed model and makes it difficult to separate genuine perceptual improvement from reward-specific optimization.
-
The quality of the caption supervision is an important dependency that is not sufficiently characterized. Training captions are automatically generated from video-audio pairs and further expanded into structured descriptions, so hallucinated sound sources, materials, or temporal events may propagate into the student through distillation.
Questions for authors
Please see weaknesses. If the authors adequately address the above concerns, I would consider raising my score.