- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LocalSoundBench to evaluate whether video-to-audio models can correctly update the target sound following a local visual replacement while preserving previously correct sounds associated with unchanged events. The authors evaluate 21 released settings and diagnose failures through source-isolation tests, same-class replacements, shot reordering, seed variation, and internal representation analysis. They also propose Regional Flow as an inference-time intervention and investigate temporally aligned training. The results identify sound placement in multi-event videos as an important weakness of existing models and show that explicit sound–time correspondence can improve some models.
Critical strengths
- The paper addresses a distinctive and valuable research question.
- The paper is clearly written and provides extensive analysis and explanation.
- The experimental evaluation is comprehensive.
Critical weaknesses
See Questions.
Questions for authors
-
Most videos in the main evaluation are constructed by concatenating source clips. This may introduce a distribution shift relative to the models’ original training data and contribute to poor synthesis quality. To what extent could this affect the experimental findings and conclusions?
-
Regarding the automatic metrics, do they assess whether sounds occur at the correct times, including temporal alignment within each interval? How reliably can the classifiers identify the sound categories in generated audio?
-
Could the authors report human and automatic evaluation results on the same set of examples to determine whether there are significant discrepancies between them?
-
I would appreciate a discussion of the directionality of PR/JS. Their eligibility sets include only unchanged events that were correctly generated in the designated original output, excluding incorrect-to-correct transitions. However, both silent videos in a benchmark pair are used independently to generate complete soundtracks, so the original-to-edited direction is not inherently unique. Would it be useful to score each pair in both directions and report a correctness transition matrix over all unchanged intervals, distinguishing stable correctness, degradation, recovery, and persistent failure? This would provide a more complete assessment of cross-event effects and test whether the conclusions are robust to the choice of reference direction.
Response to Reviewer fMNQ (P1)
We sincerely thank the reviewer for the positive assessment and the questions about distribution shift, metric properties and scoring direction.
[Q1]: Effect of concatenating unrelated shots
Thank you for raising the effect of concatenation. Low JS also occurs without combining unrelated scenes. We test this by replacing one action with another from the same continuous EPIC-SOUNDS recording, keeping the other frames unchanged. Standard sampling results (%):
The main paper also keeps cut positions fixed while repeating clips or changing their order, showing that cuts alone do not explain the errors. We also shift a placement action smoothly in continuous running-water footage without adding a cut. Regional Flow reduces the unchanged background's level change from 0.85 to 0.64 dB and spectral change from 3.30 to 2.69 dB in HunyuanVideo-Foley XL. Replacing an action tests EA, PR and JS within one scene, while shifting its time tests whether the background sound changes without inserting a shot.
[Q2]: Temporal alignment and classifier reliability
We appreciate the question about timing. EA, PR and JS check whether the intended sound occurs in its event interval, rather than its precise onset, and DeSync provides a separate audiovisual timing check. Section 3.3 and Appendix D.4 show why both are needed: placing sounds in wrong intervals reduces interval accuracy from 99.6% to 0.4%, while whole-clip similarities remain nearly unchanged. DeSync responds to a moving energy envelope and can miss wrong sounds when that envelope stays fixed. For generated audio, correlations with human decisions range from 0.38 to 0.53 for EA, 0.21 to 0.43 for PR and 0.21 to 0.47 for JS across CLAP, BEATs and PANNs. FoleyBench [A] separates sound identity, timing and quality.
[Q3]: Human and automatic evaluation on identical audio
Thank you for suggesting a direct comparison on the same examples. Human and automatic evaluation both find preservation easier than updating the requested sound, with different absolute rates. We use identical audio and count the same initially correct unchanged intervals as the listeners for PR and JS:
The table shows how often classifiers and listeners give the same yes/no answer (%).
Same answers as listeners (%)
[Q4]: PR and JS with the two videos exchanged
Thank you for suggesting this additional check. JS remains low when either the original or edited audio is used first. We calculate PR and JS twice from the same generated outputs, first checking the original audio's correct unchanged sounds in the edited audio, then reversing the comparison. JS also requires the requested sound to be correct. The table reports medians across model settings:
The table covers all unchanged intervals (%) rather than JS over edit pairs, with ✓ for a correct sound and × for an incorrect sound:
The percentages of unchanged sounds that become wrong or become correct after editing are similar.
[A] Dixit et al., FoleyBench, 2025.