acceptodds
Withdrawn from ICLR 2027

LocalSoundBench: Do Video-to-Audio Models Respond Locally to Visual Edits?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Videos with several visible events need each sound to occur with its event. A local edit gives a controlled test: the changed event’s audio should update while sounds elsewhere stay as they were. Global fidelity and synchronization metrics can miss this error because the expected sounds still occur somewhere in the clip. We introduce LocalSoundBench to test whether current video-to-audio (V2A) models can do both. Across 21 released settings, when models produce the requested sound, they often also alter the audio of footage that was never edited. This is not because they cannot produce the affected sounds: when we test these events in isolation and probe the models' internal features, the sound identities are still present. The models instead do not reliably keep each sound at the right time. We therefore propose Regional Flow, which, given the event regions, lets the model see the whole scene but uses information from each event only when generating audio at that event’s time. With Regional Flow, interval accuracy rises by 5.9 to 39.1 points on three continuous-latent generators. Regional Flow also improves both edit accuracy and joint success on paired edits for all three models. For MMAudio, training with sounds aligned to their event times and supplying annotated sound and time pairs on continuous video both give gains in the same direction. These results suggest that when a model generates one soundtrack from the full video, scene context alone is not enough: the model needs explicit control over when each sound is produced. Our code is available in the Supplementary Material.

Primary area:
datasets and benchmarks
Keywords:
Video-to-Audio Generation, Foley Sound Synthesis, Local Visual Editing, Temporal Assignment, Datasets and Benchmarks, Flow Matching, Audiovisual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fMNQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LocalSoundBench to evaluate whether video-to-audio models can correctly update the target sound following a local visual replacement while preserving previously correct sounds associated with unchanged events. The authors evaluate 21 released settings and diagnose failures through source-isolation tests, same-class replacements, shot reordering, seed variation, and internal representation analysis. They also propose Regional Flow as an inference-time intervention and investigate temporally aligned training. The results identify sound placement in multi-event videos as an important weakness of existing models and show that explicit sound–time correspondence can improve some models.

Critical strengths
  1. The paper addresses a distinctive and valuable research question.
  2. The paper is clearly written and provides extensive analysis and explanation.
  3. The experimental evaluation is comprehensive.
Critical weaknesses

See Questions.

Questions for authors
  1. Most videos in the main evaluation are constructed by concatenating source clips. This may introduce a distribution shift relative to the models’ original training data and contribute to poor synthesis quality. To what extent could this affect the experimental findings and conclusions?

  2. Regarding the automatic metrics, do they assess whether sounds occur at the correct times, including temporal alignment within each interval? How reliably can the classifiers identify the sound categories in generated audio?

  3. Could the authors report human and automatic evaluation results on the same set of examples to determine whether there are significant discrepancies between them?

  4. I would appreciate a discussion of the directionality of PR/JS. Their eligibility sets include only unchanged events that were correctly generated in the designated original output, excluding incorrect-to-correct transitions. However, both silent videos in a benchmark pair are used independently to generate complete soundtracks, so the original-to-edited direction is not inherently unique. Would it be useful to score each pair in both directions and report a correctness transition matrix over all unchanged intervals, distinguishing stable correctness, degradation, recovery, and persistent failure? This would provide a more complete assessment of cross-event effects and test whether the conclusions are robust to the choice of reference direction.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fMNQ (P1)

We sincerely thank the reviewer for the positive assessment and the questions about distribution shift, metric properties and scoring direction.

[Q1]: Effect of concatenating unrelated shots

Thank you for raising the effect of concatenation. Low JS also occurs without combining unrelated scenes. We test this by replacing one action with another from the same continuous EPIC-SOUNDS recording, keeping the other frames unchanged. Standard sampling results (%):

ModelEvaluatorEA ↑PR ↑JS ↑
MMAudioCLAP3.4558.330.00
MMAudioBEATs17.2487.5023.08
MMAudioPANNs13.7955.5612.50
HunyuanVideo-Foley XLCLAP13.7953.337.14
HunyuanVideo-Foley XLBEATs13.7973.6811.76
HunyuanVideo-Foley XLPANNs13.7980.000.00
Omni2SoundCLAP20.6966.6730.77
Omni2SoundBEATs27.5968.7523.08
Omni2SoundPANNs13.7966.6712.50

The main paper also keeps cut positions fixed while repeating clips or changing their order, showing that cuts alone do not explain the errors. We also shift a placement action smoothly in continuous running-water footage without adding a cut. Regional Flow reduces the unchanged background's level change from 0.85 to 0.64 dB and spectral change from 3.30 to 2.69 dB in HunyuanVideo-Foley XL. Replacing an action tests EA, PR and JS within one scene, while shifting its time tests whether the background sound changes without inserting a shot.

[Q2]: Temporal alignment and classifier reliability

We appreciate the question about timing. EA, PR and JS check whether the intended sound occurs in its event interval, rather than its precise onset, and DeSync provides a separate audiovisual timing check. Section 3.3 and Appendix D.4 show why both are needed: placing sounds in wrong intervals reduces interval accuracy from 99.6% to 0.4%, while whole-clip similarities remain nearly unchanged. DeSync responds to a moving energy envelope and can miss wrong sounds when that envelope stays fixed. For generated audio, correlations with human decisions range from 0.38 to 0.53 for EA, 0.21 to 0.43 for PR and 0.21 to 0.47 for JS across CLAP, BEATs and PANNs. FoleyBench [A] separates sound identity, timing and quality.

[Q3]: Human and automatic evaluation on identical audio

Thank you for suggesting a direct comparison on the same examples. Human and automatic evaluation both find preservation easier than updating the requested sound, with different absolute rates. We use identical audio and count the same initially correct unchanged intervals as the listeners for PR and JS:

EvaluatorEA ↑PR ↑JS ↑
Human46.3990.3242.27
CLAP35.0564.9819.59
BEATs42.2769.5926.80
PANNs28.8735.023.09

The table shows how often classifiers and listeners give the same yes/no answer (%).

Same answers as listeners (%)

EvaluatorEA ↑PR ↑JS ↑
CLAP76.2971.8973.20
BEATs69.0777.4274.23
PANNs70.1043.7860.82

[Q4]: PR and JS with the two videos exchanged

Thank you for suggesting this additional check. JS remains low when either the original or edited audio is used first. We calculate PR and JS twice from the same generated outputs, first checking the original audio's correct unchanged sounds in the edited audio, then reversing the comparison. JS also requires the requested sound to be correct. The table reports medians across model settings:

EvaluatorMetricOriginal audio firstEdited audio first
CLAPPR ↑54.3756.05
CLAPJS ↑4.165.72
BEATsPR ↑52.0154.52
BEATsJS ↑5.135.30
PANNsPR ↑62.1564.58
PANNsJS ↑3.906.35

The table covers all unchanged intervals (%) rather than JS over edit pairs, with ✓\checkmark for a correct sound and ×\times for an incorrect sound:

Original audioEdited audioCLAPBEATsPANNs
✓\checkmark✓\checkmark8.959.247.90
✓\checkmark×\times6.196.624.38
×\times✓\checkmark6.016.224.06
×\times×\times78.8577.9283.65

The percentages of unchanged sounds that become wrong or become correct after editing are similar.

[A] Dixit et al., FoleyBench, 2025.

Reviewer_iXjzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether video-to-audio (V2A) models respond locally to visual edits. It introduces LocalSoundBench, built from VGGSound: each 8-second test video is a concatenation of four unrelated 2-second shots joined by hard cuts (1,024 clips, 64 classes, 256 groups), and each edit pair replaces exactly one shot (2,048 pairs). Per-interval audio classifiers (CLAP, BEATs, PANNs) score Edit Accuracy (EA, the edited interval contains the new sound), Preservation Rate (PR, initially correct unchanged intervals stay correct), and Joint Success (JS, both in the same pair). Across 21 released V2A settings, median EA/PR/JS are 14.0%/54.5%/4.2%. Through a series of controls (isolated generation, reruns, same-class donors, shot permutations, probes, input reversals), the authors argue that models can generate the sounds but place them at the wrong events. They propose Regional Flow, a training-free sampling modification that, given known event boundaries, blends the velocity predicted from the full video with velocities predicted from each temporal segment under temporal masks, reporting interval-accuracy gains of 5.9 to 39.1 points on three generators. Additional experiments cover aligned fine-tuning of MMAudio and continuous EPIC-SOUNDS video.

Critical strengths
  1. Broad empirical coverage. The evaluation spans 21 released V2A settings, three interval-level audio classifiers, an audio-only LLM judge, and a human listening study, with source-group bootstrap confidence intervals. This makes the reported measurements on the proposed benchmark reasonably reliable as measurements, even though I have concerns about what the benchmark measures (see weaknesses).
  2. Code release. The supplementary material includes the evaluation code, an implementation of Regional Flow, the MMAudio training script, and a listening demo, which supports reproducibility and follow-up work.
Critical weaknesses

1. The problem setting does not justify generating the whole soundtrack in a single V2A pass. Every benchmark video is a concatenation of four independent, unrelated 2-second clips joined by hard cuts, and the reference audio is likewise four independent audio segments concatenated (as the supplementary demos show). The cut positions are known, and the proposed method itself requires them. In this setting there is no evident reason not to generate audio for each shot independently and concatenate the results. The most natural baseline would be to run V2A on each 2-second shot separately and stitch; the paper's Window Stitch runs V2A on each clip's original 8-second source and crops it, which is already unnecessary and also exposes the model to frames outside the sequence. The paper's own results support this concern: Window Stitch outperforms Regional Flow in interval accuracy on all three generators (MMAudio 67.7 vs. 59.5, HunyuanVideo-Foley XL 43.9 vs. 31.7, Omni2Sound 61.4 vs. 23.2; Tables 13 and 19), and listeners prefer it in 96.7% (event matching), 93.2% (continuity), and 97.6% (overall) of decisive judgments (Table 15). The authors acknowledge that Window Stitch is appropriate when shots are independent with known cuts (Appendix C.4), but that is precisely the benchmark's setting. The only evidence offered for the value of whole-video context is the EPIC-SOUNDS cropping experiment (Table 24), which is not run on the benchmark and shows a drop only for MMAudio, while other models are flat or improve. The settings where whole-video context would plausibly matter (continuous shots, unknown boundaries, overlapping events, shared ambience across cuts) are not evaluated with the edit protocol and are left as open questions. Because the motivation of both the benchmark and the method rests on this point, I consider it decisive for my recommendation.

2. Regional Flow is closely related to existing region-wise composition methods, and the experiments do not isolate the contribution of its global term. With non-overlapping masks (qn=1q_n = 1), Eq. 9 reduces to v~t[n]=(1−αt) vg[n]+αt vk(n)[n]\tilde v_t[n] = (1-\alpha_t)\, v_g[n] + \alpha_t\, v_{k(n)}[n], i.e., a position-wise weighted average of the velocity predicted from the full video and the velocity predicted from the corresponding segment, with a fixed sigmoid schedule (t⋆=0.70t^\star = 0.70, β=0.08\beta = 0.08). For αt=1\alpha_t = 1 this is mask-weighted fusion of per-region predictions as in MultiDiffusion (Bar-Tal et al., ICML 2023), and the guidance form in Eq. 10 closely mirrors Composable Diffusion (Liu et al., ECCV 2022) with masks; neither is cited or discussed. The method's claimed advantage over stitching is that it keeps whole-video context through vgv_g, yet the available evidence suggests that the global term does not bring a clear benefit, since Window Stitch is stronger on every generator. Window Stitch is also not a clean comparison, because it differs from Regional Flow in both its input (8-second source clips) and its sampling trajectories. The most informative control, Regional Flow with αt≡1\alpha_t \equiv 1 (identical inputs, masks, model calls, and solver, with only the global term removed), is not reported, and no ablation of t⋆t^\star or β\beta is given. The "wrong event time" control, which assigns each segment to the wrong interval, cannot answer this question. Without this ablation, the paper does not establish that its technical contribution goes beyond applying an existing region-wise composition scheme along the time axis.

Questions for authors
  1. Motivation. In the benchmark, each video consists of four unrelated clips joined by hard cuts at known positions, and Regional Flow itself requires these positions. In this setting, why is it necessary to generate the whole soundtrack in a single pass rather than generate each shot independently and concatenate? Please describe a concrete real-world use case in which per-shot generation is not an option.

  2. Per-shot baseline. Please report the most direct baseline: running each model on each 2-second shot separately and concatenating the outputs, with interval accuracy and EA/PR/JS on LocalSoundBench. Please also report EA/PR/JS for the existing Window Stitch (8-second sources), for which only interval accuracy is currently given. If 2-second inputs are infeasible for some models, please explain why.

  3. Contribution of the global term. Please report Regional Flow with αt≡1\alpha_t \equiv 1 (no global velocity), with interval accuracy and EA/PR/JS, together with an ablation over t⋆t^\star and β\beta. This is the comparison that isolates the contribution of whole-video context in your method.

  4. Relation to prior work. Please discuss how Regional Flow relates to MultiDiffusion and Composable Diffusion, and clarify what is technically new beyond applying mask-weighted composition along the temporal axis.

Suggestion (would strengthen the paper but is beyond this rebuttal):

  1. Evaluating the edit protocol in a setting where whole-video context is plausibly necessary, such as multi-camera edits of the same scene or local edits in continuous video, would show whether the phenomenon and the method matter beyond concatenations of independent clips.
4 comments
AuthorsOpenReview ↗

Response to Reviewer iXjz (P1)

We thank the reviewer for the detailed review and recognition of our empirical work.

[W1, Q1]: Continuous generation and Window Stitch

We agree that Window Stitch is useful for independent shots. The continuous video experiment below shows a practical advantage for Regional Flow: it preserves the running water more often, runs faster and gives the most continuous water sound in our blind listening comparison on HunyuanVideo-Foley XL.

We use continuous EPIC-SOUNDS footage with running water and a placement action, shifting the action by 0.5 seconds without adding a cut. Window Stitch generates the complete eight second test video four times and joins the corresponding pieces with 500 ms overlap, while Regional Flow generates one complete track from the same video. We count a soundtrack as retaining water when the audio classifier detects it in at least three of the four intervals. Runtime includes generation and assembly on the shared server, excluding model loading:

MethodWater detected (%) ↑Time (s) ↓
Window Stitch28.686.9
Regional Flow35.733.1

Window Stitch takes 2.63× as long as Regional Flow in this run. With independently generated two second pieces, 0.0% of soundtracks meet the same water detection rule. This tests whether changing the foreground action preserves the background sound.

[W1]: Human comparison in Table 15

We would like to clarify that Table 15 reports listeners' preference for Window Stitch over standard sampling. Our direct exploratory blind comparison with Regional Flow gives Regional Flow higher boundary continuity ratings (1 to 5). Window Stitch uses the original eight second source videos with 50 ms fades on each side:

MethodMMAudio ↑HunyuanVideo-Foley XL ↑Omni2Sound ↑
Regional Flow3.634.133.63
Window Stitch3.253.383.13
AuthorsOpenReview ↗

Response to Reviewer iXjz (P2)

[Q2]: Direct shot generation and Window Stitch baselines

Thank you for requesting this direct baseline. Regional Flow achieves higher EA than direct two second Window Stitch under all three classifiers in HunyuanVideo-Foley XL. The tables report interval accuracy and EA, PR and JS (%) for each model, comparing direct generation of test shots, Window Stitch from original eight second source videos, and Regional Flow.

The two second version uses only test shots, while the eight second Window Stitch version uses the original source videos, including frames outside the test sequence, and the paper's 50 ms join setting. Regional Flow uses the complete eight second test video. Video duration identifies the length used for each generation, and interval accuracy covers all intervals of edited outputs.

MMAudio

EvaluatorMethodVideo duration (s)Interval acc. ↑EA ↑PR ↑JS ↑
CLAPWindow Stitch264.6566.99100.0067.67
CLAPWindow Stitch868.4167.8798.4066.56
CLAPRegional Flow859.1659.6783.9443.16
BEATsWindow Stitch267.1971.48100.0071.21
BEATsWindow Stitch870.1473.6399.1971.92
BEATsRegional Flow862.7767.0985.9049.68
PANNsWindow Stitch256.4558.69100.0058.15
PANNsWindow Stitch858.7258.5099.8958.31
PANNsRegional Flow852.3454.7983.3539.25

HunyuanVideo-Foley XL

EvaluatorMethodVideo duration (s)Interval acc. ↑EA ↑PR ↑JS ↑
CLAPWindow Stitch221.3622.27100.0024.40
CLAPWindow Stitch843.4143.6597.9642.17
CLAPRegional Flow830.6632.0376.1921.28
BEATsWindow Stitch223.8526.46100.0027.75
BEATsWindow Stitch847.9748.0598.3746.86
BEATsRegional Flow830.6231.5478.6520.62
PANNsWindow Stitch214.5815.43100.0016.92
PANNsWindow Stitch832.3532.1399.4030.49
PANNsRegional Flow824.3422.6679.4316.01

Omni2Sound

EvaluatorMethodVideo duration (s)Interval acc. ↑EA ↑PR ↑JS ↑
CLAPWindow Stitch251.1752.54100.0050.95
CLAPWindow Stitch863.8264.7598.3163.27
CLAPRegional Flow826.0726.8644.6512.40
BEATsWindow Stitch254.9859.38100.0059.81
BEATsWindow Stitch869.1274.3299.2373.37
BEATsRegional Flow830.3234.1846.7714.60
PANNsWindow Stitch243.2943.65100.0042.32
PANNsWindow Stitch852.8356.8499.3755.69
PANNsRegional Flow821.7823.1442.438.94

Reusing unchanged audio gives Window Stitch 100% PR on independent test shots. Regional Flow preserves the running water more often and runs faster in the HunyuanVideo-Foley XL continuous video experiment below. Window Stitch generates four pieces independently, using either each two second shot or the complete eight second test video for each piece. Regional Flow generates one complete track. We count water as retained if the audio classifier detects it in at least three of the four intervals. Runtime includes generation and assembly on the shared server, excluding model loading:

MethodVideo duration (s)Water detected (%) ↑Time (s) ↓
Window Stitch20.074.5
Window Stitch828.686.9
Regional Flow835.733.1

Window Stitch takes 2.63× as long as Regional Flow when each piece uses the complete video. These timings are measured on the continuous-water edits, separately from the classification results above.

AuthorsOpenReview ↗

Response to Reviewer iXjz (P3)

[W2, Q3]: Contribution of the complete video prediction

We appreciate the suggestion to isolate the complete video prediction. While αt=1\alpha_t=1 gives higher PR, retaining this prediction improves EA under all three classifiers in Omni2Sound. We compare interval accuracy and EA, PR and JS (%) for αt=1\alpha_t=1 and the paper's parameters (t∗,β)=(0.70,0.08)(t_*,\beta)=(0.70,0.08) with the inputs, masks, initial noise, steps and model calls held fixed. Where the masks sum to one, αt=1\alpha_t=1 removes the explicit vgv_g term. In both versions, all predictions at a sampling step use the same full audio latent state ztz_t. The higher value in each comparison is shown in bold:

EvaluatorParametersInterval acc. ↑EA ↑PR ↑JS ↑
CLAPαt=1\alpha_t=125.5925.3946.5410.87
CLAP(t∗,β)=(0.70,0.08)(t_*,\beta)=(0.70,0.08)26.0726.8644.6512.40
BEATsαt=1\alpha_t=130.2033.1147.2514.46
BEATs(t∗,β)=(0.70,0.08)(t_*,\beta)=(0.70,0.08)30.3234.1846.7714.60
PANNsαt=1\alpha_t=122.4622.3652.0310.85
PANNs(t∗,β)=(0.70,0.08)(t_*,\beta)=(0.70,0.08)21.7823.1442.438.94

[Q3]: Sampling parameter ablations

Thank you for asking about the parameter choices. Regional Flow improves EA, PR and JS over standard sampling across all tested settings in MMAudio. We vary t∗t_* and β\beta separately in αt=σ((t∗−t)/β)\alpha_t=\sigma((t_*-t)/\beta). Here t∗t_* controls how long local predictions retain a high weight, and β\beta controls how gradually that weight decreases:

CLAP

Methodt∗t_*β\betaEA ↑PR ↑JS ↑
Standard sampling--10.9451.282.55
Regional Flow0.100.0829.3062.2213.32
Regional Flow0.500.0857.6284.6142.62
Regional Flow0.700.0858.9884.0442.44
Regional Flow0.900.0860.9484.7444.70
Regional Flow0.700.0459.7784.2242.80
Regional Flow0.700.1659.7786.2643.75

BEATs

Methodt∗t_*β\betaEA ↑PR ↑JS ↑
Standard sampling--13.0957.052.33
Regional Flow0.100.0831.6463.4415.40
Regional Flow0.500.0861.3382.9443.86
Regional Flow0.700.0864.8484.8447.00
Regional Flow0.900.0864.6585.3047.08
Regional Flow0.700.0465.2384.9247.91
Regional Flow0.700.1664.8485.1648.43

PANNs

Methodt∗t_*β\betaEA ↑PR ↑JS ↑
Standard sampling--9.1861.054.98
Regional Flow0.100.0824.2265.7113.17
Regional Flow0.500.0851.1783.2436.36
Regional Flow0.700.0855.0883.8340.94
Regional Flow0.900.0856.2582.5839.73
Regional Flow0.700.0455.2784.4440.72
Regional Flow0.700.1653.3283.9739.20

[Q4]: Relation to MultiDiffusion and Composable Diffusion

We thank the reviewer for pointing out these related methods [A, B]. Regional Flow shares their principle of combining regional predictions, which we use here to study when visible events affect the audio. Our contribution is the paired V2A benchmark and the controlled diagnosis of this timing error. With inputs, noise, steps and model calls matched, applying each local prediction at its event time outperforms averaging the same predictions across the track by 14.5 to 53.1 EA points across model and classifier comparisons. We also apply the same predictions at wrong event times. These comparisons show that each prediction should change the audio at its event time.

[A] MultiDiffusion, ICML 2023. [B] Composable Diffusion, ECCV 2022.

AuthorsOpenReview ↗

Response to Reviewer iXjz (P4)

[W1, Q1]: Audio quality alongside EA, PR and JS

We thank the reviewer for raising audio quality. On the same HunyuanVideo-Foley XL edited outputs, Regional Flow improves EA, PR and JS (%) together with Audiobox production quality (PQ). Audiobox Aesthetics [A] predicts production quality, including clarity and fidelity, independently of the interval classifiers:

EvaluatorMetricStandard samplingRegional Flow
CLAPEA ↑21.0932.03
CLAPPR ↑66.8276.19
CLAPJS ↑12.2421.28
BEATsEA ↑21.8831.54
BEATsPR ↑71.6578.65
BEATsJS ↑12.9420.62
PANNsEA ↑16.4122.66
PANNsPR ↑73.5679.43
PANNsJS ↑10.0216.01
AudioboxPQ ↑6.2126.428

[Suggestion]: Background preservation in continuous video

Thank you for suggesting a test beyond independent shots. Regional Flow produces smaller background level and spectral changes than standard sampling and Window Stitch, together with the most continuous water sound in our blind listening comparison. We test HunyuanVideo-Foley XL on continuous EPIC-SOUNDS footage, shifting a foreground placement action by 0.5 seconds through a smooth local change in playback speed. Frame order and frames outside this timing edit remain unchanged.

We compare RMS level and log STFT magnitude differences only on the unchanged background, excluding foreground placement sounds. Window Stitch receives the complete test video for each of four generations and blends the corresponding pieces with 500 ms overlap:

MethodLevel change (dB) ↓Spectral change (dB) ↓
Standard sampling0.853.30
Window Stitch0.783.32
Regional Flow0.642.69

[A] Tjandra et al., Meta Audiobox Aesthetics, 2025.

Reviewer_P5bxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces LocalSoundBench, a paired-video benchmark that evaluates whether video-to-audio models respond correctly to a local visual edit while preserving sounds associated with unchanged events.

Critical strengths

LocalSoundBench examines an important failure mode that conventional whole-clip metrics can overlook: whether changing one visual event produces the corresponding sound while preserving correct sounds elsewhere. The paired construction and complementary Edit Accuracy, Preservation Rate, and Joint Success metrics provide a useful framework for separating these requirements.

Critical weaknesses

Table 2 reports substantially higher human-assessed success than the main automatic evaluation. These results may use different subsets and therefore cannot directly quantify evaluator bias. Nevertheless, this discrepancy makes matched-subset calibration essential, particularly because the paper emphasizes the low automatic median Joint Success. The main text do not provide enough information about sample sizes, agreement, and uncertainty to establish how strongly the numerical failure rates reflect perceptually meaningful errors.

As explicitly defined on page 4, a sound receives credit if its class remains correct, even when its waveform changes. Consequently, an unchanged event could acquire a substantially different timbre, loudness, rhythm, or onset structure without being penalized. This is a reasonable semantic consistency metric, but it supports a narrower claim than keeping the unaffected soundtrack intact. A complementary perceptual comparison between the original and edited outputs would better connect the benchmark to practical local revision.

The main benchmark assembles four-shot sequences from VGGSound clips, providing clean, identifiable event intervals. The additional duration, insertion/deletion, and continuous-video experiments are useful, but they do not fully establish performance on local edits within continuous scenes, especially with overlapping sources or persistent background sound. Moreover, the continuous-video aligned-training experiment is not equivalent to the benchmark’s paired visual-edit evaluation. The paper’s broader implications would be stronger with real continuous-video edit pairs evaluated using the same preservation criteria.

Regional Flow assumes event intervals, while the continuous-video training experiments use annotated sound labels paired with their timing. These interventions demonstrate that explicit temporal supervision can help, but they do not establish how reliably the approach works when event boundaries must be inferred from video. The shuffled-label and reversed-timing controls are informative; however, robustness to realistic boundary errors, missed events, and overlapping intervals remains unclear. This limits the practical interpretation of the proposed remedy.

Equation (1) combines full-video predictions with multiple cell-conditioned predictions, introducing additional inference work. Although the paper discusses Window Stitch and clip-level quality metrics, the main paper do not present a consolidated comparison of accuracy gains, inference cost, and perceptual continuity. Improved interval classification alone does not establish that Regional Flow preserves natural transitions, background consistency, or overall listening quality. A compute-matched comparison with direct perceptual evaluation would make its practical advantages more convincing.

Questions for authors

On the exact samples used for Table 2, how closely do the automatic evaluators agree with human EA, PR, and JS judgments? Please clarify the sample sizes, inter-rater agreement, confidence intervals, and whether the model rankings and failure-rate conclusions remain consistent.

How does Regional Flow perform with automatically estimated or perturbed event boundaries, and how does it compare with the original generator and Window Stitch under a matched inference budget, considering both local accuracy and human-rated audio continuity?

6 comments
AuthorsOpenReview ↗

Response to Reviewer P5bx (P1)

We thank the reviewer for the constructive comments on human and automatic evaluation, preservation of unchanged audio and continuous scenes.

[W1, Q1]: Human and automatic evaluation on identical audio

We appreciate the request to compare human and automatic evaluation on identical audio. Both evaluations find preservation easier than updating the requested sound, despite differences in their absolute percentages. We report EA, PR and JS (%) for CLAP, BEATs and PANNs on the audio rated by listeners, using the same initially correct unchanged intervals as the listeners for PR and JS:

EvaluatorEA ↑PR ↑JS ↑
Human46.3990.3242.27
CLAP35.0564.9819.59
BEATs42.2769.5926.80
PANNs28.8735.023.09

For generated audio, correlations with human decisions range from 0.38 to 0.53 for EA, 0.21 to 0.43 for PR and 0.21 to 0.47 for JS across CLAP, BEATs and PANNs. The table shows how often classifiers and listeners give the same yes/no answer (%). Human-CLAP [A] also compares embedding scores and human judgments.

Same answers as listeners (%)

EvaluatorEA ↑PR ↑JS ↑
CLAP76.2971.8973.20
BEATs69.0777.4274.23
PANNs70.1043.7860.82

[W2]: Acoustic preservation of unchanged intervals

Thank you for highlighting changes beyond the sound class. Regional Flow also reduces changes in level, envelope and spectrum in MMAudio and HunyuanVideo-Foley XL. We directly compare the unchanged audio before and after editing using RMS level difference (dB), RMS envelope correlation (temporal energy) and log STFT magnitude difference (dB, spectrum) [B, C, D]. Values show standard sampling → Regional Flow:

ModelLevel change ↓Envelope correlation ↑Spectral change ↓
MMAudio4.31 → 2.230.488 → 0.6349.21 → 7.66
HunyuanVideo-Foley XL1.98 → 1.390.698 → 0.7706.67 → 5.80

[W3, Q2]: Local edits with continuous background sound

We appreciate the suggestion to evaluate continuous scenes. Regional Flow preserves the running water most often and gives the most continuous water sound in our blind listening comparison on HunyuanVideo-Foley XL. We use continuous EPIC-SOUNDS kitchen footage with running water and a foreground placement action, shifting the action by 0.5 seconds through a smooth local change in playback speed. Frame order and unedited frames stay fixed.

Window Stitch generates the complete eight second test video four times and blends the corresponding audio pieces with 500 ms overlap, while Regional Flow generates one complete track. We count a soundtrack as retaining water when the audio classifier detects it in at least three of the four intervals. Spectral change compares the unchanged background outside the placement sounds. With independently generated two second pieces, 0.0% of soundtracks meet the same water detection rule.

MethodWater detected (%) ↑Spectral change (dB) ↓
Standard sampling21.43.30
Window Stitch28.63.32
Regional Flow35.72.69

[A] Takano et al., Human-CLAP, 2025. [B] Lee et al., Video-Foley, 2025. [C] Peeters et al., The Timbre Toolbox, 2011. [D] Yamamoto et al., Parallel WaveGAN, 2020.

AuthorsOpenReview ↗

Response to Reviewer P5bx (P2)

[W4, Q2]: Robustness to imperfect event boundaries

We agree that imperfect event times are an important practical test. Regional Flow retains higher JS than standard sampling with shifted masks, a removed mask and event times estimated from silent video. We report EA, PR and JS (%) for these comparisons below.

1. Shifting mask boundaries. Regional Flow improves EA, PR and JS over standard sampling at every tested boundary shift in MMAudio. We shift the internal mask boundaries by ±0.10, ±0.25 and ±0.50 seconds while keeping the video and evaluation intervals fixed:

CLAP

MethodBoundary shift (s)EA ↑PR ↑JS ↑
Standard sampling-10.9451.282.55
Regional Flow058.9884.0442.44
Regional Flow-0.5050.0079.4330.97
Regional Flow-0.2556.2583.6540.38
Regional Flow-0.1059.1884.5143.16
Regional Flow+0.1059.3884.1542.53
Regional Flow+0.2553.5281.1136.63
Regional Flow+0.5045.9075.9926.86

BEATs

MethodBoundary shift (s)EA ↑PR ↑JS ↑
Standard sampling-13.0957.052.33
Regional Flow064.8484.8447.00
Regional Flow-0.5056.0578.4035.48
Regional Flow-0.2562.8982.2942.71
Regional Flow-0.1060.9485.9846.36
Regional Flow+0.1063.0984.9946.04
Regional Flow+0.2560.1681.7640.83
Regional Flow+0.5053.9178.9133.85

PANNs

MethodBoundary shift (s)EA ↑PR ↑JS ↑
Standard sampling-9.1861.054.98
Regional Flow055.0883.8340.94
Regional Flow-0.5049.8080.8535.42
Regional Flow-0.2550.9880.9635.15
Regional Flow-0.1053.3285.4540.50
Regional Flow+0.1055.2783.6840.09
Regional Flow+0.2550.9881.9738.04
Regional Flow+0.5045.5181.5134.42

2. Removing one mask. Regional Flow retains higher JS than standard sampling under all three classifiers when one audio mask is disabled. That mask covers one second, and the complete video prediction remains active in that interval:

EvaluatorEA ↑PR ↑JS ↑
CLAP56.0580.5737.80
BEATs61.9183.0643.20
PANNs53.1279.8736.36

3. Event times from silent video. Event times estimated by Qwen3-VL-8B give higher JS than standard sampling under all three classifiers. We test local action replacements within continuous EPIC-SOUNDS recordings, comparing Qwen event times with human event times. Qwen receives only silent frames, without sound labels or human times, and all other Regional Flow settings are held fixed:

EvaluatorMethodEvent times fromEA ↑PR ↑JS ↑
CLAPStandard sampling-3.4558.330.00
CLAPRegional FlowHuman6.9064.297.69
CLAPRegional FlowQwen3-VL-8B6.9064.298.33
BEATsStandard sampling-17.2487.5023.08
BEATsRegional FlowHuman17.2484.2118.75
BEATsRegional FlowQwen3-VL-8B17.2472.2225.00
PANNsStandard sampling-13.7955.5612.50
PANNsRegional FlowHuman17.2450.0010.00
PANNsRegional FlowQwen3-VL-8B17.2460.0022.22

FoleyCrafter [A] also separates event timing from generation. Shifting or removing masks tests errors in supplied event times. The Qwen comparison uses times estimated from silent video.

[A] Zhang et al., FoleyCrafter, 2024.

AuthorsOpenReview ↗

Response to Reviewer P5bx (P3)

[W5, Q2]: Inference cost, audio quality and boundary continuity

Thank you for requesting a joint comparison of accuracy, runtime and listening quality. Additional standard sampling steps do not reproduce Regional Flow's gains in EA, PR and JS (%). The comparisons below also show higher Audiobox quality and boundary continuity ratings.

1. Increasing sampling steps. To compare at similar runtime, we increase the original MMAudio sampler from 25 to 38 steps. This approximates the runtime of Regional Flow with 25 steps, which takes about 1.5× the original 25-step time:

EvaluatorMethodStepsEA ↑PR ↑JS ↑
CLAPStandard sampling2514.4552.554.68
CLAPStandard sampling3815.2354.174.78
CLAPRegional Flow2559.6783.9443.16
BEATsStandard sampling2514.6555.192.81
BEATsStandard sampling3816.6053.544.32
BEATsRegional Flow2567.0985.9049.68
PANNsStandard sampling2511.1359.823.57
PANNsStandard sampling3812.3060.444.09
PANNsRegional Flow2554.7983.3539.25

2. Audio quality. On the same HunyuanVideo-Foley XL edited outputs, Regional Flow improves EA, PR and JS together with Audiobox production quality (PQ). Audiobox [A] evaluates the audio independently of the interval classifiers:

EvaluatorMetricStandard samplingRegional Flow
CLAPEA ↑21.0932.03
CLAPPR ↑66.8276.19
CLAPJS ↑12.2421.28
BEATsEA ↑21.8831.54
BEATsPR ↑71.6578.65
BEATsJS ↑12.9420.62
PANNsEA ↑16.4122.66
PANNsPR ↑73.5679.43
PANNsJS ↑10.0216.01
AudioboxPQ ↑6.2126.428

3. Boundary continuity. Regional Flow also receives higher ratings for boundary continuity (1 to 5) in our exploratory blind comparison with the paper's Window Stitch. Window Stitch generates audio from each original eight second source video and joins the required pieces with 50 ms fades on each side, while Regional Flow generates one complete track:

MethodMMAudio ↑HunyuanVideo-Foley XL ↑Omni2Sound ↑
Regional Flow3.634.133.63
Window Stitch3.253.383.13

[A] Tjandra et al., Meta Audiobox Aesthetics, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

Dear Reviewers and AC,

We sincerely thank the reviewers for their constructive feedback and recognition of the research question (P5bx, fMNQ), empirical coverage (iXjz, fMNQ) and reproducibility (iXjz).

LocalSoundBench evaluates whether a visual edit updates the requested sound while preserving correct sounds at unchanged events. Alongside the benchmark, we use controlled tests to explain these errors and study Window Stitch, Regional Flow and training with sounds aligned to their event times.

In response to your feedback, we have completed the following experiments and analyses:

  • Human and automatic evaluation on the same audio, with sample sizes and confidence intervals.
  • Level, envelope and spectral changes in unchanged audio.
  • Regional Flow with shifted masks, one removed mask and Qwen3-VL-8B event times.
  • Sampling steps, inference time, Audiobox quality and boundary continuity.
  • Window Stitch from two second test shots and original eight second source videos.
  • Regional Flow without the complete video prediction, plus t∗t_* and β\beta comparisons.
  • Local edits in continuous video, with background sound and runtime comparisons.
  • PR and JS with the two videos exchanged, including all unchanged intervals.

We sincerely hope these results address your concerns and look forward to the discussion.

Best regards, Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.