acceptodds
Under review as a conference paper at ICLR 2027

DepthFoley: Depth-Guided Video-to-Audio Generation for Spatial Consistency

Abstract

Current Video-to-Audio (V2A) approaches have achieved impressive semantic quality but often treat the auditory environment as a spatially-static and distance-agnostic plane. Neglecting fundamental distance-dependent acoustic decay, such as sound attenuation and spectral filtering, results in audio that fails to reflect spatial dynamics and shatters the immersive illusion. To address this, we propose DepthFoley, a framework that shifts the paradigm from purely data-driven fitting to a depth-guided alignment strategy. By leveraging metric depth estimation, we extract metric depth trajectories to guide a pre-trained backbone model via a Depth ControlNet. We further introduce a training-free object-wise inference strategy for multi-object scenes. To ensure spatial consistency without sacrificing generative priors, we introduce Sound Pressure Level (SPL) and Spectrally Perceptual Energy (SPE) losses. Augmented by depth-informed regularization, these objectives guide the model to internalize realistic decay trends while preserving disentangled acoustic dynamics, balancing veridical spatial realism with artistic flexibility. To evaluate this, we introduce DITC and DIGA, two novel metrics experimentally validated to align with human spatial perception. Extensive experiments demonstrate that our architecture achieves superior spatial-acoustic alignment and strong zero-shot generalizability to complex, out-of-distribution environments, establishing a new standard for spatial perceptual realism.

Primary area:
generative models
Keywords:
Depth-Guided, Foley Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_H1cmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DepthFoley adds a depth-conditioned branch to a frozen CAFA backbone and uses sound pressure level and spectral energy losses to model distance-dependent acoustic changes, with object-wise generation and mixing for multi-source scenes. Objective evaluations on VGGSound and AVE show improved distance–loudness consistency while largely preserving the backbone’s audio quality.

Critical strengths
  1. The paper addresses a practical aspect of video-to-audio generation that is often overlooked: how sound should change as a visible source moves closer or farther away.

  2. The experiments provide perceptual evidence beyond the proposed objective metrics. The method achieves a spatial alignment rating of 4.18, compared with 2.67 for CAFA and 2.82 for Post-hoc, while FAD changes only from 7.20 to 7.35 relative to CAFA. These results support improved distance control without a substantial loss in overall audio quality.

Critical weaknesses
  1. Post-hoc achieves a DITC of 0.5684, slightly higher than the proposed method’s 0.5529, and its DIGA is also close. However, Post-hoc applies a single global gain envelope to the entire audio mixture, whereas DepthFoley generates separate tracks for individual sources and then mixes them. The comparison therefore differs in both distance modulation and inference procedure. The single-pass result in Table 9 still improves FAD over Post-hoc, so the gains cannot be attributed entirely to object-wise generation. Nevertheless, a CAFA + Post-hoc baseline using the same object-specific inputs and mixing procedure is needed to better separate the contributions of learned distance control and object-wise processing.

  2. The multi-object ablation reports aggregate results over the full test set, with spatial metrics computed from the final audio mixture against a single reference distance trajectory. This does not establish whether each source is controlled correctly. For example, when one object approaches and another moves away, the mixture’s loudness curve may be dominated by just one source.

Questions for authors
  1. Section 3.5 introduces a text prompt T(k)T^{(k)} and a visual input V(k)V^{(k)} for each object, but does not explain how they are obtained. Section 4.1 gives the prompt template “The sound of <label>,” but how are the source-specific labels determined? Does this require an additional video-understanding model or a large language model?
  2. Is each visual input an object crop, a masked video, or the full video?
  3. Since the tracks are generated independently and then summed, how does the method avoid duplicating background sounds and maintain appropriate relative levels across sources?
4 comments

We are grateful to the reviewer for highlighting these important concerns, which we address below.

W1: Unfair Comparison with Post-hoc Scaling.

A1: We agree that the original comparison does not fully disentangle learned distance modulation from object-wise inference. To address this, we implement an Object-wise Post-hoc baseline using the same object-centric visual inputs, individual distance trajectories, multi-pass generation, and mixing procedure as DepthFoley. The primary difference lies in the modulation mechanism: DepthFoley incorporates distance information during generation, whereas Object-wise Post-hoc applies rule-based attenuation to CAFA-generated stems.

We compare the final mixtures under the same evaluation protocol:

MethodDITC ↑DIGA ↑FAD ↓KL ↓
Global Post-hoc0.56840.75449.481.467
Object-wise Post-hoc0.50770.69538.421.744
DepthFoley (Ours)0.55290.76857.351.472

Under matched inference conditions, DepthFoley consistently outperforms Object-wise Post-hoc in both spatial consistency (DITC: 0.5529 vs. 0.5077; DIGA: 0.7685 vs. 0.6953) and audio fidelity (FAD: 7.35 vs. 8.42; KL: 1.472 vs. 1.744). These results demonstrate that the improvements cannot be attributed solely to multi-pass processing, supporting the effectiveness of learned distance-aware generation. We will include this controlled comparison in the revised manuscript.

W2: Insufficient Validation of Independent Multi-object Control.

A2: We agree that mixture-level DITC/DIGA cannot fully establish object-specific distance control, as the final mixture may be dominated by a single source. We therefore introduce a per-object evaluation, measuring each generated stem against its corresponding object-specific distance trajectory.

For each video, we average the scores across object stems, followed by averaging across videos. We compare DepthFoley with Object-wise Post-hoc under the same protocol:

Method (Per-object Evaluation)DITC ↑DIGA ↑
Object-wise Post-hoc0.51310.6686
DepthFoley (Ours)0.55650.7002

DepthFoley outperforms Object-wise Post-hoc on both metrics, demonstrating improved alignment with individual object trajectories. Unlike mixture-level evaluation, this protocol assesses each stem independently, providing complementary evidence of object-specific distance modulation beyond rule-based attenuation. We will incorporate these results and clarify the distinction between the two evaluation protocols in the revised manuscript.

Q1/2: Construction of Object-specific Text and Visual Inputs.

A3: Our prompts are derived from dataset-provided sound-event labels, while web-collected videos are manually annotated with a single sound category. These labels guide Grounded-SAM2 in detecting and tracking multiple instances of the same category. Thus, our multi-object setting primarily involves same-category object instances, rather than arbitrary multi-category sound sources. All instances share the template "The sound of <label>," without additional LLM-based captioning or instance-specific descriptions.

To minimize background sound duplication, we adopt a primary-secondary object strategy. The primary object (selected based on visibility duration and proximity) retains the original video and prompt, allowing background sounds to be generated primarily through this pass. Single-object scenes use the same unmodified inputs. For secondary objects, we apply Gaussian blurring and darkening to non-target regions, preserving scene context while emphasizing the target. We also append "without any sound in the background" to discourage background sound generation.

Thus, object-specific conditioning relies on individual visual inputs and distance trajectories while sharing the same sound category. We will revise the manuscript to clarify this setting, refine ambiguous terminology, and distinguish same-category instances from arbitrary multi-category sound sources.

Q3: Background Sound Duplication and Relative Source Levels.

A4: As described in A3, we use a primary-secondary object strategy to reduce background duplication. The primary object retains the original visual and text inputs, while secondary objects use blurred, darkened backgrounds and prompts discouraging background sounds. This encourages background audio to originate mainly from the primary-object pass.

Regarding relative loudness, each stem is generated under its own distance trajectory for source-specific intensity modulation. During mixing, we preserve generated amplitudes and apply only global scaling to prevent clipping, without inter-source gain normalization. This retains model-generated relative levels but does not ensure physically accurate loudness relationships. We will clarify this limitation and explore distance-aware inter-source gain calibration in future work.

Thank you for the response and additional experiments. The Object-wise Post-hoc comparison alleviates my concern about the fairness of the baseline comparison, and the explanation of the primary–secondary strategy makes the inference procedure clearer. However, I still have several specific questions.

  1. Data and evidence for the per-object evaluation.
    I understand that the new per-object evaluation compares each generated audio stem with the distance trajectory of its corresponding object. Could you clarify whether this evaluation uses the full VGGSound and AVE test set from the paper or only its multi-object subset, and report the numbers of videos and objects evaluated? Since these datasets do not provide ground-truth audio stems, how were the stems and associated object-level information used in this evaluation obtained? Will this benchmark be made publicly available?

  2. Mapping sound events to tracking targets.
    In this text-conditioned video-to-audio setting, how are the supplied sound-event labels converted into detection and tracking targets for Grounded-SAM2? For example, when a person plays a drum, does the tracker follow the person, the drum, or the contact region? How are the target object and its distance trajectory defined for diffuse background sounds such as wind, or sounds caused by wind-induced glass vibration? Are these cases within the current scope of the method? Have you also examined sensitivity to tracker choice or tracking errors, for example through an alternative tracker?

  3. Effectiveness of the background-suppression strategies.
    What are the individual contributions of background blurring/darkening in the primary–secondary strategy and the additional prompt, “without any sound in the background”? Could you provide a small ablation showing whether these interventions reduce duplicated background and non-target sounds while preserving the intended source? This would also help establish whether both interventions are necessary.

  4. Differences between training and object-wise inference.
    My understanding is that training uses original videos, mixed audio, and composite distance trajectories, whereas object-wise inference may use modified visual inputs, individual object trajectories, and additional background-suppression prompts. Could you confirm whether these visual and text modifications are also used during training? If not, have you examined how this potential input-distribution mismatch affects source correspondence, audio quality, and the reliability of distance control?

Overall, the rebuttal has clarified several of my initial questions. My main remaining concern is whether the current evidence sufficiently supports the paper’s contribution to reliable object-specific distance control. If these points can be clarified with appropriate evidence, I would be inclined to raise my recommendation. I would also welcome alternative analyses or validation approaches that directly address these concerns.

We thank the reviewer for their constructive feedback and are pleased that some concerns have been addressed. Below, we provide further clarifications and additional experimental evidence in response to the follow-up questions.

Q1: Data and Evidence for the Per-object Evaluation.

A1: Our previous evaluation covers the full VGGSound and AVE test set used in our paper, comprising 612 single-object videos, 347 two-object videos, and 41 videos with three or more objects, totaling 1,461 object instances. We separately report per-object results for the multi-object subset (388 videos, 849 instances), averaging scores across objects within each video and then across videos:

MethodDITC ↑DIGA ↑
Object-wise Post-hoc0.38320.4788
DepthFoley (Ours)0.55410.5858

The evaluated stems are generated independently during object-wise inference, rather than extracted from ground-truth audio. Object masks and distance trajectories are obtained using Grounded-SAM2 and Depth Anything V2 (Metric), respectively. Each stem is evaluated against its corresponding trajectory, requiring no ground-truth source separation.

We view this as an object-level evaluation protocol built upon existing datasets, rather than a separate benchmark. We plan to publicly release our implementation and evaluation settings to facilitate reproducibility.

Q2: Mapping Sound Events to Tracking Targets.

A2: Our pipeline directly uses dataset-provided sound-event labels as text prompts for Grounded-SAM2 to detect and track corresponding object instances. The resulting masks are used to construct object-specific videos and extract distance trajectories by averaging Depth Anything V2 (Metric) estimates within each mask.

We primarily select motion-driven categories with clearly identifiable sound-producing objects, such as cars, trains, boats, and aircraft, reducing ambiguity in target selection. For manually collected videos, we explicitly specify the target object (e.g., the drum rather than the performer). Diffuse sounds such as wind are treated as background audio, relying on the pretrained backbone rather than independent distance control.

Regarding tracking robustness, we have not conducted a systematic sensitivity analysis using alternative trackers. Instead, we mitigate tracking errors through trajectory quality filtering, excluding unreliable results with unnatural fluctuations, intermittent detections, insufficient visibility, or negligible distance variation. While this reduces obvious tracking failures, sensitivity to tracker choice remains to be evaluated. We will clarify these criteria and limitations in the revised manuscript.

Q3: Effectiveness of Background-suppression Strategies.

A3: We conduct a six-configuration ablation on the multi-object subset (388 videos) to examine the effects of background blurring, darkening, white-background replacement, and additional background-suppression prompting. We report per-object DITC/DIGA for distance alignment and FAD/KL on the final mixtures for audio fidelity.

We evaluate per-object DITC/DIGA to assess individual distance alignment, and FAD/KL on the final mixed audio to assess overall audio fidelity.

ConfigurationPer-object DITC ↑Per-object DIGA ↑Mix FAD ↓Mix KL ↓
Original Video0.54830.571513.101.62
Blur Only0.53980.560913.481.65
Dark Only0.54660.571812.631.61
White Background0.54240.573512.951.60
Blur + Dark (Base Prompt)0.54270.570810.841.54
DepthFoley (Ours)0.55410.58589.381.47

The results show that individual background modifications yield modest and varying improvements, while their combination, together with prompt-based suppression, provides a more substantial improvement in overall audio quality and distance consistency.

Q4: Differences Between Training and Object-wise Inference.

A4: We confirm that training uses original videos, mixed audio, and composite distance trajectories, without visual or textual background suppression. During object-wise inference, the primary object retains the original inputs, while secondary objects use modified videos, individual distance trajectories, and background-suppression prompts.

This inference strategy enables object-specific conditioning without additional training. We acknowledge the potential input-distribution mismatch, but preserve some background context to reduce the differences between training and inference inputs.

As shown in A3, our ablation results demonstrate that the full strategy achieves better distance alignment and audio fidelity than the alternative input configurations, supporting its effectiveness despite the training-inference differences. We will clarify these differences and their limitations in the revised manuscript.

Thank you for the detailed responses and additional experiments. The results on the multi-object subset and the background-suppression ablations help clarify the method and strengthen the experimental evidence.

I still have some reservations about the training–inference mismatch and the extent to which the evaluation metrics establish source-specific control. Nevertheless, I consider the overall study reasonably complete and will raise my recommendation to Weak Accept. Please incorporate the additional experiments, implementation details, and discussion of the remaining limitations from the rebuttal into the revised manuscript.

Reviewer_WJydOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents DepthFoley, an approach for Video-to-Audio sound generation for objects in silent scenes, conditioned on predicted metric depth of detected objects. It exploits distance-dependent acoustic decay as supervision to learn to modulate a depth-residual for a frozen CAFA V2A backbone in ControlNet fashion. Object-wise spatial trajectories are extracted from videos to condition the latent for audio diffusion.

Critical strengths

The approach seems to perform average on audio quality but the user study and qualitative examples show this is a small trade-off for having significantly better spatial-audio consistency.

Adding samples from another dataset only to the test set should make the task more difficult so it is interesting that the paper performs well even in this more difficult setting.

The paper presents a user study of 30 participants which is important given the average performance on audio fidelity metrics. Very good subjective evaluation results indicate the trade-off seems justified.

The DITC and DIGA metrics are key in showing the performance of the approach so the effort of testing alignment with human perception and moderate to high correlation scores are reassuring.

The approach seems very efficient. 9000 clips trained for two days on an RTX 4090 is a very accessible demand of resources, even though the backbone is not trained from scratch.

Critical weaknesses

DITC computes correlation between estimated metric distance (estimated by Depth Anything V2) and the audio intensity. DIGA evaluates correlation between an estimated distance trajectory and the audio intensity profile. Therefore, both metrics do compare if the audio behaves according to the laws of physics but compared against an estimated metric depth which may have a characteristic error for these videos. This seems hard to isolate and could be another reason why the ground truth underperforms (Appendix G).

In the training set, the mean SPL has a pearson correlation of 0.37 and spearman correlation of 0.4 which is weak to moderate. Claims of a consistent macroscopic distance-dependent attenuation trend are therefore weakened and a large part of the audio variability is not explained by spatial proximity alone. This will limit the performance of the approach.

It would have been nice to have examples of MMAudio in the supplementary video as the 2nd strong baseline, maybe even some original scenes from the user study.

The worse DITC score for SA consistency is hypothesized to be confounding noise and speech and it is argued that the proposed approach would perform better on scenes with many audio sources compared to applying an attenuation curve to the waveform. This is never tested. It should have been tested with dedicated multi-object scenes. In the qualitative examples and the user study examples, only simple scenes with one moving object are shown/heard.

Furthermore, the robustness to and handling of multiple objects in general could be clearer. Because the amount of objects detected should be easy to extract, there should be a study into the performance against the number of objects in the scene. It is not completely clear why multi-source motions were excluded during training. It may be because otherwise supervision would be ambiguous when training on a per-object basis in the same scene.

Selection of w_d and its evaluation should be clearer. In table 8, 0.4 is selected under the single-pass composite-trajectory protocol. In table 9 it is found that object-wise multi-pass performs better, but w_d was not selected under that protocol. There may be a better w_d for the multi-pass protocol. Then, in table 9 results were given on the test set and are identical to table 8 results for w_d = 0.4. It is not clearly stated if the test set was used for for table 8 but if so, w_d = 0.4 was chosen based on the test set performance. That may skew the results in Table 1, w_d should have been selected on a validation set.

In summary, I see a need for important clarifications. If the details surrounding the choice of w_d and its impact on the results in table 1 are explained well and other concerns are addressed I see this as a paper that can be accepted but not before.

Questions for authors

Was there an investigation how well the approach performs on the VGGSound vs. AVE part of the test set? It seems like an easy test of generalization.

Why was w_d not selected on the multi-pass protocol?

Is it accurate that w_d was selected on the test set?

Which VGGSound categories were used?

What is b_0, the band cut off? Was using it ablated or was the concrete choice estimated?

4 comments

We thank the reviewer for this valuable feedback and the opportunity to clarify our approach.

W1: Reliability of Estimated Metric Depth and Spatial Evaluation.

A1: We acknowledge that monocular metric depth estimation inevitably introduces errors. We adopt Depth Anything V2 (Metric), a widely used model providing distance estimates in meters. Our framework primarily focuses on relative distance variations and corresponding acoustic trends, rather than requiring perfectly accurate absolute distances. Thus, moderate estimation errors do not necessarily undermine distance-intensity consistency, as further supported by our depth-noise robustness experiments in Appendix D.

Moreover, our modular framework can accommodate alternative depth estimators as they improve. We acknowledge that validation against ground-truth geometry would provide stronger evidence and clarify that DITC/DIGA measure alignment with estimated distance trajectories, rather than independently verified geometric distances. We will explicitly discuss this limitation in the revised manuscript.

W2: Weak Distance-SPL Correlation in Real-world Data.

A2: The reported Pearson (0.37) and Spearman (0.40) correlations are computed from aggregated distance-bin statistics, reflecting a measurable distance-dependent attenuation trend in unconstrained recordings. We do not assume that distance alone explains audio intensity, which is also influenced by source-intrinsic dynamics, background noise, and recording conditions. Accordingly, our SPL loss employs relative intensity constraints with acoustic slack, allowing the model to learn spatial trends without enforcing rigid inverse-square attenuation.

Regarding the evaluation, DITC/DIGA measure audio-distance alignment using estimated metric depth trajectories. While depth estimation errors may introduce uncertainty, the same distance references are used to evaluate GT and all generated outputs, ensuring a consistent comparison of their alignment with the estimated spatial cues. These metrics are intended to assess distance-conditioned acoustic consistency rather than independently establish absolute geometric accuracy.

Furthermore, our Appendix G analysis shows that GT DITC/DIGA improve from 0.3750/0.5252 to 0.4941/0.6684 after AudioSep source separation, suggesting that interfering audio components contribute to the lower GT alignment. We will clarify the interpretation of these metrics and the role of relative physical constraints in the revised manuscript.

W3: Limited Supplementary Qualitative Examples.

A3: We agree that more diverse examples would better demonstrate our approach. In the revision, we will expand the supplementary video with additional MMAudio comparisons, representative user-study samples, and more scenes involving multiple moving objects for a more comprehensive qualitative evaluation of DepthFoley.

W4: Insufficient Multi-object Evaluation and Training Justification.

A4: Our user study was not restricted to single-object scenes: 11 of the 30 evaluated videos (36.7%) contain multiple objects. We acknowledge that the current qualitative examples primarily feature simpler scenes and will include additional multi-object demonstrations, including representative user-study samples, in the revised supplementary material.

Regarding training, multi-object scenes were not excluded. However, VGGSound and AVE provide mixed audio without source-separated ground-truth stems, making independent source-level supervision ambiguous. We therefore aggregate object trajectories into a composite condition during training, while applying the same trained model independently to each object at inference.

To directly examine performance as scene complexity increases, we stratify the test set by the number of valid object trajectories and evaluate the final mixtures under the original mixture-level protocol. We include both Global and Object-wise Post-hoc baselines to assess whether learned distance modulation offers advantages over rule-based waveform attenuation in multi-object scenes.

MethodMetric ↑1 Object (61%)2 Objects (35%)3+ Objects (4%)
Global Post-hocDITC0.59610.51580.6057
DIGA0.78890.69100.7828
Object-wise Post-hocDITC0.59610.34850.5521
DIGA0.78890.52460.7609
DepthFoley (Ours)DITC0.55800.53280.6502
DIGA0.77270.75010.8653

DepthFoley outperforms both Post-hoc variants on DITC and DIGA in the multi-object subsets, whereas Global Post-hoc performs slightly better in single-object scenes. These results provide direct evidence of the benefit of learned distance modulation in multi-object scenarios, although the 3+ object subset remains relatively small. We will incorporate this analysis and additional multi-object demonstrations in the revised manuscript.

W5/Q2/Q3: Selection and Evaluation of Distance Guidance Weight wdw_d.

A5: We thank the reviewer for highlighting this ambiguity. We clarify that (w_d=0.4) was empirically selected through preliminary listening comparisons, rather than optimized using the quantitative results in Table 8.

Specifically, we conducted an informal listening evaluation on approximately 50 videos, including VGGSound clips without automatically extracted distance trajectories (for which distance variations were manually annotated) and additional web-collected videos covering both familiar and previously unseen scene types. These preliminary listening samples had no video-level overlap with the final testset. Our research team compared different guidance weights (0.3-0.6), and (w_d=0.4) was perceptually preferred in these examples, providing an empirical basis for our final setting. This was an internal preliminary assessment rather than a formal user study.

Table 8 provides a subsequent quantitative sensitivity analysis of the selected weight under the single-pass protocol. For object-wise multi-pass inference, we retained the same (w_d=0.4) to ensure a controlled comparison without additional parameter tuning. While protocol-specific optimization may yield further improvements, our results demonstrate the effectiveness of multi-pass inference under this fixed setting.

We will clarify the empirical selection process and distinguish it from the quantitative sensitivity analysis in the revised manuscript.

Q1: Insufficient Dataset-specific Generalization Analysis.

A6: We thank the reviewer for this question. We have conducted separate evaluations on the VGGSound and AVE test subsets to further examine cross-dataset generalization. As shown below, DepthFoley consistently achieves improved spatial-acoustic consistency over the CAFA baseline on both datasets, demonstrating that the learned distance-aware modulation generalizes across different data distributions rather than being limited to VGGSound.

DatasetMethodDITC ↑DIGA ↑FAD ↓KL ↓
VGGSoundCAFA0.03840.14077.021.453
VGGSoundDepthFoley0.52730.78947.231.492
AVECAFA0.07580.13597.431.863
AVEDepthFoley0.57850.74767.421.328

Importantly, AVE is used only for evaluation and is not included in our training data. The consistent improvements on both datasets therefore provide additional evidence of the cross-dataset generalization capability of DepthFoley. We will include these separate evaluation results in the revised manuscript.

Q4: VGGSound Category Selection.

A7: We thank the reviewer for this question. We selected 20 motion-driven sound categories from VGGSound, primarily involving moving vehicles, transportation, and other sound-producing objects with noticeable spatial motion. Specifically, these categories include: airplane; race car, auto racing; subway, metro, underground; car passing by; skateboarding; horse clip-clop; missile launch; driving buses; motorboat, speedboat acceleration; railroad car, train wagon; train whistling; helicopter; train horning; tractor digging; driving snowmobile; sailing; skidding; airplane flyby; train wheels squealing; and car engine starting. These categories were selected to provide diverse motion-related acoustic events for learning distance-dependent sound variations. We will explicitly include the complete category list in the revised manuscript for reproducibility.

Q5: Definition and Selection of the SPE Frequency Cutoff b0b_0.

A8: We thank the reviewer for this question. In our implementation, b0=21b_0=21, selecting bands 21-31 from 32 linearly spaced frequency bands. This cutoff was determined by analyzing the spectral energy distribution across the dataset, where bands below index 21 account for over 98% of the total energy. We therefore focus SPE supervision on the remaining high-frequency bands, which contain relatively little energy but are important for preserving spectral characteristics.

We further conducted an ablation comparing full-band and high-frequency SPE supervision:

SPE ConfigurationFAD ↓DITC ↑DIGA ↑
All Bands13.710.4400.657
High-frequency (Ours)7.350.55290.7685

High-frequency supervision achieves better audio fidelity and spatial consistency, while full-band supervision may overconstrain dominant low-frequency energy variations and interfere with SPL-guided modulation. These results support our energy-distribution-based cutoff selection. We will clarify the selection criterion and ablation results in the revised manuscript.

I thank the authors for the clarifications and added evaluations. I am satisfied with the responses and especially relieved that multi-object scenes were not excluded. I will increase my rating to weak accept and can not see remaining major questions at this point.

Thank you for your comments and recognition. We are glad that our clarification about multi-object scenes addresses your major concern, and we appreciate you raising your score. We will carefully revise the manuscript accordingly.

Reviewer_g1FxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper points out one limitation of current video-to-audio (V2A) models as overlooking fundamental distance-dependent acoustic decay and proposes DepthFoley as a depth-guided alignment strategy. DepthFoley extracts per-object metric depth trajectories using Grounded-SAM2 and Depth Anything V2 and then feeds them through a ControlNet-style adapter into a frozen CAFA backbone. Two auxiliary training losses are proposed: Sound Pressure Level (SPL) and Spectrally Perceptual Energy (SPE) losses. To handle multi-object scenes, this work runs the model once per object independently and then sums the resulting stems. Two rank-correlation metrics as DITC and DIGA are proposed, and large gains are reported on them over baselines including Seeing&Hearing , CAFA, MMAudio, and FoleyCrafter. The authors also conduct listening study as well as generalizability and robustness analysis. The five architectural and objective variants are also ablated.

Critical strengths
  1. It is a valid claim that general V2A models still lack modeling of distance-dependent level changes. This work is towards a valuable direction of modeling it explicitly and making it controllable.

  2. The proposed framework uses efficient adaptation as it freezes the backbone and only trains the adapter. This framework should be generalizable to other backbones, but this point is not discussed nor validated.

Critical weaknesses
  1. For the methodology, there are following concerns.

  a. As part of the foundation of the work, whether absolute or relative SPL can track source distance, with noisy YouTube-based datasets such as VGGSound and AVE, needs to be clearly analyzed.

  b. The SPE loss seems basically a spectral reconstruction loss with no distance term, so it is unclear how it can encode distance-dependent spectral filtering. A spectral attenuation analysis would help.

  c. It is unclear how the cross-object relative levels are determined in the multi-pass inference, and the effectiveness of the approach on objects with quite large distances between them are not evaluated.

  d. The accuracy, robustness to noise and OOD inputs to the predictor should be clearly evaluated and discussed.

  1. For experiments, there are some serious concerns on baseline comparisons.

  a. In Table 1, all baselines receive no distance input and score very low (near 0) on DITC and DIGA. It would be important to include more direct baselines that actually support envelope or loudness control driven by inverse-square law of distance attenuation as a directional guide.

  b. The Post-hoc alternative is underspecified. As a trivial alternative, it is already notably better than DepthFoley on DITC and DeSync, yet FAD receiving a large degradation. However, neither gains nor degradations from Post-hoc are clearly discussed or explained. The gains from this trivial alternative could weaken the central claim of the paper. Also, I think Post-hoc should also be able to extend to multi-object scenes by running through the same multi-pass pipeline.

  1. The evaluation depth comes from the same monocular estimator used as the conditioning signal; no metric is computed against true distance. There is no validation of the approach on data with known geometry.

  2. The Generalizability and Robustness Analysis are quite brief and unclear. For example, for indoor-OOD evaluation, again the baseline does not support depth, and human evaluations are not presented. For depth-noise robustness, realistic failure modes are not tested. For controllability, DITC and DIGA both drop under edits, so it is unclear whether the model indeed strictly adheres to user-defined spatial constraints. These results need to be analyzed and explained.

Questions for authors
  1. Please address all points under Weaknesses.

  2. Most citations should be formed in parenthetical citations with parentheses around both the authors and the year, e.g., “Foley synthesis (Iashin & Rahtu (2021); …)” (using \citep in latex), instead of the current “Foley synthesis Iashin & Rahtu (2021); …”. The current citations in the paper reduce readability.

4 comments

W1: Reliability of Distance-SPL Correspondence in Real-world Data.

A1: We thank the reviewer for this comment. We acknowledge that sound intensity in unconstrained recordings is affected by source-intrinsic dynamics, background noise, and recording conditions. Our SPL loss therefore focuses on within-video relative variations, using the inverse-square law as a directional prior with acoustic slack to accommodate natural deviations.

Our analysis in Appendix G further shows that GT DITC/DIGA increase from 0.3750/0.5252 to 0.4941/0.6684 after AudioSep source separation, suggesting that interfering sounds partially obscure distance-dependent acoustic trends. These observations support using relative distance-intensity relationships as flexible supervisory signals rather than enforcing absolute physical attenuation.

W2: Distance-dependent Spectral Modeling with SPE Loss.

A2: We clarify that SPE does not explicitly impose a distance-dependent spectral attenuation law. Instead, it complements depth conditioning and SPL supervision by constraining high-frequency spectral energy to preserve realistic spectral characteristics.

To examine spectral fidelity, we measure the MAE between generated and GT spectral feature changes using three metrics:

MAE to GT ↓High/low-frequency ratio (dB)Spectral centroid (Hz)High-frequency energy ratio
CAFA3.153178.331.741 × 10⁻⁵
Post-hoc3.101183.881.748 × 10⁻⁵
DepthFoley2.674116.081.395 × 10⁻⁵

DepthFoley achieves the lowest error on 72.9%, 72.5%, and 71.5% of samples across these metrics, respectively. Paired Wilcoxon tests against Post-hoc confirm significant differences (p < 0.001 for all three).

These results show that DepthFoley better preserves GT spectral variations than CAFA and frequency-independent Post-hoc Scaling. We will clarify SPE's complementary role and include this analysis in the revised manuscript.

W3: Cross-object Relative Loudness and Large Distance Differences.

A3: We thank the reviewer for this concern. In object-wise inference, each object instance is conditioned on its own distance trajectory. Generated stems are summed, with global scaling applied only to prevent clipping, preserving their model-generated relative amplitudes without additional inter-source normalization.

However, our framework primarily models within-source distance-dependent variations and does not explicitly calibrate absolute loudness across sources. Thus, physically accurate relative levels between objects at substantially different distances are not guaranteed. We will clarify this limitation and discuss inter-source loudness calibration as future work.

W4: Accuracy, Noise Robustness, and OOD Generalization of Acoustic Predictors.

A4: We clarify that the SPL/SPE predictors are pretrained auxiliary modules, frozen during DepthFoley training and not used during inference. They enable acoustic supervision directly in the latent space without repeated waveform decoding.

We evaluate their prediction accuracy on VGGSound and AVE samples using identical preprocessing:

PredictorVGGSound (In-domain) MAE ↓AVE (OOD) MAE ↓
SPL0.8100.893
SPE0.4820.527

The relatively small cross-dataset differences suggest limited degradation in prediction accuracy. However, these results do not directly establish noise robustness. We acknowledge that prediction errors and latent distribution shifts may affect training supervision, although the SPL loss incorporates slack to tolerate deviations. We will clarify these limitations in the revised manuscript.

W5: Comparison with Distance-aware Baselines.

A5: We thank the reviewer for this suggestion. While existing approaches explore loudness-controllable or spatial audio generation, their conditioning and output formulations differ from our distance-guided V2A setting. To provide a direct and controlled comparison, we evaluate two distance-aware baselines: Global Post-hoc Scaling, which applies inverse-square-law-based attenuation to CAFA outputs, and Object-wise Post-hoc, which follows the same object-centric generation and mixing pipeline as DepthFoley, with distance-based attenuation independently applied to each stem.

We evaluate the final mixtures under the same protocol:

MethodDITC ↑DIGA ↑FAD ↓KL ↓
CAFA0.05710.13837.201.665
Global Post-hoc0.56840.75449.481.467
Object-wise Post-hoc0.50770.69538.421.744
DepthFoley0.55290.76857.351.472

Under matched object-wise inference, DepthFoley outperforms Object-wise Post-hoc across all four metrics. Per-object evaluation further shows higher DITC (0.5565 vs. 0.5131) and DIGA (0.7002 vs. 0.6686), supporting improved object-specific distance alignment beyond rule-based attenuation. We will include these comparisons in the revision.

W6: Analysis and Extension of Post-hoc Scaling.

A6: We acknowledge that Global Post-hoc Scaling is an effective rule-based alternative, achieving slightly higher DITC (0.5684 vs. 0.5529) and lower DeSync (1.01 vs. 1.13) than DepthFoley. However, its uniform gain envelope modulates the entire waveform, including background components, without adapting the underlying spectral structure. Such modulation may contribute to its worse FAD (9.48 vs. 7.35).

More importantly, the matched Object-wise Post-hoc comparison in A5 shows that DepthFoley achieves better DITC, DIGA, FAD, and KL under the same multi-pass procedure. Thus, our contribution lies in integrating distance modulation into generation while balancing spatial consistency and audio fidelity, rather than simply maximizing distance-intensity correlation. We will clarify the baseline implementation and this distinction in the revision.

W7: Lack of Validation against Ground-truth Distances or Known Scene Geometry.

A7: We use Depth Anything V2 (Metric) rather than its relative-depth variant to estimate object-centric distances in meters. However, estimated metric depth is not equivalent to ground-truth geometry. Our spatial metrics therefore assess alignment with estimated trajectories, rather than independently verified physical distances. We acknowledge that evaluation using known camera-source distances would provide stronger geometric validation and will clarify this limitation.

W8: Generalizability, Robustness, and Controllability Analysis.

A8: We thank the reviewer for highlighting these concerns.

(1) OOD Generalization. On 200 unseen indoor videos, DepthFoley substantially outperforms CAFA in distance-intensity consistency (DITC: 0.463 vs. 0.041; DIGA: 0.647 vs. 0.062). Leave-5-out results on unseen categories (DITC: 0.5465; DIGA: 0.7537) further support generalization beyond the training distribution. We acknowledge that additional OOD human evaluation would provide complementary perceptual evidence.

(2) Depth-noise Robustness. Appendix D evaluates scale bias, distance-dependent noise, and sparse frame errors. Mild perturbations have only a limited impact on performance, suggesting that our method is reasonably robust to minor depth estimation errors. These tests do not fully capture failures such as sustained occlusions or tracking errors, which we will acknowledge as limitations.

(3) Controllability. We clarify that DITC/DIGA are computed against edited trajectories under the earlier single-pass composite-trajectory protocol. Reduced scores likely reflect conflicts between original visual motion and counterfactual depth guidance: the frozen backbone retains visual cues, while Depth ControlNet introduces potentially contradictory spatial conditions. Larger edits may further disrupt the acoustic dynamics implied by the video.

These results indicate partial controllability rather than strict adherence to arbitrary edits. We will clarify the protocol and limitations and moderate the corresponding claims.

Q2: Citation Formatting and Readability.

A9: We thank the reviewer for pointing this out. We will standardize parenthetical citation formatting throughout the revised manuscript to improve readability.

Reviewer_g1FxOpenReview ↗

The authors' detailed response has addressed all of my concerns

I appreciate the authors' detailed response, which has addressed all of my Weaknesses points and questions. The clarifications are important, and the additional evaluations and analyses are helpful, such as spectral fidelity, the accuracy and generalizability of SPL and SPE predictors, comparisons with the two distance-aware baselines Global Post-hoc Scaling and Object-wise Post-hoc as well as deeper analysis and comparisons between DepthFoley and these two baselines, and the clarifications on OOD generalizability, depth-noise robustness, and controllability. Please make sure to incorporate the clarifications, additional evaluations and analyses, and discussions of Limitations in the author response into the revised paper. I increased my rating to Weak acceptance.

Thank you very much for your careful review and valuable feedback. We are glad our clarifications and additional evaluations address your concerns. We will incorporate all these discussions and analyses into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.