- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DepthFoley adds a depth-conditioned branch to a frozen CAFA backbone and uses sound pressure level and spectral energy losses to model distance-dependent acoustic changes, with object-wise generation and mixing for multi-source scenes. Objective evaluations on VGGSound and AVE show improved distance–loudness consistency while largely preserving the backbone’s audio quality.
Critical strengths
-
The paper addresses a practical aspect of video-to-audio generation that is often overlooked: how sound should change as a visible source moves closer or farther away.
-
The experiments provide perceptual evidence beyond the proposed objective metrics. The method achieves a spatial alignment rating of 4.18, compared with 2.67 for CAFA and 2.82 for Post-hoc, while FAD changes only from 7.20 to 7.35 relative to CAFA. These results support improved distance control without a substantial loss in overall audio quality.
Critical weaknesses
-
Post-hoc achieves a DITC of 0.5684, slightly higher than the proposed method’s 0.5529, and its DIGA is also close. However, Post-hoc applies a single global gain envelope to the entire audio mixture, whereas DepthFoley generates separate tracks for individual sources and then mixes them. The comparison therefore differs in both distance modulation and inference procedure. The single-pass result in Table 9 still improves FAD over Post-hoc, so the gains cannot be attributed entirely to object-wise generation. Nevertheless, a CAFA + Post-hoc baseline using the same object-specific inputs and mixing procedure is needed to better separate the contributions of learned distance control and object-wise processing.
-
The multi-object ablation reports aggregate results over the full test set, with spatial metrics computed from the final audio mixture against a single reference distance trajectory. This does not establish whether each source is controlled correctly. For example, when one object approaches and another moves away, the mixture’s loudness curve may be dominated by just one source.
Questions for authors
- Section 3.5 introduces a text prompt and a visual input for each object, but does not explain how they are obtained. Section 4.1 gives the prompt template “The sound of <label>,” but how are the source-specific labels determined? Does this require an additional video-understanding model or a large language model?
- Is each visual input an object crop, a masked video, or the full video?
- Since the tracks are generated independently and then summed, how does the method avoid duplicating background sounds and maintain appropriate relative levels across sources?
4 comments
Thank you for the response and additional experiments. The Object-wise Post-hoc comparison alleviates my concern about the fairness of the baseline comparison, and the explanation of the primary–secondary strategy makes the inference procedure clearer. However, I still have several specific questions.
-
Data and evidence for the per-object evaluation.
I understand that the new per-object evaluation compares each generated audio stem with the distance trajectory of its corresponding object. Could you clarify whether this evaluation uses the full VGGSound and AVE test set from the paper or only its multi-object subset, and report the numbers of videos and objects evaluated? Since these datasets do not provide ground-truth audio stems, how were the stems and associated object-level information used in this evaluation obtained? Will this benchmark be made publicly available? -
Mapping sound events to tracking targets.
In this text-conditioned video-to-audio setting, how are the supplied sound-event labels converted into detection and tracking targets for Grounded-SAM2? For example, when a person plays a drum, does the tracker follow the person, the drum, or the contact region? How are the target object and its distance trajectory defined for diffuse background sounds such as wind, or sounds caused by wind-induced glass vibration? Are these cases within the current scope of the method? Have you also examined sensitivity to tracker choice or tracking errors, for example through an alternative tracker? -
Effectiveness of the background-suppression strategies.
What are the individual contributions of background blurring/darkening in the primary–secondary strategy and the additional prompt, “without any sound in the background”? Could you provide a small ablation showing whether these interventions reduce duplicated background and non-target sounds while preserving the intended source? This would also help establish whether both interventions are necessary. -
Differences between training and object-wise inference.
My understanding is that training uses original videos, mixed audio, and composite distance trajectories, whereas object-wise inference may use modified visual inputs, individual object trajectories, and additional background-suppression prompts. Could you confirm whether these visual and text modifications are also used during training? If not, have you examined how this potential input-distribution mismatch affects source correspondence, audio quality, and the reliability of distance control?
Overall, the rebuttal has clarified several of my initial questions. My main remaining concern is whether the current evidence sufficiently supports the paper’s contribution to reliable object-specific distance control. If these points can be clarified with appropriate evidence, I would be inclined to raise my recommendation. I would also welcome alternative analyses or validation approaches that directly address these concerns.
We thank the reviewer for their constructive feedback and are pleased that some concerns have been addressed. Below, we provide further clarifications and additional experimental evidence in response to the follow-up questions.
Q1: Data and Evidence for the Per-object Evaluation.
A1: Our previous evaluation covers the full VGGSound and AVE test set used in our paper, comprising 612 single-object videos, 347 two-object videos, and 41 videos with three or more objects, totaling 1,461 object instances. We separately report per-object results for the multi-object subset (388 videos, 849 instances), averaging scores across objects within each video and then across videos:
| Method | DITC ↑ | DIGA ↑ |
|---|---|---|
| Object-wise Post-hoc | 0.3832 | 0.4788 |
| DepthFoley (Ours) | 0.5541 | 0.5858 |
The evaluated stems are generated independently during object-wise inference, rather than extracted from ground-truth audio. Object masks and distance trajectories are obtained using Grounded-SAM2 and Depth Anything V2 (Metric), respectively. Each stem is evaluated against its corresponding trajectory, requiring no ground-truth source separation.
We view this as an object-level evaluation protocol built upon existing datasets, rather than a separate benchmark. We plan to publicly release our implementation and evaluation settings to facilitate reproducibility.
Q2: Mapping Sound Events to Tracking Targets.
A2: Our pipeline directly uses dataset-provided sound-event labels as text prompts for Grounded-SAM2 to detect and track corresponding object instances. The resulting masks are used to construct object-specific videos and extract distance trajectories by averaging Depth Anything V2 (Metric) estimates within each mask.
We primarily select motion-driven categories with clearly identifiable sound-producing objects, such as cars, trains, boats, and aircraft, reducing ambiguity in target selection. For manually collected videos, we explicitly specify the target object (e.g., the drum rather than the performer). Diffuse sounds such as wind are treated as background audio, relying on the pretrained backbone rather than independent distance control.
Regarding tracking robustness, we have not conducted a systematic sensitivity analysis using alternative trackers. Instead, we mitigate tracking errors through trajectory quality filtering, excluding unreliable results with unnatural fluctuations, intermittent detections, insufficient visibility, or negligible distance variation. While this reduces obvious tracking failures, sensitivity to tracker choice remains to be evaluated. We will clarify these criteria and limitations in the revised manuscript.
Q3: Effectiveness of Background-suppression Strategies.
A3: We conduct a six-configuration ablation on the multi-object subset (388 videos) to examine the effects of background blurring, darkening, white-background replacement, and additional background-suppression prompting. We report per-object DITC/DIGA for distance alignment and FAD/KL on the final mixtures for audio fidelity.
We evaluate per-object DITC/DIGA to assess individual distance alignment, and FAD/KL on the final mixed audio to assess overall audio fidelity.
| Configuration | Per-object DITC ↑ | Per-object DIGA ↑ | Mix FAD ↓ | Mix KL ↓ |
|---|---|---|---|---|
| Original Video | 0.5483 | 0.5715 | 13.10 | 1.62 |
| Blur Only | 0.5398 | 0.5609 | 13.48 | 1.65 |
| Dark Only | 0.5466 | 0.5718 | 12.63 | 1.61 |
| White Background | 0.5424 | 0.5735 | 12.95 | 1.60 |
| Blur + Dark (Base Prompt) | 0.5427 | 0.5708 | 10.84 | 1.54 |
| DepthFoley (Ours) | 0.5541 | 0.5858 | 9.38 | 1.47 |
The results show that individual background modifications yield modest and varying improvements, while their combination, together with prompt-based suppression, provides a more substantial improvement in overall audio quality and distance consistency.
Q4: Differences Between Training and Object-wise Inference.
A4: We confirm that training uses original videos, mixed audio, and composite distance trajectories, without visual or textual background suppression. During object-wise inference, the primary object retains the original inputs, while secondary objects use modified videos, individual distance trajectories, and background-suppression prompts.
This inference strategy enables object-specific conditioning without additional training. We acknowledge the potential input-distribution mismatch, but preserve some background context to reduce the differences between training and inference inputs.
As shown in A3, our ablation results demonstrate that the full strategy achieves better distance alignment and audio fidelity than the alternative input configurations, supporting its effectiveness despite the training-inference differences. We will clarify these differences and their limitations in the revised manuscript.
Thank you for the detailed responses and additional experiments. The results on the multi-object subset and the background-suppression ablations help clarify the method and strengthen the experimental evidence.
I still have some reservations about the training–inference mismatch and the extent to which the evaluation metrics establish source-specific control. Nevertheless, I consider the overall study reasonably complete and will raise my recommendation to Weak Accept. Please incorporate the additional experiments, implementation details, and discussion of the remaining limitations from the rebuttal into the revised manuscript.
We are grateful to the reviewer for highlighting these important concerns, which we address below.
A1: We agree that the original comparison does not fully disentangle learned distance modulation from object-wise inference. To address this, we implement an Object-wise Post-hoc baseline using the same object-centric visual inputs, individual distance trajectories, multi-pass generation, and mixing procedure as DepthFoley. The primary difference lies in the modulation mechanism: DepthFoley incorporates distance information during generation, whereas Object-wise Post-hoc applies rule-based attenuation to CAFA-generated stems.
We compare the final mixtures under the same evaluation protocol:
Under matched inference conditions, DepthFoley consistently outperforms Object-wise Post-hoc in both spatial consistency (DITC: 0.5529 vs. 0.5077; DIGA: 0.7685 vs. 0.6953) and audio fidelity (FAD: 7.35 vs. 8.42; KL: 1.472 vs. 1.744). These results demonstrate that the improvements cannot be attributed solely to multi-pass processing, supporting the effectiveness of learned distance-aware generation. We will include this controlled comparison in the revised manuscript.
A2: We agree that mixture-level DITC/DIGA cannot fully establish object-specific distance control, as the final mixture may be dominated by a single source. We therefore introduce a per-object evaluation, measuring each generated stem against its corresponding object-specific distance trajectory.
For each video, we average the scores across object stems, followed by averaging across videos. We compare DepthFoley with Object-wise Post-hoc under the same protocol:
DepthFoley outperforms Object-wise Post-hoc on both metrics, demonstrating improved alignment with individual object trajectories. Unlike mixture-level evaluation, this protocol assesses each stem independently, providing complementary evidence of object-specific distance modulation beyond rule-based attenuation. We will incorporate these results and clarify the distinction between the two evaluation protocols in the revised manuscript.
A3: Our prompts are derived from dataset-provided sound-event labels, while web-collected videos are manually annotated with a single sound category. These labels guide Grounded-SAM2 in detecting and tracking multiple instances of the same category. Thus, our multi-object setting primarily involves same-category object instances, rather than arbitrary multi-category sound sources. All instances share the template "The sound of <label>," without additional LLM-based captioning or instance-specific descriptions.
To minimize background sound duplication, we adopt a primary-secondary object strategy. The primary object (selected based on visibility duration and proximity) retains the original video and prompt, allowing background sounds to be generated primarily through this pass. Single-object scenes use the same unmodified inputs. For secondary objects, we apply Gaussian blurring and darkening to non-target regions, preserving scene context while emphasizing the target. We also append "without any sound in the background" to discourage background sound generation.
Thus, object-specific conditioning relies on individual visual inputs and distance trajectories while sharing the same sound category. We will revise the manuscript to clarify this setting, refine ambiguous terminology, and distinguish same-category instances from arbitrary multi-category sound sources.
A4: As described in A3, we use a primary-secondary object strategy to reduce background duplication. The primary object retains the original visual and text inputs, while secondary objects use blurred, darkened backgrounds and prompts discouraging background sounds. This encourages background audio to originate mainly from the primary-object pass.
Regarding relative loudness, each stem is generated under its own distance trajectory for source-specific intensity modulation. During mixing, we preserve generated amplitudes and apply only global scaling to prevent clipping, without inter-source gain normalization. This retains model-generated relative levels but does not ensure physically accurate loudness relationships. We will clarify this limitation and explore distance-aware inter-source gain calibration in future work.