- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the utility of synthetic vehicle pass-by audio for training downstream acoustic motion inference models (specific to vehicle speed estimators here), in response to the lack of motion-annotated real pass-by recordings. It introduces two complementary motion-conditioned synthesis frameworks, respectively PassByDiT, a physics-conditioned Diffusion Transformer that integrates 3-source kinematics and vehicle metadata into diffusion blocks, and DopplerSim, a physics-based pipeline that recovers an intrinsic vehicle source power spectral density (PSD) by inverting original Doppler and distance attenuation from a single reserved recording.
Critical strengths
-
The paper provides a clear and valuable conceptual distinction between acoustic realism, kinematic control, and downstream training utility.
-
The "transfer asymmetry" is insightful.
-
The methodology and evaluation look rigorous and solid.
-
The paper shows scientific honesty by emphasizing negative findings and limitations, as well as presenting detailed diagnostics.
Critical weaknesses
-
The experiment is entirely based on the VS13 dataset, which comprises only 13 vehicle models (all re-occur across splits) and 378 benchmark conditions. The generalization ability to unseen vehicles or conditions is uncertain.
-
Downstream utility is assessed only on scalar vehicle speed estimation using only the SE-ResNet architecture, which limits the effectiveness of the conclusions on other downstream tasks.
-
The benchmark protocol generates only one frozen random realization per condition with a fixed seed. Since diffusion models have inherent randomness, it would be better to explore the impact of sampling diversity.
-
While the diagnostic eliminates the frontend vocoder shift and flat ambient noise, why PassByDiT experiences a gap from 2.54 km/h in-domain MAE to 14.51 km/h on real audio remains essentially unknown.
Questions for authors
See weaknesses pls.
2 comments
Thank you for the detailed response and additional results. I appreciate the honest discussion on limitations. I would maintain my positive rating. Good luck!
Transfer Asymmetry Across Unseen Vehicles, Sampling Seeds and Training Conditions
Thank you for highlighting the transfer asymmetry. Shared concerns about unseen vehicles and the source of the gap are addressed in General Response G1 and G2 and summarized in points 1 and 4.
1. Generalization to unseen vehicle models
The original five-fold VS13 benchmark holds out recordings while vehicle models recur across splits. We ran a 13-fold leave-one-vehicle-model-out evaluation, excluding each held-out model from generator and estimator training and checkpoint selection (G1). Averaged over the 13 models, MAE was 8.11 km/h for real-to-real, 8.81 for real-to-PassByDiT and 15.95 for PassByDiT-to-real.
The paired directional gap was 7.13 km/h (95% vehicle-bootstrap CI [3.99, 10.71]), positive in 10 of 13 holdouts (two-sided exact sign test, p = 0.092). This supports average directional asymmetry across unseen vehicle models within VS13.
2. Scope of the downstream evaluation
Vehicle motion changes the recorded spectrum through the Doppler effect, with the shift depending on velocity relative to the microphone. Estimating speed tests whether generated clips preserve cues associated with the requested motion.
VS13 provides reference speeds and PassByDiT conditions on them. We use the same SE-ResNet architecture across training domains to compare audio sources under a consistent protocol. Our conclusions are limited to this task and architecture, which we state in the revision (G3).
3. Sampling diversity
We repeated the downstream evaluation with sampling seeds 1234, 2234 and 3234. Within each fold, we fixed the PassByDiT checkpoint, test conditions and requested vehicle motion. Each seed produced one clip per condition, keeping dataset size fixed. We retrained SE-ResNet on each resulting synthetic training set.
Columns list training audio first and test audio second. Each MAE pools 378 test predictions across five folds. All values are in km/h. The directional gap is synthetic-to-real MAE minus real-to-synthetic MAE. It remained positive for all three seeds, ranging from 7.2 to 11.5 km/h. Synthetic-to-real MAE varied from 12.5 to 18.6 km/h, so sampling affected the gap's size but not its direction in these runs.
We also varied the PassByDiT and SE-ResNet training seeds while fixing the sampling seed. Replacing real training audio with synthetic audio increased real-test MAE by 7.46 and 7.23 km/h in the two runs.
4. Why is there a gap between in-domain and real-recording MAE?
We cannot yet name a single cause. The diagnostics below narrow it down.
Frontend. In the original clip-holdout benchmark, PassByDiT-trained MAE was 2.54 km/h on synthetic audio and 14.51 on real recordings. On the same real test recordings, training on mel/Vocos-reconstructed real audio raised MAE from 7.05 to 8.05 km/h. Passing the real test recordings through mel/Vocos raised PassByDiT-trained MAE to 25.49 km/h (Appendix G). The reconstruction control increased MAE by about 1 km/h. The gap persisted when the reconstruction pipeline was shared.
Vehicle coverage (G2). We trained PassByDiT and its estimator on 2, 4, 7 and 10 vehicle models and tested on real recordings of held-out models. The real-trained curve was much flatter than the PassByDiT-trained curve (14.9 to 9.0 km/h against 47.7 to 17.7). The gap narrowed from 32.8 km/h with 2 models to 6.0 with 4, suggesting that limited vehicle coverage contributes strongly when few models are available. From 4 to 13 models the gap stayed around 5 to 9 km/h. It varied widely with which models were in training. We hypothesize that it would shrink further with more models than VS13 provides.
Direction. A real-trained estimator achieved 3.8 to 8.7 km/h MAE on PassByDiT audio across all vehicle counts and 3.8 to 4.1 once the generator had seen at least 4 models. Generated audio carries speed cues that a real-trained estimator can use. The reverse-direction loss is consistent with the PassByDiT-trained estimator relying on cues that are absent or different in real recordings.
Constant offset. The PassByDiT-trained estimator overestimates speed on real recordings. With 2 training models, mean error was +43 km/h against 47.7 km/h MAE. With 13 models (fold 0), mean error was +5.3 km/h and correlation with reference speed was 0.75, against 0.94 for real training. Removing the mean error reduced MAE from 13.0 to 12.2 km/h, about 0.8 km/h of the 7.6 km/h gap.
Per-vehicle variation. On the original real recordings, excess MAE relative to real training ranged from −0.34 to +20.46 km/h across vehicles and varied nonmonotonically across speed bands (Appendix I).
We will add these analyses, per-vehicle results, protocols and the vehicle-count figure to the revision.