- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces PhyAudioBench, a three-level benchmark for text-to-audio (T2A) generation. L1 covers acoustic and prompt fidelity (10 metrics: format checks, noise, distortion, continuity, plus observer-judged presence, irrelevant-sound, count, continuity and periodicity). L2 covers temporal structure (completeness, ordering, interval). L3 covers physical consistency. The benchmark has 240 synthetic prompts covering 8 scenes, 3 difficulty levels, 7 properties and 12 intervention types. Qwen3.5-omni-plus detects events and localizes them in time. Seven hand-designed DSP evaluators then measure distance, Doppler, reverberation, decay, resonance, echo and occlusion inside the localized windows. Five evaluators are checked with a positive/negative AUC test, distance with a near/mid/far ordering test.
Critical strengths
-
The topic is a good one. Current T2A evaluation (FAD, CLAP score, AudioCaps-style captions, newer event/count/timing benchmarks) does not test whether generated audio follows the physical conditions in the prompt.
-
Testing controlled response instead of absolute judgment is the right framing. Asking whether f_a(x′) − f_a(x) moves in the expected direction avoids needing a ground-truth waveform, and it cancels many source-dependent confounds that make absolute "is this far away?" judgments meaningless. This is the strongest idea in the paper.
Critical weaknesses
-
The related work section is placed in the Appendix. I think this may violate the page limitation. I do not think this is appropriate for a submission to ICLR.
-
The measuring instruments are not validated well enough to support a benchmark, and the obvious validation route is never used. All L3 conclusions depend on seven hand-tuned DSP heuristics:
- Three of the five validated evaluators sit at AUC 0.66–0.68. Roughly one in three positive/negative pairs is ranked the wrong way, even when the positives were deliberately prompted to show the effect.
- Distance, the flagship example of the paper, scores AUC 0.422 under the binary protocol. Under the ordinal protocol the means move only 0.542 → 0.588 → 0.662, and just 6 of 10 scenes are monotonic.
- Echo has no validation at all but still feeds into L3.
-
More fundamentally, physical acoustics is the one generative domain where controlled ground truth is cheap. Distance, occlusion, reverberation, echo and Doppler can all be produced with known parameters: inverse-square attenuation plus air absorption, RIR convolution at controlled RT60 or DRR (pyroomacoustics, measured RIR sets), delayed copies with known delay and gain, resampling along a known trajectory, low-pass and transmission-loss filters. Real recordings with acoustic annotations also exist. The paper uses none of these. Its validation positives and negatives are generated clips, presumably from T2A models, possibly the same ones being benchmarked. That makes the validation circular: the evaluator is validated on whatever generators produce when asked for the property, which is the very thing the benchmark claims generators fail at.
-
The headline finding that responses go in the wrong direction is what a model that ignores the intervention would produce. Take a generator that completely ignores "far away" vs. "nearby." Then x and x′ are two near-independent draws of a phone ringing, and Δ_a = f_a(x′) − f_a(x) is pure measurement and sampling noise with a sign close to 50/50. Such a model would show about as many opposite-direction failures as correct-direction responses, almost no exact zero-change cases (a continuous measurement is essentially never exactly zero), and a non-trivial M302. The positive half of the noise is often larger than the small margin 0.2·G_a, so it earns substantial or even full credit. The reported pattern fits this null: opposite 36–51%, below margin 13–27%, zero 0.8–5%, M302 between 0.23 and 0.41. Without a null control (I mean a control condition, not a significance test), the paper cannot separate "models respond in the wrong direction" from "models do not respond at all and the evaluator is noisy." These are opposite scientific conclusions. The obvious control is to use p for both sides of the pair and score M302 on two generations of the same prompt. The seeds {0,1,2} needed for this are already generated. If TangoFlux at 0.41 is not clearly above that floor, the M302 leaderboard measures noise.
-
There is no ceiling or reference point, so L3 numbers cannot be interpreted. M301 is clip((u − μ⁻)/(μ⁺ − μ⁻)), calibrated so that the mean of the positive set maps to 1. A clip as "physical" as the validation positives would therefore score about 0.5 after clipping, not 1. M302's margin is an arbitrary 20% of the reference gap. The paper never shows what L3 a perfect physics-respecting system would get. An "oracle generator" is easy to build: take x and apply the intervention to it with DSP (attenuate, low-pass and add reverb for "far," for example). Report its M301/M302 and the scores of real recordings. Until then, an L3 of 0.33 could mean "models are terrible" or "the metric saturates near 0.4."
-
240 prompts over 12 intervention types and 7 properties leaves very few pairs per intervention type and per property. The paper does not even give per-property or per-intervention pair counts. Results broken down by property, which is what a practitioner would want, are missing.
-
The model pool is mostly 2023–2024 open checkpoints. AudioX and Omni2Sound are cited but not evaluated, commercial SFX generators are absent, and MMAudio, a V2A model, is run text-only, which is not a fair test.
-
All evaluation is monaural. Several properties (distance, motion) are mainly spatial and binaural phenomena, and the paper never discusses this.
-
From my perpective, I think that the insights are limited. Beyond "scores are low," the paper does not explain why models fail. It offers no analysis of how often distance, occlusion or reverberation descriptors appear in AudioCaps, WavCaps or Clotho captions, no correlation with training data or text encoder (T5 vs. CLAP vs. FLAN), and no per-property failure analysis. TangoFlux, trained with CLAP-ranked preference optimization, wins L3. Is that physics or a CLAP-alignment effect?
Questions for authors
-
What is M302, and what is the opposite / below-margin / zero breakdown, when both sides of the pair come from the same prompt p (e.g., seed 0 vs. seed 1)? Is any model clearly above that floor?
-
Oracle ceiling. If x′ is made by applying the intended intervention to x with DSP (attenuation + low-pass + DRR change for distance, filtering for occlusion, RIR convolution for reverberation, delayed copy for echo, resampled trajectory for Doppler), what M301 and M302 does it get? What do real recordings score?
-
Which generator(s) produced the 226 positive and 170 negative validation clips? Are any of them among the ten evaluated models (TangoFlux in particular)? Were the evaluator constants tuned on these clips?
-
Why were simulated acoustics with known ground truth (controlled RT60, DRR, distance, delay, velocity) or annotated real RIR and recording datasets not used to validate the evaluators? What AUC and monotonicity do you get on them?
-
The benchmark relies on a proprietary API observer. What exactly will be released (prompts, specs, evaluator code, generated audio, cached observer outputs), and how will the leaderboard stay reproducible when the API version changes?
-
Can you show the evaluator is hard to game? For example, a trivial post-processor that attenuates and low-passes whenever the prompt contains "far/behind/outside": what L3 does it achieve?