acceptodds
Under review as a conference paper at ICLR 2027

PhyAudioBench: Benchmarking Physical Consistency in Audio Generation through Controlled Interventions

Abstract

Recent text-to-audio generation models have achieved substantial progress in perceptual quality and semantic alignment, yet existing evaluation largely focuses on whether generated audio sounds plausible, matches the prompt, and follows the requested temporal structure. Whether generated sounds behave consistently with their described physical conditions remains much less characterized. We introduce PhyAudioBench, a hierarchical benchmark for evaluating physical consistency in general audio generation. PhyAudioBench evaluates acoustic and prompt fidelity, temporal structure, and physical consistency across 240 structured prompts spanning eight acoustic scenes, three complexity levels, seven physical properties, and twelve controlled intervention types. Rather than judging physical realism from isolated waveforms, we construct minimally modified prompt pairs that alter a target physical condition and test whether the corresponding acoustic evidence changes in the expected direction. A shared multimodal observer grounds acoustic events in time, while property-specific signal evaluators perform physical measurement. Evaluating ten representative audio generators with fifteen metrics reveals that physical-consistency scores are consistently lower than lower-level scores, all ten models change rank between Level1 and Level3, and opposite-direction responses constitute a prominent failure mode under controlled interventions. These results expose a gap between generating audio that sounds plausible and generating audio that behaves consistently in physical world.

Primary area:
datasets and benchmarks
Keywords:
text-to-audio generation, physical consistency, controlled intervention, diagnostic benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iboGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PhyAudioBench, a three-level benchmark for text-to-audio (T2A) generation. L1 covers acoustic and prompt fidelity (10 metrics: format checks, noise, distortion, continuity, plus observer-judged presence, irrelevant-sound, count, continuity and periodicity). L2 covers temporal structure (completeness, ordering, interval). L3 covers physical consistency. The benchmark has 240 synthetic prompts covering 8 scenes, 3 difficulty levels, 7 properties and 12 intervention types. Qwen3.5-omni-plus detects events and localizes them in time. Seven hand-designed DSP evaluators then measure distance, Doppler, reverberation, decay, resonance, echo and occlusion inside the localized windows. Five evaluators are checked with a positive/negative AUC test, distance with a near/mid/far ordering test.

Critical strengths
  1. The topic is a good one. Current T2A evaluation (FAD, CLAP score, AudioCaps-style captions, newer event/count/timing benchmarks) does not test whether generated audio follows the physical conditions in the prompt.

  2. Testing controlled response instead of absolute judgment is the right framing. Asking whether f_a(x′) − f_a(x) moves in the expected direction avoids needing a ground-truth waveform, and it cancels many source-dependent confounds that make absolute "is this far away?" judgments meaningless. This is the strongest idea in the paper.

Critical weaknesses
  1. The related work section is placed in the Appendix. I think this may violate the page limitation. I do not think this is appropriate for a submission to ICLR.

  2. The measuring instruments are not validated well enough to support a benchmark, and the obvious validation route is never used. All L3 conclusions depend on seven hand-tuned DSP heuristics:

  • Three of the five validated evaluators sit at AUC 0.66–0.68. Roughly one in three positive/negative pairs is ranked the wrong way, even when the positives were deliberately prompted to show the effect.
  • Distance, the flagship example of the paper, scores AUC 0.422 under the binary protocol. Under the ordinal protocol the means move only 0.542 → 0.588 → 0.662, and just 6 of 10 scenes are monotonic.
  • Echo has no validation at all but still feeds into L3.
  1. More fundamentally, physical acoustics is the one generative domain where controlled ground truth is cheap. Distance, occlusion, reverberation, echo and Doppler can all be produced with known parameters: inverse-square attenuation plus air absorption, RIR convolution at controlled RT60 or DRR (pyroomacoustics, measured RIR sets), delayed copies with known delay and gain, resampling along a known trajectory, low-pass and transmission-loss filters. Real recordings with acoustic annotations also exist. The paper uses none of these. Its validation positives and negatives are generated clips, presumably from T2A models, possibly the same ones being benchmarked. That makes the validation circular: the evaluator is validated on whatever generators produce when asked for the property, which is the very thing the benchmark claims generators fail at.

  2. The headline finding that responses go in the wrong direction is what a model that ignores the intervention would produce. Take a generator that completely ignores "far away" vs. "nearby." Then x and x′ are two near-independent draws of a phone ringing, and Δ_a = f_a(x′) − f_a(x) is pure measurement and sampling noise with a sign close to 50/50. Such a model would show about as many opposite-direction failures as correct-direction responses, almost no exact zero-change cases (a continuous measurement is essentially never exactly zero), and a non-trivial M302. The positive half of the noise is often larger than the small margin 0.2·G_a, so it earns substantial or even full credit. The reported pattern fits this null: opposite 36–51%, below margin 13–27%, zero 0.8–5%, M302 between 0.23 and 0.41. Without a null control (I mean a control condition, not a significance test), the paper cannot separate "models respond in the wrong direction" from "models do not respond at all and the evaluator is noisy." These are opposite scientific conclusions. The obvious control is to use p for both sides of the pair and score M302 on two generations of the same prompt. The seeds {0,1,2} needed for this are already generated. If TangoFlux at 0.41 is not clearly above that floor, the M302 leaderboard measures noise.

  3. There is no ceiling or reference point, so L3 numbers cannot be interpreted. M301 is clip((u − μ⁻)/(μ⁺ − μ⁻)), calibrated so that the mean of the positive set maps to 1. A clip as "physical" as the validation positives would therefore score about 0.5 after clipping, not 1. M302's margin is an arbitrary 20% of the reference gap. The paper never shows what L3 a perfect physics-respecting system would get. An "oracle generator" is easy to build: take x and apply the intervention to it with DSP (attenuate, low-pass and add reverb for "far," for example). Report its M301/M302 and the scores of real recordings. Until then, an L3 of 0.33 could mean "models are terrible" or "the metric saturates near 0.4."

  4. 240 prompts over 12 intervention types and 7 properties leaves very few pairs per intervention type and per property. The paper does not even give per-property or per-intervention pair counts. Results broken down by property, which is what a practitioner would want, are missing.

  5. The model pool is mostly 2023–2024 open checkpoints. AudioX and Omni2Sound are cited but not evaluated, commercial SFX generators are absent, and MMAudio, a V2A model, is run text-only, which is not a fair test.

  6. All evaluation is monaural. Several properties (distance, motion) are mainly spatial and binaural phenomena, and the paper never discusses this.

  7. From my perpective, I think that the insights are limited. Beyond "scores are low," the paper does not explain why models fail. It offers no analysis of how often distance, occlusion or reverberation descriptors appear in AudioCaps, WavCaps or Clotho captions, no correlation with training data or text encoder (T5 vs. CLAP vs. FLAN), and no per-property failure analysis. TangoFlux, trained with CLAP-ranked preference optimization, wins L3. Is that physics or a CLAP-alignment effect?

Questions for authors
  1. What is M302, and what is the opposite / below-margin / zero breakdown, when both sides of the pair come from the same prompt p (e.g., seed 0 vs. seed 1)? Is any model clearly above that floor?

  2. Oracle ceiling. If x′ is made by applying the intended intervention to x with DSP (attenuation + low-pass + DRR change for distance, filtering for occlusion, RIR convolution for reverberation, delayed copy for echo, resampled trajectory for Doppler), what M301 and M302 does it get? What do real recordings score?

  3. Which generator(s) produced the 226 positive and 170 negative validation clips? Are any of them among the ten evaluated models (TangoFlux in particular)? Were the evaluator constants tuned on these clips?

  4. Why were simulated acoustics with known ground truth (controlled RT60, DRR, distance, delay, velocity) or annotated real RIR and recording datasets not used to validate the evaluators? What AUC and monotonicity do you get on them?

  5. The benchmark relies on a proprietary API observer. What exactly will be released (prompts, specs, evaluator code, generated audio, cached observer outputs), and how will the leaderboard stay reproducible when the API version changes?

  6. Can you show the evaluator is hard to game? For example, a trivial post-processor that attenuates and low-passes whenever the prompt contains "far/behind/outside": what L3 does it achieve?

Reviewer_ScGxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces PhyAudioBench, a benchmark that evaluates whether text-to-audio (T2A) outputs are consistent with the physical conditions described in the prompt, using a three-level hierarchy, controlled prompt interventions, and property-specific signal evaluators.

The paper introduces four research goals:

-R1: Establishing a structured evaluation that separates what is generated, when it occurs, and how it behaves physically;

-R2: Evaluating physical consistency through controlled response rather than absolute judgment of isolated waveforms;

-R3: Measuring physical behavior with explicit acoustic measurements rather than a learned judge for final scoring;

-R4: Characterizing the gap between acoustic plausibility and physical consistency in current T2A generators.

The corresponding proposed methods are:

-M1: A three-level hierarchy (L1 acoustic/prompt fidelity, L2 temporal structure, L3 physical consistency) over 240 structured prompts covering 7 physical properties and 12 intervention types;

-M2: Minimal prompt pairs (p, p′) scored by M302 (direction-gated response with a 0.2·G_a margin), together with M301 (absolute attribute realization);

-M3: An MLLM observer (Qwen3.5-omni-plus) for event localization, combined with seven physics-aware DSP evaluators, five of which are validated by ROC–AUC;

-M4: Benchmarking ten T2A models with three seeds, with level-wise scores, L1–L3 rank correlation, and an M302 failure-mode decomposition.

Critical strengths

Regarding the contribution and validation,

-R1: Separating content, timing, and physical behavior is a sensible way to localize failures, and the factorized specification is designed to be extensible (App. H.1);

-R2: M2 is the most valuable idea of the paper. It needs no ground-truth waveform, reduces dependence on source-specific baselines, and targets a gap left by existing what/when-oriented T2A benchmarks;

-R3: Separating perception from measurement is principled, and the observer prompt and most formulas are disclosed (App. B–D);

-R4: Ten diverse generators are evaluated under fixed configurations, and the reported scores are internally consistent.

Critical weaknesses

-R1: The ordering L1 > L2 > L3 is difficult to interpret as a capability gap. Each level averages differently normalized metrics: L1 includes two metrics that are 1.000 for every model (M101, M102), the scale of L3 depends on ρ and µ±, and no floor (null generator) or ceiling (real or DSP-rendered audio) anchors are provided;

-R2: The opposite-direction finding is not separated from noise. "Opposite" covers every Δ_signed < 0, whereas "below margin" covers only (0, 0.2·G_a) (≈0.025–0.03 for occlusion/reverberation/resonance if G_a uses the Table 2 means). Under symmetric seed noise, a generator that ignores the modifier would place roughly half of its measurable pairs in "opposite", so without a placebo baseline "wrong direction" cannot be distinguished from "no response". Fig. 3b rows also sum to only 55–70%, and the sentence that appears to define the remainder is truncated (p. 9). In addition, M301 rewards stronger evidence regardless of the specified level: under Eqs. 49–51, a "nearby" phone would score higher if it sounds farther away;

-R3: M3 is not shown to measure the physical properties it is named after. (i) Doppler (App. D.3) expects frequency to rise on approach and rise–fall on a pass-by, but at constant velocity the received frequency is constant on-axis and monotonically decreasing through a pass-by; a rising pitch on approach is the intensity-driven "Doppler illusion" (Neuhoff & McBeath, 1996). Its spectral-centroid feature also shifts with source-to-background level and spectral tilt. (ii) Distance (App. D.2) is 70% absolute level, which also varies with source amplitude and output normalization, as the authors note in D.5; the late-energy cue receives only 10%. (iii) Validation (Sec. 4.4, App. E) has only 26–40 positives per AUC-validated property, described only as "generated" (negatives "without explicitly requesting" the effects), with no generator named; if it is a T2A model, the validation is partly circular. The same clips appear to also define µ±. Three AUCs are only 0.66–0.68, distance scores AUC 0.42 under the binary protocol (which the authors consider ill-posed for distance), echo is not validated, and there is no human study;

-R4: The rank-shift finding is sensitive to L1 composition and has limited statistical support. From Table 4, M103 accounts for ~60% of L1's cross-model variance, and removing M101–M103 raises Spearman(L1, L3) from 0.37 to 0.66. M301 rankings closely track event-level metrics (Spearman(M201, M301) = 0.96; (M203, M301) = 0.99), raising the possibility that it partly reflects event-detection success; how undetected targets are scored is not specified. ρ = 0.37 with n = 10 is not significant (p ≈ 0.29), and no seed variance is reported.

Questions for authors

-[Section 5.4, Figure 3b] How are unmeasurable pairs and undetected ("x"/"u") targets scored in M301 and M302, and what is the missing category in Figure 3b? Do x and x′ share seeds?

-[Section 5.4] Please report M302 on placebo pairs (p, p) generated with different seeds, with bootstrap CIs.

-[Section 5.3] Please report L3 restricted to instances and pairs where the target is detected.

-[Section 4.4, Appendix E] What generated the validation clips? Are the Table 2 means the µ± used in Eqs. 48–59, and were the evaluator constants fit on these clips?

-[Appendix D, E] Please validate each evaluator on DSP-rendered interventions (known RT60/DRR, low-pass occlusion, fractional-delay Doppler) and with a small listening test on (x, x′) pairs.

-[Section 3.3] Please list the 12 interventions, their expected directions, and the number of pairs per property.

-[Section 5.1, Table 5] What are the native vs. evaluated sample rates, and were outputs resampled or loudness-normalized?

Missing references:

M2 is closely related to behavioral testing with directional expectations in NLP, and R1/R2 parallel physical-commonsense benchmarks for video generation; neither line of work is cited. Also, Tian et al., 2026 is cited both as AudioX (Appendix A.1) and as a T2A benchmark ("T2A-Bench", Appendix A.2; also Section 1), but the reference list contains only AudioX.

[CheckList] Ribeiro, Marco Tulio, et al. "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList." Proceedings of ACL, 2020.

[VideoPhy] Bansal, Hritik, et al. "VideoPhy: Evaluating Physical Commonsense for Video Generation." 2024.

[Physics-IQ] Motamed, Saman, et al. "Do Generative Video Models Understand Physical Principles?" 2025.

Reviewer_mRFaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

PhyAudioBench tests whether audio generators respond correctly to changes in physical conditions. It changes a condition in the prompt and checks whether acoustic cues change in the expected direction. A Qwen observer identifies and localizes events, while signal-based evaluators measure physical properties. The benchmark also scores acoustic and prompt fidelity and timing. The paper reports weaker physical scores and frequent responses in the opposite direction to the requested change.

Critical strengths
  1. I like the intervention-based setup. A sound can be recognizable yet respond incorrectly to a change in distance or motion. Testing these responses could reveal failures that a general realism score misses.
  2. The diagnostic breakdown is useful. Separating event observation from property measurement makes the pipeline easier to inspect. The model coverage, individual metrics, and inference settings are also helpful.
Critical weaknesses
  1. The physical evaluators lack reliable ground truth. Validation uses generated examples prompted for a property, without independently checking that they realize it correctly. Some evaluators separate positives and negatives only weakly, and echo is unvalidated. The level-heavy distance score also needs controls for source loudness and normalization.
  2. A constant-pitch source approaching directly at constant speed can have a steady upward pitch shift. A standard pass-by moves from higher to lower frequency. These do not match the stated increasing and rise-fall templates.
  3. Qwen's errors can affect every evaluation level. Its event judgments and timestamps feed both direct scores and the physical measurements. I found no human-validated accuracy results. Since Qwen sees the reference prompt, it also needs testing on requested events that are absent.
  4. Timbre, loudness, background, and event identity can also change, even with shared seeds. Without repeated generations under unchanged prompts, an opposite-direction cue change cannot reliably be attributed to the intervention.
  5. The paper gives weights, thresholds, and a response margin, but does not show how sensitive the results are to them. I would like ranking-stability tests and evidence that the scores agree with human judgments.
  6. The benchmark omits explicit interaction and material consistency. FlatSounds (https://arxiv.org/abs/2605.30339) already tests single-factor physical responses and compares its metrics with human judgments. The paper should explain what it adds beyond changing the conditioning modality.
Questions for authors

See Weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.