- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper’s focus is on the evaluation of systems that generate audio “soundscapes” (including, e.g., music, speech, and sound effects) from natural language descriptions. The argument is that existing evaluations do not take complexity of soundscapes into account, and mainly aggregate evaluation in an uninterpretable way. The approach uses annotation-grounded items (around 2K) with “semantic” rubrics that enable analyzing generated soundscapes in terms of event density and “structural complexity” allowing stratification of performance scores. The approach uses an audio-language model and an automated speech recognition system to carry out the evaluation. The experiments with 13 open-weight models reveal richer distinctions among model capability than simple similarity scoring approaches. A small study against human evaluations shows stronger correlation to humans than other methods, though there is no claim of fine-grained accuracy in the overall approach.
The main novelty compared to earlier benchmarks, as I understand it, is in the combination of structured soundscape annotations, fixed evaluation rubrics, and complexity-stratified analysis. The experiments, while showing promise for the method, don’t really show the accuracy or value of the semantic decomposition that inspired the design of the benchmark.
Critical strengths
The overall pipeline for constructing the benchmark is mostly clearly explained and thoughtfully constructed.
The main evaluation of systems is careful and includes two variants of the overall metric, taking into account which capabilities are “in scope” for each model.
The inclusion of a comparison of the method to human judgments is a good start.
Critical weaknesses
The event density and structural complexity constructs are not especially well motivated or explained. Why do evaluators of text-to-audio systems want to measure these quantities? The definition of complexity isn’t unreasonable, but there’s not much intuition given for why this particular measurement is interesting or useful. (Q1)
While the introduction motivates the work and describes the approach well, it says almost nothing about how the dataset is constructed. Specifically of interest is helping the reader form a judgment about the quality and staying power of the benchmark. Is it purely synthetic? Is it derived from existing sources? What kinds of specific use cases motivated the data collection? One is left unable to judge the relevance of the benchmark to any particular real-world use case. (Q2)
The items in the benchmark are 10s clips; this is extremely short and likely not especially applicable to many application settings in creative production. Is there something I'm missing? (Q3)
The benchmark is “positive requirements” only — it doesn’t check (for example) that the TTA system hasn’t added unwanted elements. Is this justified? (Q4)
Discussion of what language(s) the benchmark’s included speech samples are in is deferred to appendix, and not really justified (it skews heavily English with some Chinese). Perhaps more notable is the relative paucity of speech examples in the benchmark (only a couple hundred). Why not more? Was there some motivation for these proportions? (Q5)
The evaluation doesn’t make any inferences about how performance differences are explained by different aspects of the evaluated systems’ designs. This leaves us with no real takeaways or directions for future TTA capability advances. The complexity-stratified evaluation, while reasonable in principle, doesn’t deliver any surprises.
The human evaluation is very small scale, so there is low confidence around the human quality scores (figure 3). It really only shows higher correlation overall for the new benchmark than one predecessor; it’s not clear why others are not included in this analysis. The human experiment would be much stronger if it validated the different semantic axes separately.
Minor:
line 107 — “it does not certify individual questions or Yes/No decisions.” This puzzling remark needs more context. It reads like an AI-injected hedge; why would the reader have been thinking about “certifying” individual questions or yes/no decisions? I think the point the authors want to make is perhaps that the experiments validate the approach in aggregate but do not provide any assurance that the automated evaluation correctly answers rubric questions in individual cases. The issue of over-defensiveness repeats at lines 150 and 431.
Questions for authors
See five questions under "weaknesses."
Update following the authors' response:
I appreciate the authors' careful and substantive responses to my questions. The clarifications about the intended applications, data construction, and scope of the benchmark address several of my concerns. The additional human annotation and experiments comparing evaluators provide new evidence for the value of the approach. Of course, I remain concerned about the reliability of the fine-grained semantic judgments, especially given the substantial error rates and sensitivity of model rankings to the choice of evaluator. The human validation is also still limited in scale. Nonetheless, I think the authors have made a stronger case that this benchmark will be a useful resource for the community, and that its approach to evaluation has merit. Readers will learn from this paper. On balance, I now lean toward acceptance.
9 comments
Response to Reviewer MYN7, Q2: Data provenance, construction, and applications
We thank the reviewer for this helpful comment. We clarify the benchmark’s provenance, construction, and intended use below.
1. Data provenance. AudioScape-TTA is not purely synthetic; it is built from audio clips in existing film and TV content and everyday recordings, where speech, SFX, and BGM naturally co-occur.
2. Construction, quality, and staying power. Clips are retrieved with scene- and event-oriented queries, annotated by Gemini 2.5 Pro, and verified by trained annotators to remove ambiguous or unsupported cases. Fixed, annotation-grounded rubrics provide a consistent target for future generators, while the documented provenance and evaluation code support reproducible comparison.
3. Real-world use case. AudioScape-TTA targets automatic soundtrack generation for short video scenes, where a model must jointly generate speech, SFX, and BGM consistent with the scene description. Existing benchmarks do not systematically evaluate this combined capability; our real mixed soundscapes test all three components and their combinations.
We have added these data sources, construction and verification steps, and motivating applications to the main text.
Response to Reviewer MYN7, Q3: The role of 10-second clips
We thank the reviewer for this helpful comment. The 10-second duration standardizes the evaluation horizon across systems, matches the generation length supported by the evaluated models, and avoids confounding semantic performance with output duration.
Within this short-form setting, AudioScape-TTA evaluates event presence, attribute fidelity, and coordination among speech, SFX, and BGM for video-scene soundtrack generation. Long-form temporal planning and cross-segment consistency remain future work.
We have clarified this scope explicitly in the revised manuscript.
Response to Reviewer MYN7, Q1: Event density and structural complexity
We thank the reviewer for this helpful comment. These constructs separate two sources of difficulty that aggregate scores can obscure: content load and cross-component coordination. Event density counts the required audio events, while structural complexity groups clips by combinations of SFX, speech, and BGM, providing an annotation-grounded proxy for coordination demand. They are complementary: many events may involve one component, whereas a few events may still require cross-component coordination.
As shown in Figure 1(b), this stratification reveals distinct robustness patterns: all 13 models score lower on Hard than on Easy samples, but the decline ranges from 22.03 points for TangoFlux to 6.78 points for Dasheng AudioGen. Thus, these measures stratify model robustness under increasing content and coordination demands. We have clarified this motivation in the revised manuscript.