acceptodds
Under review as a conference paper at ICLR 2027

AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation

Abstract

Text-to-audio (TTA) generation has made substantial progress in synthesizing realistic audio from natural-language descriptions. However, determining whether generated audio faithfully satisfies complex instructions remains challenging. Widely used global text–audio similarity metrics provide limited insight into which specific semantic requirements are satisfied or omitted. We introduce AudioScape-TTA, a structured and complexity-aware benchmark for fine-grained TTA evaluation. The benchmark represents soundscapes through verified annotations of scene context, sound effects, background music, and speech, and characterizes sample complexity using annotation-derived event density and audio-component structure. From these annotations, we derive fixed semantic rubrics covering event presence, acoustic attributes, and specified speech content. Event and attribute requirements are evaluated by an audio-language model, while speech content is assessed through a separate ASR-based coverage pathway. AudioScape-TTA contains 2,258 audio–text pairs and 25,707 semantic rubrics, enabling scalable and interpretable analysis of TTA systems. Evaluation of 13 open-source models reveals distinct profiles in event realization, attribute control, speech-content coverage, and compositional soundscape coverage. On a human-study subset, the proposed scores achieve higher model-level rank correlation with human semantic judgments than CLAP similarity, supporting structured semantic assessment beyond global text–audio alignment. Anonymous project page:https://anonymous.4open.science/w/AudioScape-T2A-426A/.

Primary area:
datasets and benchmarks
Keywords:
Text-to-Audio Generation, Fine-Grained Semantic Evaluation, Audio-Text Alignment, Multimodal Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MYN7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper’s focus is on the evaluation of systems that generate audio “soundscapes” (including, e.g., music, speech, and sound effects) from natural language descriptions. The argument is that existing evaluations do not take complexity of soundscapes into account, and mainly aggregate evaluation in an uninterpretable way. The approach uses annotation-grounded items (around 2K) with “semantic” rubrics that enable analyzing generated soundscapes in terms of event density and “structural complexity” allowing stratification of performance scores. The approach uses an audio-language model and an automated speech recognition system to carry out the evaluation. The experiments with 13 open-weight models reveal richer distinctions among model capability than simple similarity scoring approaches. A small study against human evaluations shows stronger correlation to humans than other methods, though there is no claim of fine-grained accuracy in the overall approach.

The main novelty compared to earlier benchmarks, as I understand it, is in the combination of structured soundscape annotations, fixed evaluation rubrics, and complexity-stratified analysis. The experiments, while showing promise for the method, don’t really show the accuracy or value of the semantic decomposition that inspired the design of the benchmark.

Critical strengths

The overall pipeline for constructing the benchmark is mostly clearly explained and thoughtfully constructed.

The main evaluation of systems is careful and includes two variants of the overall metric, taking into account which capabilities are “in scope” for each model.

The inclusion of a comparison of the method to human judgments is a good start.

Critical weaknesses

The event density and structural complexity constructs are not especially well motivated or explained. Why do evaluators of text-to-audio systems want to measure these quantities? The definition of complexity isn’t unreasonable, but there’s not much intuition given for why this particular measurement is interesting or useful. (Q1)

While the introduction motivates the work and describes the approach well, it says almost nothing about how the dataset is constructed. Specifically of interest is helping the reader form a judgment about the quality and staying power of the benchmark. Is it purely synthetic? Is it derived from existing sources? What kinds of specific use cases motivated the data collection? One is left unable to judge the relevance of the benchmark to any particular real-world use case. (Q2)

The items in the benchmark are 10s clips; this is extremely short and likely not especially applicable to many application settings in creative production. Is there something I'm missing? (Q3)

The benchmark is “positive requirements” only — it doesn’t check (for example) that the TTA system hasn’t added unwanted elements. Is this justified? (Q4)

Discussion of what language(s) the benchmark’s included speech samples are in is deferred to appendix, and not really justified (it skews heavily English with some Chinese). Perhaps more notable is the relative paucity of speech examples in the benchmark (only a couple hundred). Why not more? Was there some motivation for these proportions? (Q5)

The evaluation doesn’t make any inferences about how performance differences are explained by different aspects of the evaluated systems’ designs. This leaves us with no real takeaways or directions for future TTA capability advances. The complexity-stratified evaluation, while reasonable in principle, doesn’t deliver any surprises.

The human evaluation is very small scale, so there is low confidence around the human quality scores (figure 3). It really only shows higher correlation overall for the new benchmark than one predecessor; it’s not clear why others are not included in this analysis. The human experiment would be much stronger if it validated the different semantic axes separately.

Minor:

line 107 — “it does not certify individual questions or Yes/No decisions.” This puzzling remark needs more context. It reads like an AI-injected hedge; why would the reader have been thinking about “certifying” individual questions or yes/no decisions? I think the point the authors want to make is perhaps that the experiments validate the approach in aggregate but do not provide any assurance that the automated evaluation correctly answers rubric questions in individual cases. The issue of over-defensiveness repeats at lines 150 and 431.

Questions for authors

See five questions under "weaknesses."

Update following the authors' response:

I appreciate the authors' careful and substantive responses to my questions. The clarifications about the intended applications, data construction, and scope of the benchmark address several of my concerns. The additional human annotation and experiments comparing evaluators provide new evidence for the value of the approach. Of course, I remain concerned about the reliability of the fine-grained semantic judgments, especially given the substantial error rates and sensitivity of model rankings to the choice of evaluator. The human validation is also still limited in scale. Nonetheless, I think the authors have made a stronger case that this benchmark will be a useful resource for the community, and that its approach to evaluation has merit. Readers will learn from this paper. On balance, I now lean toward acceptance.

9 comments
AuthorsOpenReview ↗

Response to Reviewer MYN7, Q1: Event density and structural complexity

We thank the reviewer for this helpful comment. These constructs separate two sources of difficulty that aggregate scores can obscure: content load and cross-component coordination. Event density counts the required audio events, while structural complexity groups clips by combinations of SFX, speech, and BGM, providing an annotation-grounded proxy for coordination demand. They are complementary: many events may involve one component, whereas a few events may still require cross-component coordination.

As shown in Figure 1(b), this stratification reveals distinct robustness patterns: all 13 models score lower on Hard than on Easy samples, but the decline ranges from 22.03 points for TangoFlux to 6.78 points for Dasheng AudioGen. Thus, these measures stratify model robustness under increasing content and coordination demands. We have clarified this motivation in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer MYN7, Q2: Data provenance, construction, and applications

We thank the reviewer for this helpful comment. We clarify the benchmark’s provenance, construction, and intended use below.

1. Data provenance. AudioScape-TTA is not purely synthetic; it is built from audio clips in existing film and TV content and everyday recordings, where speech, SFX, and BGM naturally co-occur.

2. Construction, quality, and staying power. Clips are retrieved with scene- and event-oriented queries, annotated by Gemini 2.5 Pro, and verified by trained annotators to remove ambiguous or unsupported cases. Fixed, annotation-grounded rubrics provide a consistent target for future generators, while the documented provenance and evaluation code support reproducible comparison.

3. Real-world use case. AudioScape-TTA targets automatic soundtrack generation for short video scenes, where a model must jointly generate speech, SFX, and BGM consistent with the scene description. Existing benchmarks do not systematically evaluate this combined capability; our real mixed soundscapes test all three components and their combinations.

We have added these data sources, construction and verification steps, and motivating applications to the main text.

AuthorsOpenReview ↗

Response to Reviewer MYN7, Q3: The role of 10-second clips

We thank the reviewer for this helpful comment. The 10-second duration standardizes the evaluation horizon across systems, matches the generation length supported by the evaluated models, and avoids confounding semantic performance with output duration.

Within this short-form setting, AudioScape-TTA evaluates event presence, attribute fidelity, and coordination among speech, SFX, and BGM for video-scene soundtrack generation. Long-form temporal planning and cross-segment consistency remain future work.

We have clarified this scope explicitly in the revised manuscript.

Reviewer_C7gyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AudioScape-TTA, a benchmark for fine-grained evaluation of text-to-audio (TTA) generation. It contains 2,258 clips of about 10 seconds collected from video platforms. Each clip is paired with a human-verified structured annotation of scene context, sound effects (SFX), background music (BGM), and speech, including verbatim target utterances where specified. From these annotations, the authors derive a fixed set of 25,707 yes/no rubrics covering event presence, acoustic attributes, and speech content. Event and attribute rubrics are judged by Qwen3-Omni-Instruct, while speech content is scored separately with an ASR-based coverage metric ([email protected]). Samples are also stratified into Easy, Medium, and Hard levels using an annotation-derived complexity score. The authors evaluate 13 open-source TTA models and validate the metric with a human study (10 prompts × 13 models, 26 raters). In this study, model-level Overall SR correlates more strongly with human semantic ratings than MS-CLAP similarity (Spearman ρ = 0.879 vs. 0.312).

Critical strengths

The paper addresses a real gap. Fine-grained evaluation benchmarks for audio generation remain limited, and this work provides one. For each prompt, it derives a fixed set of yes/no rubrics from human-verified annotations, covering event presence, acoustic attributes, and speech content, with SFX, BGM, and speech handled within a single benchmark. Because such resources are scarce, I consider this a useful contribution to the community, and it is the main reason for my positive assessment.

Critical weaknesses

The main limitation I see is the small amount of human speech content. Only 252 of the 2,258 clips (11.16%) are annotated with speech, and only 83 clips include a specified target utterance, yielding 163 speech-content rubrics (Table 1). In my view this is not a reason for rejection, but it limits the benchmark’s coverage of speech generation.

Questions for authors

Plz refer to the Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer C7gy: Speech coverage and benchmark scope

We thank the reviewer for this helpful comment. We agree that lexical speech coverage is limited and have positioned AudioScape-TTA as a general soundscape benchmark with speech evaluation, not a balanced speech-generation benchmark.

Speech coverageCount
Speech clips252 / 2,258 (11.16%)
Speech-content rubrics163
Speech event-attribute rubrics931
Total speech-related rubrics1,094

The speech-content inventory contains 163 rubrics across 83 clips. For the supplementary WER/CER diagnostics, usable quoted-utterance references are available for 80 of these clips; the remaining three are retained in the speech-content benchmark but excluded from WER/CER, as detailed in Appendix B. The additional speech-attribute rubrics broaden coverage to speaker attributes, language, and source.

The observed speech proportion reflects the source-video distribution after verification and quality filtering rather than quota sampling. We have clarified this scope; quota-controlled multilingual, multi-speaker, and explicit-utterance sampling remain future extensions.

Reviewer_LHRJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AudioScape-TTA, a benchmark comprising 2,258 soundscapes and 25,707 fixed semantic requirements derived from structured annotations for fine-grained text-to-audio evaluation. It combines audio-language model judgments with ASR-based speech-content coverage to evaluate 13 open-source systems across audio-component compositions and complexity levels. Human evaluation shows that its semantic scores correlate more strongly with human judgments than CLAP, supporting structured assessment beyond global similarity.

Critical strengths
  1. Structured and reusable evaluation resource. The benchmark provides human-refined soundscape annotations and fixed semantic requirements covering sound effects, music, and speech, enabling more interpretable assessment than a single similarity score. Evaluation of 13 systems includes breakdowns by component, composition, and model capability scope. (§§3–5)

  2. Meaningful human validation and evaluator controls. Overall SR achieves model-level Spearman correlation of 0.879 with human judgments, compared with 0.312 for MS-CLAP, and clip-level correlation of 0.632 with prompt-clustered confidence intervals. Silence, mismatched-audio, contrastive-negative, and answer-order controls further support sensitivity to acoustic content. (Tables 8–9, 16–19)

Critical weaknesses
  1. Fine-grained reliability on generated audio remains insufficiently validated. Human validation covers only 10 prompts and provides aggregate ratings rather than rubric-level binary labels, while the balanced controls use Gemini-verified reference audio. Consequently, errors involving distorted events or source–attribute misattribution, and their effects on model rankings, remain insufficiently characterized. (Appendices C and E.3)

  2. Direct comparison with closely related evaluators is missing. The human-alignment study compares against MS-CLAP but does not evaluate AQAScore or AnyAudio-Judge on the same generated outputs. The practical benefit of the fixed annotation-derived protocol over existing fine-grained evaluation methods therefore remains unclear. (Appendix F)

Questions for authors
  1. Could you provide human binary labels for individual rubrics on generated audio, including distorted events and source–attribute misattribution, and report evaluator error rates on this subset?
  2. How stable are the generator rankings when the same generated outputs are scored by a different audio-language evaluator?
  3. On identical generated outputs and human-study prompts, how does the proposed protocol compare with AQAScore and AnyAudio-Judge in agreement with human judgments and semantic requirement coverage?
3 comments
AuthorsOpenReview ↗

Response to Reviewer LHRJ, Q1: Human rubric labels and ranking sensitivity

We thank the reviewer for this helpful comment. We have collected human binary labels for 1,820 audio–rubric items on generated audio, measured evaluator errors, and tested how correcting diagnostic errors changes generator rankings. Volunteers independently annotated the same 130 generated clips. Majority vote defines the human binary reference for all error rates and corrections below.

1. Rubric-level errors against human labels. The item-level error rates directly quantify reliability on generated audio:

Evaluation axisEvaluator errorsError rate ↓
Event Presence127/67618.79%
Event Attribute265/1,14423.16%
Overall392/1,82021.54%
Overall, Event Presence judgments are more reliable than fine-grained Event Attribute judgments. The diagnostic subsets below further isolate the failure modes highlighted by the reviewer:
Diagnostic subsetUnique itemsEvaluator error rate ↓
Distorted but recognizable21711.5% (25/217)
Severe/unrecognizable4630.4% (14/46)
Wrong source20437.3% (76/204)
Combined, deduplicated42723.2% (99/427)

“Wrong source” denotes cases where the requested attribute is associated with a source other than the intended one. Diagnostic tags can overlap; items are deduplicated by annotation_item_id. Across these subsets, the evaluator handles recognizable distortion better than severe acoustic degradation and source–attribute misattribution.

2. Ranking sensitivity under oracle correction. To isolate the effect of each failure mode, we replaced only the evaluator predictions for items in that diagnostic subset with the corresponding human labels. We then recomputed all 13 generator scores using the same 140 event/attribute items per generator.

Corrected subsetSpearman ρKendall τMax rank shiftTop-2 retainedTop-4 retained
Distorted but recognizable0.9750.9092YesNo
Severe/unrecognizable0.9720.9212.5YesYes
Wrong source0.9820.9422YesYes
Combined0.9590.9093YesYes

Spearman ρ and Kendall τ compare the original and corrected model-level SR vectors; average ranks are used for ties. The Top-2 is retained in all settings, and only distorted-only correction changes Top-4 membership. Thus, these diagnostic errors can reorder middle-ranked systems but do not overturn the leading generators on this subset.

3. Consistency with the original human study. Aggregating the majority-vote binary labels over the same 140 event/attribute rubrics per generator yields a 13-model ranking that agrees with the ranking derived from the original clip-level human semantic ratings: model-level Spearman ρ=0.802\rho=0.802 and Kendall τ=0.641\tau=0.641, with the same Top-3 (Foley-Omni, Dasheng AudioGen, and EzAudio-XL). This is a human-to-human ranking cross-check, distinct from automatic-evaluator-to-human alignment.

Accordingly, we have added the annotation protocol and overall error analysis to Appendix H.1, the diagnostic error analysis and complete per-generator oracle-correction ranking table to Appendix H.2, and only the summary human-to-human agreement statistics to Appendix H.3.

AuthorsOpenReview ↗

Response to Reviewer LHRJ, Q2: Stability under evaluator replacement

We thank the reviewer for this helpful comment. We replaced Qwen3-Omni with Gemini 2.5 Pro as the evaluator while keeping the generated outputs, rubrics, and scoring procedure unchanged.

Ranking-stability measureResult
Spearman ρ\rho0.693
Kendall τ\tau0.571
Top-4 order retainedYes
Maximum rank shift6

The leading group is stable across evaluators, but the moderate rank correlations and six-position maximum shift show evaluator sensitivity among middle/lower-ranked systems. Accordingly, we have added the complete per-model evaluator-replacement scores and rank changes to Appendix H.4.

AuthorsOpenReview ↗

Response to Reviewer LHRJ, Q3: AnyAudio-Judge comparison, requirement coverage, and AQAScore

We thank the reviewer for this helpful comment. We evaluate AnyAudio-Judge on the same 130 generated clips and 10 human-study prompts in two settings: (1) a controlled fixed-rubric comparison, where both evaluators score the same annotation-derived rubrics to isolate evaluator differences; and (2) a descriptive native comparison, where each method retains its original rubric construction and aggregation to compare the complete protocols.

1. Controlled fixed-rubric comparison. Both evaluators assess the same rubrics. Rubric-level agreement uses the new majority-vote human labels; clip- and model-level correlations use the original user-study ratings.

Level / metricOursAnyAudio-Judge (Fixed)
Rubric-level Cohen’s κ\kappa: Overall0.5190.382
Rubric-level Cohen’s κ\kappa: Event Presence0.5820.411
Rubric-level Cohen’s κ\kappa: Event Attribute0.4810.364
Rubric-level Accuracy: Overall0.7850.728
Clip-level Spearman ρ\rho: Overall0.6240.553
Clip-level Spearman ρ\rho: Event Presence0.6440.470
Clip-level Spearman ρ\rho: Event Attribute0.4580.409
Model-level Spearman ρ\rho: Overall0.8250.725
Model-level Spearman ρ\rho: Event Presence0.7520.731
Model-level Spearman ρ\rho: Event Attribute0.8180.590

Under identical requirements, our evaluator has higher human-agreement point estimates on all reported measures.

2. Descriptive complete-protocol comparison. Each method retains its original rubric construction, score type, and model-level aggregation, enabling a comparison of the complete protocols.

ProtocolScore type / model aggregationClip-level Spearman ρ\rhoModel-level Spearman ρ\rho
Ours, fixed protocolBinary SR; rubric-micro over 140 fixed items per generator0.6240.825
AnyAudio-Judge-30B, native protocolSoft pyesp_{\mathrm{yes}}; clip mean, then equal-weight macro over 10 prompts0.5860.852

Ours has a higher clip-level correlation point estimate, whereas AnyAudio-Judge-30B has a higher model-level point estimate under its native soft aggregation. Because rubric construction, score type, and aggregation all differ, this comparison describes the complete protocols rather than isolating evaluator quality. The binaryized native-rubric variant is reported as a sensitivity analysis in the appendix.

3. Semantic requirement coverage. On the same 10 prompts, our protocol applies 142 fixed, annotation-derived requirements to every generator, whereas native AnyAudio-Judge generates 82 dynamic rubrics.

Protocol and scopeRequirements / rubricsJudgments across 13 generators
Ours, fixed 10-prompt subset1421,846
└ Event Presence / Attribute1401,820
└ Speech Content226
AnyAudio-Judge-30B, native 10-prompt subset821,066

Requirement counts depend on decomposition granularity, so they describe the evaluated inventories rather than semantic recall. The practical benefit of our protocol is not the count alone, but a shared and auditable target—with explicit Event Presence, Event Attribute, and Speech Content requirements—across all generators.

4. Comparison with AQAScore. No official implementation, checkpoint, or reproducible API was available, preventing a reliable comparison on our generated outputs.

Accordingly, we have added the controlled and native-protocol AnyAudio-Judge experiments to the main evaluation discussion, with full results and the binaryized native-rubric sensitivity analysis in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.