- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GASA, a structured alignment benchmark for text-to-audio generation. It decomposes prompt requirements into 25 fields across seven dimensions and uses controlled minimal prompt pairs to evaluate whether generated audio realizes individual properties. The authors validate field-specific probes using real audio and Gaussian-noise controls, select Qwen3-Omni as the primary judge, and conduct additional human and cross-judge evaluations. The benchmark evaluates 11 generation models on 2,500 prompts, with 23 fields included in the main scoring after excluding two judge-limited fields.
Critical strengths
-
Important problem and actionable evaluation.
Holistic text–audio similarity can obscure failures in acoustic attributes, temporal relations, and other prompt details. GASA’s field-level evaluation provides a more informative capability profile and clearer targets for model improvement than a single global score.
-
Well-motivated controlled comparisons and judge validation.
Minimal prompt pairs help isolate responsiveness to individual properties. The real-audio versus Gaussian-noise comparison also explicitly addresses the risk that a judge answers from textual priors rather than listening. Selecting probe formats separately for each field is a thoughtful aspect of the design.
-
Broad experiments and transparent reporting of limitations.
The paper evaluates multiple generation architectures and includes checks across generation seeds, option orders, and alternative judges. Importantly, the authors identify failures on distance and location through human evaluation and exclude these fields from the main scores rather than attributing the failures to the generators.
Critical weaknesses
Judge reliability remains a concern. It is unclear whether the LLM judge can reliably recognize all evaluated attributes in generated audio. Low scores may reflect the judge failing to recognize an attribute rather than the generator failing to produce it. The documented failures on distance and location reinforce this concern, and further attribute-level validation against human judgments would strengthen the results.
Questions for authors
Further attribute-level validation against human judgments and using additional recent models such as Qwen3.8-Omni-Flash and Gemini 4, would help assess whether the findings are robust to the choice of judge.
2 comments
3.2 Additional (closed, API-only) judges
"Further attribute-level validation against human judgments and using additional recent models such as Qwen3.8-Omni-Flash and Gemini 4, would help assess whether the findings are robust to the choice of judge."
Thank you for the suggestion. On the stability of the judge, in the paper we re-scored the entire benchmark with two other open-weight judges, Qwen2.5-Omni and AudioFlamingo-Next. The results (in Appendix H of the paper) show the Spearman rank correlations of the model orderings with the main judge are .95 (Qwen2.5-Omni) and .89 (AudioFlamingo-Next), and the three judges all agree on the two strongest and the two weakest models. So, the findings seem robust to the judge choice.
Many thanks for recommending Qwen3.8-Omni-Flash and Gemini 4. Unfortunately, a suitable judge has to be able to return token probabilities because the score used in the paper is the probability of the judge’s answer to a question, that is, the first-token probability it assigns to "yes" and "no" (or to each option letter), and the noise floor, the ceiling, and the normalization are all built on this probability. We checked the official documentation as of November 10: Qwen3.8-Omni-Flash currently has no public weights and can only be called through the Alibaba Cloud API, which, according to its documentation, does not provide token probabilities for the omni series; Gemini 4 is likewise API-only, and models from Gemini 3 onward do not provide token probabilities either. So, currently, these two models cannot be scored under the paper's protocol. Should they provide token probabilities or release their weights in the future, we would be glad to include them in the judge comparison.
3.1 Judge reliability
Thank you for your comment. This concern is justified, and that is why the paper has three safeguards specifically against "the judge did not hear it": when selecting the probe for a field, only formats that the judge answers correctly by listening are kept (Section 4.2); for every field, the score obtainable without listening is subtracted (Section 5.2); and fields for which the judge cannot distinguish human-perceivable properties from noise are not included, which is how distance and location were excluded (Section 5.2 and Appendix J). For the other fields, the ceiling is the judge's mean score on the clips that raters confirmed as realizing the field (Section 5.2 and Appendix J).
Since the submission we have collected two further sets of human ratings. The first is an additional round for the eleven fields with the fewest confirmed clips, rated by three of the authors with the same method as the first round. All eleven fields remain separated from their control, and Table 3 now uses the ceilings from all rounds (Appendix J). The second is a re-rating of all 340 clips of the first round by five listeners who are not authors of the paper, with the same questions and interface (Appendix K). The judge agrees with these raters as it does with the authors: its decision matches their majority on 71% of the clips of the 23 scored fields (73% for the authors), and it rates 91% of the clips they confirmed as realized (89% for the authors).