- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LongAudioSpan evaluates long-form speech understanding in English and Chinese across perception, understanding, and reasoning. It combines caption-derived Native QA with edit-based Anchor QA, introduces multiple evaluation formats, and evaluates 12 audio-language models.
Critical strengths
-
Relevant focus on localization and reasoning over long audio across languages and durations.
-
Complementary QA paths, paired answer formats, and prefix-based chain scoring.
-
Multi-stage quality control combining automated verification, adversarial filtering, refinement, and human review.
Critical weaknesses
- The construction of structured captions and the rationale for using them to generate questions require further explanation.
- The paper notes that prior caption-then-QA pipelines depend on caption quality, yet adopts a similar approach. Please clarify how transcripts, semantic segments, acoustic descriptions, and cross-segment dependencies are converted into the final structured captions, which model or program performs each step, and whether final assembly is rule-based merging or LLM processing. If LLMs are used, provide their names, specific versions or API snapshots, and complete prompt templates, including input fields and output formats. Also explain why questions are generated from the assembled captions rather than directly from the preceding outputs.
- Anchor placement and surrounding-context requirements lack quantitative characterization.
- Please explain how selected anchor locations are ensured to contain sufficient context for understanding and reasoning questions. Report the distribution of anchor positions and how much surrounding audio is needed to answer these questions, including the distribution of context lengths and representative examples.
- The actual evidence scope required by the questions is insufficiently analyzed.
- Recording duration is not equivalent to the context needed to answer a question. For Native QA and Anchor QA, separately report the cumulative evidence duration, the span from the earliest to the latest evidence, and the proportion of questions requiring multiple segments at each cognitive level. These analyses should distinguish locating local evidence within long audio from integrating distant information, clarifying which long-audio abilities the benchmark measures.
- The information advantage of the text baselines needs clarification, and an independently executed chunk-based cascade is missing.
- The Caption baseline achieves the highest average scores in Table 3, but receives the same captions used to construct the questions. The table or its notes should identify the models generating the transcripts and captions, any manual revisions, and their relationship to the question-authoring materials. Results using the authoring captions should be labeled as a diagnostic reference. More importantly, add a practical cascade: divide audio into approximately 30–60-second chunks, independently generate transcripts and captions, and use a text LLM to aggregate them and answer questions. A same-model pipeline that first generates text and then answers using only that text could also be evaluated. Report Native QA and Anchor QA results separately, and explain qualitatively or quantitatively which additional long-range abilities the benchmark requires beyond short-chunk processing.
- The overlap between the judge and the evaluated text model introduces a potential self-preference risk.
- GPT-5.4 both answers the text-baseline questions and judges open-ended responses, potentially favoring its own outputs in Rubric scoring. Rescore representative responses using independent judges from other model families to check whether GPT-5.4’s relative advantage and the model rankings remain stable. This concern primarily applies to Rubric and does not directly explain Accuracy or Chain, which use exact option matching.
Questions for authors
See Weakness