- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the temporal event grounding and reasoning capabilities of audio-language models (ALMs) and proposes a benchmark consisting of two types of tasks and covering three domains (speech, sound, and music). Task 1 tests the ALM's ability to identify acoustic events and their temporal relationship; Task 2 benchmarks the ALM's reasoning ability to distinguish events that belong to different temporal intervals. Experiments show that all evaluated ALMs are very sensitive to answer type in Task 1 and mostly fail to solve Task 2, which unveils some concrete limitations of ALMs' temporal reasoning ability.
Critical strengths
- The proposed benchmark covers three domains, with real audio rather than synthetic scenes, and two different types of tasks that investigate the temporal reasoning ability of ALMs more carefully.
- The task formulations are precise and verifiable, which provides a solid foundation for making each correct answer deterministic and unambiguous based on annotations.
- This paper identifies that ALMs are very sensitive to answer types, which may be an important finding for quantifying ALMs' performance reliably.
Critical weaknesses
- There are two missing related works: DAQA (Fayek et.al., 2020) and CLEAR (Abdelnour et al., 2018); Both works require systems to identify acoustic events by their ordinal and relative position and reason by counting, comparing, and attributing. They share many significant similarities with this paper and should be considered related.
- A major finding of Task 1 is based on the asymmetry of ALM's performance given different answer types (Yes/No accuracy averages 51.2% versus 31.8% for numeric answers); However, it is important to notice that the random baseline for the binary classification task is 50% and % for the multi-label classification task with C categories, suggesting that making a direct comparison is already unreasonable. Also, DeSTA2.5 scores about 32% on binary classification, suggesting potential formatting or evaluation issues.
- There are some serious issues and unprofessional practices in Appendix A.
- A.3: C. ADDITIONAL EXAMPLES is completely empty, while Table 7 is left uncited in the text.
- A.7: G. Model Coverage is completely empty, while Table 8 is left uncited in the text.
- A.4: D. MUSICNET CATEGORY RESULTS cites a non-existent Figure 5, and I assume the authors meant Table 5 as a typo.
- Table 5 and Table 6 share exactly the same table content, while their captions are confusingly different.
- Table 8 says that "R1-AQA, Voxtral, Gemma 3n, OmniVinci, Qwen2.5-Omni-3B" are NOT covered for task 2. However, these models spookily appear in Figure 3 and disappear in Table 4. Please correct me if I misunderstood Table 8 or Figure 3.
Questions for authors
- Given that temporal event localization, let alone reasoning, is already a known limitation of ALMs, what is the extra contribution of this work compared with DAQA (Fayek et.al., 2020) and CLEAR (Abdelnour et al., 2018)?
- The formulation of multi-constraint temporal reasoning with "two temporal intervals" in Task 2 is interesting. However, according to your experiments (in Table 4), most ALMs cannot achieve 1% accuracy for even single-interval counting (also in Figure 3). Under these circumstances, how can we distinguish the "set-composition failure" from simply inaccurate counting failure?
- In Table4, why do you report only partial results for two proprietary models: Gemini 3.8 Flash (249 configs) and GPT-Audio 1.5 (60 configs), without any explanation or interpretation?
Thank you for the thorough reading, including the appendix.
W1 and Q1. Prior work and contribution. Section 2 now discusses CLEAR and DAQA, and Table 1 includes them. Both pose ordinal and relative event references with counting on synthetic scenes assembled from isolated sounds, and both were built to train task-specific models. TEROS studies this query family on real recordings in three domains, with pretrained audio-language models evaluated zero-shot. DAQA reports question-only baselines for trained models. TEROS compares the same pretrained ALM on matched questions with real and silent audio, and scores the timestamps it gives for supporting events.
W2. Chance baselines and DeSTA2.5. Results are now reported against the relevant trivial baseline. On VoxConverse (Yes/No in a new appendix table, integer in Table 4):
The two columns use different question subsets. Section 4.2 states that the raw Yes/No advantage reflects chance levels and is not evidence about reasoning. Output format contributes to the low DeSTA2.5-Audio score, as the reviewer suggested: about half of its Yes/No responses on VoxConverse are "1", "0", "FIRST" or "SECOND", and it answers "yes" in fewer than 1% of cases. A new appendix table reports the share of unusable responses per model (33.8% for DeSTA2.5-Audio on VoxConverse).
W3. Appendix. The appendix is corrected. A.3 and A.7 are complete and their tables are cited, the reference to Figure 5 is removed, and the two duplicate tables are merged. The reviewer's reading of Table 8 in the submission is correct. Figure 3 came from an earlier export, and Task 2 results are now reported in tables only.
Q2. Composition versus counting. With single-interval accuracy at most 7.95% on the 591 configurations, the two cannot be separated, and the paper presents Task 2 as an end-to-end test without a composition-specific claim. Two analyses are consistent with a counting failure. On Task 1, accuracy does not fall as temporal reference points are added (FireRedAudio on DataSED: 48.7, 48.2, 49.4 and 46.3 for 0 to 3 references). On Task 2, supplying the annotated notes as text gives single-interval accuracy at or below 10% for Qwen2-Audio and Gemma 3n E2B, so supplying the events is insufficient for accurate interval counts. This control leaves occurrence resolution, interval construction and counting jointly under test.
Q3. Partial results for Gemini 3.8 Flash and GPT-Audio 1.5. These models require paid API calls, so they were evaluated on subsets: 249 Task 2 configurations (1,245 questions) and 60 configurations (300 questions). These are the configurations of the first 85 and the first 20 clips of the Task 2 clip list. The paper states the sizes, the selection rule and the reason, marks both models as partial in the Task 2 tables, and does not compare them directly with full-coverage models.