acceptodds
Under review as a conference paper at ICLR 2027

After the Second Bell: Can Audio-Language Models Reason with Temporally Grounded Events?

Abstract

Audio-language models (ALMs) increasingly answer questions about speech, environmental sounds, and music, but temporal operations such as ordering, counting, and localization are often evaluated as separate skills. We study a more specific dependency: whether a model can identify a particular event occurrence, use its temporal position as a reference, and execute a downstream computation conditioned on that reference. We introduce (Temporal Event-Reference and Relational Reasoning), a diagnostic benchmark spanning speech, environmental sound, and music. The benchmark contains 67,466 questions in Task\ 1 across four source datasets and 2,955 matched questions in Task\ 2 over short MusicNet clips. Task\ 2 fixes two temporal intervals and asks models to count within each interval, determine interval overlap, and compute their intersection and union without double counting. Ground truth is generated from structured annotations and retained using cross-model agreement, followed by expert verification. Across the evaluation suite, models show a consistent gap between coarse temporal judgments and exact temporal counting: binary questions are often answered more reliably than numerical counting, while Task\ 2 intersection and union counting remain difficult even when interval-overlap recognition is substantially higher. We further find that Task\ 1 category scores are strongly affected by answer type and therefore should not be interpreted as a difficulty hierarchy. exposes a narrower failure mode: temporally grounded information can be recognized locally without being reliably carried into a subsequent relational computation.

Primary area:
datasets and benchmarks
Keywords:
Compositional Reasoning, Audio-Language Models, Audio Reasoning, Temporal Reasoning, Multimodal Learning, Long-Form Audio, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_JF17OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the temporal event grounding and reasoning capabilities of audio-language models (ALMs) and proposes a benchmark consisting of two types of tasks and covering three domains (speech, sound, and music). Task 1 tests the ALM's ability to identify acoustic events and their temporal relationship; Task 2 benchmarks the ALM's reasoning ability to distinguish events that belong to different temporal intervals. Experiments show that all evaluated ALMs are very sensitive to answer type in Task 1 and mostly fail to solve Task 2, which unveils some concrete limitations of ALMs' temporal reasoning ability.

Critical strengths
  • The proposed benchmark covers three domains, with real audio rather than synthetic scenes, and two different types of tasks that investigate the temporal reasoning ability of ALMs more carefully.
  • The task formulations are precise and verifiable, which provides a solid foundation for making each correct answer deterministic and unambiguous based on annotations.
  • This paper identifies that ALMs are very sensitive to answer types, which may be an important finding for quantifying ALMs' performance reliably.
Critical weaknesses
  • There are two missing related works: DAQA (Fayek et.al., 2020) and CLEAR (Abdelnour et al., 2018); Both works require systems to identify acoustic events by their ordinal and relative position and reason by counting, comparing, and attributing. They share many significant similarities with this paper and should be considered related.
  • A major finding of Task 1 is based on the asymmetry of ALM's performance given different answer types (Yes/No accuracy averages 51.2% versus 31.8% for numeric answers); However, it is important to notice that the random baseline for the binary classification task is 50% and (1/C)×100(1/C)\times 100 % for the multi-label classification task with C categories, suggesting that making a direct comparison is already unreasonable. Also, DeSTA2.5 scores about 32% on binary classification, suggesting potential formatting or evaluation issues.
  • There are some serious issues and unprofessional practices in Appendix A.
    • A.3: C. ADDITIONAL EXAMPLES is completely empty, while Table 7 is left uncited in the text.
    • A.7: G. Model Coverage is completely empty, while Table 8 is left uncited in the text.
    • A.4: D. MUSICNET CATEGORY RESULTS cites a non-existent Figure 5, and I assume the authors meant Table 5 as a typo.
    • Table 5 and Table 6 share exactly the same table content, while their captions are confusingly different.
    • Table 8 says that "R1-AQA, Voxtral, Gemma 3n, OmniVinci, Qwen2.5-Omni-3B" are NOT covered for task 2. However, these models spookily appear in Figure 3 and disappear in Table 4. Please correct me if I misunderstood Table 8 or Figure 3.
Questions for authors
  • Given that temporal event localization, let alone reasoning, is already a known limitation of ALMs, what is the extra contribution of this work compared with DAQA (Fayek et.al., 2020) and CLEAR (Abdelnour et al., 2018)?
  • The formulation of multi-constraint temporal reasoning with "two temporal intervals" in Task 2 is interesting. However, according to your experiments (in Table 4), most ALMs cannot achieve 1% accuracy for even single-interval counting (also in Figure 3). Under these circumstances, how can we distinguish the "set-composition failure" from simply inaccurate counting failure?
  • In Table4, why do you report only partial results for two proprietary models: Gemini 3.8 Flash (249 configs) and GPT-Audio 1.5 (60 configs), without any explanation or interpretation?
1 comment

Thank you for the thorough reading, including the appendix.

W1 and Q1. Prior work and contribution. Section 2 now discusses CLEAR and DAQA, and Table 1 includes them. Both pose ordinal and relative event references with counting on synthetic scenes assembled from isolated sounds, and both were built to train task-specific models. TEROS studies this query family on real recordings in three domains, with pretrained audio-language models evaluated zero-shot. DAQA reports question-only baselines for trained models. TEROS compares the same pretrained ALM on matched questions with real and silent audio, and scores the timestamps it gives for supporting events.

  • With the audio replaced by silence, raw accuracy changes by less than 5 points in 15 of 20 model-dataset pairs.
  • On the chance-controlled set, models score between 20.8% and 35.2% with silence, and in 12 of 18 pairs the score with audio is within 5 points of that. Audio helps by more than 8 points in five pairs: MiMo-Audio on three datasets, FireRedAudio on MUStARD (47.4% versus 26.1%) and OmniVinci on DataSED (37.4% versus 25.4%).
  • Gemma 3n E4B gave the same answer on 89.3% of MusicNet questions with a prediction in both conditions, across all answer types.
  • Asked to list the events behind an answer, models identify the correct sound classes but rarely localize them in time: Qwen2-Audio has a label F1 of 63.2% and 3.5% with timestamps.

W2. Chance baselines and DeSTA2.5. Results are now reported against the relevant trivial baseline. On VoxConverse (Yes/No in a new appendix table, integer in Table 4):

ModelYes/No accuracy (majority 57.5)Integer 0 to 3, equal-weight (chance 25)
Qwen3-Omni-30B66.742.9
Voxtral-Small-24B63.842.0
FireRedAudio61.042.5
Kimi-Audio58.534.8
Gemma 3n E2B50.136.5
DeSTA2.5-Audio25.228.0

The two columns use different question subsets. Section 4.2 states that the raw Yes/No advantage reflects chance levels and is not evidence about reasoning. Output format contributes to the low DeSTA2.5-Audio score, as the reviewer suggested: about half of its Yes/No responses on VoxConverse are "1", "0", "FIRST" or "SECOND", and it answers "yes" in fewer than 1% of cases. A new appendix table reports the share of unusable responses per model (33.8% for DeSTA2.5-Audio on VoxConverse).

W3. Appendix. The appendix is corrected. A.3 and A.7 are complete and their tables are cited, the reference to Figure 5 is removed, and the two duplicate tables are merged. The reviewer's reading of Table 8 in the submission is correct. Figure 3 came from an earlier export, and Task 2 results are now reported in tables only.

Q2. Composition versus counting. With single-interval accuracy at most 7.95% on the 591 configurations, the two cannot be separated, and the paper presents Task 2 as an end-to-end test without a composition-specific claim. Two analyses are consistent with a counting failure. On Task 1, accuracy does not fall as temporal reference points are added (FireRedAudio on DataSED: 48.7, 48.2, 49.4 and 46.3 for 0 to 3 references). On Task 2, supplying the annotated notes as text gives single-interval accuracy at or below 10% for Qwen2-Audio and Gemma 3n E2B, so supplying the events is insufficient for accurate interval counts. This control leaves occurrence resolution, interval construction and counting jointly under test.

Q3. Partial results for Gemini 3.8 Flash and GPT-Audio 1.5. These models require paid API calls, so they were evaluated on subsets: 249 Task 2 configurations (1,245 questions) and 60 configurations (300 questions). These are the configurations of the first 85 and the first 20 clips of the Task 2 clip list. The paper states the sizes, the selection rule and the reason, marks both models as partial in the Task 2 tables, and does not compare them directly with full-coverage models.

Reviewer_WHJSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TEROS, a benchmark for evaluating whether audio-language models can identify a particular occurrence of an event and use its temporal position for subsequent counting or relational reasoning. Task 1 covers speech, environmental sounds, and music. Task 2 compares basic counting, overlap, and intersection/union operations over the same musical events and time intervals. The experiments reveal difficulties with exact counting and set operations and examine how answer type and audio duration affect performance.

Critical strengths
  1. Using a particular event occurrence as the temporal reference for a subsequent computation gives the benchmark a clear and practically meaningful objective. The structured temporal programs also make the operations required by each question easier to inspect.

  2. Related questions in Task 2 use the same audio and time intervals, allowing basic counting and compositional operations to be compared under matched conditions. This design offers a useful basis for diagnosing where errors arise.

  3. The paper explicitly defines event occurrences in each audio domain and reports results by answer type. These choices make the comparisons easier to interpret and reduce the risk of mistaking differences in question composition for differences in domain or reasoning difficulty.

Critical weaknesses
  1. Nearly all overlap labels in Task 2 are positive, and the models also frequently make errors in basic counting within the two intervals. These factors make it difficult to conclude that models successfully recognize temporal relations but struggle specifically with the subsequent compositional computation. Balancing the overlap labels and adding controls that supply correct event lists or temporal boundaries would help isolate difficulties at the computation stage.

  2. Responses in which a model states that the audio is unavailable are excluded from evaluation. It is not clear, however, whether all such responses correspond to verified input-delivery failures. Similar responses could reflect a refusal or an incorrect judgment about input availability, in which case each model's evaluation denominator would depend partly on its own behavior. It would be helpful to distinguish these cases using the call logs and to report results over the full evaluation sample.

  3. I would appreciate clarification of several apparent inconsistencies in the Task 2 results. Figure 3 includes Gemma, Voxtral, and OmniVinci, whereas Table 8 marks these models as not evaluated on Task 2, and Table 4 reports a different set of models. Clarifying which evaluation versions underlie these results, reconciling the figures and tables, and providing the corresponding prediction records would help establish that the central comparisons come from the same set of experiments.

  4. Question validation relies primarily on agreement among models, and I would welcome more detail on the number of items reviewed by humans, how those items were stratified, and the errors found. For fine-grained note counting in MusicNet, an answer determined by the annotations may not always be reliably recoverable from the audio mixture. Human validation based on the actual audio, together with an explanation of how ambiguities in event boundaries and counting are handled, would strengthen confidence in the benchmark.

Questions for authors

I would appreciate the authors' responses to the questions raised in the weaknesses above.

1 comment

Thank you for the constructive review.

W1. Overlap labels and controls. The revision adds both controls. On a balanced set of 40 overlapping and 40 disjoint configurations, three models score 78.8 to 85.0% balanced accuracy, three score exactly 50.0%, and Phi-4 scores 52.5%, while interval counts remain at the majority-answer baselines. We also ran the control the reviewer proposed: the complete annotated note list of each clip was supplied as text together with the audio. Single-interval accuracy is at or below 10% for Qwen2-Audio and Gemma 3n E2B, with or without the audio, and below the majority baselines for MiMo-Audio. Supplying the annotated notes was therefore insufficient for accurate interval counts. The control leaves occurrence resolution, interval construction and counting jointly under test. A related analysis on Task 1 shows that perception also fails: when asked to list the events behind an answer, models identify the correct sound classes but rarely localize them in time (label F1 of 63.2% versus 3.5% with timestamps for Qwen2-Audio on DataSED). The paper presents Task 2 as an end-to-end test and reports raw overlap accuracy on the 591 configurations only in the appendix, with the always-Yes baseline of 99.83%.

W2. Excluded responses. The silent-audio control addresses this directly. MiMo-Audio returns an empty response on 1,304 VoxConverse questions with the real audio and on the same 1,304 questions with silence, which indicates model behaviour. Delivery could not be verified for every historical call, so Tables 3, 5 and 10 count empty responses and responses stating that the audio is unavailable as incorrect on the stated evaluation set. For MiMo-Audio on VoxConverse the score is 48.1% over all 9,520 questions, against 55.9% over answered questions. Task 2 was recomputed for all models with one parser and this rule. There are 29 such responses: 20 from MiMo-Audio, 2 from GPT-Audio 1.5, 2 from DeSTA2.5-Audio, 4 from Qwen3-Omni-Thinking and 1 from Gemini 3.8 Flash. These 29 are separate from responses that contain text but no final answer, which are also incorrect: for Qwen3-Omni-Thinking, 1,542 of 2,955 responses are reasoning traces that end before a final answer, including responses that mention unavailable audio and then continue reasoning. Some cells differ from the submission as a result, for example MiMo-Audio's overall accuracy is 20.95% (22.10% before) and Qwen3-Omni-Thinking's is 2.13% (3.71% before). The table now covers 19 models, and the conclusions are unchanged.

W3. Figure and table consistency. Task 2 results are now reported in tables only, all computed with one parser and one scoring rule, and Figure 3 is removed. Table 3 is recomputed over the released question files (9,520, 15,319, 20,376 and 22,441 questions). The appendix lists model coverage for Tables 3 and 5, and raw responses, extracted answers and per-question scores are released so each number can be reproduced.

W4. Validation and recoverability from audio. The revision adds a human audit. For 30 questions from VoxConverse, DataSED and MUStARD, annotators answering from the audio matched the ground truth on 22. Annotators answering from the annotation files matched on 15 of 20, and all five disagreements were annotator errors. An independent re-derivation of VoxConverse answers from the RTTM files matches 7,808 of 7,935 parsed questions (98.4%). For MusicNet, two trained musicians each answered 15 questions spanning eight reasoning categories from the audio alone, without access to the ground truth, and both matched it on all 15. These are Task 1 questions with small counts. For large note counts in dense mixtures, an annotation-determined answer may not be recoverable by a listener, and the paper states this as a limitation together with MusicNet's estimated 4% label error rate. The chance-controlled Task 1 analysis is restricted to answers 0 to 3.

Reviewer_b7T5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TEROS, a diagnostic benchmark to study temporal event‑reference reasoning for audio‑language models: identifying an event occurrence, using its temporal boundary as reference to execute downstream counting or relational computation. TEROS consists of Task 1 (67 466 questions across speech, environmental sound, music) and Task 2 (controlled multi‑constraint reasoning over MusicNet clips). Evaluations across many ALMs show a large performance gap: binary yes/no temporal judgements substantially outperform exact numeric counting.

Critical strengths
  1. The target capability (ground event and propagate its temporal boundary for subsequent computation) is a real under‑studied failure point for modern audio‑language models.
  2. Task‑2 matched‑question design is conceptually appealing, enabling within‑instance comparison of different reasoning operations given identical audio input.
  3. Large‑scale programmatically generated question corpus with multi‑model validation plus stratified human spot‑check.
Critical weaknesses

1.590 out of total 591 configurations have positive overlap. A trivial always‑yes baseline reaches 99.83% accuracy. Although authors acknowledge this caveat, they still report raw overlap accuracy as part of main results, without constructing a balanced evaluation subset. We cannot draw reliable conclusions about models’ true overlap‑detection ability from this skewed setup. 2.Evaluation heavily relies on strict exact‑match. Minor semantic‑equivalent rephrasings or number‑word variants risk being marked wrong. Only ad‑hoc partial relaxation is applied. There is no controlled study quantifying how much reported accuracy fluctuates under looser semantic‑aware matching. 3.Speech domain merges speaker turns with silence filtering; music and environmental sound directly adopt raw annotated event instances. Different occurrence granularities are baked into benchmark construction. Observed cross‑domain performance gaps may stem from annotation conventions rather than genuine differences of model temporal reasoning capacity.

Questions for authors
  1. Will you produce a balanced overlap subset for Task 2 and report overlap performance on this balanced hold‑out partition? Without this, the overlap sub‑task yields little scientific value.
  2. How much does overall accuracy change if you adopt semantic‑aware matching instead of strict exact‑match? Please supply metric‑sensitivity statistics.
  3. Could you run ablation with unified occurrence‑merging rules across all three domains, to separate annotation‑artifact effects from genuine cross‑domain reasoning differences?
1 comment

Thank you for the review.

W1 and Q1. Balanced overlap subset. The revision adds this subset as a paired diagnostic. The balanced set has 40 overlapping and 40 disjoint configurations with five questions each. It is built on 40 clips from the Task 2 pool, and each clip contributes one configuration of each kind.

ModelYes on overlappingNo on disjointBalanced accuracy
Qwen2-Audio70.010085.0
Gemma 3n E2B60.010080.0
MiMo-Audio10057.578.8
Phi-4-Multimodal30.075.052.5
FireRedAudio0.010050.0
Qwen2.5-Omni-3B0.010050.0
Audio-Flamingo-30.010050.0

Three models separate the two cases (78.8% to 85.0%). Three score exactly 50% by answering No throughout, and Phi-4 scores 52.5%. Interval counts on this set remain at the majority-answer baselines: on the disjoint set the baselines are 32.5% and 42.5%, and the best model, Phi-4, scores 35.0% and 42.5%. On the disjoint set the intersection question is replaced by a count in the gap between the intervals. Raw overlap accuracy on the original 591 configurations, where always answering Yes scores 99.83%, now appears only in the appendix. With 40 configurations per class the intervals are wide, and the paper states this as a limitation.

W2 and Q2. Sensitivity to answer matching. Section 3.6 now gives the full scoring procedure, and a new appendix table compares strict exact match with rule-based normalization for 15 models on Yes/No and integer questions. Ten models change by under 2.5 points on every dataset. Models that answer in sentences change more: Qwen2-Audio gains 8 to 13 points and Kimi-Audio 9 to 18. The largest effect is for Qwen3-Omni-30B on VoxConverse, which rises from 11.2% to 61.7%. About half of DeSTA2.5-Audio's Yes/No responses on VoxConverse are "1", "0", "FIRST" or "SECOND" instead of Yes or No. A second new table reports the share of responses with no usable answer per model: 33.8% for DeSTA2.5-Audio on VoxConverse and under 3% for the other six models there. Sentence-form answers are extracted by an LLM under fixed rules, and a manual check of a 10% sample per dataset found no extraction errors. Raw and extracted responses are released.

W3 and Q3. Occurrence granularity. The unit of reasoning is the annotated occurrence: a speaker turn, a sound instance or a note. A single merging rule does not transfer across them. Merging adjacent notes would remove the occurrences that musical questions ask about, and leaving speaker segments unmerged would turn pauses into new turns. For this reason the paper makes no cross-domain difficulty claim, and every analysis is computed within a dataset, so none depends on a shared convention. Where a unified rule is well defined we tested it: on DataSED, merging same-class events that overlap or touch changes none of the 3,602 count answers checked.

Reviewer_QpN8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TEROS, a benchmark for evaluating whether audio-language models can identify a particular event occurrence and use it as a reference for subsequent temporal computation. It covers speech, environmental sound, and music, and pairs interval counts, overlap judgments, and intersection/union counts within the same musical clips. The evaluation reports the testing results and highlights how answer format affects comparisons across question categories.

Critical strengths
  1. This work addresses a useful evaluation problem: whether models can use a specific event occurrence to constrain a subsequent answer. Applying a common representation across speaker turns, environmental events, and musical notes gives the benchmark broader coverage than a single-domain test.

  2. The matched questions in Task 2 are a promising diagnostic design. Asking about constituent intervals and their intersection and union within the same configuration creates opportunities to examine dependencies between basic counting and more complex operations.

Critical weaknesses
  1. The contribution needs clearer positioning relative to earlier compositional audio reasoning benchmarks. Sections 1–2 and Table 1 distinguish TEROS through the use of a grounded occurrence as a reference for downstream computation. However, CLEAR [1] already includes questions combining ordinal event references with temporal relations and subsequent attribute queries, as well as reference-conditioned counting. DAQA [2] also includes ordinal and relative event references, conditional counting, and comparisons between event sets. Neither is discussed. These precedents do not establish that TEROS's matched interval tasks are redundant, but they make the conceptual distinction less clear than presented.

  2. The interpretation of Task 2 needs to distinguish overall difficulty from a specific composition failure. Section 4.4 acknowledges that the overlap labels are highly imbalanced, and Section 3.6 explains why too few constituent counts are correct for stable conditional estimates. Nevertheless, Section 4.6 describes failures “even when models recognize two intervals.” Neither high overlap accuracy under this imbalance nor the low constituent-count scores establishes successful recognition of those intervals. The authors appropriately avoid claiming a single internal mechanism, but this particular interpretation still goes beyond the evidence. An annotation-assisted control could help distinguish errors in recovering events from errors in computing over them; alternatively, narrowing the interpretation to end-to-end task difficulty would address the concern.

  3. The scope and results of the human validation are underreported. Section 3.5 and Appendix A.8 describe expert review of a stratified subset and validator disagreements, but do not give the sample size, outcomes, or what information experts used. Section 3.2 explicitly defines occurrences from annotations rather than perceptual segments.

I am open to revising this assessment during rebuttal if the authors clarify the benchmark's added value and address or appropriately qualify these conclusions.

[1] Abdelnour, J., Salvi, G., & Rouat, J. (2018). CLEAR: A dataset for compositional language and elementary acoustic reasoning. NeurIPS 2018 Workshop on Visually Grounded Interaction and Language (ViGIL). arXiv:1811.10561.

[2] Fayek, H. M., & Johnson, J. (2019). Temporal reasoning via audio question answering [Preprint]. arXiv. doi:10.48550/arXiv.1911.09655.

Questions for authors
  1. Relationship to prior benchmarks. Which TEROS question structures or diagnostic capabilities are not covered by CLEAR and DAQA?

  2. Human validation protocol. How many examples were checked by experts, what domains and tasks did they cover, and what errors or corrections resulted?

  3. Dataset construction and coverage. Could the authors clarify how source recordings map to the “audio units” in Table 2, including the number of distinct source recordings, segment selection, and whether segments overlap? Please also describe how event-reference configurations are sampled and summarize how many questions represent different computations versus alternative phrasings of the same computation.

4 comments

Thank you for identifying CLEAR and DAQA as the closest precedents.

W1 and Q1. CLEAR and DAQA. Section 2 now discusses both, and Table 1 includes them. They establish the query structure of an ordinal or relative event reference followed by counting or comparison. TEROS builds on that structure and differs in five respects, now listed in Section 2 and in the contribution list.

  • Natural audio. CLEAR scenes concatenate 10 recorded notes from a bank of 56, and DAQA clips concatenate 5 to 12 events with at most 500 ms of overlap. TEROS uses real recordings in which overlap and simultaneity occur naturally.
  • Evaluation target. CLEAR and DAQA train task-specific models on their own splits. TEROS evaluates pretrained audio-language models zero-shot.
  • One program abstraction across speaker turns, sound events and musical notes.
  • Matched multi-constraint questions. Task 2 asks five operations over the same pair of intervals in the same audio: two interval counts, overlap, intersection and union.
  • Grounding diagnostics. DAQA reports question-only baselines. TEROS compares the same pretrained ALM with real and silent audio and scores the timestamps it gives for supporting events.

W2. Task 2 interpretation. The revision follows the reviewer's second option. Single-interval accuracy on the 591 configurations is at most 7.95%, so the counts cannot separate interval recognition from composition, and the paper now presents Task 2 as an end-to-end test without a composition-specific claim. The revision adds two analyses. On a balanced set of 40 overlapping and 40 disjoint configurations, three models score 78.8 to 85.0% balanced accuracy, three score exactly 50.0%, and Phi-4 scores 52.5%. We also ran the annotation-assisted control the reviewer suggested: models received the audio and the complete annotated note list of the clip as text, without the interval boundaries. Single-interval accuracy is at or below 10% for Qwen2-Audio and Gemma 3n E2B, with or without the audio, and below the majority baselines for MiMo-Audio. Supplying the annotated notes was therefore insufficient for accurate interval counts, so the errors cannot be attributed to perception alone. The control leaves occurrence resolution, interval construction and counting jointly under test.

W3 and Q2. Human validation. The revision adds a human audit. For 30 questions from VoxConverse, DataSED and MUStARD, annotators answering from the audio matched the ground truth on 22. Annotators answering from the annotation files matched on 15 of 20, and all five disagreements were annotator errors, confirmed by re-derivation or a manual recount. For MusicNet, two trained musicians each answered 15 questions from the audio alone, without access to the ground truth, and both matched it on all 15. An independent re-derivation of VoxConverse answers from the RTTM files matches 7,808 of 7,935 parsed questions (98.4%). The audit is small and does not cover Task 2; both limits are stated in the paper. Occurrences are defined from annotations, not perceptual segments.

Q3. Construction and coverage. This table is in the appendix.

DatasetRecordingsAudio unitsQuestionsUnique question stringsUnique programs
VoxConverse4464469,5204,3483,359
MUStARD69069015,3196,0905,101
DataSED71771720,37612,3246,805
MusicNet32086722,44114,40114,123

For MusicNet an audio unit is a (recording, duration bin) pair: 230 recordings appear in all three bins, 87 in two and 3 in one. Segments in different bins are distinct and non-overlapping but share a recording. A bootstrap over the 320 recordings gives 95% intervals of at most ±0.9 points for the MusicNet accuracies in Table 3. Task 2 uses the first 200 clips, in sorted filename order, of the 319 in the 0 to 30 s bin. Three configurations per clip were generated automatically from its annotation file, each with four anchor events. An anchor must exist in the clip and be uniquely identifiable, as a particular note or instrument or as its first, second, third or last occurrence. Intervals are defined by events, never by timestamps. Nine of the 600 attempts gave no valid configuration, leaving 591, each asked in five forms. Section 3.4 states this rule.

Ethics. The release contains questions, answers, temporal programs, source identifiers and extraction scripts, and no audio. VoxConverse is released under CC BY 4.0, MusicNet recordings are Creative Commons licensed or public domain, the MUStARD repository is MIT licensed, and DataSED is released under CC BY-NC-SA 4.0. The paper includes a statement with these terms.

Thank you for identifying CLEAR and DAQA as the closest precedents.

W1 and Q1. CLEAR and DAQA. Section 2 now discusses both, and Table 1 includes them. They establish the query structure of an ordinal or relative event reference followed by counting or comparison. TEROS builds on that structure and differs in five respects, now listed in Section 2 and in the contribution list.

  • Natural audio. CLEAR scenes concatenate 10 recorded notes from a bank of 56, and DAQA clips concatenate 5 to 12 events with at most 500 ms of overlap. TEROS uses real recordings in which overlap and simultaneity occur naturally.
  • Evaluation target. CLEAR and DAQA train task-specific models on their own splits. TEROS evaluates pretrained audio-language models zero-shot.
  • One program abstraction across speaker turns, sound events and musical notes.
  • Matched multi-constraint questions. Task 2 asks five operations over the same pair of intervals in the same audio: two interval counts, overlap, intersection and union.
  • Grounding diagnostics. DAQA reports question-only baselines. TEROS compares the same pretrained ALM with real and silent audio and scores the timestamps it gives for supporting events.

W2. Task 2 interpretation. The revision follows the reviewer's second option. Single-interval accuracy on the 591 configurations is at most 7.95%, so the counts cannot separate interval recognition from composition, and the paper now presents Task 2 as an end-to-end test without a composition-specific claim. The revision adds two analyses. On a balanced set of 40 overlapping and 40 disjoint configurations, three models score 78.8 to 85.0% balanced accuracy, three score exactly 50.0%, and Phi-4 scores 52.5%. We also ran the annotation-assisted control the reviewer suggested: models received the audio and the complete annotated note list of the clip as text, without the interval boundaries. Single-interval accuracy is at or below 10% for Qwen2-Audio and Gemma 3n E2B, with or without the audio, and below the majority baselines for MiMo-Audio. Supplying the annotated notes was therefore insufficient for accurate interval counts, so the errors cannot be attributed to perception alone. The control leaves occurrence resolution, interval construction and counting jointly under test.

W3 and Q2. Human validation. The revision adds a human audit. For 30 questions from VoxConverse, DataSED and MUStARD, annotators answering from the audio matched the ground truth on 22. Annotators answering from the annotation files matched on 15 of 20, and all five disagreements were annotator errors, confirmed by re-derivation or a manual recount. For MusicNet, two trained musicians each answered 15 questions from the audio alone, without access to the ground truth, and both matched it on all 15. An independent re-derivation of VoxConverse answers from the RTTM files matches 7,808 of 7,935 parsed questions (98.4%). The audit is small and does not cover Task 2; both limits are stated in the paper. Occurrences are defined from annotations, not perceptual segments.

Q3. Construction and coverage. This table is in the appendix.

DatasetRecordingsAudio unitsQuestionsUnique question stringsUnique programs
VoxConverse4464469,5204,3483,359
MUStARD69069015,3196,0905,101
DataSED71771720,37612,3246,805
MusicNet32086722,44114,40114,123

For MusicNet an audio unit is a (recording, duration bin) pair: 230 recordings appear in all three bins, 87 in two and 3 in one. Segments in different bins are distinct and non-overlapping but share a recording. A bootstrap over the 320 recordings gives 95% intervals of at most ±0.9 points for the MusicNet accuracies in Table 3. Task 2 uses the first 200 clips, in sorted filename order, of the 319 in the 0 to 30 s bin. Three configurations per clip were generated automatically from its annotation file, each with four anchor events. An anchor must exist in the clip and be uniquely identifiable, as a particular note or instrument or as its first, second, third or last occurrence. Intervals are defined by events, never by timestamps. Nine of the 600 attempts gave no valid configuration, leaving 591, each asked in five forms. Section 3.4 states this rule.

Ethics. The release contains questions, answers, temporal programs, source identifiers and extraction scripts, and no audio. VoxConverse is released under CC BY 4.0, MusicNet recordings are Creative Commons licensed or public domain, the MUStARD repository is MIT licensed, and DataSED is released under CC BY-NC-SA 4.0. The paper includes a statement with these terms.

Thank you for the detailed response and the substantial revision. The revision resolves my concerns about dataset construction, human validation, ethics, the evaluation protocol, and the positioning relative to CLEAR and DAQA.

My remaining concern is about the diagnostic claim, not the low scores, which are a legitimate finding. Since the query structure follows CLEAR and DAQA, the contribution of TEROS rests on measuring event-reference reasoning in pretrained ALMs on real audio. With current models, however, every constituent step is at or near floor: single-interval counts in Task 2 are near zero, and Task 1 accuracy does not change with the number of reference points. The benchmark therefore cannot yet show where models fail, or whether the event-reference step contributes to the failure at all. For a benchmark presented as diagnostic, a difficulty gradient would address this, for example sparse targets or small counts, so that basic steps are partly solved and the effect of the reference structure becomes measurable.

I appreciate the authors’ effort, but since this central concern is only partially addressed, I keep my rating at 2.

Thank you for the follow-up. The open point led to a new analysis, and the pattern across the analyses supports the central claim of the paper: the event-reference step is where models fail.

Difficulty gradient. A diagnostic needs steps that are partly solved, so that the effect of the reference structure can be measured, and the existing outputs provide this. In a boundary analysis on the same questions for every model, we grouped Task 1 questions by the number of temporal reference points the model must locate (0 to 3) and report equal-weight accuracy, where a constant answer scores 25%. With no reference point, MiMo-Audio reaches 48.6% on VoxConverse, and FireRedAudio reaches 51.0% on VoxConverse and 48.7% on DataSED, so the basic step is partly solved for these models. On VoxConverse, adding references lowers accuracy for both. MiMo-Audio falls from 48.6% to 28.7% with one reference point, and FireRedAudio falls from 51.0% to 31.5%, 34.3% and 35.3% for 1, 2 and 3 references. On DataSED, FireRedAudio’s answer accuracy stays flat (48.7%, 48.2%, 49.4% and 46.3%), as reported in the paper, while the quality of its enumeration falls, as shown below. The paper’s finding of no consistent decline in accuracy across all models therefore reflects that answer accuracy alone underreports the reference effect.

Where the failure occurs. Models name the correct sound classes but cannot place them in time (Table 14). On correctly answered questions, label F1 is 47 to 88 for the models in Table 14, while timestamp-sensitive F1 is only 3.5 for Qwen2-Audio, 2.2 for Gemma 3n E4B and 12.8 for MiMo-Audio. FireRedAudio places events best (46.3), and it is also much better when its answer is correct than when it is wrong (46.3 against 30.3), as are Audio-Flamingo-3 (30.4 against 12.9) and MiMo-Audio (12.8 against 2.9). In the boundary analysis, FireRedAudio’s enumeration F1 falls as references are added, from 34.1 with no reference to 17.5 with two on VoxConverse and from 50.7 to 31.6 on DataSED, while its DataSED answer accuracy does not move. The reference step shows up in whether the model finds and places the right events. For three of the five models in Table 14, correct answers come with almost no correctly placed events.

Silent audio. Replacing the audio with silence leaves many answers unchanged. Gemma 3n E4B gives the same answer with real and silent audio on 89.3% of MusicNet questions, and in 12 of 18 model-dataset pairs chance-controlled accuracy with audio is within five points of accuracy with silence. Answers that do not depend on the recording, together with correct answers that come without correctly placed events, point to the same conclusion: models return a plausible answer without locating the reference event.

Across these analyses the pattern is consistent. The reference structure changes what models do, the enumeration of the reference events fails and degrades as references are added, and many answers do not depend on the audio. This is the failure TEROS is designed to expose, and it exposes it.

Reviewer_tvJsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces a benchmark called TEROS that aims to test temporally grounded acoustic event reasoning in ALMs. The contributions are 67,466 questions across 3 domains - speech, music and acoustic environments with multiple temporal reasoning categories and 2,955 questions in music domain that tests counting within intervals and their intersection and union. The results show a large gap between coarse temporal reasoning and fine-grained temporal counting. The paper also finds that answer format type strongly affects the targeted scope of the evaluation.

Critical strengths
  • The formulation of event-referenced temporal reasoning is sound.
  • The question generation pipeline from audio dataset annotations is sensible.
Critical weaknesses
  • The motivation for this work is not clearly established. A few examples of temporally grounded event reasoning are provided followed by the discussion of existing benchmarks and the methodology of the work. It needs a stronger problem and impact framing.

  • In Section 3.6 explaining the evaluation protocol, it's unclear how the responses were normalized. Was an LLM used here, was it regex rules? And the methodology for penalizing empty and unambiguous responses while shrinking the denominator for responses that say that the audio is unavailable is inconsistent.

  • Based on the analysis in Section 4.2, answer format bias contaminates task 1 scoring and is not remediated in this work. This limits the contributions of the paper to 2955 questions in Task 2 which are only about music.

  • Discussion of source dataset annotation quality is missing. Since the benchmark of questions and answers is constructed on these annotations and not raw audios/transcripts, noise in the annotations can significantly impact the validity of the reference response labels.

Questions for authors
  • Were there any steps taken to filter out any incorrect annotations from the source datasets?
  • How are responses normalized for scoring?
3 comments

Thank you for the review.

W1. Motivation. The introduction now motivates the task with practical queries that depend on one specific occurrence, such as "how many interruptions follow speaker B's second turn" or "how many alarms follow the first siren". Answering them requires finding that occurrence and using its time as a boundary. The new controls show why this needs dedicated measurement. On the chance-controlled set, models score between 20.8% and 35.2% when the audio is replaced by silence, and in 12 of 18 model-dataset pairs the score with audio is within 5 points of the score with silence. When asked to list the events behind an answer, models identify the correct sound classes but rarely localize them in time: on DataSED, Qwen2-Audio reaches a label F1 of 63.2% and 3.5% once timestamps must match. Aggregate scores on existing benchmarks do not measure this.

W2 and Q2. Normalization and denominators. Section 3.6 now gives the full procedure. Responses in the requested format are scored by a script (trimming, lowercasing, number words to digits). Sentence-form responses are mapped to an answer by an LLM extractor (Claude Sonnet 5.5) under fixed rules, using only the model output. A manual check of a 10% sample per dataset (952 VoxConverse, 1,532 MUStARD, 2,038 DataSED and 2,244 MusicNet questions) found no extraction errors. A response with no extractable answer is incorrect. Tables 3, 5 and 10 count every question in each model's evaluation set, with empty responses and responses stating that the audio is unavailable scored as incorrect. The only partial set in Table 3 is Kimi-Audio on MUStARD (7,443 of 15,319 questions), which is marked. The rule affects the scores: MiMo-Audio on VoxConverse obtains 55.9% over answered questions and 48.1% over all 9,520. A new appendix table reports the share of unusable responses per model, and the raw responses and extracted answers are released.

W3. Answer-format bias. The revision controls for it in the metric. Counting is evaluated with equal-weight accuracy on integer questions with answers 0 to 3, after removing questions answerable from the text, so always predicting one of these four counts scores 25%. Yes/No results are reported against majority baselines of 57.5% (VoxConverse), 51.3% (MUStARD), 56.0% (DataSED) and 59.1% (MusicNet). The chance-controlled set covers 2,297 VoxConverse, 6,605 MUStARD, 8,159 DataSED and 9,611 MusicNet questions, so Task 1 remains usable across speech, sound and music. No model exceeds 48% on speech or environmental sound, and MusicNet scores range from 22.1% to 30.2%.

W4 and Q1. Source annotation quality. Source annotations are used unmodified (Section 3.1). The revision adds three checks.

  • An independent re-derivation of answers from the source files matches the stored answer on 7,808 of 7,935 parsed VoxConverse questions (98.4%) and 9,019 of 9,207 DataSED questions (98.0%). The 127 mismatching and 78 tie-ambiguous VoxConverse questions (2.2%) and the 188 mismatching DataSED questions remain in the files and in the reported results. A mismatch is a disagreement between two implementations, not a confirmed labeling error.
  • In a human audit of 30 questions from VoxConverse, DataSED and MUStARD, annotators answering from the audio matched the ground truth on 22. Annotators answering from the annotation files matched on 15 of 20, and all five disagreements were annotator errors.
  • For MusicNet, two trained musicians each answered 15 questions from the audio alone, and both matched the ground truth on all 15.

MusicNet's authors estimate a 4% label error rate, which the paper reports. MUStARD provides transcripts and speaker labels without timestamps, so its questions are defined over speaker-turn order.

Thank you for the response.

The added examples and new controls explain the need for better evaluation of temporally grounded event reasoning, including the deficiencies of current benchmarks. However, the paper still doesn’t motivate real world use cases or explain the impact of failing to address this capability.

Normalization of responses in the evaluation protocol is now sufficiently explained. Empty and audio unavailable responses are marked incorrect in the main results tables.

Equal-weight accuracy in Task 1 addresses the format bias for the counting questions. But Yes/No questions are only compared against majority baselines.

Thank you for adding the three new checks. While the re-derivation and human audit checks from annotations show that the answers are correctly derived from the source annotations, the 8 annotator disagreements on VoxConverse, DataSED and MUStARD when answering from the source audio are not explained which was the original intention of the question.

I appreciate the substantial additional analysis and this improves the paper's soundness and informativeness, and I am accordingly raining those scores. As several of my concerns are only partially addressed, I will retain my original overall score.

Thank you for the follow-up. Each of the three open points led to a new analysis, and each analysis sharpens a claim in the paper.

Real-world use and the cost of failure. Event-referenced queries arise whenever recorded audio is reviewed after the fact. A compliance auditor asks whether a disclosure was read before the agent’s second offer. A clinician monitoring cough asks how many coughs followed a dose. A sound-alert system for deaf and hard-of-hearing users must decide whether an alarm sounded again after the first alert. In each case the user receives a number or a Yes/No answer, with no sign of whether it came from the audio. Our controls show that current models often answer without using the audio. Gemma 3n E4B gives the same answer with real and silent audio on 89.3% of MusicNet questions. In 12 of 18 model-dataset pairs, chance-controlled counting accuracy with audio is within 5 points of accuracy with silence. The cost of failure is therefore a plausible answer that is not grounded in the recording. TEROS is built to detect this.

Yes/No questions. The check you asked for gives a clear result. We report balanced accuracy, the mean of recall on Yes and recall on No, where a constant answer scores 50%. We give it with unusable responses counted as incorrect, and over usable responses only. The second form separates reasoning from output format. Our prompt did not state the answer type, and some models answered temporal Yes/No questions with labels such as BEFORE.

On MusicNet, Yes/No accuracy is at chance. No model exceeds 52.1%, or 53.2% over usable responses. Replacing the audio with silence changes the score by at most 2.1 points for the nine models with paired runs.

On VoxConverse, 8 of 14 full-coverage models exceed 55%, with a best score of 66.7%. We tested where this advantage comes from. Removing 382 questions answerable from the text alone lowers the top scores by 1 to 2 points. Replacing the audio with silence changes twelve models by 4.3 points or less. Only FireRedAudio and Qwen3-Omni-30B lose more, by 15.4 and 11.5 points, the latter on 857 questions. The above-chance Yes/No accuracy on VoxConverse therefore appears to come mostly from the question text, not the audio. This is the same pattern we find for counting, and it shows why the silent-audio control is part of the benchmark.

Source annotations and the eight disagreements. We did not correct or remove any source annotations. The re-derivation verifies our generation code against the annotation files. The audio-based audit tests the annotations themselves, so we examined each of its eight disagreements against both the audio and the annotation files. None traced to an error in the source annotations.

In four cases, the annotator gave the same wrong answer when working from the annotation files, so the error lay in reading the question. Three of these require multi-step counting, the kind of computation TEROS targets. This suggests the difficulty lies in the computation, not in the labels. In the other four cases, only the answer from audio differed. One involves an unclear event boundary in DataSED. The other three are perceptual. On VoxConverse, the annotator counted an extra speaker during overlapping speech. On DataSED, the annotator reported a sound before the first annotated event, and on MUStARD the annotator missed a short utterance.

With 30 items, the 95% interval on agreement from audio is about 55% to 86%, so this audit bounds annotation noise only loosely. One grounding finding does not depend on the labels at all: the 89.3% agreement between predictions on real and silent audio. The revised version will include a larger audio-based audit, reporting these error categories and human accuracy from audio as a reference.

Across the three points, the new analyses agree. Current audio-language models often answer event-referenced questions from the question text, and TEROS with its controls measures this directly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response

Global response

We thank the reviewers for their detailed and constructive reviews. This note lists what is new in the revised PDF (changes are in red).

  1. Chance-controlled evaluation. Task 1 is now evaluated with equal-weight accuracy on integer questions with answers 0 to 3, after removing text-answerable questions, where always predicting one of these four counts scores 25%. Yes/No results are reported against majority baselines.
  2. Controls for audio grounding. A silent-audio control, an event-enumeration analysis with timestamps, and an analysis by number of temporal reference points test whether answers depend on the audio and on localized events.
  3. Task 2. A balanced set of 40 overlapping and 40 disjoint configurations, and a control that supplies the annotated notes as text. Task 2 is presented as an end-to-end test.
  4. Protocol and validation. One scoring rule for Tables 3, 5 and 10, a table of unusable responses, an independent re-derivation of answers, and a human audit that includes trained musicians for MusicNet.
  5. Positioning and novelty. CLEAR and DAQA are discussed in Section 2 and included in Table 1. Relative to them, TEROS is new in four respects: real recordings, pretrained models evaluated zero-shot across three domains, matched multi-constraint questions, and controls that test whether answers depend on the audio. The contribution list in the introduction now states these.

Tables 3, 5 and 10 were recomputed under the single scoring rule over every question in each model's evaluation set, so some cells differ from the submission. Partial sets are marked: Kimi-Audio on MUStARD in Table 3, and Gemini 3.8 Flash and GPT-Audio 1.5 in Task 2. The released question counts are VoxConverse 9,520, MUStARD 15,319, DataSED 20,376 and MusicNet 22,441 (67,656 in total). Raw responses, extracted answers and per-question scores are released.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.