acceptodds
Under review as a conference paper at ICLR 2027

MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing

Abstract

Long-form audio performance is often summarized by context length and aggregate accuracy, obscuring how language, evidence, and task jointly shape difficulty. We introduce MuLA-Bench, a challenging benchmark of 5,038 open-ended questions over 1,732 in-the-wild recordings totaling 1,341 hours, covering 16 languages and eight domains. A balanced LanguageDomain semantic track supports controlled comparisons, while a complementary acoustic track preserves naturally occurring non-speech evidence. Evidence-grounded generation, shortcut checks, and language-expert review provide auditable questions without translating a shared source set or injecting target sounds. We evaluate eleven audio-language models and conduct targeted diagnostics of language, evidence, and temporal performance. Language rankings change across domains and tasks; acoustic-semantic performance gaps vary with the requested operation; and later evidence is associated with lower temporal accuracy but nearly flat long-range accuracy in a four-hosted-model cohort. Long-range retrieval is comparatively strong, while precise clock alignment and factual grounding of natural acoustic events remain fragile. MuLA-Bench thus exposes conditional failure patterns that a single long-context score does not capture. Anonymous code and benchmark artifacts are included in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
audio-language models, long-form audio understanding, multilingual audio, audio benchmarks, temporal grounding, acoustic event understanding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6gWXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a long audio benchmark including evaluation of various open and source models on a combination of different languages and target domains, on tracks of both semantic and acoustic evidence. These tracks include 4 task families each: factual/completeness, reasoning, temporal localization and long-range retrieval.

Critical strengths

The authors release a benchmark on longer audio, where they focus on a wider variety of languages within various target domains and tasks. They provide an extensive construction pipeline and evaluate also open-weight models. The decoupling of languages over domains is a great addition. The appendix contains many other interesting experiments done in the additional results section.

Critical weaknesses

The authors mention many times (27 times) throughout the work the word "clock": “recording clock” or “clock alignment” or “clock localization” or “clock grounding” or other terminology using “clocks”. I have never heard anywhere in the field of this term being applied like this. To my understanding this "clock" means describing events using a timestamp, and I think its better to just use terminology also described in related work (as described in lines 146-158), and refrain from a clock analogy, “time markers” or “temporal markers” etc fit better.

Questions for authors
  • Line 44: "Benchmark design faces a tension between control and authenticity" Why?

  • Section 3.1 "Audio Collection": what is the origin source of the audio in the benchmark? Sources seem to be YouTube id’s, any information on how the dataset was gathered? In particular one issue I find with some modern datasets is that they are often derived from sources such as Audioset and VGGSound, which are datasets often used in training of these models (train on test). Would be good to specifically describe any deduplication or precautions related to this.

  • There is a huge discrepancy between closed source models and open source models, more than other benchmarks would show (i.e. say MMAU-Pro or MMAR ). In Table 2 open source models seem to be far behind in terms of performance. Do the authors also recognize this discrepancy?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6gWX

Thank you for recognizing the multilingual domain design, construction pipeline, and appendix analyses. We have standardized the temporal terminology and clarified data collection and result interpretation.

1. “Clock” terminology and readability

We replaced “clock localization/grounding” with time-point localization for the task and use timestamp for annotations and predictions. We also specify that times refer to the original recording's timeline. These changes are highlighted throughout the abstract, main text, appendices, tables, and examples.

The distinction is now explicit: event ordering asks which event came first; time-point localization asks when it occurred in minutes and seconds. Section 4 states that time-point scoring requires both the correct event and a timestamp within ±3 seconds of the reference.

Q1. Why is there a tension between control and authenticity?

We did not make this motivation sufficiently concrete. When languages use different original recordings, their domain and task proportions may also differ. Overall scores can then reflect differences in the mix of questions. Translating shared material or inserting target sounds can make some conditions more comparable, but changes the language content or acoustic environment being evaluated.

MuLA retains original recordings while fixing the number of semantic questions and the proportions of tasks in each cell of the 16-language × 8-domain grid. The acoustic track is built from verified natural events. The Introduction now explains this problem and design in place of the abstract “control versus authenticity” framing, making clear what is controlled and why.

Q2. Audio sources, collection, and dataset overlap

We first use GPT-5.6 Sol to generate 100 search keywords per domain, then search YouTube for candidate recordings covering the 16 target languages. Sources must have usable original-language audio; we use only human-created VTT subtitles in the same language as the audio. We cross-check recording language against the source catalog and subtitle content, and retain source links to map questions to their original recordings.

We check subtitle coverage and timestamp validity. Subtitle coverage duration must be 80%–105% of video duration. We filter invalid records with a start time at or after the end time, negative timestamps, an end time more than 3 seconds beyond the video, or a start time at or beyond the video's end.

When metadata suggests dubbing, simultaneous interpretation, or a language mismatch, we inspect the subtitles and audio further and exclude sources confirmed to be invalid. We do not exclude a recording solely for occasional code-switching. GPT-5.6 Sol then assigns each transcript to one of eight domains according to its dominant function. Section 3.1 and Appendix B.2 describe source requirements and domain assignment; B.2 also documents exclusion of known predecessor sources and leakage lists before QA generation.

For the datasets you specifically mentioned, our additional source-level comparisons found zero overlapping recordings with either AudioSet [1] or VGGSound [2]. This result is now highlighted in Section 3.1.

Q3. Why is the gap between open-weight and hosted models so large?

The gap is substantial: in Table 2, the eight hosted models achieve 33.78%–73.48% overall accuracy, versus 13.42%–24.77% for the three open-weight models. Section 5.1 now discusses this result in relation to their actual input conditions.

MuLA recordings have a median duration of 36.7 minutes. Under their native context limits, MOSS, Qwen3-Omni, and Audio Flamingo Next receive at most the first 54.6, 40, and 30 minutes of each recording, respectively. Evidence beyond these limits is unavailable to them (Section 4; Appendix C.3). Their valid-response rates are nevertheless 99.98%–100% (C.4), so widespread empty responses or execution failures do not explain the low scores. This rate measures whether a valid answer is returned, not whether the full recording was received.

Table 2 therefore measures how the full systems perform on multilingual long audio. Scores reflect both whether models receive the evidence they need and whether they can answer from it. Comparisons with gaps on MMAU-Pro or MMAR must account for input scope and scoring rules. We did not separately measure how much truncation contributes to our reported gap.

References

[1] Gemmeke et al. Audio Set: An ontology and human-labeled dataset for audio events. ICASSP, 2017. Paper.

[2] Chen, Xie, Vedaldi, and Zisserman. VGGSound: A Large-scale Audio-Visual Dataset. ICASSP, 2020. Paper.

I think the authors did great work, especially a benchmark where the discrepancy is so big between closed and open source is a good sign for a benchmark, for academic and/or open models to catch up. I was mistaken about the "clock" terminology, seems like this is becoming standard terminology now having seen it also in other works, it still sounds strange to me though. I would loved for it to be not just a benchmark of evaluating others, but also setting a baseline or own model as a reference, for example through post-training an existing model. I will maintain my score of weak acceptance.

Thank you for your encouraging feedback and continued support. We also appreciate your suggestion to develop a model baseline through post-training and see this as a useful direction for future work.

Reviewer_Gk1eOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MuLA-Bench, a multilingual long-form audio benchmark comprising of 5038 open-ended questions across 1732 in-the-wild recordings. The benchmark is comparably extensive spanning over 1341 hours in 16 languages and eight domains. It is comprise of a dual-track architecture, with a a Semantic track balanced across language-domain pairs without altering natural content, and an Acoustic track evaluating naturally occurring non-speech events without artificial injection.

The benchmark construction is extensive, LLM verification (Gemini) and paid expert human review. Evaluations across 11 audio-LLMs reveal critical insights beyond standard leaderboard scores, such as a sharp dissociation between event ordering vs. precise timestamp alignment, and persistent failures when given ground-truth localized audio crops.

Overall, I recommend accepting this paper. It provides a principled, high-quality community resource backed by commendable statistical discipline and human auditing.

Critical strengths
  • Principled dual-track design: Successfully balances experimental control and audio authenticity by controlling question allocation rather than synthesizing artificial acoustic scenes.
  • Decoupled Q/A generation, shortcut filtering, and most notably human expert review with 94.2% judge-human agreement.
  • Valuable diagnostic insights: Uncovers fine-grained model behaviors beyond simple accuracy, including temporal order vs. clock-alignment gaps and operation-dependent evidence failures.
Critical weaknesses
  • GPT-5.6 Sol and Gemini are used for source curation, verification, and scoring, while also being evaluated models. This might lead to a biased evaluation setup. Even though human evaluation helps mitigating this, it is somewhat unclear why authors choose to use this particular setup.
  • While not particulalry problematic, the length of the benchmark is somewhat long with over 1000 hours, meaning that participants might take hours or days to evaluate their model.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer Gk1e

Thank you for recognizing the two-track design, expert review, and diagnostic analyses, and for raising the risks of model involvement across stages and the practical cost of evaluation.

1. Model roles and potential bias

We understand this concern. Benchmark construction and scoring can reflect the preferences of models or human annotators. We assign them distinct roles, use expert review, and compare scores with external judgments to reduce and examine these effects. GPT-5.6 Sol handles textual evidence, generates questions and reference answers, and scores answers for semantic correctness. Gemini helps verify candidate acoustic events. A language expert must review each question before inclusion. Sections 3.3–3.4 and Appendix B describe the inputs, decisions, and review requirements at each stage.

To reduce bias from using the same model to generate and score answers, we generate each question without supplying a prewritten answer, then derive its reference answer in a separate call with a fresh context. We check that the recording supports the answer and look for shortcuts that allow answering without audio. Experts review all 5,038 questions. Scoring accepts semantically equivalent answers without requiring the wording of the reference.

We also added external scoring comparisons. Expert judgments agree with the original judge on 867/909 responses (95.38%), and the original GPT-based and Gemini-based scoring pipelines agree on 934/990 responses (94.34%). Thus, the original scores broadly agree with judgments from both humans and another model family. The key results are now in Section 4, with details in Appendix C.8.

For acoustic screening, we additionally checked 400 events using Muse Wonder 1.0, from another model family. On the 40-question subset corresponding to Muse-confirmed events in this sample, there were no pairwise rank reversals among the 11 models relative to the full acoustic track. In a sensitivity analysis, we removed the two questions identified for deletion by this check from the full acoustic track (1,198 to 1,196 questions). All 11 model rankings remained unchanged, and every model's accuracy changed by less than 0.1 percentage points. Scores and settings are reported in Appendix B.4.

Final inclusion also requires audible evidence, audibility checks of the relevant audio segments, and expert review through audio playback (Section 3.4; Appendices B.4 and B.8). These steps keep the reference answers grounded in the audio itself.

2. Evaluation cost

Thank you for considering the benchmark's practical accessibility. For hosted models, our API configuration was 60 requests per minute, and evaluation took approximately 2 hours in practice. Runtime varies with server processing speed, request quotas, and retries. For open-weight models, it depends on model size, available GPUs, and batching. As a concrete measurement, evaluating Audio Flamingo Next 8B on all 5,038 questions took 12 hours using 8 A100 80GB GPUs.

The benchmark covers 16 languages, 8 domains, and different tasks, allowing users to examine performance across each. Section 4 and Appendices C.1–C.4 specify model inputs, native context limits, and scoring settings to help users plan evaluations.

Reviewer_Gk1eOpenReview ↗

Thanks

Thanks for clearing that up.

Thank you for taking the time to read our responses and reconsider your assessment. We appreciate your constructive feedback, which helped us improve the paper.

Reviewer_kWrZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MuLA-Bench is a multilingual long-form audio QA benchmark: 5,038 open-ended questions over 1,732 native, in-the-wild recordings (1,341 h, median 36.7 min) across 16 languages and 8 domains. A semantic track fixes 30 questions per Language×Domain cell with a constant task mix; an acoustic track (1,198 items) follows naturally occurring non-speech events without injection or translation. Construction relies on GPT-5.6 Sol (fact mining, decoupled Q/A generation, verification, judging), TimeAudio + Gemini 3.1 Pro (acoustic event verification), shortcut ablations, and one expert reviewer per language. Eleven models are evaluated with routed binary rubrics and deterministic ±3 s temporal matching. Main findings: large Language×Domain residuals, operation-dependent acoustic–semantic gaps, position-dependent decline concentrated in event ordering, and persistent acoustic-factual errors even under gold-localized crops.

Critical strengths

Scale and coverage. 16 languages, 8 domains, 1,732 recordings, 5,038 QA; among the broadest multilingual long-audio benchmarks to date.

Naturalistic sources. Native recordings with naturally occurring acoustic events rather than translated variants or inserted sounds, closer to deployment conditions.

Balanced semantic design. Fixed question count and task composition per Language×Domain cell enables the conditional analyses (Sec. 5.2) that most benchmarks cannot support.

Fine-grained taxonomy and routed scoring. Separating factual, reasoning, temporal (order vs. clock) and long-range, with route-specific rubrics and code-enforced temporal tolerance, isolates distinct capabilities; the order-vs-point decomposition (Table A18) is a genuinely useful result.

Extensive, carefully analyzed evaluation. Eleven open and commercial models; recording-clustered inference, simultaneous intervals, and candidly reported null results (e.g., relative-position interaction, Table A16).

Critical weaknesses

Limited novelty over existing long-audio benchmarks. Overlap with AudioSpan, VoiceGiraffe and BLAB is substantial; Table 1 asserts differences but no empirical comparison (e.g., item overlap, difficulty, or diagnostic power) substantiates the claimed advantage.

Cross-lingual comparability is weaker than the balanced design suggests. Languages use different recordings, so content difficulty, audio quality, speaker and cultural factors remain confounded with language if no further discussion provided. With 30 questions and a median of 11 recordings per cell (some <5), residuals of ±10–16 pp are plausibly recording-level noise; Fig. A4 shows cross-model residual SDs comparable to the residuals themselves. The abstract’s “language rankings change” claim overreaches relative to this evidence.

LLM-in-the-loop circularity. GPT-5.6 Sol writes, answers, verifies and judges; Gemini 3.1 Pro gates acoustic events and is also evaluated, and its verification uses video although the benchmark is audio-only. Will this risks systematic phrasing, language and model-family biases?

Insufficient human validation. One expert per item; inter-annotator agreement is measured on only 150 of 5,038 items, and the statistical resampling interval and fraction of items repaired or replaced is not reported.

Acoustic–semantic comparison is not matched. The two tracks differ in recordings, question difficulty and task distribution (867/1,198 acoustic items are factual; only 97–122 for the other tasks). Gaps in Fig. 5 therefore need not reflect acoustic understanding per se, and per-task acoustic estimates for reasoning/temporal/long-range will have wide resampled statistical intervals that the paper does not report.

Reasoning and long-range definitions are shallow. Reasoning requires only two linked facts with no verification of multi-step difficulty; 68.5% of long-range items are single late facts, which also explains why long-range accuracy exceeds factual accuracy in Table 2 (a route-composition effect, not robustness).

Limited actionable insight. The diagnostics establish correlations (position, operation, language) but no mechanistic experiment explains why models fail (e.g., attention over long inputs, acoustic encoder resolution), so guidance for model improvement is thin.

Questions for authors

Report expert–judge agreement by evaluated model and by route; is judge error asymmetric?

What fraction of time-point references were corrected by experts, and by what median magnitude relative to the ±3 s tolerance?

Provide recording-clustered 95% intervals for the per-task acoustic accuracies in Table A13 (n = 97–122).

Any contamination check given public sources with public subtitles?

Will audio be redistributed, or only links? Current source availability?

6 comments
AuthorsOpenReview ↗

Response to Reviewer kWrZ (1/3): Comparisons and contribution

Part 1/3 — Cross-language comparisons and contribution (W2; W1/W7)

Thank you for the careful review. We have clarified the claims and added sampling, annotation, and scoring checks. Changes are in blue; Part 3 answers Q1–Q5.

W2: What can comparisons of different native recordings establish?

MuLA tests models on native recordings across languages and domains, preserving local content, speakers, and recording conditions. INCLUDE's regional examinations and GlobeAudio's naturalistic audio share this motivation. Our Thai questions retain native terminology and culture in explanations and chanting about “unwholesome karma.” Each language covers the same eight domains, with 30 questions per cell and fixed task proportions (§3.5). This controls the domain and task mix. The scores reflect these settings; they do not isolate the effect of language. This reveals domain-specific weaknesses that overall language scores can hide.

Recording sampling and Figure A4. The analysis resamples recordings, reporting simultaneous 95% intervals over 120 eligible cells (Appendix C.9). Chinese–Entertainment and Chinese–Product residuals are +10.6/−16.0 percentage points (pp), or +13.7/−16.1 pp after adjusting for task, scoring route, and evidence features. All four intervals exclude zero. Figure 4 marks cells with fewer than five recordings and does not highlight them as significant.

Removing each recording in turn leaves residuals of +9.97 to +12.03 pp for Entertainment (12 recordings) and −18.41 to −12.98 pp for Product (7): neither example depends on a single recording (D.6). We also doubled four cells from 30 to 60 questions: English–Belief, Spanish–Lifestyle, Japanese–How-to, and Russian–How-to. The original 30 questions are included in each set of 60, with answers and scores from the same run. Overall accuracy across the four cells changes by +0.83 pp for Gemini 3.1 Pro and −2.92 pp for Gemini 3.8 Flash; all eight model–cell changes are at most 5 pp in magnitude (D.6).

Figure A4 shows variation between models, not uncertainty from sampling recordings. Despite varying magnitudes, Chinese–Entertainment is positive for 11/11 models and Chinese–Product negative for 10/11. Sparsely recorded cells still warrant caution.

Language-rank changes. Figure 4(b) compares each model's domain/task language rankings with its overall semantic ranking (§5.2). Qwen3.8-Omni-Flash scores Chinese/Japanese at 86.7/60.0% in News but 66.7/90.0% in Product. Both relations hold jointly in 97.208% of 50,000 recording-bootstrap replicates (§5.2; D.6). This reversal persists under recording resampling. The abstract now limits this claim to the sampled native recordings and questions.

W1/W7: What does MuLA add?

MuLA provides 1,732 native long recordings and 5,038 open questions, with a balanced 16-language × 8-domain semantic grid and naturally occurring non-speech events. AudioSpan and VoiceGiraffe mainly cover Chinese/English; BLAB covers English. Table 1 now reports total hours: MuLA 1,341, BLAB 833, AudioSpan 296, and VoiceGiraffe 113. MuLA also reveals differences hidden by overall scores: Gemini 3.1 Pro and Qwen3.5-Omni-Plus score similarly overall (61.39/60.92%) but differ by 15.61 pp on acoustic tasks (26.54/42.15%; Table 2).

Open questions require producing complete lists, relations, or timestamps rather than choosing an option, unlike VoiceGiraffe. AudioSpan also uses open questions; MuLA extends open QA to these languages, domains, and natural acoustic tasks. Revised §2 details these distinctions.

Temporal diagnosis. Time-point localization scores are lower than ordering scores for 9/11 models. Yet in the four-model analysis, adjusted ordering accuracy is 9.88 pp lower for evidence 30 minutes later, while localization changes by +0.28 pp (§§5.4–5.5; Table A18). This distinguishes low timestamp precision from ordering deterioration later in context, extending AudioSpan's content-versus-temporal-reference comparison.

Acoustic diagnosis. With the same three models, 200 acoustic factual questions, and scoring references, replacing full recordings by annotated gold evidence crops reduces median input from 36.5 minutes to 31.1 seconds. Still, 278/347 original errors (80.1%) persist (§5.3; D.9). Better retrieval alone therefore leaves most errors unresolved. Future work can test whether encoders lose fine acoustic detail or later attention fails to use it. AudioSpan's caption baseline changes model and input, VoiceGiraffe compares end-to-end and caption cascades, and BLAB-mini modifies some task targets. MuLA fixes model, question, and reference to test whether correct localization repairs errors.

AuthorsOpenReview ↗

Response to Reviewer kWrZ (2/3): Validation and LLM involvement

Part 2/3 — Validation, definitions, and LLM involvement (W4; W5/W6; W3)

W4: Human review and question corrections.

Experts checked all 5,038 questions and references against the recordings, repairing 318 questions (6.31%) and replacing 16 (0.32%). Four questions underwent both, so 330 distinct questions were corrected or replaced (6.55%).

Two experts independently scored the same model responses, agreeing on 145/150 originally (96.7%; Wilson 95% CI 92.4–98.6%) and 857/909 in the additional audit (94.28%; Appendix C.8). This supports consistent application of the scoring rules.

W5/W6: Acoustic–semantic gaps and task definitions.

Does question composition explain the acoustic deficit? Figure 5 compares the two tracks within each task. Different proportions of factual questions cannot explain the gap within that task. Recordings and item difficulty are not matched one-to-one, as acknowledged in F.3. Adjusting for model, language, domain, recording duration, and evidence features still gives much lower acoustic factual accuracy (Table A17: OR 0.093; recording-cluster 95% CI [0.080, 0.109]). These differences in composition do not explain the factual gap.

Part 3 provides recording-cluster 95% intervals for all eleven models. Smaller tasks have wider intervals, but large differences remain: MOSS achieves 40.98% [32.41, 49.59] on acoustic reasoning versus 10.84% [8.96, 12.77] on semantic reasoning. Acoustic questions are not uniformly harder. The gold-crop experiment also leaves 80.1% of original errors unresolved on the same questions (§5.3; D.9).

Do two facts constitute reasoning? Two evidence items are the minimum. The task requires inferring a relation from the recording, not necessarily complex multi-step reasoning (§3.2; A.4). The question plan must link the evidence through a causal, comparative, or other concrete relation (B.3; E.1), and scoring requires both the conclusion and relation to be correct (C.5). Retrieving or restating two disconnected facts does not satisfy this rubric: the evidence must support the required inference.

Do single-fact questions explain strong long-range scores? Yes, composition affects pooled scores: 614/896 semantic long-range questions retrieve one late fact, meeting the definition of late or separated evidence (§3.2). Within the semantic track, factual accuracy exceeds long-range accuracy for all eleven models (Table A12). The separate position analysis adjusts for scoring routes and other covariates: per 30 minutes, temporal accuracy declines 5.83 pp more than long-range accuracy (95% CI for this additional decline [1.92, 9.73] pp; Table A18). This finding uses adjusted slopes, not pooled accuracy.

W3: Shared generation, scoring, and screening preferences.

Both human annotators and LLMs can introduce their own preferences when constructing or scoring questions.

Question/answer generation. GPT-5.6 Sol is not among the eleven evaluated models. Questions are generated from evidence plans without a prewritten answer; a separate call with a fresh context generates each reference answer. Thus, questions are not worded around a supplied answer. We check that answers have audio support and questions require listening; experts review all questions (§§3.3–3.4; B.5–B.8).

Evaluation. Scoring accepts answers with the same meaning, without requiring the reference wording. We check GPT scoring against experts and a different model family:

ComparisonAgreement
Original expert audit807/844 (95.6%)
Additional expert vs. original judge867/909 (95.38%)
Independent second expert vs. original judge875/909 (96.26%)
GPT vs. Gemini scoring pipelines934/990 (94.34%)

Original audit: Wilson 95% CI 94.0–96.8%; the additional audit spans eleven models and ten languages. These external checks support scoring reliability. Part 3 gives results by model and scoring rule, and the direction of errors. Revised §4/C.8 report the results.

Gemini screening and video assistance. We checked 400 acoustic events with Muse Wonder 1.0 from another family (B.4). We selected 40 questions corresponding to events confirmed by Wonder: 40 recordings, all sixteen languages, and ten questions per task. No pair of models reverses order compared with the full acoustic track; B.4 reports all model scores. Removing the two questions identified for deletion leaves 1,196/1,198 acoustic questions, preserves all eleven model ranks, and changes each accuracy by less than 0.1 pp. The model comparisons remain stable under this additional screening.

Final inclusion also requires audible evidence, local audibility checks, and expert audio replay (§3.4; B.4; B.8). Video may help identify an event's source, but details supported only by images cannot enter the reference answer. The retained questions must be answerable from audio alone.

AuthorsOpenReview ↗

Response to Reviewer kWrZ (3/3): Answers to Q1–Q5

Part 3/3 — Answers to Q1–Q5

Q1. Expert–judge agreement by model and scoring route; error direction.

The new audit covers 909 responses from eleven models and ten languages. Expert–judge agreement is 867/909 (95.38%); the independent second expert agrees with the judge on 875/909 (96.26%), and the experts agree with each other on 857/909 (94.28%). Appendix C.8 reports results by model. Q3 uses the same model names.

ModelAgreed/reviewed%
Audio Flamingo Next 8B86/9095.56
Doubao 2.0 Lite84/9093.33
Gemini 3.1 Pro84/9093.33
Gemini 3.7 Flash86/9095.56
Gemini 3.8 Flash86/9095.56
MOSS 8B Thinking61/6396.83
Muse Spark 1.259/6393.65
Qwen3-Omni 30B-A3B85/9094.44
Qwen3.5-Omni Flash87/9096.67
Qwen3.5-Omni Plus61/6396.83
Qwen3.8-Omni Flash88/9097.78
Total867/90995.38

The route breakdown uses the separate original 844-response audit, with overall agreement of 807/844 (95.62%). The time-point route includes semantic judgment and programmatic temporal checks, matching the original end-to-end audit definition (C.8).

Scoring routeAgreed/reviewed%
Single-fact155/15898.10
Multi-slot154/15798.09
Reasoning154/15996.86
Time-order145/15593.55
Exhaustive-list53/5792.98
Time-point pipeline146/15892.41
Total807/84495.62

The judge rejects correct answers more often than it accepts incorrect ones. Against the first expert in the additional 909-response audit, TP=490, TN=377, FP=11 (judge accepts; expert rejects), and FN=31 (judge rejects; expert accepts). Thus, the proportion marked correct by the judge is 2.20 pp below the expert's; these counts are now in C.8. Separately, GPT/Gemini scoring pipelines agree on 934/990 (94.34%): 989 Gemini outputs plus one programmatic fallback retained in the denominator.

Q2. Timestamp corrections relative to ±3 seconds.

Experts corrected 15/453 single-time-point localization references (3.31%). The median absolute correction among these 15 is 6.5 seconds, exceeding the ±3-second scoring tolerance. All corrections are incorporated into the final references, resolving timing errors large enough to affect scoring.

Q3. Recording-cluster 95% CIs for Table A13.

We resample whole recordings 50,000 times, keeping their questions and all models' responses together and recomputing accuracy. Below are pointwise percentile 95% CIs in percent, without adjustment for multiple comparisons. Point estimates are in Table A13. F/R/T/L denote factual/reasoning/temporal/long-range. Their question/recording counts are 867/587, 122/83, 112/77, and 97/66, respectively.

ModelFRTL
Gemini 3.8 Flash45.27–52.5457.14–73.8764.15–81.8975.51–89.29
Gemini 3.7 Flash43.15–50.2361.21–78.9565.52–82.2473.27–88.89
Qwen3.8-Omni Flash38.90–46.1964.29–79.7667.54–83.7077.78–92.38
Gemini 3.1 Pro14.53–20.1247.57–66.9020.15–37.4058.44–77.01
Qwen3.5-Omni Plus31.70–38.6955.12–72.4137.59–55.2863.01–80.72
Qwen3.5-Omni Flash16.39–22.4444.17–60.5022.45–38.6756.70–76.92
Doubao 2.0 Lite30.36–37.9050.42–69.034.10–14.4070.87–87.37
Muse Spark 1.215.65–21.5055.00–72.500.00–2.9445.45–65.77
MOSS 8B Thinking7.27–11.7632.41–49.592.38–12.8210.67–27.40
Qwen3-Omni 30B-A3B5.84–9.7328.36–46.920.00–5.9314.78–31.46
Audio Flamingo Next 8B4.03–7.6516.95–34.260.00–4.506.67–21.18

Smaller tasks have wider intervals, so small score differences may not imply reliable rank differences.

Q4. Public audio/subtitle contamination.

We excluded known sources from earlier datasets and leakage lists before question generation (B.2). We found zero overlapping recordings with either AudioSet or VGGSound, now reported in §3.1. We also remove questions answerable from the question alone, without audio or subtitles (B.6). These checks cover source overlap and shortcuts that bypass audio. Complete pretraining corpora of closed models remain unavailable, so prior use of public audio or subtitles cannot be directly verified.

Q5. Release and current source availability.

Upon acceptance, we will release the complete question set, source links, and all evaluation audio (F.2). On November 10, 88.0% of source players were accessible; others were unavailable or restricted. Researchers can reproduce the evaluation using the released audio even if source links fail. Anonymous supplementary materials already include examples and links for all sixteen languages and 200 gold-crop diagnostic audio clips.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.