- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a long audio benchmark including evaluation of various open and source models on a combination of different languages and target domains, on tracks of both semantic and acoustic evidence. These tracks include 4 task families each: factual/completeness, reasoning, temporal localization and long-range retrieval.
Critical strengths
The authors release a benchmark on longer audio, where they focus on a wider variety of languages within various target domains and tasks. They provide an extensive construction pipeline and evaluate also open-weight models. The decoupling of languages over domains is a great addition. The appendix contains many other interesting experiments done in the additional results section.
Critical weaknesses
The authors mention many times (27 times) throughout the work the word "clock": “recording clock” or “clock alignment” or “clock localization” or “clock grounding” or other terminology using “clocks”. I have never heard anywhere in the field of this term being applied like this. To my understanding this "clock" means describing events using a timestamp, and I think its better to just use terminology also described in related work (as described in lines 146-158), and refrain from a clock analogy, “time markers” or “temporal markers” etc fit better.
Questions for authors
-
Line 44: "Benchmark design faces a tension between control and authenticity" Why?
-
Section 3.1 "Audio Collection": what is the origin source of the audio in the benchmark? Sources seem to be YouTube id’s, any information on how the dataset was gathered? In particular one issue I find with some modern datasets is that they are often derived from sources such as Audioset and VGGSound, which are datasets often used in training of these models (train on test). Would be good to specifically describe any deduplication or precautions related to this.
-
There is a huge discrepancy between closed source models and open source models, more than other benchmarks would show (i.e. say MMAU-Pro or MMAR ). In Table 2 open source models seem to be far behind in terms of performance. Do the authors also recognize this discrepancy?
3 comments
I think the authors did great work, especially a benchmark where the discrepancy is so big between closed and open source is a good sign for a benchmark, for academic and/or open models to catch up. I was mistaken about the "clock" terminology, seems like this is becoming standard terminology now having seen it also in other works, it still sounds strange to me though. I would loved for it to be not just a benchmark of evaluating others, but also setting a baseline or own model as a reference, for example through post-training an existing model. I will maintain my score of weak acceptance.
Thank you for your encouraging feedback and continued support. We also appreciate your suggestion to develop a model baseline through post-training and see this as a useful direction for future work.
Response to Reviewer 6gWX
Thank you for recognizing the multilingual domain design, construction pipeline, and appendix analyses. We have standardized the temporal terminology and clarified data collection and result interpretation.
1. “Clock” terminology and readability
We replaced “clock localization/grounding” with time-point localization for the task and use timestamp for annotations and predictions. We also specify that times refer to the original recording's timeline. These changes are highlighted throughout the abstract, main text, appendices, tables, and examples.
The distinction is now explicit: event ordering asks which event came first; time-point localization asks when it occurred in minutes and seconds. Section 4 states that time-point scoring requires both the correct event and a timestamp within ±3 seconds of the reference.
Q1. Why is there a tension between control and authenticity?
We did not make this motivation sufficiently concrete. When languages use different original recordings, their domain and task proportions may also differ. Overall scores can then reflect differences in the mix of questions. Translating shared material or inserting target sounds can make some conditions more comparable, but changes the language content or acoustic environment being evaluated.
MuLA retains original recordings while fixing the number of semantic questions and the proportions of tasks in each cell of the 16-language × 8-domain grid. The acoustic track is built from verified natural events. The Introduction now explains this problem and design in place of the abstract “control versus authenticity” framing, making clear what is controlled and why.
Q2. Audio sources, collection, and dataset overlap
We first use GPT-5.6 Sol to generate 100 search keywords per domain, then search YouTube for candidate recordings covering the 16 target languages. Sources must have usable original-language audio; we use only human-created VTT subtitles in the same language as the audio. We cross-check recording language against the source catalog and subtitle content, and retain source links to map questions to their original recordings.
We check subtitle coverage and timestamp validity. Subtitle coverage duration must be 80%–105% of video duration. We filter invalid records with a start time at or after the end time, negative timestamps, an end time more than 3 seconds beyond the video, or a start time at or beyond the video's end.
When metadata suggests dubbing, simultaneous interpretation, or a language mismatch, we inspect the subtitles and audio further and exclude sources confirmed to be invalid. We do not exclude a recording solely for occasional code-switching. GPT-5.6 Sol then assigns each transcript to one of eight domains according to its dominant function. Section 3.1 and Appendix B.2 describe source requirements and domain assignment; B.2 also documents exclusion of known predecessor sources and leakage lists before QA generation.
For the datasets you specifically mentioned, our additional source-level comparisons found zero overlapping recordings with either AudioSet [1] or VGGSound [2]. This result is now highlighted in Section 3.1.
Q3. Why is the gap between open-weight and hosted models so large?
The gap is substantial: in Table 2, the eight hosted models achieve 33.78%–73.48% overall accuracy, versus 13.42%–24.77% for the three open-weight models. Section 5.1 now discusses this result in relation to their actual input conditions.
MuLA recordings have a median duration of 36.7 minutes. Under their native context limits, MOSS, Qwen3-Omni, and Audio Flamingo Next receive at most the first 54.6, 40, and 30 minutes of each recording, respectively. Evidence beyond these limits is unavailable to them (Section 4; Appendix C.3). Their valid-response rates are nevertheless 99.98%–100% (C.4), so widespread empty responses or execution failures do not explain the low scores. This rate measures whether a valid answer is returned, not whether the full recording was received.
Table 2 therefore measures how the full systems perform on multilingual long audio. Scores reflect both whether models receive the evidence they need and whether they can answer from it. Comparisons with gaps on MMAU-Pro or MMAR must account for input scope and scoring rules. We did not separately measure how much truncation contributes to our reported gap.
References
[1] Gemmeke et al. Audio Set: An ontology and human-labeled dataset for audio events. ICASSP, 2017. Paper.
[2] Chen, Xie, Vedaldi, and Zisserman. VGGSound: A Large-scale Audio-Visual Dataset. ICASSP, 2020. Paper.