- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper present 3MSPEECHBENCH, a multi-speaker (50 speakers), multilingual (10 languages), multi-task (covering multiple-choice QA, spoken visual QA, article summarization, and dialogue summarization) benchmark that keeps linguistic content fixed while varying the speaker. Nine open-weight MLLMs are evaluated on the benchmark, showing that top average performance does not necessarily imply stable behavior across speaker. These findings are also tested on another benchmark, MCIF, showing their generalizability.
Critical strengths
- The idea is interesting and, to the best of my knowledge, this is the first multilingual benchmark built with the concept "hold linguistic content fixed and vary only the speaker".
- Although built on top of existing corpora, it covers 10 languages and multiple QA and summarization tasks.
- The benchmark design is very robust although exclusively based on an automatic pipeline (no human supervision, post-editing, or judgment).
- The paper is clear, easy to follow, and with a lot of analyses and ablations.
Critical weaknesses
- At least a small manual analysis on a restricted number of samples would have been useful to further demonstrate the quality of the synthesized voices, as the entire pipeline is without human supervision (the analysis in Section 5 helps, but still relies on a fully automatic pipeline).
- Although metadata as speaker's gender, coarse age group, acoustic scene, and self-stated nationality are collected and released, there is no analysis involving these groups. This would have been an interesting aspect to investigate, useful to understand whether the instability is more pronounced in a specific demographic group.
- There is no analysis on frontier models, therefore it is not know if the finding about top-performance on average != stability across speakers generalizes to top MLLMs, nor an upperbound is set (as also human performance is not reported on this benchmark).
Not a critical weakness but some figures are too small (e.g., Figure 4, axes are barely visible).
Questions for authors
- What is the result of closed models on this benchmark? At least on some selected languages/restricted number of samples? It is always useful to have comparisons with frontier MLLMs to test the benchmark broad usefulness and set an upperbound for open-weight models.
- Do you observe any difference between the different gender, age group, acoustic scene, and nationality? For instance, is the speaker-wise performance instability more prominent in one gender instead of the other? Does it relates to age or nationality? What is the effect of the acoustic scene?
2 comments
I thank the authors for their reply, and their additional results provided in the rebuttal, which further confirms my already very positive assessment of the work.
Reply to reviewer VLQn.
Thank you for this suggestion. We additionally evaluated Gemini 3 Flash on all four tasks, covering 9 languages for MMLU, 3 for CVQA, 7 for CrossSum, and English for SAMSum, using 200 questions per language for QA, 50 articles per language for CrossSum, and 50 SAMSum dialogues, with all 10 speakers per item.
Gemini 3 Flash achieves the highest accuracy and consistency on both QA tasks, while Voxtral Small achieves higher BERTScore and consistency on both summarization tasks (Table below).
Even for Gemini 3 Flash, different speakers can still lead to different model responses, with consistency scores of 91.7 on MMLU and 46.5 on CrossSum. This suggests that evaluating consistency alongside performance remains relevant for frontier models.
Table: Performance and consistency of Gemini 3 Flash and representative open-weight models on the same evaluation subset. Scores are macro-averaged across the evaluated languages. We evaluated the open-weight models on the same subsets for comparison.