acceptodds
Under review as a conference paper at ICLR 2027

3MSpeechBench: Measuring Behavioral Consistency of Multimodal LLMs across Speakers and Languages

Abstract

Existing benchmarks for multimodal large language models (MLLMs) with speech input primarily evaluate aggregate task performance, providing limited insight into whether model behavior remains stable when the same content is spoken by different speakers. We present 3MSpeechBench, a multi-speaker, multilingual, multi-task benchmark that holds linguistic content fixed while varying the speaker. Built from four established corpora, it covers spoken multiple-choice QA, spoken visual QA, article summarization, and dialogue summarization, with 50 speakers per language and up to 10 languages. We evaluate MLLMs using both task performance and within-instance consistency across speakers. Our results show that strong average performance does not necessarily imply stable behavior across speakers, highlighting the need to evaluate speech-facing MLLMs beyond aggregate task scores. A demo with data examples is available at https://anonymous.4open.science/w/3mspeechbench-demo-27A3/.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Benchmarking, Robustness Evaluation, Multilingual Evaluation, Speaker Variation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VLQnOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper present 3MSPEECHBENCH, a multi-speaker (50 speakers), multilingual (10 languages), multi-task (covering multiple-choice QA, spoken visual QA, article summarization, and dialogue summarization) benchmark that keeps linguistic content fixed while varying the speaker. Nine open-weight MLLMs are evaluated on the benchmark, showing that top average performance does not necessarily imply stable behavior across speaker. These findings are also tested on another benchmark, MCIF, showing their generalizability.

Critical strengths
  • The idea is interesting and, to the best of my knowledge, this is the first multilingual benchmark built with the concept "hold linguistic content fixed and vary only the speaker".
  • Although built on top of existing corpora, it covers 10 languages and multiple QA and summarization tasks.
  • The benchmark design is very robust although exclusively based on an automatic pipeline (no human supervision, post-editing, or judgment).
  • The paper is clear, easy to follow, and with a lot of analyses and ablations.
Critical weaknesses
  • At least a small manual analysis on a restricted number of samples would have been useful to further demonstrate the quality of the synthesized voices, as the entire pipeline is without human supervision (the analysis in Section 5 helps, but still relies on a fully automatic pipeline).
  • Although metadata as speaker's gender, coarse age group, acoustic scene, and self-stated nationality are collected and released, there is no analysis involving these groups. This would have been an interesting aspect to investigate, useful to understand whether the instability is more pronounced in a specific demographic group.
  • There is no analysis on frontier models, therefore it is not know if the finding about top-performance on average != stability across speakers generalizes to top MLLMs, nor an upperbound is set (as also human performance is not reported on this benchmark).

Not a critical weakness but some figures are too small (e.g., Figure 4, axes are barely visible).

Questions for authors
  • What is the result of closed models on this benchmark? At least on some selected languages/restricted number of samples? It is always useful to have comparisons with frontier MLLMs to test the benchmark broad usefulness and set an upperbound for open-weight models.
  • Do you observe any difference between the different gender, age group, acoustic scene, and nationality? For instance, is the speaker-wise performance instability more prominent in one gender instead of the other? Does it relates to age or nationality? What is the effect of the acoustic scene?
2 comments
AuthorsOpenReview ↗

Reply to reviewer VLQn.

At least a small manual analysis on a restricted number of samples would have been useful to further demonstrate the quality of the synthesized voices, as the entire pipeline is without human supervision (the analysis in Section 5 helps, but still relies on a fully automatic pipeline).

  • We agree that quality assessments based on automatic metrics cannot be considered equivalent to human evaluations.
  • To support qualitative inspection, we provide a demo page (https://anonymous.4open.science/w/3mspeechbench-demo-27A3/) with ~800 randomly sampled audio examples across languages. We also manually listened to sampled recordings to check their intelligibility and overall quality.
  • We would be happy to provide additional audio examples for inspection if helpful, and plan to release the full dataset with the final version.

In response to the questions regarding speaker metadata and group-level differences in model instability.

  • We have included a speaker attribute analysis in Appendix E.7 (Fig. 23 and Table 31), covering gender, age group, and acoustic scene.
  • We observe gender-related differences in consistency, although the direction varies across tasks. In particular, male speakers exhibit higher consistency across most model–language combinations on CVQA.
  • We find no clear evidence that instability is concentrated in any particular age group or acoustic scene. Within-group consistency differs from the overall consistency by at most 1.3 percentage points.
  • We do not draw conclusions about nationality because it is strongly confounded with language in our speaker pool. Given the small number of speakers per group. We will state the key findings in the main text.

In response to the questions regarding frontier MLLM performance and consistency.

  • Thank you for this suggestion. We additionally evaluated Gemini 3 Flash on all four tasks, covering 9 languages for MMLU, 3 for CVQA, 7 for CrossSum, and English for SAMSum, using 200 questions per language for QA, 50 articles per language for CrossSum, and 50 SAMSum dialogues, with all 10 speakers per item.

  • Gemini 3 Flash achieves the highest accuracy and consistency on both QA tasks, while Voxtral Small achieves higher BERTScore and consistency on both summarization tasks (Table below).

  • Even for Gemini 3 Flash, different speakers can still lead to different model responses, with consistency scores of 91.7 on MMLU and 46.5 on CrossSum. This suggests that evaluating consistency alongside performance remains relevant for frontier models.

Table: Performance and consistency of Gemini 3 Flash and representative open-weight models on the same evaluation subset. Scores are macro-averaged across the evaluated languages. We evaluated the open-weight models on the same subsets for comparison.

TaskDatasetMetric (%)Gemini 3 FlashQwen3-Omni 30BVoxtral Small 24B
QAMMLUAccuracy88.879.068.2
CqaC^{\mathrm{qa}}91.785.173.8
CVQAAccuracy94.576.0–
CqaC^{\mathrm{qa}}96.384.1–
SummarizationCrossSumBERTScore89.288.589.6
CsumC^{\mathrm{sum}}46.540.649.7
SAMSumBERTScore91.891.492.1
CsumC^{\mathrm{sum}}60.161.567.2

Not a critical weakness but some figures are too small (e.g., Figure 4, axes are barely visible).

  • Thank you for pointing this out. We will enlarge the axis labels and tick marks in Figure 4 and check the remaining figures at print size.

I thank the authors for their reply, and their additional results provided in the rebuttal, which further confirms my already very positive assessment of the work.

Reviewer_WgBQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces 3MSpeechBench, a multi-speaker, multilingual, multi-task speech benchmark. It holds linguistic content fixed and varies only the speaker. Four existing datasets (Global MMLU, CVQA, CrossSum, SAMSum) are re-synthesized in up to 10 languages, using 50 cloned real-speaker voices per language, with ASR- and speaker-verification-based quality control. Besides task performance, the authors define within-instance cross-speaker consistency: pairwise answer agreement for QA, and pairwise Jaccard over LLM-extracted information units for summarization. Across nine open MLLMs, they report that (1) models with similar performance can differ substantially in consistency, especially on summarization; (2) the instability is not explained by synthesis quality or a few atypical speakers, and it carries over to human speech; (3) answer shifts follow the models’ existing option preferences.

Critical strengths
  1. Controlled, large-scale, carefully validated construction. The benchmark covers 500 real-speaker voices × 10 languages × 4 tasks, including open-ended generation. Both content fidelity and speaker fidelity are verified. This makes “same content, different voice” comparisons genuinely controlled, at a scale well beyond prior work (e.g., C3T). This is the paper’s core value as a benchmark.

  2. The analysis are systematic and very interesting. The authors test synthesis quality, human vs. TTS speech, individual speakers, and the structure of the instability. This makes the claim that consistency is a separate dimension of model behavior more convincing than in typical benchmark papers.

Critical weaknesses
  1. There is no same-speaker baseline, so the instability cannot be attributed to speaker differences. The paper never measures consistency when the same speaker’s audio is re-synthesized with a different TTS seed or param. Without this baseline, speaker sensitivity cannot be separated from general sensitivity to any small acoustic perturbation. The paper’s own results actually favor the latter: instability is mostly model-driven, and human-vs-TTS agreement is similar to TTS-vs-TTS agreement. This affects how the title and main claims should be framed.

  2. The consistency metrics are not validated and can reward bias. C_qa gives 100% to a model that always outputs the same option. The key example in the paper (Kimi-Audio vs. Gemma 3n E4B) may therefore reflect Kimi-Audio’s strong preference for option C rather than robustness. C_sum relies on an LLM extraction pipeline with no human validation and no reference scale, and lead-only summaries naturally score higher. Both issues weaken the claim that consistency should be reported alongside performance.

Questions for authors
  1. Can you report model consistency across multiple TTS renditions of the same speaker and text, as baseline? How far below this baseline does cross-speaker consistency fall?
  2. Can you provide a C_qa baseline that accounts for per-item accuracy and option bias (e.g., a chance-corrected agreement measure)? Does the Kimi-Audio vs. Gemma 3n E4B comparison still hold after this correction?
3 comments
AuthorsOpenReview ↗

Reply to reviewer WgBQ (1/2)

In response to the concerns regarding consistency metric validity and sensitivity to bias.

  • We agree that a model's preference for certain answer options can lead to higher consistency at the expense of accuracy. If a model were less biased toward particular options, its consistency might decrease while its accuracy improves, illustrating why the two metrics should be considered together.

  • To examine whether our findings depend on option bias, we compute chance-corrected agreement based on each model's answer distribution.

chance corrected agreement formula: κ=Cqa−Pe1−Pe,Pe=∑k∈{A,B,C,D}pk2,\kappa = \frac{C^{\mathrm{qa}} - P_e}{1 - P_e}, \qquad P_e = \sum_{k \in \{\mathrm{A,B,C,D}\}} p_k^2, where pkp_k is the proportion of the model’s answers that select option kk, computed per model and language over all items and speakers.. PeP_e is the agreement expected if the model ignored the audio and picked options at its own rates; κ\kappa is close to 0 when the agreement comes only from such option preference.

  • Model rankings remain unchanged after correction (ρ = 1.00 on both MMLU and CVQA). Kimi-Audio also remains more consistent than Gemma 3n E4B (+8.3 points), despite their similar accuracy (52.5% vs. 52.1%).

  • For summarization, model rankings remain largely stable after excluding lead content or replacing LLM-based alignment with exact matching.

  • We will report chance-corrected agreement and include these sensitivity analyses in the revised paper.

Table: Effect of chance correction on consistency for Kimi-Audio and Gemma 3n E4B on Spoken Global MMLU. Scores are percentages. Δ denotes Kimi-Audio minus Gemma 3n E4B in percentage points.

MetricKimi-AudioGemma 3n E4BΔ
Original CqaC^{\mathrm{qa}}63.856.3+7.4
Chance-corrected κ\kappa49.941.7+8.3

We are also running additional experiments for other questions, and will share the results in this thread as they become available.

AuthorsOpenReview ↗

Reply to reviewer WgBQ (2/2)

In response to the concerns regarding the missing same-speaker baseline.

  • We thank the reviewer for suggesting this baseline and added it on MCQA and summarization tasks (MMLU and CrossSum) in four languages. For two speakers per language, we synthesized six renditions of each item (three reference clips of the same speaker × two TTS seeds) and evaluated Qwen3-Omni, Gemma 3n E4B, and Kimi-Audio. Consistency is computed between renditions that differ only in seed, only in reference clip, or in speaker.

  • Models are unstable even under the same-speaker baseline, as shown in Table 1. On the MCQA task, cross-speaker consistency falls at most 2 points below the baseline, so re-synthesis alone changes answers almost as often as a speaker change. On the summarization task, it falls by 6–8 points, more than changing only the reference clip, so speaker identity adds further instability.

  • This supports the premise of our benchmark that a single rendition of an item cannot characterize how a model responds to speech, so performance should be reported together with consistency across renditions. Cross-speaker evaluation covers realization variation and adds speaker variation, and it ranks the three models in the same order as the same-speaker baseline.

  • We will add this decomposition to the paper and frame the main claims around sensitivity to both acoustic realization and speaker identity.

Table 1: Consistency on Spoken Global MMLU (CqaC^{\mathrm{qa}}) and Spoken CrossSum (CsumC^{\mathrm{sum}}) between renditions that differ only in TTS seed (same speaker and reference clip), in reference clip (same speaker), or in speaker.

CqaC^{\mathrm{qa}}┊CsumC^{\mathrm{sum}}
Modelother seedother clipother speaker┊other seedother clipother speaker
Qwen3-Omni83.883.883.2┊43.639.336.0
Gemma 3n E4B56.456.255.2┊36.032.029.6
Kimi-Audio68.166.966.1┊46.441.938.1

In response to the concerns regarding human validation of CsumC^{\mathrm{sum}}.

  • To validate CsumC^{\mathrm{sum}}, we audited 27 CrossSum summary pairs per language in Chinese and English. For three articles per language, one pair of summaries from different speakers for each of the nine models.

  • The annotator judges whether each aligned unit pair states the same fact, with hidden control rows of unrelated units, and classifies each unmatched unit against the other summary and the article. We then correct the alignment by splitting the pairs judged different and merging the units judged to be the same fact, and recompute CsumC^{\mathrm{sum}}.

  • The aligner rarely merges different facts, and correcting its errors leaves the model ordering largely unchanged (ρ = 0.97).

  • Most unmatched units reflect complementary details or the same facts expressed differently, and the human audit finds even fewer contradictions than the LLM classification, as shown in Table 2.

Table 2: Classification of the unmatched units in the audited CrossSum pairs.

Unmatched unitLLMHuman
Complementary44.5%50.0%
Same fact, unmatched33.8%31.7%
Unsupported by article11.1%15.3%
Contradictory9.4%2.9%
Invalid unit1.3%0.2%

The dataset releases cloned voices of 500 identifiable YouTube speakers. A CC BY 4.0 license does not imply consent to voice cloning, and voice may count as biometric data under GDPR. The authors describe some mitigations, but consent and legal compliance should be reviewed before release.

  • We thank the reviewer for raising this concern and agree that a CC BY 4.0 license does not imply consent to voice cloning.

  • We will release the dataset through gated access on Hugging Face for research use only, and any speaker can request removal of their audio from the release.

  • We will not release the reference recordings or speaker embeddings, which could be used to clone the speakers' voices, and these safeguards will be described in the revised Ethics Statement.

solid experiments

Reviewer_osk3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

3MSpeechBench studies whether speech-capable multimodal models respond consistently to the same linguistic content spoken in different cloned voices. It converts four existing datasets into spoken multiple-choice QA, visual QA, article summarization, and dialogue summarization tasks, covering up to ten languages and 50 voices per language. Nine models are evaluated, generally with a selected subset of ten voices per language. In addition to task scores, the paper measures pairwise answer agreement for QA and Jaccard overlap of LLM-extracted, canonicalized information units for summaries. Analyses examine synthesis quality, speaker-level effects, response preferences, and a comparison between human recordings and TTS renditions of scientific talks.

Critical strengths
  1. Holding text fixed across multiple voices allows instance-level analysis that aggregate accuracy can hide. Extending this design across languages and to open-ended tasks is useful. The distinction between consistently correct, consistently wrong, and fragile QA items is particularly interpretable and helps avoid equating consistency with correctness.

  2. It checks content fidelity using two ASR systems, checks speaker similarity, reports parsing failures, uses greedy decoding, and analyzes speaker-level variation rather than reporting only a leaderboard. The MCIF comparison also adds evidence beyond the main TTS system. These are meaningful controls, even though they do not fully establish the stronger attribution discussed below.

Critical weaknesses
  1. Section 3.2 and Appendix B.3 rely on one model, Ministral-3-14B, for both unit extraction and canonicalization across languages. The appendix provides prompts and an example, but I did not find a human audit of extraction/alignment accuracy or evidence that rankings are stable across extractors and extraction limits. This is consequential because the main open-ended result depends on this pipeline. Jaccard overlap also decreases when two correct summaries select different complementary details; this is a valid notion of content-selection consistency, but it does not establish factual disagreement or degraded utility. The discussion of summaries stating different facts should therefore be calibrated against human judgments of contradiction, omission, and acceptable variation, rather than interpreting all unit differences alike.

  2. The main design uses one synthesized realization per text–voice pairing, while Appendix A.4 describes stochastic initial synthesis and quality-dependent regeneration. Changing voices consequently also changes uncontrolled realization properties. Low correlation with ASR error cannot rule out prosody, timing, or segmentation effects that preserve the transcript. More importantly, in the MCIF experiment each talk has one original speaker and six TTS renditions cloned from that same speaker (Section 5.1; Appendix E.5). That experiment changes the synthesis condition while approximately preserving identity; it is useful evidence of sensitivity to audio realization, but is not a within-text comparison across different human speakers. The paper's stronger speaker-induced and human-speech generalization claims therefore need either a stronger control or more careful scope.

Overall, I lean toward weak rejection. The paired evaluation resource is useful, but the distinctive open-ended measurement and the attribution of its variation are not yet sufficiently validated to support the main conclusions.

Questions for authors
  1. Can you provide a language-stratified human audit of the extracted/aligned information units and the resulting consistency ordering? How sensitive are scores and rankings to a second extractor, the unit caps, and summary length? Please separate complementary fact selection from contradictory content in this validation.

  2. How large is output variation across repeated syntheses of the same text in the same voice, compared with variation across voices? A nested text × speaker × realization control would directly address the attribution issue; greedy downstream decoding alone does not control TTS variation.

  3. Does any experiment hold the text fixed across different human speakers? If not, can the MCIF conclusions be limited to human-versus-synthetic realization robustness, rather than direct validation of cross-speaker consistency on natural recordings?

4 comments
AuthorsOpenReview ↗

Reply to reviewer osk3 (1/3)

In response to the questions regarding the scope and interpretation of the MCIF experiment.

  • We agree that the MCIF experiment does not establish cross-speaker consistency on natural recordings. Its purpose is to examine whether evaluating models on synthesized speech yields results comparable to those obtained from human recordings.

  • We use TTS renditions cloned from each talk's original speaker to minimize speaker-identity differences between the two conditions.

  • In Sec. 5.1, we observe similar BERTScore and broadly consistent model rankings between human and synthesized speech, supporting their comparability in this evaluation setting.

  • We will clarify the experiment's intended scope and revise the corresponding claims accordingly.


We are also running additional experiments for other questions, and will share the results in this thread as they become available.

AuthorsOpenReview ↗

Reply to reviewer osk3 (2/3)

In response to the concerns regarding the reliability and interpretation of summarization consistency.

  • We agree that relying on a single LLM raises concerns about evaluation reliability, and that low Jaccard overlap does not necessarily indicate factual disagreement.

  • To examine these concerns, we repeated the extraction and alignment pipeline with Claude Opus 5.5 on CrossSum in English, Chinese, and Hindi, 50 articles per language, and on 50 SAMSum dialogues, analyzed the types of information differences between summaries, and conducted a human audit on27 CrossSum summary pairs per language in Chinese and English.

  • Details of the settings:

    • Second extractor: Claude Opus 5.5 replaces Ministral-3-14B for both unit extraction and alignment, and CsumC^{\mathrm{sum}} is recomputed on the same items.
    • Reduced unit caps: the paper keeps at most 10 entities, 10 time/number expressions, and 8 propositions per summary. Here we keep only the first 4/4/3 units of each type and recompute CsumC^{\mathrm{sum}}.
    • Length-adjusted consistency: we estimate how CsumC^{\mathrm{sum}} changes with summary length within the same model and article, subtract this length effect from each score, and re-rank the models.
    • Human audit: The annotator judges whether each aligned unit pair states the same fact, with hidden control rows of unrelated units, and classifies each unmatched unit against the other summary and the article. Human-corrected alignment splits the pairs judged different and merges the units judged to be the same fact before recomputing CsumC^{\mathrm{sum}}.
  • Model rankings remain largely stable across evaluation settings in Table 1. Most unmatched information units reflect complementary details or different expressions of the same facts, rather than contradictions in Table 2. The human audit finds even fewer contradictions than the LLM classification, and correcting the alignment errors it identifies leaves the model ordering largely unchanged.

  • For example, the following summaries contain different valid details and overlapping information, but disagree on the number of migrants.

Example. Two summaries of the same CrossSum article generated from different speakers (Gemma 3n E2B).

  • Summary A

More than 2,000 migrants have reached the Italian island of Lampedusa since Saturday, raising concerns about a potential surge in arrivals ... prompting calls for greater responsibility from the European Union ...

  • Summary B

More than 1,000 migrants, including over 13,000 in Italy so far this year, have reached the southern Italian island of Lampedusa since Saturday, raising concerns about a potential surge in arrivals ... The influx has strained the island's resources ...

  • We will include these analyses in the revised paper and clarify that CsumC^{\mathrm{sum}} measures information-selection consistency rather than factual correctness.

Table 1: Robustness of CsumC^{\mathrm{sum}} model rankings under alternative evaluation settings. Entries are Spearman rank correlations against the corresponding original settings.

Evaluation settingCrossSumSAMSum
Second extractor (Claude Opus 5.5)0.980.93
Reduced unit caps (4/4/3)1.000.98
Length-adjusted consistency0.820.88
Human-corrected alignment0.97–

Table 2: Classification of the unmatched units in the audited CrossSum pairs.

Unmatched unitLLMHuman
Complementary44.5%50.0%
Same fact, unmatched33.8%31.7%
Unsupported by article11.1%15.3%
Contradictory9.4%2.9%
Invalid unit1.3%0.2%
AuthorsOpenReview ↗

Reply to reviewer osk3 (3/3)

In response to the questions regarding whether the observed instability can be attributed to speakers.

  • We thank the reviewer for suggesting this control and added it on MCQA and summarization tasks (MMLU and CrossSum) in four languages. For two speakers per language, we synthesized six renditions of each item (three reference clips of the same speaker × two TTS seeds) and evaluated Qwen3-Omni, Gemma 3n E4B, and Kimi-Audio. Consistency is computed between renditions that differ only in seed, only in reference clip, or in speaker.

  • Models are unstable even when the speaker is held fixed, as shown in Table 3. On the MCQA task, re-synthesizing the same text in the same voice changes answers almost as often as changing the speaker. On the summarization task , consistency already drops with a different seed and falls further with a different reference clip and a different speaker.

  • This supports the premise of our benchmark that, a single rendition of an item cannot characterize how a model responds to speech, so performance should be reported together with consistency across renditions. Cross-speaker evaluation covers realization variation and adds speaker variation, and it ranks the three models in the same order as the same-speaker conditions.

  • We will add this decomposition to the paper and state explicitly that cross-speaker consistency reflects both acoustic realization and speaker identity.

Table 3: Consistency on Spoken Global MMLU (CqaC^{\mathrm{qa}}) and Spoken CrossSum (CsumC^{\mathrm{sum}}) between renditions that differ only in TTS seed (same speaker and reference clip), in reference clip (same speaker), or in speaker.

CqaC^{\mathrm{qa}}┊CsumC^{\mathrm{sum}}
Modelother seedother clipother speaker┊other seedother clipother speaker
Qwen3-Omni83.883.883.2┊43.639.336.0
Gemma 3n E4B56.456.255.2┊36.032.029.6
Kimi-Audio68.166.966.1┊46.441.938.1
Area_Chair_CfctOpenReview ↗

Reminder: Discussion deadline approaching

Dear Reviewer,

Thank you for your time and effort in reviewing Submission 17524. As the discussion deadline is approaching, could you please read the authors’ response and any additional evidence, reply in the discussion to indicate whether your concerns have been addressed, and update your evaluation accordingly as soon as possible? Thank you for your contribution to the review process.

Best regards, Area Chair

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewer-Author Discussion

We thank all reviewers for their constructive and insightful feedback.

Reviewers acknowledged the following strengths of our paper:

  1. The first multilingual benchmark that holds linguistic content fixed and varies only the speaker (Reviewer VLQn).
  2. Controlled, large-scale, and carefully validated construction, at a scale well beyond prior work (Reviewer WgBQ).
  3. Instance-level analysis that aggregate accuracy can hide, with an interpretable distinction between consistently correct, consistently wrong, and fragile items (Reviewer osk3).
  4. Systematic analyses of synthesis quality, human versus synthesized speech, and speaker-level variation (Reviewers WgBQ, VLQn).

After the rebuttal, Reviewer WgBQ raised the score and described our additional analyses as “solid experiments,” and Reviewer VLQn noted that our additional results “further confirms my already very positive assessment of the work.”

During the discussion period, we provided detailed responses to concerns raised by reviewers. For each concern, we conducted additional analyses as follows:

In response to concerns about attributing instability to speakers (osk3, WgBQ):

  • We added a nested control on the MCQA and summarization tasks in four languages, synthesizing each item with three reference clips × two TTS seeds per speaker.
  • Models are unstable even when the speaker is held fixed, and changing the speaker further reduces summarization consistency by 6–8 points.
  • Model rankings stay the same across seed, reference clip, and speaker variation, supporting our premise that consistency should be reported alongside performance.

In response to concerns about the validity of the consistency metrics (osk3, WgBQ):

  • For CsumC^{\mathrm{sum}}, model rankings remain stable under a second extractor (Claude Opus 5.5), reduced unit caps, length adjustment, and lead removal (ρ = 0.82–1.00).
  • A human audit of CrossSum summary pairs in Chinese and English shows that most differences are complementary details or the same facts expressed differently. Contradictions account for only 2.9%, and rankings hold after human correction (ρ = 0.97).
  • For CqaC^{\mathrm{qa}}, chance-corrected agreement leaves model rankings unchanged, and Kimi-Audio remains more consistent than Gemma 3n E4B at similar accuracy.

In response to concerns about generality and scope (osk3, VLQn):

  • We evaluated Gemini 3 Flash on all four tasks. It achieves the highest QA accuracy but is still not fully consistent across speakers.
  • We analyzed speaker metadata, finding gender-related differences on CVQA but no concentration of instability by age group or acoustic scene.
  • We clarified that the MCIF experiment examines the comparability of human and synthesized speech.

In response to ethics concerns (WgBQ):

  • We will release the dataset through gated access for research use only, remove any speaker's audio on request, and withhold the reference recordings and speaker embeddings.

We thank all reviewers for their valuable feedback, which helped strengthen our work through additional controls and validation of our consistency metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.