acceptodds
Under review as a conference paper at ICLR 2027

Do Multi-Turn Conversational Capabilities Carry Over from Text to Speech? A Parallel Multilingual Benchmark

Abstract

Successful multi-turn voice interaction requires models to retain information from earlier turns, follow persistent instructions, and track revisions to conversational state. Yet these abilities are often evaluated from transcripts, leaving open whether successes demonstrated from text carry over when the same user turns are presented as speech, and whether this transfer is consistent across languages. We introduce a strictly parallel multilingual extension of Audio MultiChallenge to study this question under controlled conditions. The benchmark contains 354 conversations and 1,404 rubric items across ten languages, with conversational content, turn structure, cross-turn dependencies, and evaluation criteria aligned across transcript and speech conditions. Across six speech-language models, the shift from reference transcripts to direct speech input is strongly model- and language-dependent: models with similar transcript performance can diverge substantially under speech, and their relative strengths across languages can change. Paired rubric-level analysis further shows that aggregate transcript–speech gaps can obscure which conversational requirements are actually preserved, because losses of transcript-demonstrated successes can be offset by gains on other items. We then diagnose recoverable speech degradation by selectively replacing preceding user turns, the current user turn, or both with transcripts generated by the evaluated model. For the two models with the largest transcript–speech gaps, replacing preceding user turns yields a significantly larger macro-level improvement than replacing only the current turn, and the same numerical ordering holds across all ten languages; in contrast, the strongest direct-speech model gains little from either intervention. These results show that transcript performance is an unreliable proxy for multilingual spoken conversational performance and, for models most affected by speech, identify the representation of preceding spoken context as an important source of recoverable degradation.

Primary area:
datasets and benchmarks
Keywords:
Multilingual Evaluation, Multi-Turn Conversational Benchmark, Large Audio Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eYPbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends Audio MultiChallenge into a parallel benchmark of 354 conversations and 1,404 rubric items per language across ten languages, covering inference memory, instruction retention, self coherence, and voice editing. English uses human recordings; the other nine languages use localized synthetic speech. Evaluation fixes historical assistant responses as text and generates only the designated response.

Across six primary models, the paper compares reference transcripts with direct audio, examines paired rubric outcomes, and introduces a 2×2 intervention replacing historical user turns, the current turn, or both with frozen model-generated transcripts. It finds substantial model-dependent transcript–speech gaps and larger benefits from history than current-turn transcription for the two models with the largest gaps. Supporting analyses include paired bootstrap intervals, transcript-source controls, an English human/TTS comparison, and alternative judging.

Critical strengths
  • The controlled comparison demonstrates a consequential limitation of transcript-based evaluation.
  • The interventions identify substantial recoverable degradation with a reasonably careful paired design.
Critical weaknesses
  • The central claim of multilingual benchmark validity lacks reported human-validation outcomes. Appendix C.3 describes an audit of 50 conversations per language, but does not report the resulting residual defects or evidence that task-critical repairs and rubric meanings remain equivalent. For a benchmark whose main contribution is strict alignment across languages and modalities, describing the audit sample is insufficient to establish that the released examples satisfy this promise. Automated checks and the English human/TTS control are useful, but do not establish fidelity in the nine synthesized target languages. The upstream benchmark’s human validation supports the original English judging setup; it does not automatically transfer to localized rubrics. This matters because language-specific construction errors could affect both the measured modality gaps and the interpretation of language differences.

  • The paired survival and recovery analyses do not separate modality effects from ordinary response variability. RQ2 and the recovery-overlap analysis use single observed outputs without a within-modality repeated-generation baseline. Conditioning on reference success and audio failure selects cases that may succeed on another run even without an effective intervention. For illustration, if every procedure independently succeeds with probability 0.6, a new run recovers 60% of selected failures, and 57.1% of cases recovered by either of two runs are recovered by exactly one—within the paper’s reported 50–70% exclusive-recovery range. This is a counterexample to the interpretation, not an estimate of the actual models. Bootstrapping frozen outcomes or changing the judge does not measure generation variability. Appendix E acknowledges this limitation, and the observed transition counts remain valid descriptions; however, they provide weaker evidence of modality-specific capability loss or distinct recoverable failure types than the framing suggests. This limits the added insight of a central analysis, without negating the large aggregate deficits or net intervention gains.

Questions for authors
  • What did the 50-conversation-per-language human audit find on the final benchmark? Please report residual defects by language and whether spoken content, repair order, and rubric equivalence were preserved. These results would materially affect my confidence in the benchmark contribution.

  • Is repeated-generation evidence available on a representative matched subset to compare within-modality disagreement with cross-modality disagreement, and recovery from another Direct Audio run with recovery under transcription? If unavailable, can the contribution be framed explicitly around observed response pairs rather than preservation of stable per-item capabilities? A targeted control would strengthen this claim; a full rerun of every experiment is unnecessary.

Reviewer_noZKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper propose a multilingual extension of Audio MultiChallenge (Audio MC) with controlled data construction for non-English tracks. The paper analyzes the multi-turn performance mismatch of speech-language models under different presentation modality of the dialogue context.

Critical strengths

The data construction pipeline is rigorous enough. The author strictly controls the parallelism, with the task difficulties and speaker characteristics, and data where such parallelism is not achievable is filtered out. This serves as the foundation of the paper's analyses. For me, such rigorous data construction pipeline is the paper's most outstanding strength in its own right.

Critical weaknesses
  1. From my perspective, the main novelty of the paper lies in extending an existing benchmark to multilingual settings. The paper also examines how the modality of the dialogue context affects performance, in order to assess whether model capabilities transfer from text to voice interaction; however, these analyses are relatively straightforward. Moreover, the motivation for both the modality analysis and the multilingual setting is not clearly established. This is particularly evident in the discussion of language effects, which are largely described as “model-specific” without further investigation into the underlying mechanisms or possible explanations. For these reasons, I find the novelty of the paper insufficient for ICLR.
  2. Although the authors acknowledge this limitation, the comparison between replacing the dialogue history and replacing the current turn with text transcripts is not fully controlled. Since the amount of replaced audio (in terms of the number of turns or total duration) is not systematically varied, the observed differences may not purely reflect the effect of modality and could be confounded by the models’ long-context behavior. As a result, I find the current experimental design insufficient to support the conclusions drawn. One possible remedy is to replace the dialogue turn by turn, from the first turn up to the current turn, converting only one turn to text at a time. Such an analysis could help identify which parts of the context the model tracks less reliably, and would better disentangle modality effects from context-length effects.
  3. The paper may not be sufficiently reproducible as some of the setup, such as decoding strategies, is not clearly provided. I will suggest the author to provide the details to enhance reproducibility, which is especially crucial for a benchmark paper.
Questions for authors

I have listed my concerns about the novelty and experimental design in the weakness part. Please refer to the points there.

Reviewer_grr4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper checks whether multi-turn conversational capabilities demonstrated with text transcripts carry over to speech across different languages. The authors introduce a multilingual extension of Audio MultiChallenge, containing 354 conversations and 1,404 rubric items across ten languages, covering the original taxonomy of inference memory, instruction retention, self-coherence, and voice editing. They evaluate six speech-language models using matched reference transcripts and audio inputs, examining both overall performance differences and whether individual conversational requirements remain satisfied across modalities. The authors also introduce a 2 × 2 intervention that selectively replaces preceding user turns, the current turn, or both with model-generated transcripts to identify where speech-related performance losses arise.

The main finding is that transcript performance does not reliably predict spoken conversational performance. For models with the largest performance gaps, replacing preceding spoken turns with transcripts recovers substantially more performance than replacing only the current turn, suggesting that processing spoken conversation history is an important source of degradation.

Critical strengths
  1. The authors preserve conversational content, turn structure, and evaluation criteria across ten languages and both text and speech inputs, with additional translation and audio validation which is a commendable feat. Beyond these matched comparisons, the 2 × 2 transcript substitution experiments provide useful evidence about where speech-related degradation can be recovered. For example, replacing preceding spoken turns improves Muse-Spark by 20.1 points, compared with 6.2 points when replacing only the current turn (Table 5), while Qwen3.8 Omni benefits little from either intervention. These contrasting results make the analysis more informative than a simple transcript-versus-audio comparison.
Critical weaknesses
  1. There is limited novelty here beyond Audio Multichallenge. The paper largely builds on the existing benchmark, retaining its conversations, four capability categories, evaluation rubrics, and fixed-context protocol. The main extensions are multilingual localization, paired text–audio comparisons, and transcript-substitution experiments. While useful, these additions primarily extend an established evaluation framework rather than introduce a substantially new benchmark or methodology.

  2. The most compelling evidence for the importance of preceding spoken context comes from Muse-Spark and Gemini 3.1 Pro, which exhibit the largest transcript-to-audio performance drops. However, the remaining models show smaller or more language-dependent effects, and Qwen3.8 Omni gains almost nothing from transcript substitution. While the authors appropriately qualify their conclusions, the central finding that transcribing preceding conversation history is more beneficial than transcribing the current turn is primarily supported by two models, limiting its broader applicability.

Questions for authors
  1. How was the equivalence of evaluation rubrics verified across translated conversations? Could differences in linguistic ambiguity or instruction complexity affect the reported cross-language performance gaps?

  2. Appendix C is describing checks to preserve the content and order of self-corrections during translation and synthesis. However, has the validation examined whether TTS also preserves the natural timing, hesitation, and prosody of these corrections? If not, could differences between synthetic and spontaneous speech affect the difficulty of the Voice Editing tasks?

  3. The authors indicate removing the 42 AudioMC conversations involving background sounds. Since these tasks specifically require information unavailable in transcripts, does excluding them make the text-to-speech transfer problem artificially easier? How do we think the main conclusions change if such audio-dependent memory tasks were included?

  4. Curious if the transcript-to-audio performance gaps increase with the number of turns between an important instruction or correction and the response that depends on it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.