- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors investigates whether instruction-following (IF) SpeechLLMs can be trained using only Automatic Speech Recognition (ASR) data via Sequence-Level Knowledge Distillation (SeqKD), thereby eliminating the need for expensive speech-instruction tuning datasets. Using a projection-based architectur, the authors compare Prompt-Free SeqKD (PF-SeqKD) and Prompt-Conditioned SeqKD (PC-SeqKD).
Critical strengths
- While prior works have used SeqKD to align speech encoders with LLMs, they largely overlooked its detrimental effect on verbatim speech recognition. Through a manual error analysis of 166 sentences, the authors convincingly demonstrate that SeqKD's high WER stems from semantic paraphrasing rather than acoustic misrecognition.
- The paper features comprehensive ablations across data mixing ratios, prompt pool sizes, adapter capacities, prompt formats , and LLM backbones.
Critical weaknesses
- In Tables 1 and 2, the WER for the Cascade baseline (which uses the exact same
canary-1b-v2model as the ASR front-end) is omitted ("–"). Without the standalone WER ofcanary-1b-v2on these test sets, it is impossible to assess how much recognition accuracy the proposed SpeechLLM loses (or gains) relative to its own frozen speech encoder, or why the cascade underperforms on ST and SQA. Additionally, SQA is evaluated almost exclusively on a single benchmark (MCIF) using rescaled BERTScore. For open-ended QA and instruction following, lexical/embedding overlap metrics like BERTScore correlate poorly with actual answer correctness and instruction adherence compared to LLM-as-a-judge evaluations. - The title and abstract claim that ASR data is all that is needed to build instruction-following SpeechLLMs, but the empirical validation is restricted to three tasks: ASR, ST, and short-context SQA. Broader instruction-following capabilities—such as summarization (which is present in MCIF and evaluated for text LLMs in Appendix G, Table 5, but omitted for the SpeechLLMs), formatting constraints, or multi-turn dialogue—are not evaluated. Moreover, the ASR prompt pools and DeSTA 2.5 prompts are entirely in English. It is unclear how cross-lingual pairing (e.g., English prompts with Spanish/Portuguese audio and transcripts) was handled during teacher target generation and training, which could introduce language-mismatch artifacts.
Questions for authors
- What is the exact WER of the standalone
canary-1b-v2model (used in the Cascade baseline) on the evaluated EN and ES ASR benchmarks (MCIF, Europarl, Common Voice)? Including these numbers in Tables 1 and 2 is essential to contextualize the adapter's ASR performance. - During data construction for Spanish (and Portuguese) samples in PC-SeqKD and ASR augmentation, were the English prompts from DeSTA 2.5 and Appendix B used directly alongside Spanish/Portuguese transcripts? If an English prompt was fed with a Spanish transcript to the teacher LLM in PC-SeqKD, did the teacher respond in English or Spanish, and how did this affect Spanish ASR/ST performance?
- In Appendix G, candidate text LLMs are evaluated on MCIF Summarization (
SUM-BERTS.), why was Summarization excluded from the main SpeechLLM evaluations?
5 comments
BERTScore vs LLM-as-a-judge evaluation.
For the evaluation, we have used the MCIF benchmark and its own evaluation library, which computes the BERTScore for the SQA task. We thank the reviewer for the suggestion, and we have now extended the evaluation using an LLM-as-a-judge. Below, we report the results of our models and the Cascade baseline as reference, with the BERTScore for comparison.
| Method | p | k | MCIF EN | MCIF IT | MCIF DE | MCIF ZH | LibriSQA | AVG LLM-as-a-judge | AVG BertScore |
|---|---|---|---|---|---|---|---|---|---|
| Cascade | - | - | 3.5364 | 3.0636 | 3.2182 | 3.1591 | 4.4011 | 3.4757 | 0.2750 |
| PF-SeqKD Baseline | - | - | 3.4545 | 3.3909 | 3.2591 | 3.3727 | 4.2973 | 3.5549 | 0.3266 |
| PC-SeqKD Baseline | - | - | 3.3773 | 3.1182 | 3.2409 | 3.2818 | 4.0573 | 3.4151 | 0.2810 |
| PF-SeqKD - Finetuning | 5 | 20 | 3.1591 | 2.9273 | 3.0409 | 3.1773 | 3.9370 | 3.2483 | 0.3041 |
| 5 | 200 | 3.1864 | 2.9273 | 2.8727 | 3.1000 | 3.8985 | 3.1970 | 0.2979 | |
| 10 | 20 | 3.1409 | 2.8909 | 2.9409 | 3.2182 | 3.9481 | 3.2278 | 0.3043 | |
| 10 | 200 | 3.2273 | 2.9500 | 2.8773 | 3.1455 | 3.8779 | 3.2156 | 0.2973 | |
| PC-SeqKD - Finetuning | 5 | 20 | 3.1045 | 2.7409 | 2.9545 | 3.0636 | 3.8534 | 3.1434 | 0.3032 |
| 5 | 200 | 3.1273 | 2.6864 | 2.8909 | 3.0909 | 3.9229 | 3.1437 | 0.2934 | |
| 10 | 20 | 2.9727 | 2.7909 | 2.9273 | 3.0955 | 3.9500 | 3.1473 | 0.2893 | |
| 10 | 200 | 3.1455 | 2.9000 | 3.0364 | 3.2545 | 3.9733 | 3.2619 | 0.3013 | |
| PF-SeqKD (Our Approach - SeqKD+ASR+Prompt-Conditioned samples) | 5 | 20 | 3.4227 | 3.2000 | 3.2909 | 3.2864 | 4.1107 | 3.4621 | 0.336 |
| 5 | 200 | 3.2818 | 3.1500 | 3.2682 | 3.2364 | 4.1042 | 3.4081 | 0.321825 | |
| 10 | 20 | 3.2864 | 3.2045 | 3.1909 | 3.2682 | 4.1080 | 3.4116 | 0.3442 | |
| 10 | 200 | 3.3773 | 3.1636 | 3.2136 | 3.2500 | 4.1019 | 3.4213 | 0.329225 | |
| PC-SeqKD (SeqKD+ASR+Prompt-Conditioned samples) | 5 | 20 | 3.2455 | 3.0364 | 3.1727 | 3.2955 | 4.0416 | 3.3583 | 0.28365 |
| 5 | 200 | 3.3000 | 3.2045 | 3.1000 | 3.3727 | 4.0481 | 3.4051 | 0.2847 | |
| 10 | 20 | 3.3273 | 2.9818 | 3.1182 | 3.3727 | 4.0462 | 3.3692 | 0.2755 | |
| 10 | 200 | 3.3136 | 3.0818 | 3.2455 | 3.3727 | 3.9893 | 3.4006 | 0.2784 |
The scores are in line with the BERTScore indications, with our proposed approach being on par with the PF-SeqKD Baseline and Cascade, confirming our findings.
Action: We have added the LLM-as-a-judge scores in Appendix J to confirm that our findings hold also with it.
Weakness 2
Question 3 – Missing MCIF Summarization scores.
The MCIF results reported in Appendix I and in the main paper refer to two different task variants of the benchmark. MCIF features SHORT (without the summarization task) and LONG (with the summarization task) variants. The source text is available only for the LONG variant and was therefore used for our experiments reported in Appendix I for text-only models. The SpeechLLM results instead refer to the SHORT-form variant of MCIF, as SpeechLLMs commonly do not have long-form capabilities.
Action: We have clarified the distinction between the two MCIF variants in Appendix I (lines 1671-1673).
Multi-turn dialog and formatting constraints not evaluated.
Regarding the multi-turn dialogue, the SOTA models on this task are full-duplex models, which are beyond the scope of our analysis. However, we have extended our evaluation with experiments on intent classification using the SLURP dataset, asking the models to generate the output in JSON format. All the models, including external models, baselines and our models, always produce correctly formatted JSONs, with the only exception of one of our 2-stage models that produces 1 invalid JSON out of 2974 samples (0.03%) and Qwen3Omni and GPT-Audio-1.5 that produce invalid JSONs in, respectively, 3% and 0.17% of the samples. All our models, including baselines and 2-stage models, also produce valid intents (i.e., generate one intent among the 60 possible ones, provided in the prompt) in more than 99% of the cases, unlike Phi-4, which hallucinates intents in 18% of the cases. Below, we report the accuracy values, which we have included in Tables 1 and 2.
| Method | p | k | Accuracy |
|---|---|---|---|
| Qwen 3 Omni | 0.7451 | ||
| Phi-4 | 0.1846 | ||
| GPT-Audio-1.5 | 0.7532 | ||
| Cascade | - | - | 0.6318 |
| PF-SeqKD Baseline | - | - | 0.5898 |
| PC-SeqKD Baseline | - | - | 0.4331 |
| PF-SeqKD - Finetuning | 5 | 20 | 0.4173 |
| 5 | 200 | 0.4479 | |
| 10 | 20 | 0.4650 | |
| 10 | 200 | 0.4576 | |
| PC-SeqKD - Finetuning | 5 | 20 | 0.3971 |
| 5 | 200 | 0.4523 | |
| 10 | 20 | 0.4768 | |
| 10 | 200 | 0.4879 | |
| PF-SeqKD (Our Approach - SeqKD+ASR+Prompt-Conditioned samples) | 5 | 20 | 0.5992 |
| 5 | 200 | 0.6029 | |
| 10 | 20 | 0.5810 | |
| 10 | 200 | 0.5736 | |
| PC-SeqKD (SeqKD+ASR+Prompt-Conditioned samples) | 5 | 20 | 0.4734 |
| 5 | 200 | 0.4802 | |
| 10 | 20 | 0.4953 | |
| 10 | 200 | 0.5044 |
Our models demonstrate capable of performing this additional task, besides respecting the instructions. However, Qwen3-Omni emerges as the best model on the task by a large margin and also the cascade system is superior to our models. Phi-4 is instead the worst model by far with only 18.46% of accuracy. The trend across our model is consistent with the one observed for the other tasks: PF-SeqKD outperforms PC-SeqKD and our single-stage approach maintains similar scores to PF-SeqKD, unlike the models trained with a 2-stage strategy (finetuning). Overall, this confirms our conclusions: sequence KD effectively transfers a text LLM’s linguistic and instruction-following capabilities to a corresponding SpeechLLM and our proposed methodology solves its weak speech recognition ability while maintaining other capabilities.
Action: Table 1 and 2 now include the additional intent classification task, which are discussed in Section 5. We included a new subsection 5.4 that contains this analysis on formatting instruction-following capabilities.
Question 2a – Were the English-only DeSTA 2.5/Appendix B prompts used alongside Spanish/Portuguese transcripts?
The ASR prompts reported in Appendix B are all in English and are paired with both Spanish and English audio. For such ASR samples, the teacher does not generate the target of our training data, which is always the reference transcription in the audio language. The DeSTA 2.5 prompts used to build the prompt-conditioned dataset, instead, cover 22 languages, not only English. However, most of the prompts (70%) are in English, while 3% of the prompts are in Spanish. While generating the teacher’s outputs, the prompts were sampled uniformly from the DeSTA 2.5 pool regardless of the audio language.
Question 2b – If an English prompt is fed with a Spanish transcript in PC-Seq-KD, did the teacher respond in English or Spanish?.
We analysed the teacher LLM outputs to understand the language of the outputs. When the teacher received a Spanish transcript with a prompt in another language, it answered in the language of the prompt in 79% of cases. If we fed an English transcript with a prompt in another language, it followed the prompt language in 96% of the cases.
We thank the reviewer for their comments. We provide below detailed responses to each of them.
Weakness 1
Question 1 – Canary ASR.
We originally omitted these scores because Canary-1B-v2 is an ASR/ST dedicated model with no instruction-following abilities, which is the focus of our work.
Following the reviewers' advice, we computed the scores of Canary on ASR and ST. For ST, Canary supports only a subset of the language directions that we have evaluated: it supports only EN->X and X->EN, where X is one of the EU languages, while our experiments include translation into Chinese and non-English-centric directions (e.g., ES->DE and ES->IT). For this reason, for Canary we report the average scores excluding those directions, so the comparison with other systems looking only at the average scores is not entirely fair (we noted this with a footnote).
Canary-1B-v2 obtains 8.98% WER on EN and 5.92% on ES on the same test sets, as reported now in Tables 1 and 2. These are lower than those of all evaluated SpeechLLMs, including the external models, most likely due to the fact that Canary is a dedicated ASR model. Our best SeqKD+ASR+Prompt-Conditioned SpeechLLMs (p=10%, k=200 for EN; p=5%, k=20 for ES) show a gap of 1.44 and 3.38 points for English and Spanish, respectively. In ST, comparing Canary and our system on the same subset of languages shows a small gap in favor of Canary: 0.8368 vs 0.8137 for EN → X and 0.8478 vs 0.8359 for ES → X, with a similar gap in favor of Canary when compared to the cascade Canary+Qwen4B. In exchange, our models, as well as the other SpeechLLMs and the Cascade, gain the ability to follow instructions across tasks and languages and a much wider coverage of ST language pairs.
Action: We have added Canary-1B-v2’s WER and COMET scores to Tables 1 and 2, clarifying that for ST, the language coverage is limited.
Single Benchmark for SQA.
We chose MCIF for SQA because it allows us to evaluate not only SQA performance but also the multilingual abilities of our SpeechLLMs.
MCIF provides questions in four languages (English, German, Italian, and Chinese), three of which are unseen during our training. Moreover, it is out-of-domain with respect to our training data. To accommodate the reviewer's suggestion, we introduced LibriSQA as a second SQA benchmark. LibriSQA is an English benchmark based on read speech (Librispeech) from a domain not covered by our training data. The following table reports the results, which are significantly higher than MCIF’s ones, due to the simpler domain. On average, our approach maintains the SQA performance close to the PF-SeqKD baseline and Phi-4-Multimodal, superior to other baselines.
Action: We included the results obtained with LibriSQA in the paper (Table 1, Table 2, and Appendix H).