acceptodds
Under review as a conference paper at ICLR 2027

ASR Data is All You Need to Build Instruction-Following SpeechLLMs

Abstract

SpeechLLMs can follow instructions like text LLMs by connecting a pretrained speech encoder to an LLM through an adapter. Current approaches typically rely on a two-stage training procedure including fine-tuning on instruction-following (IF) speech data, which is expensive to collect. In this work, we investigate whether IF speech data can be avoided by using Sequence-Level Knowledge Distillation (SeqKD) from a pretrained LLM operating on automatic speech recognition (ASR) data, exploring both prompt-conditioned SeqKD, where the same prompt is provided to the student and teacher, and prompt-free SeqKD, where no prompt is provided. With experiments on three tasks (ASR, speech translation, and spoken question answering) and four benchmarks including text instructions and outputs in languages unseen during training, we show that SeqKD effectively transfers the LLM's linguistic and IF capabilities but suffers from poor ASR performance. We address this limitation with a single-stage training strategy combining prompt-free SeqKD with limited ASR and prompt-conditioned data. Our approach outperforms a cascade of ASR and LLM and a two-stage approach, and remains competitive across tasks with SpeechLLMs trained on substantially larger amounts of data.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
SpeechLLM, Multimodal Model, Sequence Level Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7hhmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigates whether instruction-following (IF) SpeechLLMs can be trained using only Automatic Speech Recognition (ASR) data via Sequence-Level Knowledge Distillation (SeqKD), thereby eliminating the need for expensive speech-instruction tuning datasets. Using a projection-based architectur, the authors compare Prompt-Free SeqKD (PF-SeqKD) and Prompt-Conditioned SeqKD (PC-SeqKD).

Critical strengths
  • While prior works have used SeqKD to align speech encoders with LLMs, they largely overlooked its detrimental effect on verbatim speech recognition. Through a manual error analysis of 166 sentences, the authors convincingly demonstrate that SeqKD's high WER stems from semantic paraphrasing rather than acoustic misrecognition.
  • The paper features comprehensive ablations across data mixing ratios, prompt pool sizes, adapter capacities, prompt formats , and LLM backbones.
Critical weaknesses
  • In Tables 1 and 2, the WER for the Cascade baseline (which uses the exact same canary-1b-v2 model as the ASR front-end) is omitted ("–"). Without the standalone WER of canary-1b-v2 on these test sets, it is impossible to assess how much recognition accuracy the proposed SpeechLLM loses (or gains) relative to its own frozen speech encoder, or why the cascade underperforms on ST and SQA. Additionally, SQA is evaluated almost exclusively on a single benchmark (MCIF) using rescaled BERTScore. For open-ended QA and instruction following, lexical/embedding overlap metrics like BERTScore correlate poorly with actual answer correctness and instruction adherence compared to LLM-as-a-judge evaluations.
  • The title and abstract claim that ASR data is all that is needed to build instruction-following SpeechLLMs, but the empirical validation is restricted to three tasks: ASR, ST, and short-context SQA. Broader instruction-following capabilities—such as summarization (which is present in MCIF and evaluated for text LLMs in Appendix G, Table 5, but omitted for the SpeechLLMs), formatting constraints, or multi-turn dialogue—are not evaluated. Moreover, the ASR prompt pools and DeSTA 2.5 prompts are entirely in English. It is unclear how cross-lingual pairing (e.g., English prompts with Spanish/Portuguese audio and transcripts) was handled during teacher target generation and training, which could introduce language-mismatch artifacts.
Questions for authors
  1. What is the exact WER of the standalone canary-1b-v2 model (used in the Cascade baseline) on the evaluated EN and ES ASR benchmarks (MCIF, Europarl, Common Voice)? Including these numbers in Tables 1 and 2 is essential to contextualize the adapter's ASR performance.
  2. During data construction for Spanish (and Portuguese) samples in PC-SeqKD and ASR augmentation, were the English prompts from DeSTA 2.5 and Appendix B used directly alongside Spanish/Portuguese transcripts? If an English prompt was fed with a Spanish transcript to the teacher LLM in PC-SeqKD, did the teacher respond in English or Spanish, and how did this affect Spanish ASR/ST performance?
  3. In Appendix G, candidate text LLMs are evaluated on MCIF Summarization (SUM-BERTS.), why was Summarization excluded from the main SpeechLLM evaluations?
5 comments

We thank the reviewer for their comments. We provide below detailed responses to each of them.

Weakness 1

Question 1 – Canary ASR.

We originally omitted these scores because Canary-1B-v2 is an ASR/ST dedicated model with no instruction-following abilities, which is the focus of our work.

Following the reviewers' advice, we computed the scores of Canary on ASR and ST. For ST, Canary supports only a subset of the language directions that we have evaluated: it supports only EN->X and X->EN, where X is one of the EU languages, while our experiments include translation into Chinese and non-English-centric directions (e.g., ES->DE and ES->IT). For this reason, for Canary we report the average scores excluding those directions, so the comparison with other systems looking only at the average scores is not entirely fair (we noted this with a footnote).

Canary-1B-v2 obtains 8.98% WER on EN and 5.92% on ES on the same test sets, as reported now in Tables 1 and 2. These are lower than those of all evaluated SpeechLLMs, including the external models, most likely due to the fact that Canary is a dedicated ASR model. Our best SeqKD+ASR+Prompt-Conditioned SpeechLLMs (p=10%, k=200 for EN; p=5%, k=20 for ES) show a gap of 1.44 and 3.38 points for English and Spanish, respectively. In ST, comparing Canary and our system on the same subset of languages shows a small gap in favor of Canary: 0.8368 vs 0.8137 for EN → X and 0.8478 vs 0.8359 for ES → X, with a similar gap in favor of Canary when compared to the cascade Canary+Qwen4B. In exchange, our models, as well as the other SpeechLLMs and the Cascade, gain the ability to follow instructions across tasks and languages and a much wider coverage of ST language pairs.

Action: We have added Canary-1B-v2’s WER and COMET scores to Tables 1 and 2, clarifying that for ST, the language coverage is limited.

Single Benchmark for SQA.

We chose MCIF for SQA because it allows us to evaluate not only SQA performance but also the multilingual abilities of our SpeechLLMs.
MCIF provides questions in four languages (English, German, Italian, and Chinese), three of which are unseen during our training. Moreover, it is out-of-domain with respect to our training data. To accommodate the reviewer's suggestion, we introduced LibriSQA as a second SQA benchmark. LibriSQA is an English benchmark based on read speech (Librispeech) from a domain not covered by our training data. The following table reports the results, which are significantly higher than MCIF’s ones, due to the simpler domain. On average, our approach maintains the SQA performance close to the PF-SeqKD baseline and Phi-4-Multimodal, superior to other baselines.

ModelpkAVG MCIFLibriSQAAVG SQA
Qwen-3-Omni--0.28650.50730.3307
Phi-4-Multimodal--0.35120.54030.3890
GPT Audio 1.5--0.28770.56500.3431
Cascade--0.27500.59890.3398
PF-SeqKD Baseline--0.32660.59620.3805
PC-SeqKD Baseline--0.25510.55640.3361
PF-SeqKD - Finetuning5200.30410.55950.3552
52000.29790.54910.3481
10200.30430.55540.3545
102000.29730.54390.3466
PC-SeqKD - Finetuning5200.30320.54540.3516
52000.29340.55790.3463
10200.28930.55490.3424
102000.30130.55880.3528
PF-SeqKD (Our Approach - SeqKD+ASR+Prompt-Conditioned samples)5200.33600.56990.3828
52000.32180.56460.3704
10200.34420.56990.3893
102000.32920.57240.3779
PC-SeqKD (SeqKD+ASR+Prompt-Conditioned samples)5200.283650.56000.3389
52000.28470.55760.3393
10200.27550.56060.3325
102000.27840.56010.3348

Action: We included the results obtained with LibriSQA in the paper (Table 1, Table 2, and Appendix H).

BERTScore vs LLM-as-a-judge evaluation.

For the evaluation, we have used the MCIF benchmark and its own evaluation library, which computes the BERTScore for the SQA task. We thank the reviewer for the suggestion, and we have now extended the evaluation using an LLM-as-a-judge. Below, we report the results of our models and the Cascade baseline as reference, with the BERTScore for comparison.

MethodpkMCIF ENMCIF ITMCIF DEMCIF ZHLibriSQAAVG LLM-as-a-judgeAVG BertScore
Cascade--3.53643.06363.21823.15914.40113.47570.2750
PF-SeqKD Baseline--3.45453.39093.25913.37274.29733.55490.3266
PC-SeqKD Baseline--3.37733.11823.24093.28184.05733.41510.2810
PF-SeqKD - Finetuning5203.15912.92733.04093.17733.93703.24830.3041
52003.18642.92732.87273.10003.89853.19700.2979
10203.14092.89092.94093.21823.94813.22780.3043
102003.22732.95002.87733.14553.87793.21560.2973
PC-SeqKD - Finetuning5203.10452.74092.95453.06363.85343.14340.3032
52003.12732.68642.89093.09093.92293.14370.2934
10202.97272.79092.92733.09553.95003.14730.2893
102003.14552.90003.03643.25453.97333.26190.3013
PF-SeqKD (Our Approach - SeqKD+ASR+Prompt-Conditioned samples)5203.42273.20003.29093.28644.11073.46210.336
52003.28183.15003.26823.23644.10423.40810.321825
10203.28643.20453.19093.26824.10803.41160.3442
102003.37733.16363.21363.25004.10193.42130.329225
PC-SeqKD (SeqKD+ASR+Prompt-Conditioned samples)5203.24553.03643.17273.29554.04163.35830.28365
52003.30003.20453.10003.37274.04813.40510.2847
10203.32732.98183.11823.37274.04623.36920.2755
102003.31363.08183.24553.37273.98933.40060.2784

The scores are in line with the BERTScore indications, with our proposed approach being on par with the PF-SeqKD Baseline and Cascade, confirming our findings.

Action: We have added the LLM-as-a-judge scores in Appendix J to confirm that our findings hold also with it.

Weakness 2

Question 3 – Missing MCIF Summarization scores.

The MCIF results reported in Appendix I and in the main paper refer to two different task variants of the benchmark. MCIF features SHORT (without the summarization task) and LONG (with the summarization task) variants. The source text is available only for the LONG variant and was therefore used for our experiments reported in Appendix I for text-only models. The SpeechLLM results instead refer to the SHORT-form variant of MCIF, as SpeechLLMs commonly do not have long-form capabilities.

Action: We have clarified the distinction between the two MCIF variants in Appendix I (lines 1671-1673).

Multi-turn dialog and formatting constraints not evaluated.

Regarding the multi-turn dialogue, the SOTA models on this task are full-duplex models, which are beyond the scope of our analysis. However, we have extended our evaluation with experiments on intent classification using the SLURP dataset, asking the models to generate the output in JSON format. All the models, including external models, baselines and our models, always produce correctly formatted JSONs, with the only exception of one of our 2-stage models that produces 1 invalid JSON out of 2974 samples (0.03%) and Qwen3Omni and GPT-Audio-1.5 that produce invalid JSONs in, respectively, 3% and 0.17% of the samples. All our models, including baselines and 2-stage models, also produce valid intents (i.e., generate one intent among the 60 possible ones, provided in the prompt) in more than 99% of the cases, unlike Phi-4, which hallucinates intents in 18% of the cases. Below, we report the accuracy values, which we have included in Tables 1 and 2.

MethodpkAccuracy
Qwen 3 Omni0.7451
Phi-40.1846
GPT-Audio-1.50.7532
Cascade--0.6318
PF-SeqKD Baseline--0.5898
PC-SeqKD Baseline--0.4331
PF-SeqKD - Finetuning5200.4173
52000.4479
10200.4650
102000.4576
PC-SeqKD - Finetuning5200.3971
52000.4523
10200.4768
102000.4879
PF-SeqKD (Our Approach - SeqKD+ASR+Prompt-Conditioned samples)5200.5992
52000.6029
10200.5810
102000.5736
PC-SeqKD (SeqKD+ASR+Prompt-Conditioned samples)5200.4734
52000.4802
10200.4953
102000.5044

Our models demonstrate capable of performing this additional task, besides respecting the instructions. However, Qwen3-Omni emerges as the best model on the task by a large margin and also the cascade system is superior to our models. Phi-4 is instead the worst model by far with only 18.46% of accuracy. The trend across our model is consistent with the one observed for the other tasks: PF-SeqKD outperforms PC-SeqKD and our single-stage approach maintains similar scores to PF-SeqKD, unlike the models trained with a 2-stage strategy (finetuning). Overall, this confirms our conclusions: sequence KD effectively transfers a text LLM’s linguistic and instruction-following capabilities to a corresponding SpeechLLM and our proposed methodology solves its weak speech recognition ability while maintaining other capabilities.

Action: Table 1 and 2 now include the additional intent classification task, which are discussed in Section 5. We included a new subsection 5.4 that contains this analysis on formatting instruction-following capabilities.

Question 2a – Were the English-only DeSTA 2.5/Appendix B prompts used alongside Spanish/Portuguese transcripts?

The ASR prompts reported in Appendix B are all in English and are paired with both Spanish and English audio. For such ASR samples, the teacher does not generate the target of our training data, which is always the reference transcription in the audio language. The DeSTA 2.5 prompts used to build the prompt-conditioned dataset, instead, cover 22 languages, not only English. However, most of the prompts (70%) are in English, while 3% of the prompts are in Spanish. While generating the teacher’s outputs, the prompts were sampled uniformly from the DeSTA 2.5 pool regardless of the audio language.

Question 2b – If an English prompt is fed with a Spanish transcript in PC-Seq-KD, did the teacher respond in English or Spanish?.

We analysed the teacher LLM outputs to understand the language of the outputs. When the teacher received a Spanish transcript with a prompt in another language, it answered in the language of the prompt in 79% of cases. If we fed an English transcript with a prompt in another language, it followed the prompt language in 96% of the cases.

Reviewer_FZyQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether instruction-following SpeechLLMs can be trained using existing ASR corpora as the source of speech supervision. A frozen text LLM generates targets from transcripts, and a speech encoder–adapter–LLM model learns to reproduce these targets from audio. The authors compare prompt-free and prompt-conditioned sequence-level distillation and identify a trade-off between instruction following and faithful transcription. Their proposed single-stage strategy combines prompt-free distillation with limited ASR supervision and prompt-conditioned examples. Experiments show substantial ASR improvements while largely retaining translation and spoken question-answering performance.

Critical strengths
  1. The paper addresses a practical question: how to reuse existing speech–transcript datasets to reduce dependence on separately collected speech instruction data. Training only the adapter makes the approach accessible for adapting pretrained components.

  2. The analysis of the tension between semantic response generation and verbatim transcription is useful. The comparisons across data proportions, prompt-pool sizes, adapter sizes, and an additional LLM backbone provide evidence that the proposed mixture improves this balance in the evaluated settings.

Critical weaknesses
  1. The suitability of ASR transcripts for constructing useful instruction-following supervision is insufficiently examined. The main training signal comes from prompt-free distillation, where a teacher LLM responds to a transcript without an explicit task instruction. However, ASR corpora are collected for transcription rather than conversational or instruction-following relevance. They may contain isolated statements, incomplete utterances, or context-dependent fragments that do not naturally elicit informative responses. Teacher targets for such inputs could be generic, repetitive, or dependent on unsupported assumptions. The issue is not that declarative sentences are inherently unsuitable, but that useful supervision cannot be assumed from transcript availability alone. The paper does not sufficiently characterize teacher-target quality, task diversity, or sensitivity to transcript content. This makes it difficult to assess how broadly the proposed recipe applies beyond the selected corpora.

  2. The advantage over a direct synthetic speech-instruction pipeline is not established. A natural alternative is to first generate task-appropriate text instructions, inputs, and responses with an LLM, and then synthesize the spoken inputs using TTS. This also avoids manually collecting speech instruction–response data, while allowing direct control over task coverage and content. Existing ASR recordings may provide more realistic acoustic variability, but the paper does not evaluate whether this advantage outweighs the restrictions imposed by their textual content. A matched-budget comparison using the same student architecture and comparable training resources is missing. Since sequence-level distillation is already established and the proposed method primarily modifies the training-data mixture, this missing comparison substantially limits the evidence for the significance of the contribution.

Questions for authors

What proportion of prompt-free teacher targets provides meaningful responses, rather than generic acknowledgments, restatements, or unsupported continuations? How does target quality vary across complete sentences, fragments, and context-dependent utterances?

How sensitive is performance to the semantic content of the ASR corpus? Does selecting transcripts that support informative teacher responses improve results relative to using the original mixture?

3 comments

We thank the reviewer for their comments and for recognizing the practical relevance of our work. We appreciate that the reviewer found the paper clear and easy to follow, and raised no substantial concerns about its soundness. We understand that the only concerns relate to the quality of teacher-generated targets and the comparison with TTS-based pipelines. Below, we clarify some aspects of our contribution, hoping they will be useful.

Weakness 1

Question 1 – Quality and semantic content of the ASR corpus.

In PF-SeqKD, the teacher targets are not meant to be informative like instruction-tuning data. The goal is to represent the LLM's native behaviour given the transcript and to align the audio modality with the text modality in a way that the LLM backbone can perceive speech as well as text. The LLM itself is already able to perform many tasks, including Textual Question Answering and Machine Translation, and what we want to train is an adapter that can correctly map audio into the LLM’s space, unlocking these capabilities when the input is speech instead of text.
Our results support this view: PF-SeqKD outperforms PC-SeqKD on SQA, even though PC-SeqKD targets are generated using prompts that are more task-oriented compared to the prompt-free generation. This is reflected also in our tests on out-of-domain test sets (MCIF). Our SpeechLLMs achieve good results, even though no in-domain data is used during training.

A practical consequence is that our approach does not require selecting or filtering transcripts based on how informative the teacher’s responses are. The only requirement regarding the application of this methodology to data is the correct alignment between text and speech data used for training, i.e., the quality of the underlying ASR corpora.

To provide insights on how much the teacher outputs are related and informative with respect to the source transcript, we added a dedicated analysis in Appendix A. We compared the teacher perplexity on the generated output both prepending the transcript (as in our generation phase) and without it. As shown in Figure 4 reported in the new version of the paper, the perplexity of the generated text is not reduced by conditioning it on the transcripts in 0.0043% of the cases, showing that the generated text is related to and depends on the corresponding transcript.

Action: We have clarified this aspect in the paper in Sections 3 (lines 149-157) and 5.1 (lines 289-290), and included in Appendix A an analysis of the generated outputs.

Weakness 2

Comparison with Synthetic Speech-Instructed Dataset.

The primary goal of this paper is to evaluate whether existing ASR datasets can be repurposed to train an instruction-following SpeechLLM exploiting Sequence-Level Knowledge Distillation. A TTS-based pipeline addresses a complementary research question: whether synthetic speech can be used to replace real data to train an instruction-following SpeechLLM. This research question has been the focus of other works in the field: [1] showed that synthetic speech alone provides weaker supervision than a real speech dataset in a setting which is similar to the one used in our paper (a projector-based SpeechLLM).

More broadly, recent work shows that synthetic speech is not a proxy for real speech: systems trained on TTS audio behave differently, and often worse, when exposed to real recordings [2].
The gap between TTS audio and real speech is due to the fact that synthetic speech does not cover the full variability of real speech, and it can introduce artifacts absent in real recordings and cannot generate parts of the real data distribution [3].

Action: We have discussed this in Section 1 (lines 62-66), clarifying why we did not include these comparisons in our experiments.

Question 2 – Training on a filtered dataset.

As shown in the previous analysis, a negligible fraction of samples contain text that is not related to the transcript content. As such, the impact of filtering the dataset in this way would be minimal. However, we agree with the reviewer that a data selection strategy tailored for sequence KD data is an interesting topic and direction for future work. An extensive work on this cannot be scoped within this paper, as it would require removing other content from it.

Action: We have acknowledged this future line of work in the Limitations section in Appendix N (lines 2089-2090).

In light of our clarifications and revisions (and also of the good soundness and presentation scores given to our paper), we kindly request the reviewer to reconsider their overall rating, which we believe is unduly penalising our work.

We thank again the Reviewer for their feedback but we wanted to follow up on our rebuttal and revised manuscript, as we have not yet received a response. Please let us know if our updates address your concerns or if any further clarification would be helpful before the discussion period ends.

Reviewer_8JbAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to adapt a frozen speech encoder and LLM for instruction-following speech tasks using ASR data as the source of speech supervision. It compares prompt-free and prompt-conditioned sequence-level knowledge distillation, then combines prompt-free targets with limited transcription and prompt-conditioned examples in a single training stage. Experiments on ASR, speech translation, and spoken question answering show substantial transcription improvements while retaining downstream capabilities. Additional adapter-size and LLM-backbone experiments support the practical training recipe.

Critical strengths
  1. The paper identifies a useful practical limitation of distillation: semantic task performance can coexist with poor reproduction of the words actually spoken. The proposed mixture substantially improves WER while retaining useful translation and question-answering performance, making the result relevant to building speech interfaces with limited task-specific supervision.

  2. The comparisons include single-stage and two-stage training, two adapter sizes, and a second LLM backbone. This evidence makes the result more persuasive than a single successful configuration and supports the value of the recipe within the evaluated settings.

Critical weaknesses
  1. The contribution is primarily a practical training recipe, and inconsistencies in the main comparison make its precise advantage harder to assess. For p=10%, k=200, Table 2 reports COMET scores of 0.8212/0.8030, whereas page 8 gives 0.8258/0.8124. The 9.30% Spanish WER belongs to p=5%, k=20, contrary to the configuration stated in the text. The proposed PF mixtures also remain below the cascade on Spanish-source translation, so the strict-superiority claim is too strong. These issues are correctable and do not negate the main WER improvement, but they weaken the current account of the trade-offs. The two-stage comparison should also be scoped to the tested schedule: constituent-data exposure is matched, but a separately tuned second-stage schedule is not reported.

  2. The distinction between harmless surface variation and lost information needs a more reliable annotation analysis. Table 34 labels Joe/Joel and Berry/Barry substitutions as paraphrasing only despite a named-entity category. These examples do not invalidate the entire analysis, but they weaken the explanation of baseline errors and should be corrected before drawing a strong semantic-preservation conclusion.

Questions for authors
  1. Clarification: Which page-8/Table-2 values are authoritative? Please provide the corrected comparisons for one selected configuration and state precisely which tasks it improves over the cascade.

  2. Focused validation of the existing analysis: After correcting named-entity substitutions in the manual annotation, does paraphrasing remain the dominant error type, and how does this affect the semantic-preservation interpretation?

3 comments

We thank the reviewer for their comments. However, we would like to respectfully clarify a few key aspects of our contribution:

Weakness 1

Question 1 – Inconsistencies in the text vs. Table 2.

Thanks for noticing this inconsistency. The scores reported in Table 2 are the correct ones. We inadvertently made a mistake when reporting the ranges in the text.

Action: We have now corrected the text to ensure consistency with the results of Table 2 (line 377).

PF mixtures below the cascade on Spanish-source translation.

We thank the reviewer for this observation and agree that the comparison with the cascade should be stated more precisely. While our PF mixtures achieve higher COMET for English ST and better SQA, they do not match the Spanish ST. However, as acknowledged by the reviewer, this does not invalidate the effectiveness of our method, which outperforms the baseline SpeechLLM training methods.

Lastly, we would like to highlight that our experiments do not employ a large-scale training corpus. Hence, while the cascade is reported at its ceiling performance, our methodology has margin for improvement by scaling up the training data. A fair comparison of our method with a cascade system would therefore require large-scale training, which, unfortunately, is beyond our current compute budget.

Action: We have clarified this trade-off in the paper and revised the corresponding claims accordingly in the abstract (lines 24-25), Section 1 (lines 96-97), Section 5.3 (lines 417-420), and the conclusions.

Two-stage comparison should also be scoped to the tested schedule.

We are not entirely sure we understand the reviewer’s request. We would appreciate it if the reviewer could clarify this point or provide further details, so that we can provide a satisfactory response.

Weakness 2

Question 2 – Named-entity errors in manual analysis.

The goal of Tables 33-35 is to show, highlighted in bold and red, examples of errors that we classified as “paraphrasing”. We annotated all errors for all categories in the reported sentences, but highlighted only the paraphrasing ones. Accordingly, the two samples involving Berry/Barry and Joe/Joel include both “Paraphrasing” and “Named Entities” errors in our evaluation. However, thanks to the reviewer's comment, we realize that we have not stated this clearly in the captions of the tables and in Appendix K.

Action: We have clarified this in Tables 33, 34, 35, and in Appendix K to avoid ambiguity.

We are confident that, with our actions, we have addressed all the reviewer's concerns.
We kindly ask the reviewer to reassess our work in light of them and reconsider the scores assigned to our paper.

[1] Yunfei Chu, et al. Qwen2-audio technical report, 2024.

[2] Changli Tang, et al. Salmonn: Towards generic hearing abilities for large language models, 2024.

[3] Alexander H. Liu, et al. Voxtral. 2025

[4] Ding Ding, et al. Kimi-audio technical report. 2025.

[5] Jin Xu, et al. Qwen3-omni technical report, 2025.

[6] Microsoft. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras, 2025.

[7] Ke-Han Lu, et al. DeSTA2.5-Audio: Toward general-purpose large audio language model with self-generated cross-modal alignment. IEEE Transactions on Audio, Speech and Language Processing, 2026.

[8] Matthias Sperber and Matthias Paulik. Speech translation and the end-to-end promise: Taking stock of where we are. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.

The authors have addressed the numerical inconsistencies and clarified the annotation issues. However, my concern about the two-stage comparison remains unresolved. To clarify, I mean that the two-stage baseline should have its own tuned fine-tuning schedule, rather than simply using the same training settings as the single-stage approach. Otherwise, the results only demonstrate superiority over the specific two-stage configuration tested.

We thank the reviewer for the clarification. We have not run any hyperparameter tuning for any trained solution and relied on the hyperparameters used in the SALM work available in the NeMo repository. The only hyperparameters that we tuned in our work are p and k. For the two-stage finetuning approach, we tested p = 5% and 10% and k = 20 and 200 to limit computational cost. We have now also run the experiments with p = 20% and k ∈\in {20, 200} in response to the reviewer’s concern, and we report them in the table below.
Increasing p to 20% improved English ASR with respect to the best two-stage finetuning experiment reported in the paper, which can be attributed to the additional data included in the finetuning. However, this comes at the cost of Spanish ASR. Moreover, especially with k=200, the ST performance drops dramatically. SQA remains similar.
The new configurations are also underperforming our approach on all dimensions, except for English ASR, which is very similar with p=20% and k=20 and better for p=20% and k=200, but at the expense of a severe gap on all other dimensions.
We hope that this clarifies the reviewer's doubts regarding the hyperparameter selection, showing that changing the hyperparameters can optimize a single dimension, but the overall capabilities remain lower than the single-stage strategy, regardless of the hyperparameters chosen.

ASR ENASR ESST en-XST es-XSQA
NEW CONFIGURATION TESTED
Two-stage Finetuning PF-SeqKD 20% 2010.2511.400.77640.76530.3531
Two-stage Finetuning PF-SeqKD 20% 2009.9511.170.68250.71000.3569
RESULTS FROM THE PAPER
Best Two-stage Finetuning in Paper PF-SeqKD 10% 2012.6810.440.79360.74960.3545
Best Our Approach PF-SeqKD 10% 20010.4210.130.82120.80300.3779

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.