acceptodds
Under review as a conference paper at ICLR 2027

Input-Adaptive Acoustic Condition Modulation Across Speech-to-LLM Bridges for Robust Speech Recognition

Abstract

Speech large language models (Speech LLMs) depend on a trainable bridge that compresses acoustic representations into LLM input embeddings. Even with strong frozen speech and language backbones, a bridge learned on one corpus can map unfamiliar acoustic conditions to poorly supported regions of its representation space. We introduce Input-Adaptive Acoustic Condition Modulation (IAACM), a bridge-local calibration method for robust speech recognition. IAACM routes an utterance through a bank of acoustic condition vectors, predicts token-level deviation from a frozen source-bridge manifold, and applies a risk-gated residual FiLM correction. The speech encoder and LLM remain frozen, and inference is self-conditioned on the target utterance without external reference audio. We evaluate matched conditioned and unconditioned systems across eight bridge architectures and six datasets. IAACM lowers the six-domain macro word error rate (WER) for every evaluated bridge and improves 35 of 48 bridge-domain pairs. Averaged uniformly over all pairs, macro WER decreases from 12.22% to 11.65%; the relative reduction grows from 4.68% overall to 17.15% on the highest-error domain of each bridge. Controlled noise and reverberation tests further support robustness. These results demonstrate that bridge-level acoustic calibration improves speech recognition robustness across diverse Speech-to-LLM bridges.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Robust speech recognition, speech recognition, speech large language models, speech and audio processing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4hXLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Input-Adaptive Acoustic Condition Modulation (IAACM) to improve the robustness of speech-to-LLM bridges for speech recognition. It combines a learned acoustic condition bank, token-level deviation estimates from a froze source bridge, and bounded residual FiLM corrections. At inference, the conditioning signal is derived from the target utterance, without requiring additional reference audio. Using Whisper large-v2 and Qwen3-8B, the authors evaluate eight bridge architectures across six datasets and 28 noise/reverberation conditions, where IAACM reduces overall macro WER. An analysis of representation movement also characterizes the learned corrections.

Critical strengths
  • IAACM is validated across a broad range of bridge architectures and test conditions, showing overall improvements particularly on OOD data. Matched backbones, training data, and decoding strengthen the experimental comparisons.
  • The controlled acoustic corruption experiments help isolate the effects of acoustic variation and provide additional evidence of robustness gains, which is the primary goal of IAACM.
Critical weaknesses
  • IAACM is evaluated with only a single encoder-LLM combination, leaving its generalization across different backbones unclear. Performance regressions for CIF and CTC, especially in the ID conditions, weaken robustness claims across different bridge types and need further analysis.
  • IAACM consists of several components and a multi-stage training procedure, but the paper lacks ablations to establish their individual contributions. Also, capacity-matched and augmentation-only experiments would help support genuine gains by IAACM.
Questions for authors
  • How were the objective weights (in Eq. (11)) and the multi-stage training and reference-sampling protocol (Sec. 4.3) selected? Ablation studies on these would be informative.
  • How do the unconditioned baselines perform when trained with the same manifold perturbation as IAACM? This comparison would help isolate the contribution of mere augmentation.
  • IAACM increases the trainable parameters of the linear projector-based bridge by 8x. Does it still outperform a baseline with a comparable parameter count?
  • CIF and CTC show substantial regressions on LS and TED, and VCTK, which I appreciate the authors' transparent reporting of negative results. But what explains these regressions, and are there any further analyses? Since both bridges use transcript-based supervision, could acoustic conditioning disrupt linguistically relevant information in their representations?
7 comments

We really appreciate the reviewer for the careful evaluation and for recognizing the breadth of our bridge comparisons and controlled robustness experiments. We address the backbone coverage, training protocol, comparison fairness, and CIF/CTC regressions below.

W1. Generalization of IAACM Across Different Speech Encoder and LLM Backbones

Thank you for raising this point. To examine generalization across backbones, we conducted additional experiments using Q-Former as the common bridge while keeping the IAACM architecture and training protocol unchanged. Starting from the original Whisper large-v2 + Qwen3-8B configuration, we evaluated two additional configurations: (1) replacing the LLM with Gemma-2-9B-it, and (2) replacing both the speech encoder and the LLM with WavLM-large and Gemma-2-9B-it, respectively.

The table below compares the baseline and IAACM results for all three configurations:

  • Original: Whisper large-v2 / Qwen3-8B / Q-Former
  • Extension 1: Whisper large-v2 / Gemma-2-9B-it / Q-Former
  • Extension 2: WavLM-large / Gemma-2-9B-it / Q-Former

WER is reported in percentages, with lower values indicating better performance. Bold values indicate the better result within each backbone configuration. Macro WER is the unweighted average across the six datasets.

DatasetOriginal BaselineOriginal IAACMExtension 1 BaselineExtension 1 IAACMExtension 2 BaselineExtension 2 IAACM
LS-clean9.5278.9167.7506.78916.8917.24
LS-other12.12111.9549.9299.39617.0017.83
TED10.70510.25810.1138.08530.6931.08
CV2325.09023.78522.01927.41731.0430.58
GigaSpeech17.03716.57615.76412.91537.7735.54
VCTK4.2814.1984.1764.2265.584.65
Macro WER13.12712.61511.62511.47123.16222.819
Absolute Macro WER reduction (pp)—0.512—0.154—0.343
Relative Macro WER reduction—3.90%—1.32%—1.48%

IAACM reduces Macro WER from 13.127% to 12.615% in the original Whisper–Qwen configuration, from 11.625% to 11.471% in Whisper–Gemma, and from 23.162% to 22.819% in WavLM–Gemma. Although the average improvements in the additional configurations are smaller than in the original configuration, both retain improvements over their respective baselines. This provides evidence that IAACM’s benefit extends beyond the original Whisper–Qwen combination, including when both the encoder and LLM are replaced.

In particular, Whisper–Gemma already provides a stronger baseline than the original Whisper–Qwen configuration, with a Macro WER of 11.625% versus 13.127%. IAACM further improves this stronger baseline to 11.471%. Improvements occur on LS-clean, LS-other, TED, and GigaSpeech, with absolute WER reductions of 2.028 and 2.849 percentage points on TED and GigaSpeech, respectively. Thus, IAACM provides additional gains even with a backbone combination that already achieves better baseline recognition performance.

For WavLM–Gemma, IAACM reduces Macro WER by 0.343 percentage points, with improvements on CV23, GigaSpeech, and VCTK. This demonstrates an average benefit in a configuration with a different speech encoder and a substantially different baseline performance level.

Overall, the magnitude and dataset-wise pattern of improvement depend on the backbone, while the same IAACM design and training protocol improve average recognition performance in both additional encoder–LLM configurations.

W2, Q1. Design Rationale and Ablation Studies for Objective Weights, Multi-Stage Training, and Reference Sampling

The protocol was designed to establish a source reference before introducing calibration, and then broaden the conditioning sources during calibration training. We selected the objective weights heuristically and used the same values across all eight bridges: 0.1 for Risk loss, 0.1 for Restoration loss, 0.001 for Balance loss, and 0.1 for Diversity loss. Below, we report Q-Former results for loss ablations, weight sensitivity, and retrained reference-policy controls.

Objective weights. Risk supervision teaches the head to estimate each token's deviation from its clean Stage-A reference; restoration trains the corrected representation toward that reference. Balance and diversity regularize bank usage and vector redundancy, respectively. Removing Risk loss or Restoration loss increases Q-Former Macro WER by 0.355 or 0.914 points, respectively; removing all four auxiliary losses increases it by 1.403 points.

Training objectiveQ-Former Macro WER (Δ)
IAACM12.615 (0.000)
w/o Risk loss12.970 (+0.355)
w/o Restoration loss13.529 (+0.914)
w/o Risk loss + Restoration loss13.253 (+0.638)
w/o Balance loss12.601 (−0.014)
w/o Diversity loss12.498 (−0.117)
w/o Risk loss + Restoration loss + Balance loss + Diversity loss (ASR-only)14.018 (+1.403)
Uniform restoration weighting13.825 (+1.210)

We varied the Risk and Restoration loss weights one at a time around the shared defaults. Parentheses show ΔWER in percentage points relative to the original IAACM result reported in the paper (12.615%).

ConfigurationQ-Former Macro WER (Δ)
Risk 0.03 + Restoration 0.112.546 (−0.069)
IAACM: Risk 0.1 + Restoration 0.112.615 (0.000)
Risk 0.3 + Restoration 0.112.392 (−0.223)
Risk 0.1 + Restoration 0.0313.291 (+0.676)
Risk 0.1 + Restoration 0.312.001 (−0.614)

These findings highlight the importance of restoration supervision and demonstrate that performance is sensitive to the choice of loss weights. Although the shared heuristic defaults provide a consistent configuration across all eight bridges, the sensitivity results suggest that further improvements are possible through hyperparameter tuning.

Training stages and reference sampling. Stage A trains the bridge with calibration disabled and provides the frozen source anchor. Stage B activates IAACM and jointly trains the current bridge and calibration module, initially using target-derived conditioning in 50% of cases. Stage C continues with the same anchor, objectives, and optimizer state while broadening reference sources and retaining 20% target-derived conditioning. Each stage lasts one epoch; the baseline also receives three epochs.

Reference sourceStage BStage C
Same-speaker50%60%
Same-session0%15%
Target prefix25%10%
Target fallback25%10%
Wrong-speaker0%5%

These proportions prioritize relevant reference audio while retaining the self-conditioning path used at inference. Stage C additionally exposes the model to variation in reference source and suitability. The target transcript remains the ASR supervision even when the reference comes from a different speaker; a fallback uses the target itself as the conditioning source.

To test the policy, we retrained Q-Former models with target-only conditioning or a single shared condition template, holding the Stage-A checkpoint, B/C training budget, losses, and optimizer transitions fixed.

Training policyDescriptionQ-Former Macro WER (Δ)
IAACM: mixed-referenceUses diverse reference sources during training, including same-speaker, same-session, and target-derived references. At inference, conditioning is derived from the target utterance.12.615 (0.000)
Target-onlyUses only the target utterance to derive the condition during both training and inference, without mixed-reference sampling.12.786 (+0.171)
Shared condition for all utterancesUses the same condition input for every utterance, rather than deriving a separate condition from each utterance. The shared condition is constructed by averaging representations from 1,024 training utterances.12.691 (+0.076)

The shared condition was constructed from 1,024 training utterances; the conditioning modules remained trainable. IAACM improves Q-Former Macro WER by 0.171 points over target-only and 0.076 points over the shared template, with improvements on all six datasets in both comparisons. These controls evaluate the mixed-reference policy as a whole under matched training conditions. We will clarify the selection rationale and include the ablations in the revision.

W2, Q2. Isolating the Contribution of IAACM from Manifold Perturbation Augmentation

To make the baseline comparable with the IAACM protocol, we applied perturbation to the encoder features during baseline epochs 2 and 3, corresponding to IAACM Stages B and C. We used the Q-Former bridge and kept the original training protocol unchanged except for perturbation. The original IAACM achieves a Macro WER of 12.615%, outperforming both the baseline without perturbation (13.127%) and the baseline with feature perturbation (13.269%). These results show that the improvement cannot be attributed to perturbation alone and support the contribution of the IAACM module.

DatasetBaseline, no perturbation WER (Δ)Baseline, feature perturbation WER (Δ)IAACM, no perturbation WER (Δ)IAACM, feature perturbation WER (Δ)
LS-clean9.527 (+0.611)9.898 (+0.982)9.035 (+0.119)8.916 (0.000)
LS-other12.121 (+0.167)12.055 (+0.101)11.516 (−0.438)11.954 (0.000)
TED10.705 (+0.447)11.449 (+1.191)10.776 (+0.518)10.258 (0.000)
CV2325.090 (+1.305)24.226 (+0.441)22.015 (−1.770)23.785 (0.000)
GigaSpeech17.037 (+0.461)17.449 (+0.873)17.276 (+0.700)16.576 (0.000)
VCTK4.281 (+0.083)4.539 (+0.341)4.338 (+0.140)4.198 (0.000)
Macro WER13.127 (+0.512)13.269 (+0.654)12.493 (−0.122)12.615 (0.000)

W2, Q3. Parameter-Matched Comparison of IAACM and a Larger MLP Baseline for the Linear Projector Bridge

We conducted this experiment using the Linear projector, which incurs the largest parameter increase from IAACM. To match the parameter count of Linear + IAACM, we replaced the Linear projector with a parameter-matched MLP and evaluated its performance. IAACM retained its advantage in Macro WER even after matching the parameter count.

Dataset / quantityOriginal Linear projectorParameter-matched MLPFull IAACM: Linear + IAACM
Total trainable parameters (M)5.25544.12044.118
LS-clean WER (Δ)2.455 (+0.062)2.465 (+0.072)2.393 (0.000)
LS-other WER (Δ)5.671 (−0.200)5.440 (−0.431)5.871 (0.000)
TED WER (Δ)4.906 (−0.365)5.444 (+0.173)5.271 (0.000)
CV23 WER (Δ)18.826 (+0.304)20.920 (+2.398)18.522 (0.000)
GigaSpeech WER (Δ)14.479 (+1.237)13.914 (+0.671)13.243 (0.000)
VCTK WER (Δ)3.452 (−0.218)3.609 (−0.061)3.670 (0.000)
Macro WER (Δ)8.298 (+0.137)8.632 (+0.471)8.162 (0.000)
Reviewer_xopiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The main focus of the paper is to calibrate the connectors/bridges used in Speech + LLM integration better. The authors claim that the bridge performs subpar when exposed to out-of-domain examples, and hence an input-conditioned adaptive mechanism called IAACM can help address this issue.

Critical strengths

The paper does a good job of laying out prior work and where its contributions stand.

Critical weaknesses

My biggest criticism of the paper is that section 3, which is the most critical aspect of the manuscript, is very poorly written. It feels like a dump of components from Figure 1, without actually motivating the need for any of those alterations. At this point, I have read the draft numerous times, and still haven't understood what the conditioning sequence (line 152) means and where it suddenly comes from, what line 154 is trying to tell about training, why Gaussian noise is used in section 3.3, etc etc.

When it comes to the results (Table 2), IAACM has an unfair parameter and experimental advantage over the baseline. The authors mention in line 59 and subsequently in line 294 that they train the model only for 3 epochs, and the parameter overhead added by IAACM spans anywhere from 3M to 38M additional parameters (elucidated in section 5.4). For such a small number of epochs, this parameter overhead will definitely give an unfair advantage to IAACM over the baseline, resulting in superior performance.

In conclusion, the manuscript needs a massive overhaul to even be reconsidered for a change of score.

Questions for authors
  1. Have you conducted experiments where the number of parameters between IAACM and the baseline matches?
  2. How much additional computation does IAACM introduce over baseline?
3 comments

We sincerely appreciate the reviewer for the careful review and for recognizing the positioning of our work relative to prior studies. We agree that Section 3 needs a clearer explanation of the inputs, training procedure, and purpose of each component. So we’ve rewritten Section 3 and added Appendix B to provide clearer details on conditioning and training (newly added or modified parts are highlighted in blue text), and subsequently uploaded new version of paper pdf on OpenReview. We address these points below and provide the requested parameter-matched comparison and computational measurements.

W1. Conditioning sequence, training procedure, and Gaussian perturbation

The conditioning sequence is the sequence of bridge tokens extracted from the audio selected as the condition source. It supplies the utterance-level information used to calibrate the target utterance. To make this explicit, let EE denote the frozen speech encoder, GθG_{\theta} the current bridge, and G0G_0 the frozen Stage-A bridge. The three sequences have different roles:

SequenceConstructionPurpose
Target tokens BBTarget audio through EE and GθG_{\theta}Tokens to be calibrated and passed to the LLM
Conditioning sequence BcB^cSelected condition-source audio through the same EE and GθG_{\theta}Input for constructing the utterance-level condition
Anchor tokens B0B^0Clean target encoder features through G0G_0Fixed training targets for risk prediction and restoration

Thus, the conditioning sequence in line 152 comes from an audio input; it is not an additional set of learned query tokens. During training, the condition source is selected by the reference-sampling policy and can be a reference utterance or a target-derived input. Its transcription does not need to match the target transcription: the ASR loss always uses the target utterance's transcript. At inference, the target utterance itself supplies the condition, and its already computed bridge tokens are reused. No external reference audio or second encoder–bridge pass is needed.

Gaussian feature perturbation in Section 3.3 creates paired perturbed and clean representations of the same utterance. We compute H=E(x)H = E(x), feed perturbed HH to the current bridge, and feed clean HH to the frozen anchor. Their token-level cosine distance provides the risk target, while restoration trains the calibrated representation toward the clean anchor. This gives the calibration module a controlled deviation to learn from, directly in its input space.

Q1. Have you conducted experiments where the number of parameters between IAACM and the baseline matches?

We conducted this experiment using the Linear projector, which incurs the largest parameter increase from IAACM. To match the parameter count of Linear + IAACM, we replaced the Linear projector with a parameter-matched MLP and evaluated its performance, keeping all settings unchanged. IAACM retained its advantage in Macro WER even after matching the parameter count.

Dataset / quantityOriginal Linear projectorParameter-matched MLPIAACM: Linear + IAACM
Total trainable parameters (M)5.25544.12044.118
LS-clean WER (Δ)2.455 (+0.062)2.465 (+0.072)2.393 (0.000)
LS-other WER (Δ)5.671 (−0.200)5.440 (−0.431)5.871 (0.000)
TED WER (Δ)4.906 (−0.365)5.444 (+0.173)5.271 (0.000)
CV23 WER (Δ)18.826 (+0.304)20.920 (+2.398)18.522 (0.000)
GigaSpeech WER (Δ)14.479 (+1.237)13.914 (+0.671)13.243 (0.000)
VCTK WER (Δ)3.452 (−0.218)3.609 (−0.061)3.670 (0.000)
Macro WER (Δ)8.298 (+0.137)8.632 (+0.471)8.162 (0.000)

Q2. How much additional computation does IAACM introduce over baseline?

To assess IAACM's inference cost, we measured total GPU runtime, including transcription generation, and peak memory usage. Compared with the baseline, inference time increased by 4.68–8.54%, and peak allocated memory increased by 0.051–0.278 GiB.

BridgeBaseline median GPU time, msIAACM median GPU time, msIncreasePeak allocated memory, GiB
Q-Former576.829626.0688.54%16.840 → 17.118
Causal Conv655.520707.8247.98%16.647 → 16.698
Linear613.971642.7324.68%17.011 → 17.243

For each bridge, we compared the baseline and IAACM on the same GPU using 64 LS-clean utterances. After four warm-up utterances, we repeated the measurements three times. The inference times in the table are medians.

At inference, IAACM reuses the already computed target bridge tokens to construct the condition. No additional encoding of external reference audio is therefore required. The frozen anchor and restoration targets are computed only during training; inference uses the paths that construct the condition and calibrate the bridge tokens.

I appreciate the authors for rewriting section 3 to put forth the idea better. Section 3.4 is much clearer now. But it also flags a major problem with the current pitch of the authors. Their claim that “a bridge learned on one corpus can map unfamiliar acoustic conditions to poorly supported regions of its representation space and IAACM helps address this problem” is not entirely true. From section 3.4, it seems that the confidence scores learned by IAACM are on small noise/perturbations to the reference. So at max, IAACM makes the model robust to salt-and-pepper noises or tiny fluctuations. It cannot effectively handle out-of-domain examples. Also, while the updated manuscript gave more clarity on few components, it still needs a little polishing. For example, in section 3.2, what is the rationale behind three stages? What is the significance of each stage? Why not do it in 2 stages (ideally, stage B and C can be generalized into 1 stage, right)? Why only 1 epoch per stage?

I thank the authors for running the additional experiments. Regarding the experiments conducted in response to Q1, although the macro WER +0.47, it is mostly coming from CV23 where you see good gains. On all other datasets, I highly suspect that the gaps are not statistically significant. As of now, I do not see this submission as ready for ICLR, so I will retain my score.

Reviewer_YR48OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Input-Adaptive Acoustic Condition Modulation (IAACM), a lightweight module applied to the bridge between the speech encoder and the LLM in SpeechLLMs, to improve robustness on ASR tasks while keeping the speech encoder and LLM frozen. IAACM learns an acoustic condition vector bank during training and models bounded, gated residual FiLM calibration driven by token risk. The authors fix the Whisper large v2 speech encoder with the Qwen3-8B LLM, train the models with 8 different bridges on LibriSpeech, and test them on 6 datasets. The experimental results show that the macro overall average WER on the 6 datasets decreased from 12.22% to 11.65%. The authors also conduct additional noise and reverberation experiments to analyze robustness and representation movement under Gaussian noise, environmental noise, and room impulse response (RIR) conditions. However, the performance variations are inconsistent across different bridge architectures and datasets, and performance improvements are achieved only in a proportion of cases.

Critical strengths
  1. The paper presents a broad, sound empirical study, covering eight bridge architectures and six different datasets. The proposed IAACM module brings performance improvements in a large proportion of cases.
  2. The proposed method keeps the LLM frozen, which prevents the LLM's textual capabilities from degrading.
Critical weaknesses
  1. The IAACM has a complex structure with many modules and hyperparameters. But the authors do not explain the design motivation for them clearly, nor clarify how the value of each hyperparameter is determined. Ablations on the internal modules of IAACM are also missing.
  2. The additional computational resources required for training are disproportionate to the performance gains. Compared to pure bridge, adding IAACM requires an additional 2 epochs of training, but the macro overall average WER only decreases from 12.22% to 11.65%, which is not significant. Moreover, in some cases, ASR performance can even decrease significantly.
Questions for authors

Clarification requests:

  1. Why do you use Whisper-large-v2 instead of Whisper-large-v3, which is the most widely used model in related work?
  2. In Eq. (3), why is the perturbation A\mathcal{A} applied directly to the encoder representations HH rather than to the input waveform? What difference do you think there is between the two choices?
  3. The IAACM has a complex structure with many modules and hyperparameters. What is the motivation behind each design? Can some modules be replaced with simpler modules, such as linear projection or MLP? How are the values of each hyperparameter determined?
  4. This paper focuses on optimizing for ASR tasks, but will it affect the Large Audio-Language Models' (LALMs) performance on other tasks, such as speech emotion recognition, speaker identification, and environmental sound classification?

Suggestions for future work:

  1. This paper only analyzed Gaussian noise, environmental noise, and RIR, but there are many other types of perturbations in the real world, such as distortion and absorption. I suggest the authors include these perturbations in the analysis.
  2. I also recommend adding CHiME (https://www.chimechallenge.org/), an ASR challenge that includes a large amount of real-world noise, to the evaluation dataset.
6 comments

We thank the reviewer for the careful review and for recognizing the breadth of our evaluation across bridge architectures and datasets. We address the concerns about training cost, design motivation, and robustness below.

W2. Training budget and the size of the improvement

Both the baseline and IAACM were independently trained for a total of three epochs. The baseline uses its original bridge-training objective throughout, whereas IAACM allocates one epoch to each of Stages A, B, and C. For the Q-Former comparisons below, the baseline uses ASR cross-entropy for all three epochs. IAACM therefore does not receive two additional epochs beyond the baseline. We will make the baseline protocol explicit in paper.

Regarding the magnitude of improvement, the cross-corpus macro average alone does not show whether gains are concentrated in a few conditions or distributed across acoustic variations. The controlled SRB evaluation helps distinguish these cases by changing the acoustics of the same utterances. For Offline Conv, improvements occur in all 28 corruption conditions. This pattern provides evidence that IAACM can deliver gains across varied acoustic conditions, giving practical meaning to the modest improvement in aggregate WER.

Q1. Choice of Whisper large-v2

We selected Whisper large-v2 to prioritize stability in our ASR robustness study, taking into account reports of more severe hallucinations with large-v3 under challenging recognition conditions.

In particular, Yan et al. [1] observed more severe hallucinations with large-v3 in long-form recordings containing non-speech sounds and suggested that this may be related to error compounding from its semi-supervised training. Similar concerns have been reported in other ASR settings by Gao et al. [2] and Tan et al. [3].

Given our focus on robust ASR, these reported hallucination issues and their possible connection to pseudo-label-based semi-supervision informed our choice of large-v2 as an established backbone. We will clarify this selection rationale in the revised manuscript.

References

[1] Yan et al. “CMU’s IWSLT 2024 Offline Speech Translation System: A Cascaded Approach For Long-Form Robustness.” IWSLT, 2024.

[2] Gao et al. “Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts.” Interspeech, 2025.

[3] Tan et al. “CBA-Whisper: Curriculum Learning-Based AdaLoRA Fine-Tuning on Whisper for Low-Resource Dysarthric Speech Recognition.” Interspeech, 2025.

Q2. Perturbing encoder representations rather than waveforms

We apply perturbation at the input to the bridge to induce controlled variations in the resulting bridge representations, which IAACM learns to calibrate. Waveform perturbation produces E(A(x))E(A(x)), whereas feature perturbation produces A(E(x))A(E(x)). The former passes through the nonlinear encoder, so the two operations produce different learning signals even when both add Gaussian noise.

In our training procedure, H=E(x)H = E(x) is computed once. The current bridge receives the perturbed representation A(H)A(H), while the frozen Stage-A anchor receives clean HH. This creates a clean reference for token-risk supervision and restoration without a second encoder pass for the perturbed target.

We evaluated five configurations with Q-Former and seed 13. The feature-augmented baseline and the IAACM configurations with and without feature perturbation share the same Stage-A checkpoint. The table reports WER (%), with Macro denoting the equally weighted average over the six corpora.

DatasetBaseline, no perturbation WER (Δ)Baseline, feature perturbation WER (Δ)IAACM, no perturbation WER (Δ)IAACM, feature perturbation WER (Δ)IAACM, waveform perturbation WER (Δ)
LS-clean9.527 (+0.611)9.898 (+0.982)9.035 (+0.119)8.916 (0.000)8.965 (+0.049)
LS-other12.121 (+0.166)12.055 (+0.100)11.516 (−0.439)11.954 (0.000)11.434 (−0.520)
TED10.705 (+0.447)11.449 (+1.191)10.776 (+0.518)10.258 (0.000)10.712 (+0.454)
CV2325.090 (+1.306)24.226 (+0.441)22.015 (−1.770)23.785 (0.000)22.728 (−1.056)
GigaSpeech17.037 (+0.460)17.449 (+0.873)17.276 (+0.700)16.576 (0.000)16.452 (−0.124)
VCTK4.281 (+0.083)4.539 (+0.340)4.338 (+0.140)4.198 (0.000)4.424 (+0.226)
Macro WER13.127 (+0.512)13.269 (+0.655)12.493 (−0.122)12.615 (0.000)12.453 (−0.162)

Δ denotes the WER difference from the main-paper IAACM with feature perturbation, in percentage points, calculated from unrounded scores; positive values indicate higher WER. Macro WER is the equally weighted average across the six corpora.

IAACM with feature perturbation outperforms both baselines on every corpus, reducing Macro WER by 0.512 percentage points relative to the baseline without perturbation and by 0.655 points relative to the feature-augmented baseline. Feature augmentation alone does not improve the baseline’s Macro WER, whereas all three IAACM configurations achieve lower Macro WER than both baselines.

IAACM with waveform perturbation achieves the lowest Macro WER at 12.453%, followed by IAACM without perturbation at 12.493% and IAACM with feature perturbation at 12.615%. Feature perturbation achieves lower WER than waveform perturbation on LS-clean, TED, and VCTK; waveform perturbation performs better on LS-other, CV23, and GigaSpeech.

Q3. Motivation for the IAACM components, simpler alternatives

IAACM consists of three main components: the Acoustic Condition Bank, FiLM, and the Token-level Risk Prediction Head. As introduced in Sec. 2.3, the Acoustic Condition Bank adapts the Global Style Token concept, used in zero-shot TTS to improve generalization to unseen speakers, to Speech-LLM-based ASR. We selected FiLM as an established method for feature-wise conditioning. Finally, we introduced the Token-level Risk Prediction Head because an utterance-level acoustic condition alone does not indicate how far individual bridge tokens deviate from the learned reference representation.

Specifically, we freeze the bridge trained in Stage A and use it as an anchor. For the same utterance, the cosine distance between the anchor representation obtained from clean input and the current bridge representation obtained with feature perturbation serves as the training target for token-level risk.

This allows the model to predict each token's representational deviation at inference without running a separate anchor and use it in the residual FiLM gate. The Acoustic Condition Bank therefore determines the form of the correction at the utterance level, while the Token-level Risk Prediction Head provides deviation information at the token level.

In comparisons of IAACM variants with matched trainable parameter counts, the current IAACM design achieved the best performance with both Q-Former and Causal Conv, supporting the proposed design.

ComparisonConditioning and calibration configurationQ-Former parameters, MQ-Former Macro WER (Δ)Causal Conv parameters, MCausal Conv Macro WER (Δ)
IAACMBank/router-based conditioning + FiLM76.58512.615 (0.000)16.7079.524 (0.000)
MLP-conditioned FiLMReplace bank/router with an MLP; w/o Router balance loss + Bank diversity loss; retain all other settings76.58512.664 (+0.049)16.7079.739 (+0.215)
Direct bridge-token-conditioned FiLM + ASR CE onlyDerive the source condition directly from bridge tokens for FiLM calibration; train with ASR CE only76.58513.359 (+0.745)16.70710.269 (+0.745)
IAACM architecture + ASR CE only: ASR-onlyRetain the IAACM architecture and perturbation76.58514.018 (+1.403)16.7079.785 (+0.261)

The following table presents the loss ablations for IAACM. These experiments were conducted with Q-Former, keeping all settings unchanged except for the loss configuration.

Training objectiveQ-Former Macro WER (Δ)
IAACM12.615 (0.000)
w/o Risk loss12.970 (+0.355)
w/o Restoration loss13.529 (+0.915)
w/o Risk loss + Restoration loss13.253 (+0.638)
w/o Balance loss12.601 (−0.013)
w/o Diversity loss12.498 (−0.117)
w/o Risk loss + Restoration loss + Balance loss + Diversity loss (ASR-only)14.018 (+1.403)
Uniform restoration weighting13.825 (+1.210)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.