- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Input-Adaptive Acoustic Condition Modulation (IAACM) to improve the robustness of speech-to-LLM bridges for speech recognition. It combines a learned acoustic condition bank, token-level deviation estimates from a froze source bridge, and bounded residual FiLM corrections. At inference, the conditioning signal is derived from the target utterance, without requiring additional reference audio. Using Whisper large-v2 and Qwen3-8B, the authors evaluate eight bridge architectures across six datasets and 28 noise/reverberation conditions, where IAACM reduces overall macro WER. An analysis of representation movement also characterizes the learned corrections.
Critical strengths
- IAACM is validated across a broad range of bridge architectures and test conditions, showing overall improvements particularly on OOD data. Matched backbones, training data, and decoding strengthen the experimental comparisons.
- The controlled acoustic corruption experiments help isolate the effects of acoustic variation and provide additional evidence of robustness gains, which is the primary goal of IAACM.
Critical weaknesses
- IAACM is evaluated with only a single encoder-LLM combination, leaving its generalization across different backbones unclear. Performance regressions for CIF and CTC, especially in the ID conditions, weaken robustness claims across different bridge types and need further analysis.
- IAACM consists of several components and a multi-stage training procedure, but the paper lacks ablations to establish their individual contributions. Also, capacity-matched and augmentation-only experiments would help support genuine gains by IAACM.
Questions for authors
- How were the objective weights (in Eq. (11)) and the multi-stage training and reference-sampling protocol (Sec. 4.3) selected? Ablation studies on these would be informative.
- How do the unconditioned baselines perform when trained with the same manifold perturbation as IAACM? This comparison would help isolate the contribution of mere augmentation.
- IAACM increases the trainable parameters of the linear projector-based bridge by 8x. Does it still outperform a baseline with a comparable parameter count?
- CIF and CTC show substantial regressions on LS and TED, and VCTK, which I appreciate the authors' transparent reporting of negative results. But what explains these regressions, and are there any further analyses? Since both bridges use transcript-based supervision, could acoustic conditioning disrupt linguistically relevant information in their representations?
7 comments
W2, Q1. Design Rationale and Ablation Studies for Objective Weights, Multi-Stage Training, and Reference Sampling
The protocol was designed to establish a source reference before introducing calibration, and then broaden the conditioning sources during calibration training. We selected the objective weights heuristically and used the same values across all eight bridges: 0.1 for Risk loss, 0.1 for Restoration loss, 0.001 for Balance loss, and 0.1 for Diversity loss. Below, we report Q-Former results for loss ablations, weight sensitivity, and retrained reference-policy controls.
Objective weights. Risk supervision teaches the head to estimate each token's deviation from its clean Stage-A reference; restoration trains the corrected representation toward that reference. Balance and diversity regularize bank usage and vector redundancy, respectively. Removing Risk loss or Restoration loss increases Q-Former Macro WER by 0.355 or 0.914 points, respectively; removing all four auxiliary losses increases it by 1.403 points.
| Training objective | Q-Former Macro WER (Δ) |
|---|---|
| IAACM | 12.615 (0.000) |
| w/o Risk loss | 12.970 (+0.355) |
| w/o Restoration loss | 13.529 (+0.914) |
| w/o Risk loss + Restoration loss | 13.253 (+0.638) |
| w/o Balance loss | 12.601 (−0.014) |
| w/o Diversity loss | 12.498 (−0.117) |
| w/o Risk loss + Restoration loss + Balance loss + Diversity loss (ASR-only) | 14.018 (+1.403) |
| Uniform restoration weighting | 13.825 (+1.210) |
We varied the Risk and Restoration loss weights one at a time around the shared defaults. Parentheses show ΔWER in percentage points relative to the original IAACM result reported in the paper (12.615%).
| Configuration | Q-Former Macro WER (Δ) |
|---|---|
| Risk 0.03 + Restoration 0.1 | 12.546 (−0.069) |
| IAACM: Risk 0.1 + Restoration 0.1 | 12.615 (0.000) |
| Risk 0.3 + Restoration 0.1 | 12.392 (−0.223) |
| Risk 0.1 + Restoration 0.03 | 13.291 (+0.676) |
| Risk 0.1 + Restoration 0.3 | 12.001 (−0.614) |
These findings highlight the importance of restoration supervision and demonstrate that performance is sensitive to the choice of loss weights. Although the shared heuristic defaults provide a consistent configuration across all eight bridges, the sensitivity results suggest that further improvements are possible through hyperparameter tuning.
Training stages and reference sampling. Stage A trains the bridge with calibration disabled and provides the frozen source anchor. Stage B activates IAACM and jointly trains the current bridge and calibration module, initially using target-derived conditioning in 50% of cases. Stage C continues with the same anchor, objectives, and optimizer state while broadening reference sources and retaining 20% target-derived conditioning. Each stage lasts one epoch; the baseline also receives three epochs.
| Reference source | Stage B | Stage C |
|---|---|---|
| Same-speaker | 50% | 60% |
| Same-session | 0% | 15% |
| Target prefix | 25% | 10% |
| Target fallback | 25% | 10% |
| Wrong-speaker | 0% | 5% |
These proportions prioritize relevant reference audio while retaining the self-conditioning path used at inference. Stage C additionally exposes the model to variation in reference source and suitability. The target transcript remains the ASR supervision even when the reference comes from a different speaker; a fallback uses the target itself as the conditioning source.
To test the policy, we retrained Q-Former models with target-only conditioning or a single shared condition template, holding the Stage-A checkpoint, B/C training budget, losses, and optimizer transitions fixed.
| Training policy | Description | Q-Former Macro WER (Δ) |
|---|---|---|
| IAACM: mixed-reference | Uses diverse reference sources during training, including same-speaker, same-session, and target-derived references. At inference, conditioning is derived from the target utterance. | 12.615 (0.000) |
| Target-only | Uses only the target utterance to derive the condition during both training and inference, without mixed-reference sampling. | 12.786 (+0.171) |
| Shared condition for all utterances | Uses the same condition input for every utterance, rather than deriving a separate condition from each utterance. The shared condition is constructed by averaging representations from 1,024 training utterances. | 12.691 (+0.076) |
The shared condition was constructed from 1,024 training utterances; the conditioning modules remained trainable. IAACM improves Q-Former Macro WER by 0.171 points over target-only and 0.076 points over the shared template, with improvements on all six datasets in both comparisons. These controls evaluate the mixed-reference policy as a whole under matched training conditions. We will clarify the selection rationale and include the ablations in the revision.
W2, Q2. Isolating the Contribution of IAACM from Manifold Perturbation Augmentation
To make the baseline comparable with the IAACM protocol, we applied perturbation to the encoder features during baseline epochs 2 and 3, corresponding to IAACM Stages B and C. We used the Q-Former bridge and kept the original training protocol unchanged except for perturbation. The original IAACM achieves a Macro WER of 12.615%, outperforming both the baseline without perturbation (13.127%) and the baseline with feature perturbation (13.269%). These results show that the improvement cannot be attributed to perturbation alone and support the contribution of the IAACM module.
| Dataset | Baseline, no perturbation WER (Δ) | Baseline, feature perturbation WER (Δ) | IAACM, no perturbation WER (Δ) | IAACM, feature perturbation WER (Δ) |
|---|---|---|---|---|
| LS-clean | 9.527 (+0.611) | 9.898 (+0.982) | 9.035 (+0.119) | 8.916 (0.000) |
| LS-other | 12.121 (+0.167) | 12.055 (+0.101) | 11.516 (−0.438) | 11.954 (0.000) |
| TED | 10.705 (+0.447) | 11.449 (+1.191) | 10.776 (+0.518) | 10.258 (0.000) |
| CV23 | 25.090 (+1.305) | 24.226 (+0.441) | 22.015 (−1.770) | 23.785 (0.000) |
| GigaSpeech | 17.037 (+0.461) | 17.449 (+0.873) | 17.276 (+0.700) | 16.576 (0.000) |
| VCTK | 4.281 (+0.083) | 4.539 (+0.341) | 4.338 (+0.140) | 4.198 (0.000) |
| Macro WER | 13.127 (+0.512) | 13.269 (+0.654) | 12.493 (−0.122) | 12.615 (0.000) |
W2, Q3. Parameter-Matched Comparison of IAACM and a Larger MLP Baseline for the Linear Projector Bridge
We conducted this experiment using the Linear projector, which incurs the largest parameter increase from IAACM. To match the parameter count of Linear + IAACM, we replaced the Linear projector with a parameter-matched MLP and evaluated its performance. IAACM retained its advantage in Macro WER even after matching the parameter count.
| Dataset / quantity | Original Linear projector | Parameter-matched MLP | Full IAACM: Linear + IAACM |
|---|---|---|---|
| Total trainable parameters (M) | 5.255 | 44.120 | 44.118 |
| LS-clean WER (Δ) | 2.455 (+0.062) | 2.465 (+0.072) | 2.393 (0.000) |
| LS-other WER (Δ) | 5.671 (−0.200) | 5.440 (−0.431) | 5.871 (0.000) |
| TED WER (Δ) | 4.906 (−0.365) | 5.444 (+0.173) | 5.271 (0.000) |
| CV23 WER (Δ) | 18.826 (+0.304) | 20.920 (+2.398) | 18.522 (0.000) |
| GigaSpeech WER (Δ) | 14.479 (+1.237) | 13.914 (+0.671) | 13.243 (0.000) |
| VCTK WER (Δ) | 3.452 (−0.218) | 3.609 (−0.061) | 3.670 (0.000) |
| Macro WER (Δ) | 8.298 (+0.137) | 8.632 (+0.471) | 8.162 (0.000) |
We really appreciate the reviewer for the careful evaluation and for recognizing the breadth of our bridge comparisons and controlled robustness experiments. We address the backbone coverage, training protocol, comparison fairness, and CIF/CTC regressions below.
W1. Generalization of IAACM Across Different Speech Encoder and LLM Backbones
Thank you for raising this point. To examine generalization across backbones, we conducted additional experiments using Q-Former as the common bridge while keeping the IAACM architecture and training protocol unchanged. Starting from the original Whisper large-v2 + Qwen3-8B configuration, we evaluated two additional configurations: (1) replacing the LLM with Gemma-2-9B-it, and (2) replacing both the speech encoder and the LLM with WavLM-large and Gemma-2-9B-it, respectively.
The table below compares the baseline and IAACM results for all three configurations:
WER is reported in percentages, with lower values indicating better performance. Bold values indicate the better result within each backbone configuration. Macro WER is the unweighted average across the six datasets.
IAACM reduces Macro WER from 13.127% to 12.615% in the original Whisper–Qwen configuration, from 11.625% to 11.471% in Whisper–Gemma, and from 23.162% to 22.819% in WavLM–Gemma. Although the average improvements in the additional configurations are smaller than in the original configuration, both retain improvements over their respective baselines. This provides evidence that IAACM’s benefit extends beyond the original Whisper–Qwen combination, including when both the encoder and LLM are replaced.
In particular, Whisper–Gemma already provides a stronger baseline than the original Whisper–Qwen configuration, with a Macro WER of 11.625% versus 13.127%. IAACM further improves this stronger baseline to 11.471%. Improvements occur on LS-clean, LS-other, TED, and GigaSpeech, with absolute WER reductions of 2.028 and 2.849 percentage points on TED and GigaSpeech, respectively. Thus, IAACM provides additional gains even with a backbone combination that already achieves better baseline recognition performance.
For WavLM–Gemma, IAACM reduces Macro WER by 0.343 percentage points, with improvements on CV23, GigaSpeech, and VCTK. This demonstrates an average benefit in a configuration with a different speech encoder and a substantially different baseline performance level.
Overall, the magnitude and dataset-wise pattern of improvement depend on the backbone, while the same IAACM design and training protocol improve average recognition performance in both additional encoder–LLM configurations.