- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Latent Acoustic Arbitration (LAA), a parameter-efficient framework for post-ASR correction that combines linguistic preferences from a small language model with acoustic evidence from a larger speech model. LAA uses a frozen text model to rank speech-generated hypotheses and a lightweight two-stage router with phoneme-level acoustic verification to selectively correct transcription errors. Experiments across eight English ASR benchmarks show consistent improvements over constrained language-based selection with Granite Speech 8B. Additional scaling experiments suggest that smaller text models can achieve competitive correction performance without requiring comparable model sizes.
Critical strengths
- The paper addresses a meaningful problem in ASR correction by combining linguistic preferences with acoustic evidence to prevent unsupported text modifications.
- The proposed framework keeps both speech and text backbones frozen, requiring only a small number of trainable parameters.
- The experiments cover multiple ASR benchmarks, speech backbones, and text-model sizes, with additional component ablations and sensitivity analyses.
- The results demonstrate that a substantially smaller language model can effectively assist a larger speech model under the evaluated conditions.
Critical weaknesses
- LAA combines existing techniques such as N-best hypothesis selection, selective correction, and acoustic rescoring, limiting its methodological novelty. The authors should clarify its key contributions beyond prior work.
- LAA uses complete-transcript phoneme scores to determine whether local span replacements should be accepted, potentially creating a mismatch between the scope of acoustic verification and the actual modifications. Further validation is needed to establish whether this approach provides reliable acoustic evidence for local corrections.
- Although LAA consistently improves performance with Granite Speech 8B, the overall WER reduction is relatively small (0.138 percentage points), and several Whisper and SeamlessM4T configurations show no improvement or performance degradation. Further evidence is needed to establish its practical effectiveness and generalizability across speech backbones.
Questions for authors
See above
2 comments
Author Rebuttal (2/2)
W3: How practically effective is LAA, and do its gains transfer across speech backbones?
Answer: LAA improves over constrained Selection for three of four speech backbones on the common CHiME-6 evaluation. SeamlessM4T Medium shows the largest decrease, 4.2140 percentage points, with a paired 95% confidence interval of [3.3360, 5.1681]. Both Granite models also improve over native direct recognition.
To address backbone transfer, we evaluated native direct recognition, Selection and LAA on the same 1,000 utterances with Qwen3-0.6B as the text refiner. Reporting both baselines separates gains from acoustic arbitration after Selection from gains over the original recognizer.
| Speech backbone | Native direct WER (%) | Selection WER (%) | LAA WER (%) | Selection minus LAA (pp) |
|---|---|---|---|---|
| Granite Speech 8B | 72.7166 | 68.6985 | 68.2477 | 0.4508 |
| Granite Speech 2B | 70.5606 | 68.4437 | 68.3653 | 0.0784 |
| Whisper Small | 80.9878 | 108.0165 | 108.0165 | 0.0000 |
| SeamlessM4T Medium | 98.6476 | 104.1552 | 99.9412 | 4.2140 |
For SeamlessM4T, arbitration removes a net 215 word errors relative to Selection. Its interval uses 10,000 paired bootstrap resamples. Granite Speech 8B and 2B improve over Selection by 0.4508 and 0.0784 points, respectively. Against native direct recognition, their reductions are 4.4689 and 2.1953 points. Whisper's aggregate WER is unchanged. Native direct recognition remains better for Whisper and SeamlessM4T, so the 4.2140-point SeamlessM4T benefit specifically measures recovery from Selection errors. This establishes transfer of the post-selection arbitration benefit to SeamlessM4T, alongside gains over both baselines for the Granite models.
To quantify the practical tradeoff, we measured a separate 100-utterance LibriSpeech workload with Granite Speech 8B and Qwen3-0.6B, using five repetitions. Mean end-to-end time rises from 0.6594 seconds per utterance for Selection to 12.4699 for Full LAA, an 18.91-fold increase. Peak allocated GPU memory changes from 18.1752 to 18.1760 GiB. Acoustic arbitration adds little allocated memory but substantial verification time, alongside the 0.4508-point CHiME-6 gain over Selection reported above for this pipeline.
Author Rebuttal (1/2)
W1: What does LAA contribute beyond N-best selection, selective correction, and acoustic rescoring?
Answer: Thank you for recognizing the motivation and the value of combining linguistic preferences with acoustic evidence. LAA separates language-based candidate ranking from acoustic acceptance of local edits, with both backbones frozen. Its key finding is that this division lets a compact text model assist a larger speech model.
To test the compact-refiner premise beyond the original LibriSpeech analysis, we evaluated six refiners from 135M to 32B on the same 647 FLEURS en-US utterances, using Granite Speech 8B and four candidates. Across this 237-fold parameter range, LAA WER varies only from 6.2513% to 6.3066%, a 0.0553-percentage-point spread. Qwen3-0.6B achieves 6.2927% versus 6.3066% for Qwen3-32B, using 1/53.3 as many text-model parameters. These results support compact refiners for correction within the shared candidate space.
To address your concern about existing combination methods, we compared LAA with ROVER-style combination using the same candidate sets. LAA reduces WER by 0.7633 points on LibriSpeech complement, 0.9016 on CHiME-6 and 0.2556 on FLEURS relative to ROVER. These gains show the value of selecting and verifying local edits from the same available hypotheses. The acoustic controls below then test which scoring scope and acceptance rule produce useful corrections.
W2: Can complete-transcript phoneme scores provide reliable evidence for local corrections?
Answer: Complete-transcript CTC provides an overall acoustic compatibility signal, not a calibrated confidence score for each edited span. Our controlled comparison shows that local scoring with routing achieves lower WER than global scoring with routing on all three additional evaluation sets.
To address the mismatch you identified, we compared complete-transcript (global) and local CTC scoring, each with and without routing, alongside Selection and Full LAA. All conditions use Granite Speech 8B, Qwen3-0.6B and four candidates. The sets contain 1,939 LibriSpeech complement utterances, 1,000 CHiME-6 utterances and 647 FLEURS utterances. Values are WER percentages, pp denotes percentage points, and differences are computed before rounding.
Among the variants with routing, local scoring reduces WER relative to global scoring by 0.0260, 0.5096 and 0.0276 points, respectively. Adding routing to local CTC lowers WER from 3.8543% to 3.0447% on LibriSpeech and from 6.6657% to 6.2237% on FLEURS. On CHiME-6, local CTC alone is better, at 67.6009% versus 67.6597% with routing.
Full LAA achieves the lowest WER among the CTC-based variants on LibriSpeech complement, while local CTC with routing performs better on CHiME-6 and FLEURS. These controls show that matching verification to the edited region can improve correction, and quantify the contribution of selective acceptance separately from scoring scope.