- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes a way to measure benchmark optimization in ASR. It introduces three techniques to identify how the reference depends on the underlying audio, i.e. reference disagreement, masked-number recovery and orthographic agreement. The authors also show that finetuning the current models on a full epoch of the training set of a dataset, can change the leaderboard by significant margin.
Critical strengths
-
The paper does systematic experimentation to measure a well-known phenomenon of benchmaxxing for ASR. The paper does multiple forms of interventions to compute the ACCEPT-REF score to show how reliable the ASR predictions are for different open models.
-
The fine-tuning experiments shows that just a single epoch on the public training sets of the target datasets can change the leaderboard quality of open models by a significant amount. This shows that the top models in the leaderboard which often differ by a few points may be using the benchmark specific annotations or conventions and may not necessarily be good models.
-
The paper is in general easy to read and the experimental setup seems reproducible with the manuscript.
Critical weaknesses
- Reference disagreement ACCEPT-REF with acoustic context changes is missing for the fine-tuned model. The authors should reproduce Table-2 for the finetuned model from Section 4.3. This would show how much of those behaviors are shown in the fine-tuned models as well.
Questions for authors
- Can the authors share what the 1141 reference errors break down by some category of errors?
We thank the reviewer for the careful review.
We added a breakdown by error category (Section 4, Table 2). The leading models reproduce courtesy-phrase omissions most often (0.70 vs 0.18), but the gap persists for content-word errors (0.17 vs 0.07). Disfluencies are rare because the normalizer removes filled pauses and the probe does not score mid-clip omissions.
The corrected transcripts were manually reviewed and corrected as laid out by Artificial Analysis. We manually reviewed 100 edits during our experiments and agreed with the annotators' labels on 98 (the remaining 2 were ambiguous). In our original experiments, we used an ensemble of models, treating edits on which three of four models agreed as ground truth. The same six models lead under both sources.
On masked numbers, the voices read the same masked sentences, yet audio lift drops on generic voices for Cohere, Canary-Qwen, Phi-4 and Higgs, and rises for Whisper. The words around the number are the same in each voice condition. We agree that generic-voice lift does not always reach zero, and have corrected that sentence. With regards to the background silence, we tested replacing the silence with both background noise from VoxPopuli (speech removed) and Gaussian noise, and found results were within 0.06 nats per character of silence across all ten models. By contrast, real speech moves it far more (0.15–0.69 nats per character for seven of the ten models).
We agree that orthographic agreement does not change normalized WER. We treat it as supporting evidence that models recognize the benchmark, rather than evidence of transcription errors. We added a new test to the supplementary material where we re-synthesized VoxPopuli sentences, whose references originally write "Mr", in a cloned LibriSpeech narrator's voice (LibriSpeech reference transcripts use "Mister"). Cohere, Canary-Qwen, Parakeet and Higgs then write "Mister" on 28–57% of sentences, versus 0–2% in a cloned VoxPopuli voice.
In the steering table, now in the main text (Table 4), our steering vector on a single layer (estimated from 22 clips) lowers corrected-reference WER by 1.4–4.2 points for Cohere, Canary-Qwen, Parakeet and Granite (Phi-4 unchanged) on held-out clips. A random direction changes WER by at most 0.1 points. On words both references agree on, Cohere, Canary-Qwen and Granite fix more than they break; Parakeet's net change is not significant. Splicing is a diagnostic rather than a fix: a conversational donor lowers WER for Canary-Qwen, Parakeet and Phi-4 (by 1.2–2.3 points) but raises it for Higgs (by 4.4 points).
On novelty, our core insight is that how models handle ambiguity gives insight into whether a model is reacting to benchmark-specific cues rather than pure phonetic signal. We instantiate this idea with three probes, but we think that others can build on this core insight, for instance finding ambiguous cases in sequence-to-sequence tasks like translation.
To our knowledge, this is the first paper to measure benchmark optimization in ASR from model behavior on the benchmark itself, through the lens of ambiguous cases. Many in the field suspect benchmark optimization in ASR research (Bezzam et al., 2026); quantifying that suspicion is a worthy research objective, and we believe we've contributed new knowledge in this regard.
Our probes cover 11 models; a new comparison with published results (Sections S1.11 and S2.3) suggests the field faces the same question: do reported gains reflect better transcription or benchmark optimization? Section 4.3 shows that training on a benchmark's training split can lower its WER while raising WER against corrected references. Most of the 44 sampled gains over Whisper on these benchmarks come from systems trained on the benchmark's own data, and one epoch of fine-tuning beats most of them. Our probes can help measure how much benchmark-conditioned behavior these systems show.