acceptodds
Under review as a conference paper at ICLR 2027

Towards Quantifying Benchmark Optimization in ASR Models

Abstract

Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic agreement. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Models respond to narrow acoustic cues to reproduce the reference transcript instead of the audio-faithful output. This benchmark-conditioned behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

Primary area:
datasets and benchmarks
Keywords:
Automatic Speech Recognition, Benchmarking, Data Contamination, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dwcwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a way to measure benchmark optimization in ASR. It introduces three techniques to identify how the reference depends on the underlying audio, i.e. reference disagreement, masked-number recovery and orthographic agreement. The authors also show that finetuning the current models on a full epoch of the training set of a dataset, can change the leaderboard by significant margin.

Critical strengths
  1. The paper does systematic experimentation to measure a well-known phenomenon of benchmaxxing for ASR. The paper does multiple forms of interventions to compute the ACCEPT-REF score to show how reliable the ASR predictions are for different open models.

  2. The fine-tuning experiments shows that just a single epoch on the public training sets of the target datasets can change the leaderboard quality of open models by a significant amount. This shows that the top models in the leaderboard which often differ by a few points may be using the benchmark specific annotations or conventions and may not necessarily be good models.

  3. The paper is in general easy to read and the experimental setup seems reproducible with the manuscript.

Critical weaknesses
  1. Reference disagreement ACCEPT-REF with acoustic context changes is missing for the fine-tuned model. The authors should reproduce Table-2 for the finetuned model from Section 4.3. This would show how much of those behaviors are shown in the fine-tuned models as well.
Questions for authors
  1. Can the authors share what the 1141 reference errors break down by some category of errors?
Reviewer_wWbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors test eleven ASR models for benchmark-specific behavior using incorrect reference transcripts, silenced numbers, and alternative spellings that sound the same. Changes to the surrounding audio and model activations affect reference matching, and fine-tuning on public training splits can improve benchmark scores while worsening transcription against corrected references for the same audio.

Critical strengths
  1. The reference-error probe is a simple, reusable check on leaderboard WER. The leading models reproduce reference errors much more often than the rest (Fig. 1a), which helps explain their advantage on VoxPopuli.
  2. The context experiments in Table 2 (truncation, appended audio, steering) have clip-level bootstrap intervals and turn the behavior on and off within the same models, which is stronger evidence than a correlation across models.
  3. The paper is generally easy to read.
Critical weaknesses
  1. The interpretation depends on what counts as a reference error. VoxPopuli uses official parliamentary transcripts, and the paper notes systematic courtesy-phrase omissions. Without separating editing conventions from incorrect words, the results may partly measure annotation-style matching. Masked numbers can also be inferred from context, while normalized spelling choices do not change WER (VoxPopuli).
  2. Audio lift compares speech with complete silence, so it can reflect useful context rather than benchmark-specific cues. Figure 2b also shows substantial positive lift for generic voices in some leading models, weakening the claim that it collapses toward zero.
  3. Overall, the technical novelty is fairly limited, which is why I am leaning toward a weak reject.
Questions for authors
  1. How do reference-matching rates differ for courtesy omissions, disfluencies, and genuine word substitutions? What checks were used for the corrected annotations?
  2. Do steering and splicing improve corrected-reference WER on fully held-out clips without damaging correct words?
2 comments

We thank the reviewer for the careful review.

We added a breakdown by error category (Section 4, Table 2). The leading models reproduce courtesy-phrase omissions most often (0.70 vs 0.18), but the gap persists for content-word errors (0.17 vs 0.07). Disfluencies are rare because the normalizer removes filled pauses and the probe does not score mid-clip omissions.

CategoryErrorsLeading sixOther fiveDifference [95% CI]
Function word3630.480.12+0.36 [+0.32, +0.41]
Content word1310.170.07+0.10 [+0.06, +0.14]
Inflection320.340.09+0.25 [+0.14, +0.37]
Courtesy phrase310.700.18+0.51 [+0.37, +0.67]
Disfluency100.740.40+0.34 [+0.16, +0.58]
All5670.420.11+0.31 [+0.28, +0.34]

The corrected transcripts were manually reviewed and corrected as laid out by Artificial Analysis. We manually reviewed 100 edits during our experiments and agreed with the annotators' labels on 98 (the remaining 2 were ambiguous). In our original experiments, we used an ensemble of models, treating edits on which three of four models agreed as ground truth. The same six models lead under both sources.

On masked numbers, the voices read the same masked sentences, yet audio lift drops on generic voices for Cohere, Canary-Qwen, Phi-4 and Higgs, and rises for Whisper. The words around the number are the same in each voice condition. We agree that generic-voice lift does not always reach zero, and have corrected that sentence. With regards to the background silence, we tested replacing the silence with both background noise from VoxPopuli (speech removed) and Gaussian noise, and found results were within 0.06 nats per character of silence across all ten models. By contrast, real speech moves it far more (0.15–0.69 nats per character for seven of the ten models).

We agree that orthographic agreement does not change normalized WER. We treat it as supporting evidence that models recognize the benchmark, rather than evidence of transcription errors. We added a new test to the supplementary material where we re-synthesized VoxPopuli sentences, whose references originally write "Mr", in a cloned LibriSpeech narrator's voice (LibriSpeech reference transcripts use "Mister"). Cohere, Canary-Qwen, Parakeet and Higgs then write "Mister" on 28–57% of sentences, versus 0–2% in a cloned VoxPopuli voice.

In the steering table, now in the main text (Table 4), our steering vector on a single layer (estimated from 22 clips) lowers corrected-reference WER by 1.4–4.2 points for Cohere, Canary-Qwen, Parakeet and Granite (Phi-4 unchanged) on held-out clips. A random direction changes WER by at most 0.1 points. On words both references agree on, Cohere, Canary-Qwen and Granite fix more than they break; Parakeet's net change is not significant. Splicing is a diagnostic rather than a fix: a conversational donor lowers WER for Canary-Qwen, Parakeet and Phi-4 (by 1.2–2.3 points) but raises it for Higgs (by 4.4 points).

On novelty, our core insight is that how models handle ambiguity gives insight into whether a model is reacting to benchmark-specific cues rather than pure phonetic signal. We instantiate this idea with three probes, but we think that others can build on this core insight, for instance finding ambiguous cases in sequence-to-sequence tasks like translation.

To our knowledge, this is the first paper to measure benchmark optimization in ASR from model behavior on the benchmark itself, through the lens of ambiguous cases. Many in the field suspect benchmark optimization in ASR research (Bezzam et al., 2026); quantifying that suspicion is a worthy research objective, and we believe we've contributed new knowledge in this regard.

Our probes cover 11 models; a new comparison with published results (Sections S1.11 and S2.3) suggests the field faces the same question: do reported gains reflect better transcription or benchmark optimization? Section 4.3 shows that training on a benchmark's training split can lower its WER while raising WER against corrected references. Most of the 44 sampled gains over Whisper on these benchmarks come from systems trained on the benchmark's own data, and one epoch of fine-tuning beats most of them. Our probes can help measure how much benchmark-conditioned behavior these systems show.

I appreciate the authors for providing a response to the original review. While I agree that measuring the ASR benchmark optimization can be useful to some in the community, the overall contribution and novelty are why I am keeping my original scores.

Reviewer_RiMtOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper by the anonymous authors highlights an important aspect of ASR model research. It investigates how certain open source ASR models might be overstating their most important metric, namely the WER by incorporating benchmark specific artifacts ("Benchmaxxing"). It highlights the issue of training effectively on the metric and how in such a case the metric ceases to be a good measure of the model's performance.

They explore quantifying this characteristic by means of three proposed probes: reference disagreements, word/numerics silencing, and orthographic agreement. They also define an audio lift metric that quantifies how much a model's likelihood is raised for a reference transcript for real audio compared to a silences audio. They then study 11 widely used open source models and try to find out how each model responds to each intervention.

They find that the best models will be able to reproduce reference transcript's errors. But this ranking reverses when a corrected transcript is used instead, so the behaviour might be based on narrow accoustic cues inside the recordings and not the words themselves. This behavior can be activated/reduced by appending audio or steering model activations. Also, when fine tuning on public training splits,the bad models will improve WER scores but will do worse on corrected transcripts. They have three recommendations: transparency about training data, not using iid test splits, and to use multiple metrics to judge models for widespread use.

This is a important research topic in ASR Models as it helps estimate real-world performance of some of the leading models. It also adds another tool to the model user's kit who might be looking to use the best open source models for their tasks. It does so by giving a sense of direction of which way the models have been tuned and how better/worse the actual performance on new data might be. I believe it can be one of the 'soft' metrics considered when evaluating models for deployment.

Critical strengths

This paper explores in detail how bench scores might differ from actual real-world performance. They use a few useful metrics, ACCEPT-REF which looks at what fraction the model reproduces the reference as compared to the real transcript. They also define the metric audio lift, which characterizes by how much a model would have a higher likelihood of reference set as compared to silenced word. Their probes, reference disagreement, masked identity recovery and orthographic agreement are able to give a sense of how much a model is benchmaxxed and how one can identify such models. The use of corrected transcripts by using VoxPopuli-Cleaned-AA to highlight reference disagreement is an innovative technique that gives a clear evidence that the model is reproducing the benchmark's internal reference. The use of masked identity recover and orthographic agreement supports the overall point of the paper.

The protocols are well designed and the use of EP-fresh to ensure newer unseen data is evaluated suports conclusions. The fact that that the same models are being affected under the probes also gives confidence in the probes strengths. The fine tuning also shows that this behaviour can arive from jsut the public training split.

Critical weaknesses
  • I find the probe orthographic agreement a weak indicator. From a layman's point of view, a model transcribing to Mr or Mister makes no difference to the final transcription accuracy or quality. Using it as a metric might help support the other probes conclusions but it feels lacking from a practical perspective. zSince it measures the style and not accuracy, it cannot easily differentiate formatting preference from benchmaxxing
  • The methodology of the paper is strong, but it introduces another metric that might be trained for. Also, human corrected datasets like VoxPopuli-Cleaned-AA to calculate ACCEPT-REF scores will not be that simple to find.
  • The audio lift is something that can scale, but it feels like an adversarial metric. Model training would presumably not include pure silence, so a reproduction won't be language-model refenrence. The masking can still leak cues for the model which will be punished by the metric.
Questions for authors

Clarification requests:

  • For figure 1.b where WER ranking changes on VoxPopuli Cleaned AA: The difference between certain models clusters ([Canary-Qwen, Granite] or [ Higgs, Phi-4, Parakeet] ) may not be significant. Can you provide a a quantitative rank measure for all the models considered?
  • Fig 1.b: Can you provide an evaluation of the same sized VoxPopuli data for the uncorrected and corrected version? The x-axis is with the uncorrected verison and the y-axis has a smaller dataset of 628 clips only.

Followup suggestions:

  • An interesting direction to explore would be to use the three probes of the paper to build a WER score adjuster for ASR models. This would bring down the scores of benchmaxxed models while retaining scores of models that do faithful transcription. A prerequisite towards building such a tool would be human-corrected references.
  • It would be interesting to show background effects by overlaying one dataset's background to the other set and see how the model changes transcription. Also interesting would be an investigation into how accents affect the model transcription.
  • Another interesting experiment is to add background cues in gradually to see where the bench hacking gets activated across different training sets and across different models. This can help answer if this a switching behaviour or gradual behaviour.
2 comments

We thank the reviewer for the careful review and follow up suggestions. We agree that using the same 628 clips makes the figure more interpretable, and Figures 1a and 1b now both use them, scored against the original and corrected references. We also agree that the main point is that order between clusters moves significantly (four of the bottom five by original WER overtake the initial six leaders on the corrected references), while movement within clusters is noisier. We added rank intervals to Table S2, summarized below; they show which within-cluster differences are significant.

ModelRank, original [95%]Rank, corrected [95%]
Kimi-Audio9 [9–9]1 [1–1]
Voxtral7 [7–8]2 [2–2]
Qwen3-ASR8 [7–8]3 [3–4]
Moonshine10 [10–11]4 [3–5]
Parakeet5 [1–6]5 [5–8]
Whisper11 [10–11]6 [3–11]
Higgs1 [1–5]7 [5–9]
Canary-Qwen2 [1–5]8 [6–10]
Phi-46 [3–6]9 [6–10]
Granite3 [1–6]10 [7–10]
Cohere4 [1–6]11 [10–11]

We acknowledge that the orthographic switching probe is "less informative on its own." However, in addition to the other probes, we believe it provides evidence that models 'know' which benchmark they are operating on. Practitioners have anecdotally pointed to surprisingly accurate formatting choices as an early sign that a model may be tuned to a specific benchmark. We wanted to capture this signal, while being clear that high orthographic agreement is not meant as smoking-gun evidence of test-set contamination or the like. We added a new test to the supplementary material where we re-synthesized VoxPopuli sentences, whose references originally write "Mr", in a cloned LibriSpeech narrator's voice (LibriSpeech reference transcripts use "Mister"). Cohere, Canary-Qwen, Parakeet and Higgs then write "Mister" on 28–57% of sentences, versus 0–2% in a cloned VoxPopuli voice. Whisper, Phi-4 and Voxtral write "Mister" 0–1% of the time in every voice; Qwen3-ASR, Kimi and Moonshine switch weakly; and Granite writes "Mister" on at most 12% of sentences in any synthetic voice.

On gamability, ideally, a metric points towards the generalizability of a model. We think high ACCEPT-REF scores point to lower generalizability, but acknowledge there is no perfectly ungameable metric. As our conclusion recommends, practitioners should look at multiple metrics where possible, meaning reading ACCEPT-REF alongside WER. "Gaming" ACCEPT-REF also becomes harder without public i.i.d. training and test splits.

On the note of ground-truth transcripts, in our original experiments, we sorted the models by phoneme error rate and whitelisted the 4 models with the lowest phoneme error rate. We used these models as an ensemble, treating edits on which three of the four models agreed as ground truth. The same six models led, but the exact numbers changed slightly. This ensemble methodology would generalize to data without human ground truth labels.

With regards to the background silence, we tested replacing the silence with both background noise from VoxPopuli (speech removed) and Gaussian noise, and found results were within 0.06 nats per character of silence across all ten models. Any cue that survives masking is present in every voice condition, since all are masked identically, yet audio lift drops on generic voices for Cohere, Canary-Qwen, Phi-4 and Higgs, and rises for Whisper.

My review comments have been sufficiently addressed. I recommend the paper for acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of revisions

We thank the reviewers. We have uploaded a revised PDF and supplement with the following changes. New and changed text is in purple.

  • Figure 1 now scores every model on the same 628 VoxPopuli-Cleaned-AA clips in both panels, against the original and the corrected references (RiMt). Table S2 adds rank intervals and Kendall's τ between the two rankings.
  • Table 3 adds Whisper after one epoch on the public training splits, so its response to acoustic context can be compared with the stock models (dwcw).
  • Section 3.3 now states that the 1,141 reference errors are word-level edits and that ACCEPT-REF scores 567 span-level edits in 338 clips (dwcw).
  • Section 4.1 now says that generic-voice audio lift decreases for Cohere, Canary-Qwen, Phi-4 and Higgs rather than reaching zero (wWbc).
  • The steering results moved from the supplement into Section 4.2 (Table 4), including WER on held-out clips against the corrected references with and without the steering direction (wWbc). Table 3 no longer repeats the steering columns.
  • Section 4 and a new Table 2 break the reference errors down by category: function word, content word, inflection, courtesy phrase and disfluency (wWbc, dwcw).
  • Section 4 summarizes a new voice-swap test (details in Section S1.10 and Table S8): the same sentences synthesized in a cloned VoxPopuli voice and a cloned LibriSpeech voice (RiMt, wWbc).
  • Section 5 corrects the WER threshold from 6.4% to 6.3%: below it, against the original references of the 628 clips, a model must reproduce reference errors.
  • Section 5 now recommends comparing benchmark gains against a baseline fine-tuned on the same training data, and notes that three of the six leading models report VoxPopuli WER without saying whether they trained on it (wWbc, RiMt).

We also found and fixed an error in our Cohere-Transcribe inference wrapper. In batches containing clips longer than 35 seconds, it misassigned transcripts to clips. This affected 15 of the 1,842 VoxPopuli clips and 213 synthetic and spliced clips used in Tables 3. We re-decoded the affected clips and updated the results. Cohere's WER on VoxPopuli-Cleaned-AA changes from 6.86% to 5.25%. Its ACCEPT-REF remains 0.54, and the Table 3 rates change by at most 0.02. Masked recovery and audio lift are unchanged. With the corrected results, Whisper ranks fifth of seven after fine-tuning, rather than fourth.

New experiment (Sections S1.11 and S2.3; wWbc): We compared our one-epoch Whisper fine-tune with gains over pretrained Whisper reported in recent ASR papers on VoxPopuli, AMI and LibriSpeech. Of 44 reported improvements (Table S10), 39 come from systems trained on the benchmark's own corpus, and our fine-tune's gain exceeds 35 of them. This is not meant to invalidate any individual paper. However, Section 4.3 shows that training on a benchmark's training split can lower its WER while raising WER against corrected references; a gain over pretrained Whisper does not by itself distinguish benchmark optimization, which our probes help measure, from a better general method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.