acceptodds
Under review as a conference paper at ICLR 2027

LexiAdapt: Auditing Contextual Speech Recognition for Unsupported Phrase Insertions

Abstract

Contextual automatic speech recognition (ASR) uses phrase lists from documents, contacts, or domain vocabularies to recover rare terms. Yet the recognizer can copy a listed phrase that was not spoken. Word error rate (WER) measures errors across all words. Bias-word error rate (B-WER) focuses on listed rare words, while phrase recall measures recovery. None reveals by itself whether the audio supports an accepted edit or whether a safe-looking system rejects almost every edit. We introduce LexiAdapt: it selects contextual edits without the reference and audits the transcript against it afterward. LexiAdapt tracks where each phrase came from. No-leakage lists exclude the target transcript. It also fixes the acceptance rule before evaluation. Its scorecard reports separate error rates for listed rare words and all other words. It also reports phrase precision and recall, their harmonic mean, and unsupported additions. Coverage records how often the system accepts a changed contextual transcript. Overall WER and real-time factor capture transcription quality and cost. The comparison contract covers a primary modern contextual model, a larger streaming contextual model, a Whisper anchor, and a no-context control. The design requires disjoint development and test units, frozen phrase sources and gates, matched contextual interfaces, and uncertainty at the highest independent unit. This makes the central question falsifiable: under matched conditions, does context improve bias-word and phrase recovery without increasing unsupported additions, collapsing coverage, or imposing unacceptable latency? The protocol prevents lower WER or zero observed insertions under near-total abstention from being treated as sufficient evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Contextual automatic speech recognition, phrase biasing, unsupported phrase insertions, speech recognition evaluation, leakage-aware evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ETqDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LexiAdapt, an evaluation protocol for contextual ASR that distinguishes useful phrase recovery from unsupported phrase insertion. It separates phrase sourcing, reference-independent transcript selection, and reference-based auditing, and jointly reports recognition accuracy, phrase recovery, insertion counts, acceptance coverage, and latency. Whisper-family diagnostics on LibriSpeech, FLEURS, and meeting audio motivate the protocol, but their execution provenance remains unresolved, and the proposed modern-model comparison has not been run. The contribution is therefore an evaluation framework with preliminary diagnostics rather than a validated recognition method or benchmark.

Critical strengths

The strongest contribution is the explicit treatment of insertion risk alongside recovery and acceptance coverage. Reporting these quantities together prevents a gate that rejects nearly every contextual edit from appearing successful merely because it produces few unsupported additions. This is a useful evaluation principle for contextual ASR, where aggregate WER can obscure errors involving the supplied vocabulary. The separation of reference-free selection from subsequent auditing also provides a clear basis for testing whether contextual gains survive without access to the target transcript.

Critical weaknesses

The principal weakness is the absence of a verified empirical evaluation. Section 3.4 reports no values for the required B-WER/U-WER metric lanes, the modern-model comparison is unexecuted, and the numerical results lack verified connections between inputs, checkpoints, predictions, and scores. These are substantive gaps: the reported gains and transfer failures cannot establish the protocol’s empirical value, and the supplementary implementation does not reproduce them.

A second weakness concerns the validity of the central audit target. Reference absence establishes a transcript mismatch, but does not by itself establish that a phrase lacks acoustic support. The strict count further excludes near-reference variants using a fixed 0.82 string-similarity threshold without human listening validation. Because unsupported insertion is the paper’s central outcome, uncertainty in this distinction directly affects the interpretation of its safety findings, particularly across languages.

Questions for authors

For clarification, is any reported experiment supported by a complete, reproducible record of the executed phrase lists, checkpoint, predictions, gate, and scorer? If so, which results can be independently reconstructed, and do they include the full scorecard on a disjoint evaluation set?

For clarification of measurement validity, what evidence supports treating reference-absent phrases as acoustically unsupported and excluding variants at the 0.82 threshold? Do raw counts, threshold sensitivity, or any existing listening audit preserve the conclusions? These questions concern the submitted evidence rather than suggestions for future work.

3 comments

The manuscript defines an unsupported addition as a listed phrase absent from the normalized reference, then separately flags near-reference variants at a fixed 0.82 string-similarity threshold (Section 3.1, p. 3). This is a reference-based text audit; it does not by itself establish that the waveform lacks acoustic support. The paper reports no listening audit (Section 6, p. 9). Table 2 retains raw and strict counts: the AMI/QMSum prompt adds four reference-absent phrases, one treated as a near-reference variant, leaving three additional strict false positives (p. 7).

The appendix varies source-support and transcript-distance rules but does not report sensitivity to the 0.82 near-reference cutoff (Appendix Tables 8–10, pp. 16–17). These analyses do not validate the acoustic interpretation of the audit label. The available results support describing the values as reference-based audit counts, with raw and thresholded values shown separately.

A complete scorecard on a disjoint set is also unavailable in the submitted paper: run lineage remains unresolved, and the required B-WER/U-WER lanes have no reported values (Sections 3.4 and 5, pp. 4 and 6).

Thank you for your response. However, as not all of my concerns have been resolved, I will keep my score unchanged.

AuthorsOpenReview ↗

Threshold sensitivity and interpretation

Correction: the earlier sentence saying that we added the grid to the revised manuscript was inaccurate. The current PDF does not include it; we provide the full grid here. This post-hoc sensitivity check uses existing AMI/QMSum validation predictions, with no new inference (6,003 utterances and 73,186 supplied-distractor exposures per model/context condition).

Strict text-mismatch totals at cutoffs 0.70 / 0.75 / 0.80 / 0.82 / 0.85 / 0.90 / 0.95 / 1.00:

  • Whisper large-v3-turbo, no context: 0 / 0 / 0 / 1 / 1 / 1 / 3 / 4
  • Whisper large-v3-turbo, prompt: 3 / 3 / 3 / 4 / 4 / 4 / 6 / 8
  • Whisper medium, no context: 0 / 0 / 0 / 1 / 1 / 1 / 3 / 3
  • Whisper medium, prompt: 2 / 2 / 2 / 3 / 3 / 3 / 5 / 6

At 0.82, raw reference-absent phrase totals are 4 / 8 / 3 / 6 in the order above; 3 / 4 / 2 / 3 are classified as near-reference, leaving strict totals of 1 / 4 / 1 / 3. Near-reference means the best SequenceMatcher ratio against normalized reference windows within one token of phrase length.

This sweep shows how a reference-based text label changes with the cutoff. It cannot establish that a phrase lacks waveform support; no listening audit was run. The submission still lacks a complete verified input-to-checkpoint-to-prediction-to-gate-to-scorer chain and reported B-WER/U-WER values, so we do not claim acoustic validation or reconstruction of the full scorecard.

Reviewer_gzGbOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents LexiAdapt, a framework for evaluating contextual ASR systems. It focuses on whether models recover useful phrases without inserting phrases that were never spoken. The method considers phrase recovery, unsupported insertions, overall WER, and how often contextual edits are accepted. The paper presents diagnostic experiments on several speech datasets, but the planned comparison of modern ASR models remains incomplete.

Critical strengths
  • The paper highlights an important problem in contextual ASR: a model may insert phrases that were never spoken. Evaluating phrase recovery, unsupported insertions, and acceptance coverage together is useful.

  • The paper gives clear attention to data leakage and phrase provenance. Its emphasis on fixed acceptance rules and separate evaluation sets is a good direction for reliable evaluation.

Critical weaknesses
  • The reported results are difficult to verify because the exact inputs, model versions, and run details are not fully documented. The main comparison of modern ASR models has also not been completed.

  • The proposed gate accepts very few changes on AMI/QMSum. Although it avoids unsupported insertions in this setting, it is unclear whether the method can provide useful improvements at a higher acceptance rate. The transfer experiments also do not establish reliable gains on meeting data.

Questions for authors
  • Can the authors release the exact inputs, model versions, and outputs needed to reproduce the reported results?

  • The proposed gate accepts only 5 of 1,545 changed candidates on AMI/QMSum. Can the authors show that it remains useful at higher coverage while avoiding unsupported insertions?

  • Why is the comparison with Qwen3-ASR, VibeVoice, Whisper, and Parakeet still incomplete? Can the authors provide results under matched evaluation conditions?

  • Some runs do not retain multiple candidate transcripts. Which experiments genuinely evaluate acoustic selection using distinct candidates and their scores?

4 comments

The submitted manuscript and package do not contain a complete, verified input–model–prediction–score record. Section 5 (p. 6) identifies unresolved input, checkpoint, and result lineage. The comparison with Qwen3-ASR, VibeVoice, Whisper-large-v3-turbo, and Parakeet is explicitly unexecuted, so matched results for those systems are not available.

On AMI/QMSum, the source-and-distance gate accepts 5 of 1,545 changed candidates, or 0.32% coverage (Table 2, p. 7). The 129-meeting diagnostic records 132 accepted edits, including 47 unsupported additions, for the permissive gate, while the strict gate accepts two; the paper labels these results diagnostic because run lineage is unresolved (Section 5.3, p. 8; Appendix Table 11). No verified result shows safe behavior at higher coverage.

The saved AMI/QMSum validation output retains one candidate transcript per utterance, so it cannot test acoustic selection among genuine alternatives (Section 3.3, p. 4). A separate repaired-input check is incomplete and does not verify aggregate scores or candidate diversity. The supplied results therefore do not establish an acoustic-selection effect.

I wish to keep my original evaluation score, as the current rebuttal response dont adequately address my previously shared concerns.

Thank you for the follow-up. We understand that our response did not adequately address your concerns and that you will retain your original evaluation score. We appreciate the time and care you gave the submission.

AuthorsOpenReview ↗

Direct answers on evidence gaps

To answer the underlying questions directly: the submitted paper and package do not contain a complete, verified record linking executed phrase-list inputs, checkpoint revisions, predictions, gates, scorers, and reported metrics. Section 3.4 also reports no B-WER/U-WER values, so the main scorecard cannot be independently reconstructed from the submitted record.

The reported AMI/QMSum source-and-distance gate accepts 5 of 1,545 changed candidates (0.32% coverage). The disjoint ICSI replay reports unchanged phrase recall and higher WER for the fixed hybrid than its no-context anchor; no submitted result demonstrates useful behavior at higher coverage. The comparison with Qwen3-ASR, VibeVoice, Whisper, and Parakeet was not run. The saved AMI/QMSum output retains one transcript per utterance, so it does not evaluate acoustic selection among distinct candidates.

Reference absence and the fixed 0.82 text-similarity cutoff are text-audit rules. The submitted paper reports neither threshold sensitivity nor a listening audit, so these counts do not establish that the audio lacks support. The current record supports the scorecard design and finite-set text diagnostics; reproducibility, acoustic support, and a useful higher-coverage gate remain unestablished.

Reviewer_YT9mOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces LexiAdapt, an auditing protocol for contextual ASR that evaluates phrase recovery alongside unsupported insertions, overall WER, edit acceptance coverage, and latency. It separates contextual candidate generation, reference-independent acceptance, and post-hoc auditing, with emphasis on leakage control and fixed evaluation rules.

Experiments on LibriSpeech, FLEURS, and meeting audio suggest improved phrase recovery in controlled settings, but document-context prompting can substantially degrade WER. Acceptance gating mitigates this issue at the cost of extremely low edit coverage. Importantly, unresolved data provenance and execution lineage prevent the reported results from establishing reliable generalization. The paper offers a useful evaluation framework, but its empirical validation remains incomplete.

Critical strengths
  1. Clear problem formulation and evaluation design. The paper identifies an important limitation of contextual ASR evaluation: improved phrase recall or WER does not guarantee that inserted phrases are supported by the audio. Its joint evaluation of recognition quality, unsupported insertions, edit coverage, and latency provides a useful and principled framework.
  2. Practical relevance and informative diagnostics. The document-context experiments expose a meaningful trade-off between phrase recovery and insertion risk, showing why acceptance gating and coverage must be evaluated together. This insight is useful for developing more reliable contextual ASR systems, although the current empirical evidence remains preliminary.
Critical weaknesses
  1. Insufficiently verified empirical results. The paper acknowledges unresolved data provenance, model revisions, and prediction-to-result lineage. Consequently, the reported improvements cannot be reliably reproduced or attributed to the proposed protocol, substantially weakening the empirical claims.
  2. Lack of convincing generalization and validation. The planned comparison across modern ASR models remains unexecuted, while document-context experiments show either increased insertion risk or extremely low edit coverage. These limitations prevent the paper from demonstrating that LexiAdapt provides a practically useful and generalizable solution beyond a promising evaluation framework.
Questions for authors
  1. Empirical reproducibility: Can the authors provide verifiable links between the exact input data, model/checkpoint revisions, predictions, and reported metrics, and reproduce the main results from these artifacts?
  2. Generalization and utility: Can the authors provide a completed, leakage-controlled evaluation on disjoint meeting data demonstrating that the acceptance gate improves phrase recovery while controlling unsupported insertions at a practically meaningful edit-coverage level? The current results suggest that safer gating may achieve near-zero insertions primarily through abstention.
1 comment

The submitted paper states that earlier runs lack jointly verified input bytes, model revisions, and run-to-result lineage (Section 5, p. 6). It therefore does not establish a reproducible path from phrase lists and checkpoints through predictions to scores. Section 3.4 (p. 4) also reports no values for the required B-WER/U-WER lanes.

The meeting results do not show useful transfer at meaningful coverage. On AMI/QMSum validation, the source-and-distance gate accepts 5 of 1,545 changed candidates, or 0.32% coverage (Table 2, p. 7). The disjoint ICSI replay leaves recall unchanged and has higher WER for the fixed hybrid than for its no-context anchor (Section 5.3, p. 8). These diagnostics do not establish a verified held-out utility gain. The current evidence supports the scorecard design and these finite-set observations; it does not establish reproducibility or practical generalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.