- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces LexiAdapt, an evaluation protocol for contextual ASR that distinguishes useful phrase recovery from unsupported phrase insertion. It separates phrase sourcing, reference-independent transcript selection, and reference-based auditing, and jointly reports recognition accuracy, phrase recovery, insertion counts, acceptance coverage, and latency. Whisper-family diagnostics on LibriSpeech, FLEURS, and meeting audio motivate the protocol, but their execution provenance remains unresolved, and the proposed modern-model comparison has not been run. The contribution is therefore an evaluation framework with preliminary diagnostics rather than a validated recognition method or benchmark.
Critical strengths
The strongest contribution is the explicit treatment of insertion risk alongside recovery and acceptance coverage. Reporting these quantities together prevents a gate that rejects nearly every contextual edit from appearing successful merely because it produces few unsupported additions. This is a useful evaluation principle for contextual ASR, where aggregate WER can obscure errors involving the supplied vocabulary. The separation of reference-free selection from subsequent auditing also provides a clear basis for testing whether contextual gains survive without access to the target transcript.
Critical weaknesses
The principal weakness is the absence of a verified empirical evaluation. Section 3.4 reports no values for the required B-WER/U-WER metric lanes, the modern-model comparison is unexecuted, and the numerical results lack verified connections between inputs, checkpoints, predictions, and scores. These are substantive gaps: the reported gains and transfer failures cannot establish the protocol’s empirical value, and the supplementary implementation does not reproduce them.
A second weakness concerns the validity of the central audit target. Reference absence establishes a transcript mismatch, but does not by itself establish that a phrase lacks acoustic support. The strict count further excludes near-reference variants using a fixed 0.82 string-similarity threshold without human listening validation. Because unsupported insertion is the paper’s central outcome, uncertainty in this distinction directly affects the interpretation of its safety findings, particularly across languages.
Questions for authors
For clarification, is any reported experiment supported by a complete, reproducible record of the executed phrase lists, checkpoint, predictions, gate, and scorer? If so, which results can be independently reconstructed, and do they include the full scorecard on a disjoint evaluation set?
For clarification of measurement validity, what evidence supports treating reference-absent phrases as acoustically unsupported and excluding variants at the 0.82 threshold? Do raw counts, threshold sensitivity, or any existing listening audit preserve the conclusions? These questions concern the submitted evidence rather than suggestions for future work.
3 comments
Thank you for your response. However, as not all of my concerns have been resolved, I will keep my score unchanged.
Threshold sensitivity and interpretation
Correction: the earlier sentence saying that we added the grid to the revised manuscript was inaccurate. The current PDF does not include it; we provide the full grid here. This post-hoc sensitivity check uses existing AMI/QMSum validation predictions, with no new inference (6,003 utterances and 73,186 supplied-distractor exposures per model/context condition).
Strict text-mismatch totals at cutoffs 0.70 / 0.75 / 0.80 / 0.82 / 0.85 / 0.90 / 0.95 / 1.00:
- Whisper large-v3-turbo, no context: 0 / 0 / 0 / 1 / 1 / 1 / 3 / 4
- Whisper large-v3-turbo, prompt: 3 / 3 / 3 / 4 / 4 / 4 / 6 / 8
- Whisper medium, no context: 0 / 0 / 0 / 1 / 1 / 1 / 3 / 3
- Whisper medium, prompt: 2 / 2 / 2 / 3 / 3 / 3 / 5 / 6
At 0.82, raw reference-absent phrase totals are 4 / 8 / 3 / 6 in the order above; 3 / 4 / 2 / 3 are classified as near-reference, leaving strict totals of 1 / 4 / 1 / 3. Near-reference means the best SequenceMatcher ratio against normalized reference windows within one token of phrase length.
This sweep shows how a reference-based text label changes with the cutoff. It cannot establish that a phrase lacks waveform support; no listening audit was run. The submission still lacks a complete verified input-to-checkpoint-to-prediction-to-gate-to-scorer chain and reported B-WER/U-WER values, so we do not claim acoustic validation or reconstruction of the full scorecard.
The manuscript defines an unsupported addition as a listed phrase absent from the normalized reference, then separately flags near-reference variants at a fixed 0.82 string-similarity threshold (Section 3.1, p. 3). This is a reference-based text audit; it does not by itself establish that the waveform lacks acoustic support. The paper reports no listening audit (Section 6, p. 9). Table 2 retains raw and strict counts: the AMI/QMSum prompt adds four reference-absent phrases, one treated as a near-reference variant, leaving three additional strict false positives (p. 7).
The appendix varies source-support and transcript-distance rules but does not report sensitivity to the 0.82 near-reference cutoff (Appendix Tables 8–10, pp. 16–17). These analyses do not validate the acoustic interpretation of the audit label. The available results support describing the values as reference-based audit counts, with raw and thresholded values shown separately.
A complete scorecard on a disjoint set is also unavailable in the submitted paper: run lineage remains unresolved, and the required B-WER/U-WER lanes have no reported values (Sections 3.4 and 5, pp. 4 and 6).