- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the cleaner-but-wrong phenomenon in generative audio-visual speech enhancement, where enhanced speech exhibits improved acoustic quality but increased word error rates (WER). To address this issue, the authors propose AVEC, an inference-time framework that generates multiple enhancement candidates using audio- and visual-guided sampling and aggregates them based on their consistency with the original observations. Experimental results demonstrate substantial WER reductions while largely preserving speech quality, without retraining the underlying enhancement models.
Critical strengths
-
The paper addresses an interesting and practically relevant problem concerning linguistic content preservation in generative speech enhancement.
-
The proposed inference-time framework does not require retraining the enhancement backbone, making it potentially applicable to existing systems.
-
The experimental results are promising, with ablation studies, multiple enhancement backbones, and human listening evaluations supporting the effectiveness of the approach.
Critical weaknesses
-
Limited comparisons with competitive speech enhancement methods. Although multiple enhancement backbones are evaluated, the experiments primarily demonstrate improvements over their corresponding baseline outputs. Comparisons with recent competitive speech enhancement systems, particularly those designed to mitigate linguistic hallucinations, are insufficient. Such comparisons are important to establish whether AVEC offers meaningful advantages over existing alternatives. Furthermore, the SNR evaluation is limited to three discrete values (−5, 0, and +5 dB). A denser and broader SNR sweep would better characterize the robustness and applicability of the proposed approach.
-
Unclear reliability of ASR-based evaluation for linguistic hallucination. The paper relies heavily on Whisper WER to characterize content degradation. However, the listening study reveals a discrepancy: Whisper WER increases from 0.362 to 0.515 after enhancement, whereas human transcription WER slightly decreases from 0.608 to 0.586. This raises the question of whether the observed degradation reflects genuine linguistic hallucination or ASR-specific recognition biases. Although an independent ASR evaluator is included, a more detailed analysis of the disagreement between automatic and human evaluations would strengthen the central motivation.
-
Insufficient justification of the necessity of visual information. While the paper includes audio-only and visual-only evidence ablations, these experiments do not fully isolate the contribution of visual information, since the candidate pool still contains visually guided outputs. A genuinely audio-only variant, without video inputs or lip-guided candidates, would help establish whether visual evidence is essential and whether the framework can generalize to conventional audio-only speech enhancement.
-
Limited justification for waveform-level aggregation. Directly averaging generated waveforms may introduce phase inconsistencies, temporal misalignment, or speech artifacts, particularly when candidates differ in linguistic content. Moreover, hard candidate selection achieves lower WER than the proposed weighted averaging strategy. The authors should better justify the choice of waveform-level fusion and clarify its advantages over directly selecting the most reliable candidate.
-
Computational overhead and practical applicability. AVEC and AVEC+ require substantially more sampling trajectories, together with additional ASR, VSR, and guidance computations. The computational cost appears considerable relative to the improvements observed on some non-diffusion backbones. A clearer discussion of the performance–efficiency trade-off would strengthen the practical significance of the framework.
Questions for authors
-
To what extent is the cleaner-but-wrong phenomenon a general limitation of generative enhancement models, rather than a consequence of the selected backbone architectures or sampling configurations?
-
Can the authors provide a fully audio-only implementation and evaluate whether comparable gains can be obtained without visual guidance?
-
How well do ASR-based content degradation measurements correlate with actual linguistic errors perceived by human listeners, particularly on samples where automatic and human evaluations disagree?
3 comments
Response to Reviewer HNP9 (2/3)
W2/Q3. Reliability of ASR evaluation and disagreement with listeners
A. Following the reviewer's suggestion, we reanalyzed the 1,060 transcripts from 53 listeners on 30 clips, examining both error types and agreement at the word and clip levels. The analysis explains the apparent discrepancy while independently supporting AVEC's content-preservation benefit (Appendix E.3).
The principal difference is that enhancement changes the composition of listener errors:
| Condition | Human deletions | Human substitutions | Human WER ↓ |
|---|---|---|---|
| Noisy input | 0.390 | 0.198 | 0.608 |
| Always-enhance | 0.256 | 0.291 | 0.586 |
| AVEC | 0.225 | 0.200 | 0.455 |
| AVEC+ | 0.190 | 0.209 | 0.433 |
Deletion and substitution rates are per reference word; insertions are omitted from the table. Always-enhance reduces deletions but increases substitutions by 47%. These opposing changes yield no statistically significant difference in total listener WER (−0.022 [−0.075, +0.033]). Whisper and Parakeet already transcribe the noisy input more accurately than listeners (0.362 and 0.339 vs. 0.608), so their input-relative changes differ.
The improvement from AVEC is consistent across evaluators. Relative to always-enhance, listener WER decreases by −0.131 [−0.196, −0.071] for AVEC and −0.153 [−0.221, −0.093] for AVEC+. Both also improve over the noisy input; AVEC+'s difference is −0.175 [−0.255, −0.098]. These intervals pair the 30 source clips, rather than treating multiple responses to one clip as independent clips. The listener study supports both variants, although their difference in listener WER is not statistically resolved.
At the word level, 86–98% of reference words that Whisper substitutes or deletes on enhanced outputs are also missed by a strict majority of listeners, depending on the policy. For example, the reference “it's natural connective tissue” becomes “it's natural you know fisheries” for Whisper and “it's natural fishery” for a listener after always-enhance; AVEC+ allows both Whisper and at least one listener to recover the reference.
Across the 180 clip-condition outputs, absolute Whisper and listener WER correlate at r = 0.763. Agreement is weaker for changes relative to the noisy input (r = 0.322) and binary harm labels (κ = 0.123). We therefore use ASR WER to characterize recognition performance and report the per-clip indicator as ASR-measured harm, rather than equating every detected increase with audible fabrication. The human evaluation independently corroborates the improvements over always-enhance. We also clarified the WER conventions in Table 4 and corrected AVEC's listener harm to 23.3%, excluding an exact tie previously counted as harm.
W3/Q2. Is visual information necessary, and can the framework operate without video?
A. We addressed this with two complementary controls: a matched-budget ablation of the wrapper's visual roles and evaluation on genuinely audio-only backbones (Appendices F.2–F.3).
First, we independently enabled or disabled lip-guided candidates and the visual witness while keeping six candidates and 36 trajectories in every configuration:
| Lip-guided candidates | Visual witness | Whisper WER ↓ | Parakeet WER ↓ |
|---|---|---|---|
| No | No | 0.349 | 0.358 |
| No | Yes | 0.326 | 0.350 |
| Yes | No | 0.324 | 0.332 |
| Yes | Yes (AVEC+) | 0.278 | 0.306 |
Using both visual roles lowers WER relative to using neither by −0.071 [−0.088, −0.053] under Whisper and −0.052 [−0.070, −0.035] under Parakeet. Each role improves the point estimates individually, and their combination gives the largest improvement. This ablation isolates the wrapper-level visual contributions; the underlying AV backbone still receives video in all four configurations.
Second, we evaluated the acoustic-witness variant on backbones that never receive video. Without visual guidance or a visual witness, WER decreases from 0.294 to 0.286 on PASE (0.280 with hard selection), 0.573 to 0.396 on SGMSE+ WSJ0, and 0.783 to 0.570 on SGMSE+ VB-DMD. Adding the visual witness gives the corresponding WER point estimates of 0.278, 0.378, and 0.561; the final incremental difference is not statistically resolved. Thus, the evidence-scoring principle generalizes to audio-only enhancement, while video provides complementary information through candidate construction and scoring when available.
Response to Reviewer HNP9 (3/3)
W4. Why use waveform aggregation rather than hard selection?
A. We appreciate this question. The revised manuscript presents hard selection and weighted aggregation as two operating points of the same evidence-scoring framework, rather than claiming that averaging is preferable for every objective (Section 5.3; Appendix C.2).
At the frozen τ = 0.10, hard selection has lower Whisper WER (0.267 vs. 0.279), whereas weighted aggregation improves PESQ by 0.008, ESTOI by 0.006, and SI-SDR by 0.11 dB. A diagnostic temperature sweep also shows that a τ = 0.05 mixture reaches WER 0.264 while satisfying all four content–quality criteria. We retain τ = 0.10, selected on train-validation data, rather than choosing a new value on the test set.
Weighted aggregation also reduces sensitivity in the tested evidence-perturbation controls. When both witnesses are deliberately mismatched, it yields WER 0.368 versus 0.422 for hard selection. Changing the candidate recognizer shifts its WER by approximately 0.02, compared with approximately 0.047 for hard selection. Proposition 4 complements these observations by bounding waveform changes under score perturbations for a fixed pool; WER is evaluated empirically.
To address alignment directly, candidate pairs have 0-ms best global lag and 0.96–0.98 correlation. Moreover, aggregation approaches selection when the evidence strongly favors one candidate: the effective pool size is 1.07 on the 38 clips with a score margin of at least 0.30. These measurements support weighted aggregation as a quality-preserving, smoother response to uncertain evidence, while hard selection remains an available option when lower WER is the priority for the evaluated setting.
W5. Computational overhead and practical applicability
A. We have expanded Section 5.6 and Appendix D.3 to make the performance–cost trade-off explicit. On one A6000, the deployed enhancer takes 11 s per clip, a single lip-guided candidate 14 s, AVEC 37 s, and AVEC+ 105 s. AVEC already reduces WER from 0.504 to 0.313 with 12 trajectories, while the higher-budget 36-trajectory AVEC+ reaches 0.279. Repeated diffusion sampling accounts for approximately 95% of the runtime; ASR, VSR, and guidance together are about 5%, and guidance adds under 0.5 s per condition. The six conditions are independent, so running them on six GPUs gives an estimated candidate-generation latency of about 25 s per clip, computed from per-condition timings, with the total sampling work unchanged (Table X).
The value of the additional computation is preserving both content and acoustic enhancement on a fixed backbone. In the added lightweight post-processing comparison, an 805-parameter mixer reaches comparable WER but substantially lower PESQ (1.164 vs. 1.485; Appendix D.4). AVEC+ therefore targets offline use where the enhanced waveform itself matters, rather than real-time recognition.
The same evidence-guided principle also applies to faster backbones: FlowAVSE's eight-output pool takes 0.92 s of sampling, excluding ASR/VSR, and lowers WER from 0.256 to 0.242. We now report this sampling time separately from evidence-extraction cost and distinguish lower-budget configurations from the full diffusion-based AVEC+ setting.
Revisions: Sections 1, 3, and 5.3–5.6; Appendices C.2, D.3–D.4, E.3–E.4, and F.2–F.5; Table 4 caption.
Response to Reviewer HNP9 (1/3)
We sincerely thank the reviewer for recognizing the relevance of content preservation and the applicability of an inference-time approach. Following the reviewer's suggestions, we added competitive-system comparisons, a broader SNR sweep, a detailed analysis of listener–ASR agreement, and controls that isolate the two visual roles. We also expanded the justification of waveform aggregation and the discussion of computational cost. The corresponding revisions and additional results are marked in blue.
W1/Q1. Competitive comparisons, generality of the phenomenon, and SNR coverage
A. We added SGMSE+, SepFormer, and MetricGAN+ to the comparison, bringing the evaluation to eight standalone enhancers, including audio-only and discriminative systems. Appendix F.4 reports the following results on the 657 primary test clips:
The wrapper column reports the backbone-specific configurations in Table XXVI; FlowAVSE and PASE use the two-candidate configuration here. Input-relative ASR harm occurs in 15.4–69.3% of clips across these systems, including discriminative enhancers. We therefore clarify that the quality–content mismatch is not exclusive to generative models.
Importantly, AVEC also improves PASE, which is designed to mitigate linguistic hallucinations, and FlowAVSE, which has the lowest standalone WER in this comparison. This supports evidence-guided aggregation as complementary to stronger enhancement backbones. Against the discriminative SepFormer control, primary-system AVEC+ has lower WER (0.279 vs. 0.460), lower harm (18.1% vs. 40.9%), and higher PESQ (1.485 vs. 1.385), while SepFormer is slightly better on ESTOI and SI-SDR.
We further tested whether the phenomenon could be explained solely by sampling variance or domain mismatch. With 36 unguided trajectories, WER remains 0.450 versus 0.335 for the noisy input. In the enhancer's matched TCD-TIMIT + DEMAND condition, 26.0% of clips are still harmed despite improved acoustic quality. The error composition also differs across model families: substitutions or insertions account for 90–95% of additional ASR errors on harmed clips for AV-UDiffSE+ and SGMSE+ VB-DMD, versus approximately 46% for MetricGAN+. These are recognition-error measurements; their relationship to human transcription is examined separately below.
Broader SNR evaluation. We extended the sweep to −15, −10, −7.5, −5, −2.5, 0, and +5 dB, keeping the aggregation parameters fixed (Appendix F.5). AVEC remains near or below the noisy-input WER across this range. For example, at −7.5 dB, WER is 0.484 / 0.634 / 0.480 for noisy input / always-enhance / AVEC; at −2.5 dB, it is 0.169 / 0.349 / 0.164. At the added levels, AVEC substantially reduces enhancer-induced recognition errors while retaining PESQ and SI-SDR at the enhancer's level. The additional levels evaluate AVEC with K = 4, rather than AVEC+.