acceptodds
Withdrawn from ICLR 2027

Preserving What Was Said: Evidence-Guided Audio-Visual Speech Enhancement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative audio-visual speech enhancement can improve acoustic quality while increasing speech-recognition errors. Under severe noise, we observe a cleaner-but-wrong regime in which a frozen audio-visual diffusion enhancer improves acoustic quality yet increases word error rate (WER) from 0.335 for the noisy input to 0.504 after enhancement. To mitigate this failure without retraining the enhancer, we propose AVEC (audio-visual evidence consistency), an inference-time framework that evaluates multiple enhancement candidates against complementary evidence from the original observation. AVEC compares each candidate transcript with visible articulation and with the noisy-input transcript, and converts these evidence scores into input-dependent mixture coefficients for waveform aggregation. On the primary evaluation, AVEC reduces WER from 0.504 to 0.313 while keeping acoustic quality close to that of the original enhancer. The higher-budget AVEC+ further reduces WER to 0.279, below the noisy input, while improving reference-based quality metrics. A human listener study further demonstrates the effectiveness of AVEC and AVEC+.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-visual speech enhancement, generative speech enhancement, content preservation, evidence-guided aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HNP9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the cleaner-but-wrong phenomenon in generative audio-visual speech enhancement, where enhanced speech exhibits improved acoustic quality but increased word error rates (WER). To address this issue, the authors propose AVEC, an inference-time framework that generates multiple enhancement candidates using audio- and visual-guided sampling and aggregates them based on their consistency with the original observations. Experimental results demonstrate substantial WER reductions while largely preserving speech quality, without retraining the underlying enhancement models.

Critical strengths
  1. The paper addresses an interesting and practically relevant problem concerning linguistic content preservation in generative speech enhancement.

  2. The proposed inference-time framework does not require retraining the enhancement backbone, making it potentially applicable to existing systems.

  3. The experimental results are promising, with ablation studies, multiple enhancement backbones, and human listening evaluations supporting the effectiveness of the approach.

Critical weaknesses
  1. Limited comparisons with competitive speech enhancement methods. Although multiple enhancement backbones are evaluated, the experiments primarily demonstrate improvements over their corresponding baseline outputs. Comparisons with recent competitive speech enhancement systems, particularly those designed to mitigate linguistic hallucinations, are insufficient. Such comparisons are important to establish whether AVEC offers meaningful advantages over existing alternatives. Furthermore, the SNR evaluation is limited to three discrete values (−5, 0, and +5 dB). A denser and broader SNR sweep would better characterize the robustness and applicability of the proposed approach.

  2. Unclear reliability of ASR-based evaluation for linguistic hallucination. The paper relies heavily on Whisper WER to characterize content degradation. However, the listening study reveals a discrepancy: Whisper WER increases from 0.362 to 0.515 after enhancement, whereas human transcription WER slightly decreases from 0.608 to 0.586. This raises the question of whether the observed degradation reflects genuine linguistic hallucination or ASR-specific recognition biases. Although an independent ASR evaluator is included, a more detailed analysis of the disagreement between automatic and human evaluations would strengthen the central motivation.

  3. Insufficient justification of the necessity of visual information. While the paper includes audio-only and visual-only evidence ablations, these experiments do not fully isolate the contribution of visual information, since the candidate pool still contains visually guided outputs. A genuinely audio-only variant, without video inputs or lip-guided candidates, would help establish whether visual evidence is essential and whether the framework can generalize to conventional audio-only speech enhancement.

  4. Limited justification for waveform-level aggregation. Directly averaging generated waveforms may introduce phase inconsistencies, temporal misalignment, or speech artifacts, particularly when candidates differ in linguistic content. Moreover, hard candidate selection achieves lower WER than the proposed weighted averaging strategy. The authors should better justify the choice of waveform-level fusion and clarify its advantages over directly selecting the most reliable candidate.

  5. Computational overhead and practical applicability. AVEC and AVEC+ require substantially more sampling trajectories, together with additional ASR, VSR, and guidance computations. The computational cost appears considerable relative to the improvements observed on some non-diffusion backbones. A clearer discussion of the performance–efficiency trade-off would strengthen the practical significance of the framework.

Questions for authors
  1. To what extent is the cleaner-but-wrong phenomenon a general limitation of generative enhancement models, rather than a consequence of the selected backbone architectures or sampling configurations?

  2. Can the authors provide a fully audio-only implementation and evaluate whether comparable gains can be obtained without visual guidance?

  3. How well do ASR-based content degradation measurements correlate with actual linguistic errors perceived by human listeners, particularly on samples where automatic and human evaluations disagree?

3 comments
AuthorsOpenReview ↗

Response to Reviewer HNP9 (1/3)

We sincerely thank the reviewer for recognizing the relevance of content preservation and the applicability of an inference-time approach. Following the reviewer's suggestions, we added competitive-system comparisons, a broader SNR sweep, a detailed analysis of listener–ASR agreement, and controls that isolate the two visual roles. We also expanded the justification of waveform aggregation and the discussion of computational cost. The corresponding revisions and additional results are marked in blue.


W1/Q1. Competitive comparisons, generality of the phenomenon, and SNR coverage

A. We added SGMSE+, SepFormer, and MetricGAN+ to the comparison, bringing the evaluation to eight standalone enhancers, including audio-only and discriminative systems. Appendix F.4 reports the following results on the 657 primary test clips:

EnhancerStandalone WER ↓Harm (%) ↓WER with wrapper ↓
AV-UDiffSE+0.50444.60.279
AV-CA-DiffUSE0.58051.60.381
FlowAVSE0.25615.40.242
PASE0.29416.40.278
SGMSE+ WSJ00.57351.00.378
SGMSE+ VB-DMD0.78369.30.561
SepFormer0.46040.9—
MetricGAN+0.74965.8—

The wrapper column reports the backbone-specific configurations in Table XXVI; FlowAVSE and PASE use the two-candidate configuration here. Input-relative ASR harm occurs in 15.4–69.3% of clips across these systems, including discriminative enhancers. We therefore clarify that the quality–content mismatch is not exclusive to generative models.

Importantly, AVEC also improves PASE, which is designed to mitigate linguistic hallucinations, and FlowAVSE, which has the lowest standalone WER in this comparison. This supports evidence-guided aggregation as complementary to stronger enhancement backbones. Against the discriminative SepFormer control, primary-system AVEC+ has lower WER (0.279 vs. 0.460), lower harm (18.1% vs. 40.9%), and higher PESQ (1.485 vs. 1.385), while SepFormer is slightly better on ESTOI and SI-SDR.

We further tested whether the phenomenon could be explained solely by sampling variance or domain mismatch. With 36 unguided trajectories, WER remains 0.450 versus 0.335 for the noisy input. In the enhancer's matched TCD-TIMIT + DEMAND condition, 26.0% of clips are still harmed despite improved acoustic quality. The error composition also differs across model families: substitutions or insertions account for 90–95% of additional ASR errors on harmed clips for AV-UDiffSE+ and SGMSE+ VB-DMD, versus approximately 46% for MetricGAN+. These are recognition-error measurements; their relationship to human transcription is examined separately below.

Broader SNR evaluation. We extended the sweep to −15, −10, −7.5, −5, −2.5, 0, and +5 dB, keeping the aggregation parameters fixed (Appendix F.5). AVEC remains near or below the noisy-input WER across this range. For example, at −7.5 dB, WER is 0.484 / 0.634 / 0.480 for noisy input / always-enhance / AVEC; at −2.5 dB, it is 0.169 / 0.349 / 0.164. At the added levels, AVEC substantially reduces enhancer-induced recognition errors while retaining PESQ and SI-SDR at the enhancer's level. The additional levels evaluate AVEC with K = 4, rather than AVEC+.

AuthorsOpenReview ↗

Response to Reviewer HNP9 (2/3)

W2/Q3. Reliability of ASR evaluation and disagreement with listeners

A. Following the reviewer's suggestion, we reanalyzed the 1,060 transcripts from 53 listeners on 30 clips, examining both error types and agreement at the word and clip levels. The analysis explains the apparent discrepancy while independently supporting AVEC's content-preservation benefit (Appendix E.3).

The principal difference is that enhancement changes the composition of listener errors:

ConditionHuman deletionsHuman substitutionsHuman WER ↓
Noisy input0.3900.1980.608
Always-enhance0.2560.2910.586
AVEC0.2250.2000.455
AVEC+0.1900.2090.433

Deletion and substitution rates are per reference word; insertions are omitted from the table. Always-enhance reduces deletions but increases substitutions by 47%. These opposing changes yield no statistically significant difference in total listener WER (−0.022 [−0.075, +0.033]). Whisper and Parakeet already transcribe the noisy input more accurately than listeners (0.362 and 0.339 vs. 0.608), so their input-relative changes differ.

The improvement from AVEC is consistent across evaluators. Relative to always-enhance, listener WER decreases by −0.131 [−0.196, −0.071] for AVEC and −0.153 [−0.221, −0.093] for AVEC+. Both also improve over the noisy input; AVEC+'s difference is −0.175 [−0.255, −0.098]. These intervals pair the 30 source clips, rather than treating multiple responses to one clip as independent clips. The listener study supports both variants, although their difference in listener WER is not statistically resolved.

At the word level, 86–98% of reference words that Whisper substitutes or deletes on enhanced outputs are also missed by a strict majority of listeners, depending on the policy. For example, the reference “it's natural connective tissue” becomes “it's natural you know fisheries” for Whisper and “it's natural fishery” for a listener after always-enhance; AVEC+ allows both Whisper and at least one listener to recover the reference.

Across the 180 clip-condition outputs, absolute Whisper and listener WER correlate at r = 0.763. Agreement is weaker for changes relative to the noisy input (r = 0.322) and binary harm labels (κ = 0.123). We therefore use ASR WER to characterize recognition performance and report the per-clip indicator as ASR-measured harm, rather than equating every detected increase with audible fabrication. The human evaluation independently corroborates the improvements over always-enhance. We also clarified the WER conventions in Table 4 and corrected AVEC's listener harm to 23.3%, excluding an exact tie previously counted as harm.


W3/Q2. Is visual information necessary, and can the framework operate without video?

A. We addressed this with two complementary controls: a matched-budget ablation of the wrapper's visual roles and evaluation on genuinely audio-only backbones (Appendices F.2–F.3).

First, we independently enabled or disabled lip-guided candidates and the visual witness while keeping six candidates and 36 trajectories in every configuration:

Lip-guided candidatesVisual witnessWhisper WER ↓Parakeet WER ↓
NoNo0.3490.358
NoYes0.3260.350
YesNo0.3240.332
YesYes (AVEC+)0.2780.306

Using both visual roles lowers WER relative to using neither by −0.071 [−0.088, −0.053] under Whisper and −0.052 [−0.070, −0.035] under Parakeet. Each role improves the point estimates individually, and their combination gives the largest improvement. This ablation isolates the wrapper-level visual contributions; the underlying AV backbone still receives video in all four configurations.

Second, we evaluated the acoustic-witness variant on backbones that never receive video. Without visual guidance or a visual witness, WER decreases from 0.294 to 0.286 on PASE (0.280 with hard selection), 0.573 to 0.396 on SGMSE+ WSJ0, and 0.783 to 0.570 on SGMSE+ VB-DMD. Adding the visual witness gives the corresponding WER point estimates of 0.278, 0.378, and 0.561; the final incremental difference is not statistically resolved. Thus, the evidence-scoring principle generalizes to audio-only enhancement, while video provides complementary information through candidate construction and scoring when available.

AuthorsOpenReview ↗

Response to Reviewer HNP9 (3/3)

W4. Why use waveform aggregation rather than hard selection?

A. We appreciate this question. The revised manuscript presents hard selection and weighted aggregation as two operating points of the same evidence-scoring framework, rather than claiming that averaging is preferable for every objective (Section 5.3; Appendix C.2).

At the frozen τ = 0.10, hard selection has lower Whisper WER (0.267 vs. 0.279), whereas weighted aggregation improves PESQ by 0.008, ESTOI by 0.006, and SI-SDR by 0.11 dB. A diagnostic temperature sweep also shows that a τ = 0.05 mixture reaches WER 0.264 while satisfying all four content–quality criteria. We retain τ = 0.10, selected on train-validation data, rather than choosing a new value on the test set.

Weighted aggregation also reduces sensitivity in the tested evidence-perturbation controls. When both witnesses are deliberately mismatched, it yields WER 0.368 versus 0.422 for hard selection. Changing the candidate recognizer shifts its WER by approximately 0.02, compared with approximately 0.047 for hard selection. Proposition 4 complements these observations by bounding waveform changes under score perturbations for a fixed pool; WER is evaluated empirically.

To address alignment directly, candidate pairs have 0-ms best global lag and 0.96–0.98 correlation. Moreover, aggregation approaches selection when the evidence strongly favors one candidate: the effective pool size is 1.07 on the 38 clips with a score margin of at least 0.30. These measurements support weighted aggregation as a quality-preserving, smoother response to uncertain evidence, while hard selection remains an available option when lower WER is the priority for the evaluated setting.


W5. Computational overhead and practical applicability

A. We have expanded Section 5.6 and Appendix D.3 to make the performance–cost trade-off explicit. On one A6000, the deployed enhancer takes 11 s per clip, a single lip-guided candidate 14 s, AVEC 37 s, and AVEC+ 105 s. AVEC already reduces WER from 0.504 to 0.313 with 12 trajectories, while the higher-budget 36-trajectory AVEC+ reaches 0.279. Repeated diffusion sampling accounts for approximately 95% of the runtime; ASR, VSR, and guidance together are about 5%, and guidance adds under 0.5 s per condition. The six conditions are independent, so running them on six GPUs gives an estimated candidate-generation latency of about 25 s per clip, computed from per-condition timings, with the total sampling work unchanged (Table X).

The value of the additional computation is preserving both content and acoustic enhancement on a fixed backbone. In the added lightweight post-processing comparison, an 805-parameter mixer reaches comparable WER but substantially lower PESQ (1.164 vs. 1.485; Appendix D.4). AVEC+ therefore targets offline use where the enhanced waveform itself matters, rather than real-time recognition.

The same evidence-guided principle also applies to faster backbones: FlowAVSE's eight-output pool takes 0.92 s of sampling, excluding ASR/VSR, and lowers WER from 0.256 to 0.242. We now report this sampling time separately from evidence-extraction cost and distinguish lower-budget configurations from the full diffusion-based AVEC+ setting.


Revisions: Sections 1, 3, and 5.3–5.6; Appendices C.2, D.3–D.4, E.3–E.4, and F.2–F.5; Table 4 caption.

Reviewer_x5h9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the cleaner-but-wrong phenomenon in generative audio-visual speech enhancement (AVSE), where frozen diffusion enhancers improve acoustic quality but significantly degrade spoken content under severe noise. To solve this without retraining the underlying enhancer, the authors propose Audio-Visual Evidence Consistency (AVEC and AVEC+), a training-free inference-time framework. AVEC constructs multiple enhancement candidates using visual lip-reading (VSR) and noisy audio (ASR) guidance targets, scores each candidate against observation witness evidence (viseme distance and WER), and aggregates candidate waveforms via input-adaptive Gibbs weighting. On benchmark evaluations under severe noise (-5 dB SNR), AVEC+ substantially reduces word error rates and content harm while maintaining high acoustic quality.

Critical strengths

The paper clearly formulates and diagnoses the "cleaner-but-wrong" failure mode of generative AVSE, providing strong quantitative evidence across acoustic and recognition metrics. Furthermore, the proposed AVEC framework operates entirely at inference time without retraining or fine-tuning the underlying generative model, serving as a flexible plug-and-play post-hoc wrapper. The authors also conduct comprehensive empirical validations across multiple backbones, diverse noise conditions and SNRs, independent ASR models, and human listener studies.

Critical weaknesses
  • While replacing the acoustic witness scorer with Parakeet is tested in Table 12 (Appendix D.5), the audio-guided candidate generation stage still uses Whisper to derive guidance targets (txt_x, Section 4.1). This residual coupling to Whisper's language prior likely explains why WER gains are smaller under Parakeet evaluation (-0.032 in Table 12) than under Whisper (-0.056 in Table 1).

  • AVEC+ incurs severe computational latency (~105s per 2s audio clip on an A6000 GPU; Appendix D.3). Given this heavy inference overhead, it is unclear what concrete advantages this sampling-based wrapper holds over a lightweight trained post-processing/refinement module, a baseline comparison that is missing from the paper.

  • The paper lacks empirical comparisons against existing inference-time post-processing and refinement wrappers (e.g., Rethink-and-Refine [1], Trained Bridge Module [2], Observation Adding [3]). Benchmarking against these lighter alternatives is essential to justify the severe latency of AVEC+ (~105s per 2s clip on an A6000 GPU; Appendix D.3) and demonstrate its efficiency trade-offs.

  • CTC guidance cannot steer 1-step samplers directly due to missing reverse trajectories (Appendix C.1), relying instead on input perturbation which yields significantly smaller WER gains (Table 3).

Reference

  • [1] Qu, Mike, Yu-Wen Chen, and Julia Hirschberg. "Mitigating Over-Suppression in Speech Enhancement via Inference-Time Rethink-and-Refine Correction Module." arXiv preprint arXiv:2608.07781 (2026). \
  • [2] Wang, Kuan-Chen, et al. "Bridging the gap: Integrating pre-trained speech enhancement and recognition models for robust speech recognition." 2024 32nd European Signal Processing Conference (EUSIPCO). IEEE, 2024. \
  • [3] Iwamoto, Kazuma, et al. "How bad are artifacts?: Analyzing the impact of speech enhancement errors on ASR." arXiv preprint arXiv:2201.06685 (2022).
Questions for authors
  • What is the quantitative impact on WER if Whisper is completely removed from the candidate generation pipeline (i.e., using an independent ASR to generate txt_x in Section 4.1) in addition to the witness scoring test in Table 12?

  • Since CTC guidance cannot steer 1-step samplers directly (Appendix C.1) and input perturbation yields modest gains (Table 3), do the authors envision alternative guidance mechanisms for real-time / 1-step backbones?

  • When the noisy ASR transcript is empty (tx=∅t_x = \emptyset, Section 4.1), why fall back to the default diffusion output (adefa_{\text{def}}, WER ~0.504) instead of switching to a VSR-only aggregation mode (λaud=0\lambda_{\text{aud}} = 0), which achieves WER 0.295 in Table 2?

  • Given the heavy computational latency of AVEC+ (~105s per 2s clip), have the authors evaluated or considered comparing AVEC against other inference-time post-processing/refinement methods (e.g., segment-wise convex interpolation modules or generative error-correction wrappers)? What are the key performance and latency trade-offs between AVEC and these post-processing alternatives?

2 comments
AuthorsOpenReview ↗

Response to Reviewer x5h9 (1/2)

We sincerely thank the reviewer for recognizing the diagnostic value of the work and its inference-time design. Following the reviewer's suggestions, we added a complete recognizer-replacement control and comparisons with lightweight post-processing methods. We also evaluated the proposed VSR-only fallback and clarified the applicability to one-step backbones. We address these points below; revisions and additional results are marked in blue.


W1/Q1. Dependence on Whisper in generation and evidence scoring

A. Following the reviewer's suggestion, we replaced Whisper in all three inference roles: guidance target, acoustic witness, and candidate transcriber. With Parakeet in all three roles, AVEC+ still reduces Parakeet WER from 0.304 to 0.266 (−0.038 [−0.059, −0.017]) on the 624 clips with a non-empty Parakeet input transcript. Appendix D.5 reports the following controlled comparison:

Guidance / witness / candidate ASRWhisper WER ↓Parakeet WER ↓
Noisy input0.3080.304
Always-enhance0.4870.461
W / W / W0.2580.284
P / P / W0.2540.287
P / P / P0.2800.266

Here W and P denote Whisper and Parakeet. Replacing the guidance target and acoustic witness together changes the results only slightly. Replacing the candidate transcriber then shifts WER by +0.026 [+0.011, +0.042] under Whisper and −0.021 [−0.034, −0.008] under Parakeet. This identifies a recognizer-dependent component associated with candidate transcription, while the improvement remains under the other recognizer: P/P/P achieves Whisper WER 0.280 versus 0.308 for the input.

On all 657 clips, including the pre-specified fallback, the fully Parakeet-based configuration achieves Parakeet WER 0.295 versus 0.339 for the noisy input. The original train-validation-selected parameters are retained, so this control demonstrates Whisper-free inference rather than Whisper-free parameter selection. These results show that the content-preservation gain does not require Whisper in the inference pipeline.


W2/Q4. Comparison with lightweight post-processing and the computational trade-off

A. We appreciate this suggestion and added comparisons with observation adding, Rethink-and-Refine, a trained Bridge-style module, and a trained clip-wise interpolation mixer, all using the same frozen enhancer. The results on the same 656 non-fallback clips are reported in Appendix D.4, Table XIV:

MethodWER ↓Harm (%) ↓PESQ ↑ESTOI ↑SI-SDR (dB) ↑
Always-enhance0.50444.71.4700.580+3.24
Rethink-and-Refine0.47140.61.4070.573+2.47
Observation adding, α = 0.50.30618.71.2470.540−0.06
Trained Bridge-style module0.3006.41.1300.478−3.88
Trained clip-wise mixer0.27711.41.1640.501−2.58
AVEC+0.27818.11.4850.586+3.47

The named post-processing baselines are our reimplementations, as stated in the table caption. The trained 805-parameter mixer and AVEC+ have statistically indistinguishable Whisper WER (paired difference −0.001 [−0.021, +0.018]). However, the mixer assigns an average coefficient of 76% to the noisy input, whereas AVEC+ combines only enhanced waveforms. Consequently, AVEC+ retains substantially higher acoustic quality: PESQ +0.321, ESTOI +0.085, and SI-SDR +6.05 dB relative to the mixer.

This comparison clarifies AVEC+'s benefit: comparable content accuracy while retaining the acoustic enhancement, rather than a claim of better WER or lower latency in isolation. The mixer achieves a lower harm rate and is a useful lower-cost option when recognition accuracy is the primary objective. Among the post-processing alternatives in this comparison, AVEC+ is the only one that lowers WER below the noisy input while improving all three reference-based quality metrics over the deployed enhancer. Admitting the noisy input to the AVEC+ pool provides a further trade-off (WER 0.270, harm 8.5%, PESQ 1.364), which we also report.

Runtime. The lightweight mixers still require the deployed enhancer's four trajectories, taking 11 s per clip before post-processing. AVEC uses 12 trajectories and 37 s; AVEC+ uses 36 trajectories and 105 s on the same A6000. The additional cost is dominated by diffusion sampling; ASR, VSR, and guidance together are about 5% of the total. The 105 s is for one GPU running the six AVEC+ conditions sequentially. With one GPU per condition, the estimated candidate-generation latency on six GPUs is about 25 s per clip (one K = 8 condition takes 24.6 s on the same clips; this estimate is computed from per-condition timings, not measured end to end), and the total sampling work remains unchanged. Section 5.6 now positions AVEC+ as an offline content–quality operating point, not a low-latency alternative.

AuthorsOpenReview ↗

Response to Reviewer x5h9 (2/2)

W3/Q2. Applicability to one-step backbones and alternative guidance

A. The source-evidence scoring and aggregation rule is not restricted to iterative diffusion. With backbone-appropriate perturbation candidates and no retraining, it lowers WER from 0.256 to 0.242 on FlowAVSE and from 0.294 to 0.278 on PASE. Exposing the individual perturbed outputs in a nine-candidate pool further lowers WER to 0.228 and 0.270, respectively (Appendix F.2). These tested pools involve modest quality losses, which are reported alongside the content gains.

The current CTC guidance acts on intermediate diffusion estimates, so we do not apply it unchanged to one-step models. The revised limitations discuss input-space guidance for differentiable one-step models and distillation of AVEC+ outputs into a one-step student as future directions. For the evaluated FlowAVSE implementation, the eight-output pool takes 0.92 s of sampling, compared with 0.17 s for a single output, excluding ASR/VSR in both cases. This separates the cost of candidate generation from that of evidence extraction.


Q3. Why use the default-output fallback rather than VSR-only aggregation?

A. The original fallback was specified because an empty noisy-input transcript leaves both the acoustic witness and the audio-guidance target undefined. Following the reviewer's suggestion, we evaluated a VSR-only fallback, which sets λ_aud = 0 and aggregates the default and lip-guided candidates (Appendix D.5).

Under the primary Whisper configuration, only one of 657 inputs has an empty transcript, and its candidate transcripts are also empty; the two fallback rules give the same result on this clip. Under Parakeet, 33 inputs have empty transcripts. On these clips, the VSR-only alternative lowers WER from approximately 0.83 to 0.73, improving the corresponding full-set P/P/W result from 0.283 to 0.278.

This supports the reviewer's proposed fallback as a useful additional option. We report it separately while retaining the pre-specified fallback in the main tables and recognizer-replacement comparisons, so that the effect of changing the recognizer is not confounded with a change in fallback policy.


Revisions: Appendices D.3–D.5 and F.2; Section 5.6; Appendix H.

Reviewer_BTZXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AVEC, an inference-time guidance framework to address the issue of reduced ASR performance of enhanced samples in an AVSE setting. The framework includes guiding, scoring and aggregating different samples of a pretrained audio-visual speech enhancement (AVSE) model.

Critical strengths
  • The framework yields significant improvements in recognition performance at low SNRs while maintaining performance in other metrics, without requiring retraining.
  • The list of experiments and ablations is extensive, and the results include a listening experiment
Critical weaknesses
  • While the motivation of the paper is clear, the motivation of the proposed framework is not. The introduction of the proposed method in Section 4 lacks intuition. The "Guided sampling and multiple restorations" could be used here to better motivate the sampling and guidance. LipVoicer should be explicitly mentioned here and not left to the Appendix. Section 4.3 is very hard to follow, with long and complicated sentences that need rewriting. Furthermore, the role of Proposition 1 for the framework is not made clear.

  • A quantitative analysis on the contributions of the modalities via the weights wi(a)w_i(a) is missing. The value found for λaud\lambda_{aud} for the tables should be in the main paper.

  • Many sections of the paper contain numerous sentences that are missing the crispness and clearness one would expect in a scientific paper. This is even more severe in the Appendix. Examples:

    • "A reduction in measured content harm must not be presented as a per-utterance safety certificate"
    • "Candidate consensus scores each candidate by mean pairwise WER against other candidates, relying on agreement within the generated pool rather than evidence from the input"
    • "An audiovisual wrapper on an audio-only backbone: which ingredient does what"
    • Your AI use statement states that ChatGPT assisted with manuscript revision. Please elaborate what exactly that means.
Questions for authors
  • Could averaging so many samples produce artifact or result in over-smooths outputs? If not, why?

  • Do the scores adapt at different SNRs? Can the model adapt to video occlusions?

  • What is the reasoning for averaging 4 or 8 trajectories? Was an ablation done on K for AVEC? Is it better to have larger K or more candidates? Table VIII doesn't fully answer that.

  • The framework requires the generation of many (up to 36) trajectories. Although the samples per guidance can be batched, what is the impact on memory and computation time?

  • Can you prepare a project page with sound examples

  • Will you release the code?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BTZX (1/3)

We sincerely thank the reviewer for recognizing the recognition improvements, the extensive evaluation, and the benefit of preserving enhancement quality without retraining the backbone. Following the reviewer's suggestions, we have clarified the method presentation and added analyses of the evidence weights, waveform alignment, sampling budget, and computational cost. We address each concern below. Revisions and additional results are marked in blue in the revised manuscript. Unless otherwise stated, intervals below are paired 95% bootstrap confidence intervals.


W1. Intuition behind the framework, relation to LipVoicer, and the role of Proposition 1

A. We appreciate the suggestion to make the framework's motivation more explicit. The key idea is that agreement among generated candidates is not sufficient to identify which one preserves the original utterance: multiple restorations can share the same error. AVEC therefore constructs complementary default, lip-guided, and audio-guided candidates, evaluates them against evidence from the original video and noisy audio, and assigns a larger weight to candidates that better match this evidence. Section 4 now introduces this intuition before describing candidate construction, evidence scoring, and input-adaptive aggregation.

We also explicitly credit LipVoicer's transcript-guided CTC mechanism in Section 4.1. AVEC uses this mechanism for candidate construction and builds on it through source-evidence scoring and adaptive aggregation, rather than relying on a single guidance target.

Section 4.3 now explains the weighting rule before presenting the mathematics. Proposition 1 clarifies how imperfect evidence scores affect candidate selection: the regret relative to the best available candidate is bounded by the variation in score error across candidates. We explicitly distinguish this candidate-level result from the WER of the aggregated waveform, which is evaluated empirically. The revised presentation thus connects the proposition to the framework without treating it as a waveform-level WER guarantee.


W2. Quantitative modality contributions and the value of λ_aud

A. Following the reviewer's suggestion, we added a quantitative weight analysis in Appendix A.7. On the 656 clips with a defined candidate pool, the highest-weighted candidate is a lowest-reference-WER candidate, including ties, on 84.5% of clips. The evidence-weighted mean candidate WER is 0.283, compared with 0.410 under uniform weights and 0.218 for an oracle. These are candidate-level diagnostic quantities, rather than the WER of the final waveform; reference transcripts are used only for this analysis, never at inference.

The rounded mean weight shares for default, lip-guided, and audio-guided candidate types are 0.23, 0.40, and 0.38, respectively. The allocation changes with the input: the default weight decreases to 0.13 on clips harmed by always-enhance, compared with 0.30 elsewhere. To distinguish weight allocation from the effect of visual information itself, we additionally conducted a matched-budget ablation of the two wrapper-level visual roles. Removing both raises WER from 0.278 to 0.349 (+0.071 [0.053, 0.088]; Appendix F.3).

The main text and Table 1 now explicitly report λ_aud = 0.5 and τ = 0.10, selected on train-validation data and then kept fixed.


W3. Clarity of the main text and appendix

A. We thank the reviewer for identifying specific passages. We have revised these passages and simplified the exposition in Sections 4–5 and the appendix. The revised text explains that lower average harm does not guarantee correctness for every utterance, that the consensus baseline compares candidates with one another rather than with the original observation, and that the PASE ablation separates the contributions of acoustic and visual evidence. We also replaced compressed descriptions with explicit statements of the experimental setup and the conclusion supported by each analysis.


W4. AI-use disclosure

A. We have expanded the AI Use Statement to specify both the assistance provided and the authors' responsibilities. Claude Code assisted with implementing analysis and evaluation scripts, experiment execution, and preparing tables from recorded logs. ChatGPT/Codex assisted with language polishing and clarity in the manuscript and rebuttal, transcript inspection, and statistical-analysis scripting. The authors developed and determined the research ideas, methodological decisions, experimental design, and interpretation of results. All reported results, scientific claims, and citations were verified by the authors, who take full responsibility for the work.

AuthorsOpenReview ↗

Response to Reviewer BTZX (2/3)

Q1. Could waveform averaging introduce artifacts or over-smoothing?

A. We examined this concern through alignment measurements, reference-based quality metrics, and listener ratings (Appendices A.6 and C.2).

First, candidate pairs have a best global lag of 0 ms, normalized correlation of 0.96–0.98, and pairwise SI-SDR of 13–16 dB across the 656 non-fallback clips. This supports global temporal alignment, although it does not establish exact local phase alignment. Second, within-condition averaging improves reference quality: increasing K from 1 to 8 raises PESQ from 1.413 to 1.484 and SI-SDR from +2.53 to +3.43 dB.

For aggregation across candidates, the weights approach selection when one candidate is strongly supported. On the 38 clips with a top-two score margin of at least 0.30, the effective pool size is 1.07, and the output transcript matches the top-scored candidate on 92.1% of clips. Across the full non-fallback set, the mixture has higher WER than every candidate on 9 of 656 clips (1.4%). Listeners also rate AVEC+ above always-enhance (3.03 vs. 2.83), while DNSMOS is slightly lower (3.41 vs. 3.47). Together, these results support the use of averaging to retain enhancement quality, rather than indicating a systematic over-smoothing penalty.


Q2. Do the weights adapt to SNR and visual occlusion?

A. Yes. The weights are computed separately for each input, with no retraining or condition-specific parameter adjustment. As SNR increases, the share of clips with near-uniform weights rises from 24.8% at −5 dB to 43.5% at 0 dB and 66.1% at +5 dB. Thus, aggregation becomes less selective when the candidates agree more closely (Appendix A.7).

We also isolated the response to degraded visual evidence by holding the candidates fixed and degrading only the mouth video supplied to the VSR evidence extractor. Under 40% frame occlusion, the lip-guided weight share decreases from 0.40 to 0.33, while the audio-guided share increases from 0.38 to 0.42. WER remains below always-enhance (0.328 vs. 0.504). With moderate blur or a halved frame rate, WER remains significantly below the noisy input (0.291 and 0.286, respectively, vs. 0.335; Appendix D.4). These controls demonstrate adaptation at the evidence-weighting stage.


Q3. Why average K = 4 or 8 trajectories, and is a larger K preferable to more candidates?

A. We use K = 4 to match the deployed enhancer's batch mean and K = 8 to examine the benefit of additional sample averaging. The revised budget analysis directly compares the two ways of allocating computation (Appendices D.2–D.3, Tables VII–IX).

At the same 24-trajectory budget, six K = 4 candidates achieve lower WER than three K = 8 candidates (0.284 vs. 0.298). Expanding to the 36-trajectory AVEC+ pool further lowers WER to 0.279. In contrast, using all 36 trajectories for unguided generation gives WER 0.450 with uniform averaging and 0.390 with evidence weighting. Thus, in the tested setting, allocating the budget to a larger complementary candidate pool is more effective than increasing averaging alone. Table IX additionally reports the individual-candidate results for K = 1, 2, 4, and 8.

AuthorsOpenReview ↗

Response to Reviewer BTZX (3/3)

Q4. Memory usage and computational cost

A. Following the reviewer's suggestion, we report the sampling budget, runtime, and peak memory in Appendix D.3, Table X, and discuss the operating points in Section 5.6. On one RTX A6000, the reported times per clip are:

ConfigurationTrajectoriesTime per clip
Deployed enhancer411 s
Single lip-guided candidate414 s
AVEC1237 s
AVEC+36105 s

Repeated diffusion sampling accounts for approximately 95% of the cost; waveform weighting takes less than 1 ms. Generation conditions run sequentially, so peak GPU memory is 5.2 GiB for a K = 4 batch and 7.7 GiB for K = 8, rather than scaling with all 36 trajectories simultaneously. Samples within a condition are already batched; batching further on one GPU gains only about 10%, because time grows almost linearly with the batch (K = 4 to K = 8 is 1.88×, re-timed on the same 20 clips), and guidance adds under 0.5 s per condition. The six conditions are independent, so distributing them over six GPUs gives an estimated candidate-generation latency of about 25 s per clip (the longest single condition plus witness extraction, computed from the per-condition timings rather than measured end to end); the total sampling work remains unchanged. AVEC+ targets offline enhanced-audio generation, while AVEC provides a lower-budget operating point on the same backbone.


Q5/Q6. Audio examples, project page, and code release

A. Yes. The implementation code and audio demonstrations are provided in the supplementary material. We will release the code and per-clip outputs publicly upon acceptance and provide a project page with audio examples once anonymity constraints are lifted.


Revisions: Sections 4–5; Appendices A.6–A.7, C.2, D.3–D.4, and F.2–F.3; AI Use Statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.