acceptodds
Under review as a conference paper at ICLR 2027

How Much to Trust the Eyes? Reliability-Guided Inference for Diffusion-Based Audio-Visual Speech Enhancement

Abstract

Diffusion-based audio-visual speech enhancement (AVSE) iteratively recovers speech from noisy audio conditioned on temporally aligned visual cues. Its real-world deployment may confront distribution shifts, such as visual degradation and audio-visual mismatch, which misguide the reverse diffusion process. Our analysis discloses that visual quality alone does not necessarily determine whether visual conditioning remains useful or misguides enhancement since its effect is coupled with both the pretrained model and the acoustic input. Motivated by these observations, we propose Reliability-Guided Inference (RGI), which takes the departure of both internal visual representations and model responses from those of matched audio-video pairs as evidence of distribution shift inside the pretrained model and converts this evidence into a reliability score that regulates visual conditioning during reverse diffusion. Specifically, at each reverse step, RGI weights the difference between audio-visual and audio-only predictions according to the reliability score, without updating the pretrained models. To retain visual cues that can remain useful at low input signal-to-noise ratio, RGI imposes a lower bound on the conditioning coefficient, the conditioning floor. A diffusion-step schedule further modulates the retained visual contribution along the sampling trajectory. Experiments across four corpora and three frozen diffusion AVSE backbones show that RGI improves enhancement under a range of distribution shifts on the corpora with clean references, with gains that depend on the backbone and corpus. Recordings collected from the Ameca humanoid robot further show that RGI can be deployed for robot interaction, with demonstrations available at https://anonymous.4open.science/w/ameca-demo/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
audio-visual speech enhancement, diffusion models, reliability estimation, conditional guidance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ojJtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies unreliable visual conditioning in diffusion-based audio-visual speech enhancement. It argues that visual degradation alone does not determine whether visual information helps enhancement, since its utility also depends on the frozen model and acoustic input. Reliability-Guided Inference (RGI) evaluates audio-visual and audio-only score predictions at the same sampling state and adaptively weights their difference. Its weighting combines deviations in internal visual representations and relative prediction-difference magnitudes from matched reference statistics, followed by a conditioning floor and a diffusion-step schedule. RGI leaves the pretrained enhancement networks frozen, although visual reference statistics are rebuilt for each target corpus. Experiments cover three backbones, three reference-based evaluation corpora, a no-reference LRS3 evaluation, and a robot demonstration.

Critical strengths
  1. The problem formulation is practically relevant and more informative than visual-quality gating alone. The distinction between degradation relative to aligned video and utility relative to audio-only inference is useful. The analysis also recognizes that visually plausible inputs can be mismatched and that degraded video can remain helpful under weak acoustic conditions.

  2. The inference rule is explicit and applicable to several existing backbones. RGI modifies the score supplied to the sampler without retraining the enhancement networks. The implementation distinguishes separate audio-only priors from null-visual predictions and describes reference construction, cross-fitting, covariance regularization, and the guidance schedule.

  3. The ablations address several important alternative explanations. Table 2 includes both single-score variants, removal of the floor and schedule, schedule-only guidance, and a constant coefficient retaining the schedule. Full RGI improves Mean/Worst from 9.32/8.93 to 9.48/9.24 dB over the constant-coefficient variant. These controls support an incremental benefit from input-dependent reliability estimation.

  4. The evaluation includes useful controls beyond the development perturbations. On AV-UDiffSE, the held-out wrong-speaker condition improves from 8.35 dB with fixed audio-visual inference to 9.54 dB with RGI, exceeding audio-only inference at 9.03 dB. Sham padding separates boundary-padding effects from temporal misalignment. The paper also reports backbone-dependent results rather than presenting RGI as uniformly superior.

Critical weaknesses
  1. Some evaluation settings needed to interpret and reproduce the comparisons are insufficiently specified. Appendix E introduces the input-quality score (\gamma(v)) and selection threshold (\eta), but does not fully specify their implementation and values. Appendix H.1 does not report the value or selection procedure for the constant-coefficient control. These details matter because quality gating seems to have reduced to audio-only performance under all development perturbations in Table 9. Providing the exact configurations and parameter-selection protocol would make these comparisons more assessable.

  2. Evidence for perceptual and linguistic benefits is limited. The natural-speech evaluation in Table 11 shows only minimal gains: DNSMOS increases from 3.516 to 3.535 and SQUIM-SI-SDR from 15.06 to 15.12, while SQUIM-STOI remains at 0.893 and SQUIM-PESQ decreases from 2.170 to 2.122. These predicted metrics do not establish improved perceived quality or preservation of spoken content.

Questions for authors

Please see weaknesses.

Reviewer_EXthOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Reliability-Guided Inference (RGI) to dynamically adjust the weighting of visual cues during inference. The primary objective is to resolve performance degradation in diffusion-based speech enhancement under visual distribution shifts (e.g., occlusion, asynchrony) without retraining model parameters. RGI estimates visual reliability via a Representation Score and a Model-Response Score. It then controls visual guidance weights during the reverse diffusion process using a Conditioning Floor and a Diffusion-Step Schedule. The robustness of the proposed method is validated across 3 diffusion backbones, 4 datasets, and real-world experiments on an Ameca humanoid robot.

Critical strengths

This paper provides a thorough analysis of how visual distribution shifts impact audio-visual speech enhancement, demonstrating that performance depends on model architecture and input SNR while showing that frame-level degradation metrics fail to reflect actual performance drops. To address these challenges, the paper introduces RGI, a training-free inference method that combines representation and model-response scores with a conditioning floor and diffusion schedule to dynamically control visual guidance without retraining. Furthermore, the paper validates RGI across 4 datasets, 3 diffusion backbones, and real-world deployment on an Ameca humanoid robot, demonstrating robust performance improvements under diverse shift conditions.

Critical weaknesses
  • The guidance formulation (Section 3.2, Equation (6); Appendix B.1, Equation (8)) restricts visual weights to interpolation (gtct≤1.0g_t c_t \le 1.0), which prevents amplifying visual cues beyond 1.0 when audio is severely corrupted. This upper bound limits flexible adaptation in joint audio-visual degradation scenarios.

  • Performance gains under clean visual conditions are minor, particularly on strong backbones (Section 4.2, Table 1). On AV-SGMSE, average improvement is +0.10 dB (13.30 dB →\rightarrow 13.40 dB), showing that RGI acts primarily as a defensive safety net rather than an active performance booster.

  • The Model-Response Score (zDˉ,tz_{\bar{D},t}) relies solely on a scalar RMS ratio (hth_t) without considering vector directionality (Section 3.1, Equations (3) & (4)). Mismatched visual cues with normal RMS magnitudes but incorrect score directions may bypass this scalar filter.

Questions for authors
  • What are the quantitative wall-clock latency and Real-Time Factor (RTF) overheads resulting from evaluating both audio-visual and audio-only predictions at each reverse step, as outlined in Section 3.2 and Appendix G?

  • Key hyperparameters (Conditioning Floor ρ=0.1\rho=0.1, schedule range gt:1.0→0.5g_t: 1.0 \rightarrow 0.5, logistic slope κ=3\kappa=3) in Section 3.1, Equation (5) and Appendix G, Table 6 remain fixed regardless of noise severity. Have the authors considered an adaptive mechanism conditioned on SNR, or analyzed sensitivity across different SNR levels?

  • Statistics for the reference datasets (RI,RDR_I, R_D) are derived from 500 clean samples (Section 3.1; Appendix C.3, Table 6). Do these statistics generalize effectively to out-of-domain inputs with substantially different speakers, languages, or acoustic environments?

  • The step scheduler gtg_t is set to decrease linearly from 1.0 to 0.5 in Section 3.2, Equation (6). Given the distinct roles of earlier reverse steps (coarse structure formation) versus later steps (fine detail generation), what advantages does this linear decrease offer over alternative scheduling patterns?

Reviewer_UgvrOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the problem of distribution shift by enhancing existing audio-visual speech enhancement approaches with reliability guided inference. The main idea is that departure of both internal visual representations and model responses from matched audio-video pairs is evidence of distribution shift inside the pretrained model and this can be converted into a reliability score that regulates visual conditioning during reverse diffusion. This approach is tested on 3 different models and 3 different datasets and compared with other SOTA methods.

Critical strengths

The paper is easy to follow.

An interesting approach is proposed for tackling the issue of distribution shift for audio-visual speech enhancement.

Several ablation studies.

Critical weaknesses

Major weaknesses

  • There are no results reported on the LRS3 dataset. Only table 11 in the appendix shows some results on this dataset. Given that GRID and Lombard GRID are not that different, it would be more interesting if table 1 includes results on LRS3 instead of Lombard GRID.

  • The appendix contains a section which shows that the proposed approach is robust to the size of the reference video set. It would be good to do an additional ablation study which shows how robust the approach is to the different videos included in the reference set. In other words, you can sample multiple 50 video sets and then report the performance for each of them.

  • It would be very useful if a figure in the main paper or appendix is added which shows how the two scores (representation and model response) change under different appearance and distribution shifts.

  • Why audio-only inference is not included in table 1?

Other weaknesses

  • Section 4.4, Demonstration on the AMECA robot, could be moved to the appendix. Would be better if more ablation studies are included in the main paper. It’s not clear what this section adds to the main paper. It is just an application which could easily be moved to the appendix.

  • The related work section is added at the end of the paper (section 5). It would be better if it’s presented in the beginning, e.g., section 2.

  • It is not clear if noise is added in the experiments with the GRID/Lombard GRID dataset. If yes, what noise type and at what SNR levels?

Questions for authors

Please see the section major weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.