- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies unreliable visual conditioning in diffusion-based audio-visual speech enhancement. It argues that visual degradation alone does not determine whether visual information helps enhancement, since its utility also depends on the frozen model and acoustic input. Reliability-Guided Inference (RGI) evaluates audio-visual and audio-only score predictions at the same sampling state and adaptively weights their difference. Its weighting combines deviations in internal visual representations and relative prediction-difference magnitudes from matched reference statistics, followed by a conditioning floor and a diffusion-step schedule. RGI leaves the pretrained enhancement networks frozen, although visual reference statistics are rebuilt for each target corpus. Experiments cover three backbones, three reference-based evaluation corpora, a no-reference LRS3 evaluation, and a robot demonstration.
Critical strengths
-
The problem formulation is practically relevant and more informative than visual-quality gating alone. The distinction between degradation relative to aligned video and utility relative to audio-only inference is useful. The analysis also recognizes that visually plausible inputs can be mismatched and that degraded video can remain helpful under weak acoustic conditions.
-
The inference rule is explicit and applicable to several existing backbones. RGI modifies the score supplied to the sampler without retraining the enhancement networks. The implementation distinguishes separate audio-only priors from null-visual predictions and describes reference construction, cross-fitting, covariance regularization, and the guidance schedule.
-
The ablations address several important alternative explanations. Table 2 includes both single-score variants, removal of the floor and schedule, schedule-only guidance, and a constant coefficient retaining the schedule. Full RGI improves Mean/Worst from 9.32/8.93 to 9.48/9.24 dB over the constant-coefficient variant. These controls support an incremental benefit from input-dependent reliability estimation.
-
The evaluation includes useful controls beyond the development perturbations. On AV-UDiffSE, the held-out wrong-speaker condition improves from 8.35 dB with fixed audio-visual inference to 9.54 dB with RGI, exceeding audio-only inference at 9.03 dB. Sham padding separates boundary-padding effects from temporal misalignment. The paper also reports backbone-dependent results rather than presenting RGI as uniformly superior.
Critical weaknesses
-
Some evaluation settings needed to interpret and reproduce the comparisons are insufficiently specified. Appendix E introduces the input-quality score (\gamma(v)) and selection threshold (\eta), but does not fully specify their implementation and values. Appendix H.1 does not report the value or selection procedure for the constant-coefficient control. These details matter because quality gating seems to have reduced to audio-only performance under all development perturbations in Table 9. Providing the exact configurations and parameter-selection protocol would make these comparisons more assessable.
-
Evidence for perceptual and linguistic benefits is limited. The natural-speech evaluation in Table 11 shows only minimal gains: DNSMOS increases from 3.516 to 3.535 and SQUIM-SI-SDR from 15.06 to 15.12, while SQUIM-STOI remains at 0.893 and SQUIM-PESQ decreases from 2.170 to 2.122. These predicted metrics do not establish improved perceived quality or preservation of spoken content.
Questions for authors
Please see weaknesses.