From Observation-Driven to Cause-Governed: Frequency-Coupled State Space Recovery for Audio-Visual Speech Enhancement

Audio-visual speech enhancement (AVSE) has achieved substantial progress by exploiting visual articulatory information to improve speech recovery under acoustic degradation. However, two limitations persist in many existing AVSE methods. Recovery is primarily observation-driven, with vision serving mainly as an auxiliary feature or condition; frequency information is typically modeled within a unified recovery process. These limitations make reliable speech recovery more difficult, particularly at low SNRs. We therefore formulate AVSE as a cause-governed, frequency-coupled state-recovery process. Specifically, a visual articulatory prior not directly corrupted by acoustic noise governs state evolution along an explicit cause $\to$ content $\to$ detail dependency. Accordingly, we propose the Frequency-aware State Space Cause-governed Modeling (FSSCM) framework, comprising three key components. First, Visual Spectral Prior Learning (VSPL) models temporal articulatory dynamics from continuous lip movements to construct a temporally structured visual articulatory prior that governs state transitions. Second, Adaptive Progressive Frequency Mamba (APFM) realizes frequency-coupled recovery by using the visual prior to modulate low-frequency Mamba state transitions for content recovery, after which the evolving low-frequency state and the same prior jointly modulate high-frequency state transitions for detail reconstruction. Finally, Prior-anchored Frequency Consistency Learning (PFCL) is designed to enforce cross-frequency consistency under the same visual articulatory prior. Experiments on five AVSE benchmarks demonstrate the superiority of FSSCM over state-of-the-art methods, with its SDR advantage over competing methods increasing as SNR decreases.