acceptodds
Under review as a conference paper at ICLR 2027

From Observation-Driven to Cause-Governed: Frequency-Coupled State Space Recovery for Audio-Visual Speech Enhancement

Abstract

Audio-visual speech enhancement (AVSE) has achieved substantial progress by exploiting visual articulatory information to improve speech recovery under acoustic degradation. However, two limitations persist in many existing AVSE methods. Recovery is primarily observation-driven, with vision serving mainly as an auxiliary feature or condition; frequency information is typically modeled within a unified recovery process. These limitations make reliable speech recovery more difficult, particularly at low SNRs. We therefore formulate AVSE as a cause-governed, frequency-coupled state-recovery process. Specifically, a visual articulatory prior not directly corrupted by acoustic noise governs state evolution along an explicit cause content detail dependency. Accordingly, we propose the Frequency-aware State Space Cause-governed Modeling (FSSCM) framework, comprising three key components. First, Visual Spectral Prior Learning (VSPL) models temporal articulatory dynamics from continuous lip movements to construct a temporally structured visual articulatory prior that governs state transitions. Second, Adaptive Progressive Frequency Mamba (APFM) realizes frequency-coupled recovery by using the visual prior to modulate low-frequency Mamba state transitions for content recovery, after which the evolving low-frequency state and the same prior jointly modulate high-frequency state transitions for detail reconstruction. Finally, Prior-anchored Frequency Consistency Learning (PFCL) is designed to enforce cross-frequency consistency under the same visual articulatory prior. Experiments on five AVSE benchmarks demonstrate the superiority of FSSCM over state-of-the-art methods, with its SDR advantage over competing methods increasing as SNR decreases.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Speech Enhancement, State Space Models, Visual Articulatory Prior, Frequency-Coupled Recovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_29unOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that audio-visual speech enhancement (AVSE) should stop treating vision as an auxiliary feature and instead use it as a governing cause: lip movements sit upstream of acoustic production, are not corrupted by noise, and should therefore steer recovery dynamics rather than merely conditioning features. This "cause → content → detail" story is instantiated in FSSCM (Sec. 3) in three parts. VSPL (Sec. 3.2) encodes lip video with a 3D-CNN, augments the features with first- and second-order temporal differences, adaptively fuses them (Eq. 3), refines them with temporal self-attention, and produces a temporally resolved visual prior C_v plus a gate G_v (Eq. 5). APFM (Sec. 3.3) predicts a soft low/high frequency split from a global visual summary (Eqs. 6-8), modulates each Mamba branch's ∆ and B parameters with the prior (Eq. 9), and couples the branches so the low-frequency state guides the high-frequency transition signal (Eqs. 12-15). PFCL (Sec. 3.4) runs bidirectional cross-attention between the branches and scores their consistency against the broadcast prior (Eqs. 16-22). On the primary benchmark LRS3+DNS, FSSCM beats every baseline at all four SNRs, with the SDR margin over the strongest AVSE baseline CogCM growing from 1.96 dB at 0 dB to 2.99 dB at −15 dB (Table 1, Fig. 3). Four further benchmarks (GRID+CHiME3, TCD-TIMIT+NTCD-TIMIT, MEAD+DEMAND, AVSpeech+DNS) are evaluated in the supplement, and ablations cover module removal (Table 2), internal mechanisms (Table 8), intervention type (Table 10), frequency-organization direction (Table 11), and black-frame visual failure (Table 9).

As I read it, the reframing is the main conceptual sell; mechanically, most parts are familiar (3D-CNN visual encoders, FiLM-style modulation, Mamba, cross-attention), reorganized around the cause → content → detail principle and the low-to-high frequency dependency.

Critical strengths

The headline numbers are hard to dismiss. At −15 dB on LRS3+DNS, FSSCM reports 10.27 dB SDR against 7.49 for Dolphin and 7.28 for CogCM (Table 1), and the ordering holds at all four SNRs and on all three metrics. What makes this more than a leaderboard bump is that the margin widens monotonically as SNR drops (1.96 → 2.99 dB over CogCM, Fig. 3) — exactly the regime where the paper's own motivation predicts a noise-free visual prior should pay off most. That is the result the rest of the paper has to live up to, and mostly it does.

What pushes this over the line for me, though, is the ablation discipline. Table 2 removes each module wholesale, but Table 8 goes further and knocks out individual mechanisms — second-order articulatory dynamics (↓0.97 dB at −15 dB), temporal self-attention (↓1.17), global-vs-temporal C_v governance (↓0.87), ∆/B modulation (↓1.38) — so we can see which knobs do what, not just that the full model works. Tables 10-11 settle the two design claims I was most skeptical about: transition-level visual governance beats feature fusion by 1.5-2.0 dB depending on SNR (Table 10), and L→H coupling beats parallel recovery by 0.42-0.55 dB and the reverse H→L direction at every SNR (Table 11). Controlled comparisons like that are what make an architectural paper convincing.

Critical weaknesses

The paper's central narrative — the advantage grows as acoustic reliability drops — is really only established on LRS3+DNS. On TCD-TIMIT, FSSCM beats CogCM by 2.53 dB at −5 dB, but at 0 and 5 dB the two sit within ~0.1 dB and at 10 dB CogCM leads by 0.59 dB on SI-SDRi, with CogCM also taking STOIi at 0 dB (Table 5). On MEAD the SI-SDRi differences flip sign across SNRs (−10 dB: +1.23 for FSSCM; −5 dB: −1.06; Table 6), and on GRID at 0 dB CogCM wins STOIi (20.50 vs 20.16, Table 4). Fig. 4 then shows only the most favorable SNR per benchmark, all against CogCM alone, which flatters the story. None of this contradicts the main result, but Sec. 1 and the conclusion present the SNR-scaling trend as a general finding and it isn't one. A few sentences on where and why it breaks down would make the paper more credible than the current selective presentation.

No efficiency analysis, and no limitations section. The architecture stacks N APFM layers (N is never specified in the text), each with two Mamba branches scanned bidirectionally along both time and frequency, plus cross-attention, consistency heads, and two decoders — and LRS3+DNS training takes 4 A800-80G GPUs (Sec. A.1.2). Parameter count, FLOPs, and real-time factor appear nowhere, nor is there any cost comparison against CogCM or AVSEMamba. For a methods paper whose selling points are architectural, I can't tell whether the 2-3 dB SDR gains are cheap or come with a much heavier compute bill. This is the biggest gap I'd want closed.

Questions for authors
  1. FA-Mamba-AVSE (Saleem et al., 2026b) is discussed in Sec. 2.2 but never appears in any comparison table, despite being — as far as I can tell — the closest concurrent method (frequency-aware state-space modeling plus gated cross-modal fusion for the same task). Is there a head-to-head under your protocol? And beyond the L→H coupling, what do you see as the substantive difference between your transition-level governance (Eq. 9) and their gated fusion?

  2. Sec. 4 says baseline numbers mix reimplemented and previously reported results. Which rows in Tables 1-7 are which? Some differences you treat as wins are far inside run-to-run noise (Table 5, 0 dB: 18.96 vs 18.95).

  3. Why does the advantage over CogCM flatten or invert on MEAD and GRID (Tables 4, 6)? Dataset size, MEAD's emotional speech, visual quality, or the adaptive frequency routing behaving differently on smaller corpora? A sentence or two of analysis would help.

  4. Params, FLOPs, and RTF versus CogCM and AVSEMamba — even rough numbers in the rebuttal would change how I read Tables 1-7.

  5. FSSCM's −15 dB PESQ (2.949) is higher than many baselines' 0 dB PESQ (Table 1). Can you confirm identical PESQ configuration (wideband, 16 kHz, mir_eval settings) across all methods?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3: Critical Weaknesses 1–2

We sincerely thank the reviewer for the careful evaluation of our work and the thoughtful, constructive feedback. We particularly appreciate the reviewer's encouraging recognition of our contributions and valuable suggestions, which have helped us strengthen the experimental validation, methodological clarity, and reproducibility of our work. We have carefully addressed each concern and provide point-by-point responses below.

Response to Weakness 1:

We sincerely thank the reviewer for the careful examination of our cross-benchmark results and the thoughtful, constructive suggestions. We greatly appreciate this observation, which helps us clarify the scope of our findings and improve the rigor of our presentation. While LRS3+DNS exhibits a clear tendency toward larger relative gains under severe acoustic interference, this trend is less pronounced across the additional benchmarks, potentially reflecting dataset-specific characteristics and evaluation conditions. One possible explanation is that acoustic SNR characterizes the degree of noise interference but does not fully determine the complementary information provided by visual speech cues. Differences in dataset scale and diversity, speech characteristics, and articulatory–acoustic correspondence may influence the relative contribution of visual guidance. For instance, emotionally expressive speech in MEAD introduces different articulatory–acoustic characteristics, while GRID features more constrained sentence structures. To provide more comprehensive cross-benchmark evidence, we have further expanded the main-text comparisons to present the results more thoroughly and in a balanced manner. Following the reviewer's valuable suggestion, we have further clarified the observed cross-benchmark performance variations in Section 5.1. We sincerely appreciate the reviewer's constructive feedback, which helps us present our findings more accurately and rigorously.

Response to Weakness 2:

We sincerely thank the reviewer for emphasizing the importance of computational efficiency and for the thoughtful, constructive suggestions. We greatly appreciate this opportunity to provide a more comprehensive assessment of FSSCM.

1. Computational efficiency.

To directly address the reviewer's main concern, we report the computational efficiency of FSSCM alongside CogCM and AVSEMamba below.

MethodParams (M) ↓GMACs ↓GFLOPs* ↓Latency (ms) ↓RTF ↓
CogCM12.47287.69575.3865.30.065
AVSEMamba36.24104.01208.02116.90.117
FSSCM (Ours)4.0645.1990.3858.30.058

GFLOPs are approximated using two FLOPs per multiply–accumulate operation.

Compared with CogCM, FSSCM reduces the parameter count by 67.4%, GMACs by 84.3%, and inference latency by 10.7%. Compared with AVSEMamba, the corresponding reductions are 88.8%, 56.6%, and 50.1%, respectively. These results demonstrate that FSSCM achieves strong speech enhancement performance with substantially fewer parameters and lower computational cost, offering a favorable balance between enhancement quality and computational efficiency.

Inspired by the reviewer's valuable suggestion, we have further extended the computational efficiency analysis to all compared baseline methods, reporting parameter counts, GMACs, inference latency, and RTF. The complete results and measurement settings are provided in Appendix A.6. We have also included a concise discussion of current limitations and future directions for computational optimization in the Conclusion.

2. APFM depth and additional analysis.

We clarify that FSSCM employs N = 4 stacked APFM layers, each incorporating bidirectional temporal and frequency modeling. We have explicitly specified this configuration in the final paragraph of Section 3.3. We sincerely appreciate the reviewer's constructive feedback, which has helped us strengthen the completeness and rigor of our experimental evaluation.

AuthorsOpenReview ↗

Response to Reviewer 3: Questions 1–3

Response to Question 1:

We sincerely thank the reviewer for highlighting FA-Mamba-AVSE and for the thoughtful question regarding our methodological distinction. FA-Mamba-AVSE does not provide publicly available source code or pretrained checkpoints. We attempted to reproduce the method based on its published description; however, we could not reliably verify that our implementation faithfully reproduced the complete architecture. We also contacted the authors for clarification but received no response. Therefore, we did not report a direct comparison under our evaluation protocol to avoid potentially misleading results.

Beyond L→H coupling, the fundamental distinction lies in where and how visual information intervenes in speech recovery. FA-Mamba-AVSE employs cross-attentive gated fusion to integrate audiovisual representations. Although effective for cross-modal interaction, such representation-level integration may constrain how fully visual information contributes to speech recovery, particularly under severe acoustic degradation. In contrast, FSSCM constructs a temporally structured visual articulatory prior from continuous lip dynamics and uses it to directly govern acoustic state evolution, regulating how acoustic information is accumulated and updated during recovery. Our controlled comparison in Table 3(a) further supports the effectiveness of this design, showing 1.54–2.04 dB SDR improvements over feature-level fusion.

We sincerely appreciate the reviewer's valuable suggestion. We have further clarified this methodological distinction in Section 2.2 (Related Work) of the revised manuscript.

Response to Question 2:

We sincerely thank the reviewer for emphasizing the importance of transparent and consistent baseline comparisons. Across all five benchmarks, all external baseline methods except CogCM were independently run and evaluated by us using their publicly available implementations, rather than adopting previously reported results. Regarding CogCM, its official code and pretrained checkpoints were not publicly available, and our request to the authors received no response. We therefore independently reimplemented CogCM based on its published methodology and obtained results consistent with the original publication. Nevertheless, we retained the published CogCM values in the corresponding comparison tables to ensure faithful attribution of the reported results.

To ensure consistent and fair comparisons, all five benchmarks were constructed from the outset following the corresponding experimental protocols documented in CogCM. Within each benchmark, FSSCM and all baselines evaluated by us used the same fixed test samples, identical noise mixtures and SNR conditions, and unified metric implementations. We sincerely appreciate the reviewer's constructive suggestion and have clarified the baseline result provenance and benchmark-specific evaluation protocols in Section 4 of the revised manuscript.

We also appreciate the reviewer's careful observation regarding the numerical difference in Table 5. Our performance conclusions are based on the overall experimental evidence rather than isolated numerical differences. For instance, on TCD-TIMIT+NTCD-TIMIT, FSSCM achieves a 2.53 dB SI-SDRi improvement over CogCM's published result at −5 dB SNR. The 0.01 dB difference at 0 dB SNR is not used to substantiate our main performance conclusions.

We sincerely thank the reviewer for these valuable suggestions, which have helped us improve the transparency and rigor of our experimental evaluation.

Response to Question 3:

We sincerely thank the reviewer for carefully examining these performance variations and suggesting possible explanations. As discussed in our response to Critical Weakness 1, these variations may be influenced by dataset-specific characteristics. On MEAD, emotion-dependent articulatory variations may affect audiovisual correspondence and, consequently, the effectiveness of visual-guided recovery. On GRID, the constrained vocabulary and sentence structure may provide more predictable linguistic context, potentially benefiting CogCM's semantic contextual modeling and narrowing the performance gap. These factors may explain why the relative advantages vary across datasets and acoustic conditions. We sincerely appreciate the reviewer's constructive suggestion and have incorporated this discussion into Section 5.1 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 3: Questions 4–5

Response to Question 4:

We sincerely thank the reviewer for highlighting the importance of computational efficiency and for this valuable suggestion. To directly address the reviewer's question, we provide the parameter counts, computational costs, inference latency, and real-time factors of FSSCM, CogCM, and AVSEMamba below.

MethodParams (M) ↓GMACs ↓GFLOPs* ↓Latency (ms) ↓RTF ↓
CogCM12.47287.69575.3865.30.065
AVSEMamba36.24104.01208.02116.90.117
FSSCM (Ours)4.0645.1990.3858.30.058

GFLOPs are approximated using two FLOPs per multiply–accumulate operation.

Compared with CogCM, FSSCM reduces the parameter count by 67.4%, GMACs by 84.3%, and inference latency by 10.7%. Compared with AVSEMamba, the corresponding reductions are 88.8%, 56.6%, and 50.1%, respectively. FSSCM also achieves the lowest RTF among these three methods. These results demonstrate that FSSCM achieves strong enhancement performance with substantially fewer parameters and lower computational cost, offering a favorable balance between enhancement quality and computational efficiency.

Inspired by the reviewer's valuable suggestion, we have further extended the efficiency analysis to all compared baseline methods, covering parameter counts, GMACs, inference latency, and RTF. The complete results and measurement details are provided in Appendix A.6 of the revised manuscript.

Response to Question 5:

We sincerely thank the reviewer for carefully examining the PESQ results and raising this important reproducibility concern.

Across all five benchmarks, FSSCM and all independently evaluated baselines used identical metric implementations and evaluation configurations within each benchmark. Specifically, on LRS3+DNS (Table 1), PESQ was computed using the same wideband configuration on 16-kHz audio, with identical test utterances, DNS noise mixtures, and evaluation procedures. Thus, the reported PESQ of 2.949 at −15 dB was obtained under the same conditions as the independently evaluated baselines. We sincerely appreciate the reviewer's suggestion and have clarified these evaluation settings in Appendix A.2.

Reviewer_gLJjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FSSCM, an audio-visual speech enhancement framework that uses visual articulatory information to guide frequency-dependent speech recovery. The framework consists of three modules: VSPL extracts temporally structured visual priors, APFM modulates Mamba state transitions and progressively recovers low- and high-frequency components, and PFCL encourages cross-frequency consistency. Experiments on five benchmarks report improvements over existing methods, particularly under low-SNR conditions. Additional ablations investigate the contributions of the individual modules, visual intervention strategies, and frequency recovery order.

Critical strengths
  • The experimental results on LRS3+DNS are impressive, particularly under severe acoustic degradation. At -15 dB, FSSCM achieves 10.27 dB SDR, outperforming CogCM by 2.99 dB, while also improving PESQ and STOI. These results suggest that the proposed architecture can effectively exploit visual information when acoustic observations are unreliable.
  • The paper includes comprehensive ablation studies that go beyond removing individual components. Specifically Table 3 compares visual feature fusion, feature modulation, and state-transition modulation, as well as different frequency recovery orders. These controlled comparisons provide useful empirical evidence for the proposed architectural choices.
Critical weaknesses
  • The conceptual novelty appears somewhat limited relative to existing AVSE and Mamba-based enhancement methods. CSMamba (Li et al., ICASSP 2025) already explores frequency-specific state-space processing and cross-band information recovery, while FA-Mamba-AVSE (Saleem et al., Digital Signal Processing, 2026) combines frequency-aware Mamba modeling with visual guidance. Although the proposed transition-level modulation is a meaningful architectural difference, describing it as "cause-governed" recovery seems overstated, since the model does not explicitly establish a causal relationship between articulation and acoustic generation.

  • The experimental protocol raises concerns about the reliability and fairness of the comparisons. Appendix A.3 constructs the primary LRS3+DNS benchmark using an 8:1:1 partition of the LRS3 pretraining subset, but does not establish whether speakers or source recordings are disjoint across splits. Moreover, the baseline comparisons combine reproduced results with previously reported results, without sufficient evidence that all models share identical evaluation conditions. The absence of statistical variability and model complexity comparisons makes the reported performance margins difficult to assess conclusively.

  • The experiments do not sufficiently validate the proposed articulatory recovery mechanism under realistic visual uncertainty. While Table 3 supports the low-to-high recovery order, it does not establish that the low-frequency branch specifically recovers linguistic content or that the high-frequency branch primarily reconstructs details. Furthermore, the missing-visual-input analysis is difficult to interpret because Appendix A.5.1 states that the "w/o VSPL" variant retains a visual encoder, whereas Table 9 labels it audio-only. Experiments with temporally misaligned or occluded lip movements would provide stronger evidence for the claimed robustness of visual-guided state evolution.

Questions for authors
  • How were the LRS3+DNS splits constructed? Are speakers and source recordings kept separate across the training, validation, and test sets? Also, which baseline results were reproduced using the same evaluation protocol?
  • Why is FA-Mamba-AVSE not included in the main comparisons, given its relevance to the proposed approach? Could the authors also provide parameter counts, FLOPs, and inference latency for FSSCM and the closest baselines?
  • Could the authors clarify whether the audio-only variant in Table 9 is different from the w/o VSPL configuration described in Appendix A.5.1? How does the model perform when visual inputs are temporally misaligned or partially occluded?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 2: Critical Weakness 1

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We greatly appreciate these valuable suggestions, which have helped us strengthen the experimental validation, methodological clarity, and reproducibility of our work. We have carefully addressed each concern and provide point-by-point responses below.

Response to Critical Weakness 1

We sincerely thank the reviewer for highlighting these relevant studies and providing the opportunity to clarify our conceptual contribution.

CSMamba is an audio-only speech enhancement method that employs sub-band-specific processing and cross-band spectral restoration without incorporating visual articulatory information. FA-Mamba-AVSE primarily utilizes visual information through cross-modal feature fusion to enhance acoustic representations. Although it also accounts for frequency-dependent acoustic characteristics, it does not explicitly establish the visually governed low-to-high recovery dependency proposed in FSSCM. The fundamental distinction of FSSCM therefore lies not in exploiting visual or frequency information individually, but in how acoustic recovery is governed and organized through a shared articulatory prior and explicit dependencies between successive recovery stages.

Specifically, FSSCM reformulates AVSE from observation-driven reconstruction into a visually governed, frequency-coupled state-recovery process. Guided by the speech-production process, FSSCM establishes an explicit cause → content → detail recovery dependency: a noise-independent visual articulatory prior governs acoustic state evolution, the evolving low-frequency recovery state guides subsequent high-frequency detail reconstruction together with the same prior, and cross-frequency consistency is anchored to this shared articulatory reference. These interconnected mechanisms constitute a unified recovery formulation, rather than a simple combination of existing visual guidance and frequency-processing techniques. The controlled ablations in Table 3 further support these design choices: transition-level visual modulation improves SDR by 1.54–2.04 dB over feature fusion (Table 3(a)), while the proposed L→H recovery improves SDR by 0.42–0.55 dB over parallel recovery (Table 3(b)).

We sincerely thank the reviewer for raising this important question regarding the interpretation of “cause-governed.” We would like to clarify its physical motivation and intended scope. Our formulation is motivated by the physical process of speech production, in which articulatory movements contribute to acoustic speech generation. Previous studies have investigated articulatory–acoustic relationships (Conant et al., 2018) and the influence of lip movements on acoustic spectral characteristics (Plass et al., 2020). In FSSCM, visual observations provide cues to this upstream articulatory process and are used to regulate acoustic recovery through state-space parameter modulation and successive frequency-dependent reconstruction.

Importantly, “cause-governed” refers to a recovery principle motivated by the physical speech-production process, rather than causal discovery or formal causal identification. The proposed cause → content → detail dependency describes how this physical motivation is instantiated in our computational recovery framework, rather than claiming to establish a new causal model of speech generation.

We sincerely appreciate the reviewer for highlighting this important distinction. We have further clarified the discussion of CSMamba and FA-Mamba-AVSE in Related Work, so that the scope of our contribution and its relationship to prior studies are presented more precisely.

References

[1] Conant, D. F., Bouchard, K. E., Leonard, M. K., & Chang, E. F. (2018). Human sensorimotor cortex control of directly measured vocal tract movements during vowel production. Journal of Neuroscience, 38(12), 2955–2966.

https://doi.org/10.1523/JNEUROSCI.2382-17.2018

[2] Plass, J., Brang, D., Suzuki, S., & Grabowecky, M. (2020). Vision perceptually restores auditory spectral dynamics in speech. PNAS, 117(29), 16920–16927.

https://doi.org/10.1073/pnas.2002887117

AuthorsOpenReview ↗

Response to Reviewer 2: Critical Weakness 2

1. Speaker and source-recording independence.

We sincerely thank the reviewer for emphasizing the importance of data independence. Before conducting the reported experiments, we established the original LRS3+DNS partition and verified its cross-split independence. Our verification found no evidence of speaker or source-recording overlap across the training, validation, and test splits under the adopted verification criteria. Specifically, speaker-embedding checks detected no cross-split matches, while source-video identifier checks found no overlaps across the complete partition. Additional source-provenance checks identified no shared source recordings among samples with resolvable metadata. We have also added a brief clarification regarding the partition independence verification in Appendix A.3 of the revised manuscript.

Audit criterionCoverageTrain–ValTrain–TestVal–Test
Speaker-embedding matches (≥0.70)5,090/5,090000
Shared source-video IDs5,090/5,090000
Shared metadata-resolved source recordings5090/5,090000

2. Baseline comparability and evaluation consistency.

We sincerely thank the reviewer for emphasizing the importance of fair and consistent baseline comparisons. Across all five benchmarks, all baseline methods were run and evaluated by us using their publicly available implementations, rather than adopting previously reported results, with CogCM being the sole exception. Regarding CogCM, its official code and pretrained checkpoints were not publicly available, and our request to the authors received no response. We therefore independently reimplemented CogCM based on its published methodology and obtained results consistent with the original publication. Nevertheless, we retained the published CogCM values in the corresponding comparison tables to ensure faithful attribution of the reported results. To ensure consistent and fair comparisons, all five benchmarks were constructed from the outset following the corresponding experimental protocols documented in CogCM. Within each benchmark, FSSCM and all baselines evaluated by us used the same fixed test samples, identical noise mixtures and SNR conditions, and unified metric implementations. We sincerely appreciate the reviewer's constructive suggestion and have clarified the baseline result provenance and benchmark-specific evaluation protocols in Section 4 of the revised manuscript.

3. Statistical variability analysis.

We sincerely thank the reviewer for this valuable suggestion. To address this concern, we additionally report the mean, standard deviation, and 95% confidence intervals of utterance-level SDR, PESQ, and STOI across all four SNR conditions on LRS3+DNS. The corresponding results have been added in Table 8 of Appendix A.2 of the revised supplementary material.

4. Model complexity and inference efficiency

We sincerely thank the reviewer for highlighting the importance of computational complexity. To address this concern, we have added a comprehensive comparison of model parameters, GMACs, inference latency, and real-time factor (RTF) for FSSCM and all baseline methods in Table 22 of Appendix A.6. Together with the enhancement results in Table 1, this analysis demonstrates that FSSCM achieves strong enhancement performance while maintaining a relatively compact model size.

AuthorsOpenReview ↗

Response to Reviewer 2: Critical Weakness 3

Response to Critical Weakness 3

We sincerely thank the reviewer for the thoughtful and constructive suggestions regarding the interpretation of our recovery mechanism and its robustness to visual uncertainty.

We would like to clarify that “content-oriented” and “detail-oriented” characterize the intended emphasis of successive recovery stages, rather than a strict separation of linguistic content and acoustic details. This interpretation is motivated by the heterogeneous and overlapping frequency-dependent cues in speech perception (Fogerty & Chen, 2016; Kasturi et al., 2002; Li et al., 2012).

Importantly, the proposed dependency is explicitly implemented in APFM: the recovered low-frequency state and the shared visual articulatory prior jointly guide subsequent high-frequency recovery (Eqs. (12)–(15)). Table 3(b) further shows that L→H consistently outperforms both parallel L∥H and reverse H→L recovery, with 0.42–0.55 dB SDR improvements over parallel recovery. These results provide empirical support for the proposed directional low-to-high recovery dependency, which constitutes a key component of FSSCM's frequency-coupled state-recovery mechanism.

References

[1] Fogerty, D., & Chen, F. (2016). Vowel Fundamental and Formant Frequency Contributions to English and Mandarin Sentence Intelligibility. Proceedings of Interspeech 2016, 1382–1386.

https://doi.org/10.21437/Interspeech.2016-28

[2] Kasturi, K., Loizou, P. C., Dorman, M., & Spahr, T. (2002). The Intelligibility of Speech with “Holes” in the Spectrum. The Journal of the Acoustical Society of America, 112(3), 1102–1111.

https://doi.org/10.1121/1.1498855

[3] Li, F., Trevino, A., Menon, A., & Allen, J. B. (2012). A Psychoacoustic Method for Studying the Necessary and Sufficient Perceptual Cues of American English Fricative Consonants in Noise. The Journal of the Acoustical Society of America, 132(4), 2663–2675.

https://doi.org/10.1121/1.4747008

We sincerely thank the reviewer for carefully identifying this inconsistency. We apologize for the incorrect “audio-only” designation, as the w/o VSPL variant still retains video input through the 3D-CNN visual encoder. We have corrected this error by retaining the w/o VSPL results in Table 2 as a module-level ablation, revising Table 14 to focus on visual-input perturbations, and updating Appendix A.5.4 to ensure consistency with Appendix A.5.1.

We sincerely thank the reviewer for this constructive suggestion. To address this concern, we have conducted additional temporal misalignment (±2 and ±4 frames) and partial lip occlusion (25% and 50%) experiments across four SNR conditions using the same trained FSSCM checkpoint, as reported in Table 14 of Appendix A.5.4. At −15 dB, synchronized input achieves 10.27 dB SDR, compared with 9.06–9.83 dB under temporal misalignment, 9.14–9.89 dB under lip occlusion, and 8.21 dB with black-frame input. These results indicate that visual perturbations reduce enhancement performance, while FSSCM retains a measurable benefit from the remaining visual information.

Visual condition−15 dB SDR ↑−10 dB SDR ↑−5 dB SDR ↑0 dB SDR ↑
Synchronized input10.2712.5814.9817.30
Temporal shift −2 frames9.7112.0414.5116.91
Temporal shift +2 frames9.8312.0014.5516.79
Temporal shift −4 frames9.0611.4013.9416.38
Temporal shift +4 frames9.1711.3314.0016.47
Lip occlusion 25%9.8912.2414.5816.94
Lip occlusion 50%9.1411.5614.1016.61
Black-frame input8.2110.5513.1515.82
Reviewer_eecAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary
  • The paper addresses audio-visual speech enhancement: recovering clean speech from noisy audio, using video of the speaker's lips. A visual module builds per-frame features from the lips and how they move over time.
  • The audio is split into low and high frequency bands, using a soft mask chosen from the video. Each band is processed by Mamba layers whose state updates are scaled by the visual features, and the high band is also guided by the output of the low band.
  • A final module combines the two bands using cross-attention and weights computed from the visual features, before the enhanced magnitude and phase are decoded.
Critical strengths
  • Core idea: using lip movements to scale how the Mamba state updates is a simple change that performs well. With everything else fixed, it scores 1.54 to 2.04 dB above feature fusion, and the gap grows as SNR decreases (Table 3).
  • Results on the main benchmark: on LRS3+DNS, FSSCM scores above all 14 baselines, including Dolphin and CogCM, in all 12 entries of Table 1. At −15 dB it reaches 10.27 dB SDR, against 7.49 dB for Dolphin.
  • Thorough ablations: Tables 2, 3 and 8 remove or replace each module and its parts in turn, and report how much each contributes. This shows which parts of the design matter, which is useful for anyone building on the method.
Critical weaknesses
  • Results only hold on one benchmark: Table 1 shows only LRS3+DNS, where FSSCM leads. The other four benchmarks are not in the main text, and they only compare to CogCM, so performance against Dolphin is uncertain. In addition, on these benchmarks FSSCM and CogCM are essentially tied. This reduces the strength of the paper's main claims.
  • All ablations and analyses use one dataset and do not report parameter counts: the evidence that the proposed design causes the gains therefore comes from one dataset, and could possibly be due to different parameter counts or expressivity.
  • The method cannot be reproduced from the paper: many layers are written only as a function f (for example f_ω, f_trans and f_cons) without being defined or explained, and the same notation is reused for different things, which makes the method hard to follow. The architecture settings, model sizes, and how noise is mixed with speech for training, including the SNR range, are not given.
Questions for authors
  • Other benchmarks: please report the full comparison, including Dolphin, on at least two of the other benchmarks (for example GRID, TCD-TIMIT or MEAD), in the main text.
  • Ablations: please repeat Table 3 on the same additional benchmarks, and give the parameter count of each variant.
  • Reproducibility: please define each learned function, and give the architecture settings, model sizes, and how the noisy training data is built, including the SNR range. Please also say whether the w/o VSPL variant receives video, and check the Table 7 rows for DualAVSE and CogCM.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 1: Critical Weaknesses 1–2 and Questions 1–2

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We greatly appreciate these valuable suggestions, which have helped us strengthen the experimental validation, methodological clarity, and reproducibility of our work. We have carefully addressed each concern and provide point-by-point responses below.

Response to Critical Weakness 1 & Question 1: We sincerely thank the reviewer for the constructive suggestions regarding cross-benchmark evaluation and comparison with Dolphin. We agree that presenting more complete comparisons in the main text would strengthen the empirical evidence for FSSCM. First, following the reviewer's suggestion, we have extended the comparisons to include Dolphin across all four additional benchmarks. The complete multi-method comparisons on GRID+CHiME3 and MEAD+DEMAND, including Dolphin, are now presented in Section 5.1 of the revised manuscript. Due to the main-text space constraints, the complete comparisons on the remaining two benchmarks, TCD-TIMIT+NTCD-TIMIT and AVSpeech+DNS, are provided in Appendix A.4. Second, regarding the comparison with CogCM, FSSCM outperforms CogCM in 30 of the 46 metric–condition comparisons across the four additional benchmarks, demonstrating its superior performance in the majority of evaluated settings. Notably, FSSCM improves SI-SDRi by 2.53 dB on TCD-TIMIT+NTCD-TIMIT at −5 dB and by 1.23 dB on MEAD+DEMAND at −10 dB. These results provide concrete evidence of FSSCM's performance advantages beyond the primary LRS3+DNS benchmark.

Response to Critical Weakness 2 and Question 2:
We sincerely thank the reviewer for the constructive suggestions regarding the generalizability and fairness of our ablation studies. First, regarding cross-benchmark validation, we have extended both Table 3(a) and Table 3(b) to four additional benchmarks: GRID+CHiME3, TCD-TIMIT+NTCD-TIMIT, MEAD+DEMAND, and AVSpeech+DNS, covering all evaluation conditions under their respective protocols. The results consistently show that transition-level visual governance and L→HL\rightarrow H recovery achieve the highest scores among their respective ablation variants. Second, regarding model capacity, we report the trainable parameter counts of all ablation variants. The comparisons retain the same principal architectural components wherever applicable. Notably, H→LH\rightarrow L and L→HL\rightarrow H have identical parameter counts (4.06M), yet L→HL\rightarrow H improves SI-SDRi by 1.32 dB on MEAD+DEMAND at −10 dB. This parameter-matched comparison indicates that the improvement cannot be explained by differences in trainable parameter counts alone. The complete cross-benchmark ablation results and parameter statistics are provided in Appendix A.5.7 (Tables 17–21) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 1: Critical Weakness 3 and Question 3

Response to Critical Weakness 3 and Question 3

We sincerely thank the reviewer for the careful examination of our method and experimental settings. We address Critical Weakness 3 and Question 3 together below.

1. Definitions of Learnable Functions

We sincerely thank the reviewer for highlighting the need for clearer definitions of the learnable functions. We have added Table 6 in Appendix A.1.3 of the revised manuscript, explicitly specifying the inputs, layer architectures, and outputs of the principal learnable mappings in VSPL, APFM, and PFCL, including fωf_{\omega}, ftransbf_{\mathrm{trans}}^b, and fconsf_{\mathrm{cons}}. This addition clarifies their respective roles and implementations.

2. Clarification of Notation

We sincerely thank the reviewer for carefully identifying the notation ambiguities and for this valuable suggestion. Following the reviewer's advice, we have revised Eqs. (16) and (20)–(21) in Section 3.4 to eliminate symbol conflicts and clarify their respective meanings. We have also carefully reviewed the notation throughout the manuscript and corrected similar ambiguities to ensure consistent and unambiguous definitions. We believe these revisions have improved the clarity and readability of the method, and we greatly appreciate the reviewer's help in strengthening the presentation.

3. Architecture Configurations and Model Size

We thank the reviewer for emphasizing the importance of architectural reproducibility. We have added an Architecture Configurations and Model Size table in Appendix A.1.4 of the revised manuscript, as reproduced below.

ComponentConfiguration
Audio encoderDenseEncoder with 64 hidden channels and a four-layer dense block; frequency downsampling factor of 2
Visual encoderThree-layer 3D-CNN with channel dimensions 32/64/64; visual feature dimension d=64d = 64
VSPLFirst- and second-order visual dynamics, adaptive multi-order fusion, and four-head temporal self-attention
APFMFour stacked recovery layers; K=4K = 4 learnable monotonic spectral bases; progressive low-to-high frequency recovery
Mambadmodel=64d_{\mathrm{model}} = 64, dstate=16d_{\mathrm{state}} = 16, dconv=4d_{\mathrm{conv}} = 4, expansion factor of 4; bidirectional temporal and frequency scanning
PFCLFour-head cross-frequency attention and prior-anchored consistency modeling
Magnitude decoderFour-layer dense decoder with 64 hidden channels
Phase decoderFour-layer dense decoder with 64 hidden channels
Total trainable parameters4.06M

4. Training Data Construction and SNR Ranges

We sincerely thank the reviewer for highlighting the importance of specifying the noisy training-data construction procedures and SNR ranges. We have provided detailed dataset construction and partitioning protocols, noise-mixing procedures, and training SNR/SIR ranges for all five benchmarks in Appendix A.3 of the revised manuscript.

5. Clarification of the w/o VSPL Configuration

We sincerely thank the reviewer for the careful and thorough examination of our manuscript. We apologize for the incorrect “audio-only” designation, as the w/o VSPL variant still retains video input through the 3D-CNN visual encoder. We have corrected this error and revised the corresponding descriptions in the original Table 9 and Appendix A.5.4 to ensure consistency with Appendix A.5.1.

6. Verification of Table 7

We sincerely thank the reviewer for carefully examining the results reported for DualAVSE and CogCM in the original Table 7. We have rechecked the corresponding entries and corrected the affected values in Table 12 of the revised manuscript. We also reviewed the other experimental tables for consistency. We appreciate the reviewer's careful attention to the accuracy of our experimental reporting.

Additional Reproducibility Commitment

We sincerely appreciate the reviewer’s careful examination and constructive suggestions regarding reproducibility. We fully recognize the importance of providing sufficient implementation details to enable independent verification of our results. In addition to the clarifications and supplementary information provided in the revised manuscript, we commit to publicly releasing the complete source code, training configurations, data preprocessing and noise-mixing scripts, and evaluation code upon acceptance. We sincerely thank the reviewer for these valuable suggestions, which have helped us improve the clarity, transparency, and reproducibility of our work.

Reviewer_eecAOpenReview ↗

Response to rebuttal: score updated from 2 to 3

Dear Authors,

Thanks for the time taken to carefully respond to all my feedback.

Q1 - Dolphin and other benchmarks addressed.

Q2 - Additional ablations addressed, clear parameter counts rule out capacity as the driver of performance. I also commend the final model size. At 4M parameters you're now significantly smaller than Dolphin and CogCM, and despite having more MACs than Dolphin you get a lower latency. If you like, you could add an efficiency/throughput angle to the main text to strengthen your claims/arguments instead of relying on just SOTA performance. This would mean readers would be less put off by the 30/46 comparison with CogCM.

Q3 - Equations etc addressed.

Soundness 1 -> 2.

Presentation stays at 2 - still quite difficult to follow, but I understand the space is quite limited. Your plan to open-source the code on acceptance reduces reproducibility concerns.

Overall score: Updating from a 2 -> 3. Great work!

One additional question: you mention in other responses that you reimplemented CogCM but kept their published values in the paper. Could you report your reimplementation results as well/instead in the paper so that all paper results are on a level playing field?

AuthorsOpenReview ↗

Response to reviewer eecA

We sincerely thank the reviewer for the quick response and providing the useful suggestion. We agree that reporting the reimplementation results in the paper will be good for our experiments. As previously stated, we reimplemented CogCM according to its paper, and our results are broadly consistent with those reported in the original paper. However, we note that, on a few benchmarks, some results are still lower than those reported in the original paper. We cannot guarantee that our reimplementation is 100% consistent with the original CogCM implementation. Thus, in our paper, we do not adopt the results we reimplement. Instead, we use the published CogCM values in their original paper. Following your valuable suggestion, we will additionally report our CogCM reimplementation results in the supplementary material, while retaining the originally published values in the main comparison tables. We will also release our complete CogCM reimplementation code to facilitate reproducibility. To guarantee the rigor and fair comparisons, all five benchmarks in our paper are constructed following the same experimental protocols documented in CogCM. In addition, within each benchmark, FSSCM and all other baselines are evaluated by using the same fixed test samples, identical noise mixtures and SNR conditions, and unified metric implementations as those in CogCM. According to your suggestion, in the revised version we have made it clear in Section 4 (Experiments).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.