- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that audio-visual speech enhancement (AVSE) should stop treating vision as an auxiliary feature and instead use it as a governing cause: lip movements sit upstream of acoustic production, are not corrupted by noise, and should therefore steer recovery dynamics rather than merely conditioning features. This "cause → content → detail" story is instantiated in FSSCM (Sec. 3) in three parts. VSPL (Sec. 3.2) encodes lip video with a 3D-CNN, augments the features with first- and second-order temporal differences, adaptively fuses them (Eq. 3), refines them with temporal self-attention, and produces a temporally resolved visual prior C_v plus a gate G_v (Eq. 5). APFM (Sec. 3.3) predicts a soft low/high frequency split from a global visual summary (Eqs. 6-8), modulates each Mamba branch's ∆ and B parameters with the prior (Eq. 9), and couples the branches so the low-frequency state guides the high-frequency transition signal (Eqs. 12-15). PFCL (Sec. 3.4) runs bidirectional cross-attention between the branches and scores their consistency against the broadcast prior (Eqs. 16-22). On the primary benchmark LRS3+DNS, FSSCM beats every baseline at all four SNRs, with the SDR margin over the strongest AVSE baseline CogCM growing from 1.96 dB at 0 dB to 2.99 dB at −15 dB (Table 1, Fig. 3). Four further benchmarks (GRID+CHiME3, TCD-TIMIT+NTCD-TIMIT, MEAD+DEMAND, AVSpeech+DNS) are evaluated in the supplement, and ablations cover module removal (Table 2), internal mechanisms (Table 8), intervention type (Table 10), frequency-organization direction (Table 11), and black-frame visual failure (Table 9).
As I read it, the reframing is the main conceptual sell; mechanically, most parts are familiar (3D-CNN visual encoders, FiLM-style modulation, Mamba, cross-attention), reorganized around the cause → content → detail principle and the low-to-high frequency dependency.
Critical strengths
The headline numbers are hard to dismiss. At −15 dB on LRS3+DNS, FSSCM reports 10.27 dB SDR against 7.49 for Dolphin and 7.28 for CogCM (Table 1), and the ordering holds at all four SNRs and on all three metrics. What makes this more than a leaderboard bump is that the margin widens monotonically as SNR drops (1.96 → 2.99 dB over CogCM, Fig. 3) — exactly the regime where the paper's own motivation predicts a noise-free visual prior should pay off most. That is the result the rest of the paper has to live up to, and mostly it does.
What pushes this over the line for me, though, is the ablation discipline. Table 2 removes each module wholesale, but Table 8 goes further and knocks out individual mechanisms — second-order articulatory dynamics (↓0.97 dB at −15 dB), temporal self-attention (↓1.17), global-vs-temporal C_v governance (↓0.87), ∆/B modulation (↓1.38) — so we can see which knobs do what, not just that the full model works. Tables 10-11 settle the two design claims I was most skeptical about: transition-level visual governance beats feature fusion by 1.5-2.0 dB depending on SNR (Table 10), and L→H coupling beats parallel recovery by 0.42-0.55 dB and the reverse H→L direction at every SNR (Table 11). Controlled comparisons like that are what make an architectural paper convincing.
Critical weaknesses
The paper's central narrative — the advantage grows as acoustic reliability drops — is really only established on LRS3+DNS. On TCD-TIMIT, FSSCM beats CogCM by 2.53 dB at −5 dB, but at 0 and 5 dB the two sit within ~0.1 dB and at 10 dB CogCM leads by 0.59 dB on SI-SDRi, with CogCM also taking STOIi at 0 dB (Table 5). On MEAD the SI-SDRi differences flip sign across SNRs (−10 dB: +1.23 for FSSCM; −5 dB: −1.06; Table 6), and on GRID at 0 dB CogCM wins STOIi (20.50 vs 20.16, Table 4). Fig. 4 then shows only the most favorable SNR per benchmark, all against CogCM alone, which flatters the story. None of this contradicts the main result, but Sec. 1 and the conclusion present the SNR-scaling trend as a general finding and it isn't one. A few sentences on where and why it breaks down would make the paper more credible than the current selective presentation.
No efficiency analysis, and no limitations section. The architecture stacks N APFM layers (N is never specified in the text), each with two Mamba branches scanned bidirectionally along both time and frequency, plus cross-attention, consistency heads, and two decoders — and LRS3+DNS training takes 4 A800-80G GPUs (Sec. A.1.2). Parameter count, FLOPs, and real-time factor appear nowhere, nor is there any cost comparison against CogCM or AVSEMamba. For a methods paper whose selling points are architectural, I can't tell whether the 2-3 dB SDR gains are cheap or come with a much heavier compute bill. This is the biggest gap I'd want closed.
Questions for authors
-
FA-Mamba-AVSE (Saleem et al., 2026b) is discussed in Sec. 2.2 but never appears in any comparison table, despite being — as far as I can tell — the closest concurrent method (frequency-aware state-space modeling plus gated cross-modal fusion for the same task). Is there a head-to-head under your protocol? And beyond the L→H coupling, what do you see as the substantive difference between your transition-level governance (Eq. 9) and their gated fusion?
-
Sec. 4 says baseline numbers mix reimplemented and previously reported results. Which rows in Tables 1-7 are which? Some differences you treat as wins are far inside run-to-run noise (Table 5, 0 dB: 18.96 vs 18.95).
-
Why does the advantage over CogCM flatten or invert on MEAD and GRID (Tables 4, 6)? Dataset size, MEAD's emotional speech, visual quality, or the adaptive frequency routing behaving differently on smaller corpora? A sentence or two of analysis would help.
-
Params, FLOPs, and RTF versus CogCM and AVSEMamba — even rough numbers in the rebuttal would change how I read Tables 1-7.
-
FSSCM's −15 dB PESQ (2.949) is higher than many baselines' 0 dB PESQ (Table 1). Can you confirm identical PESQ configuration (wideband, 16 kHz, mir_eval settings) across all methods?
3 comments
Response to Reviewer 3: Questions 1–3
Response to Question 1:
We sincerely thank the reviewer for highlighting FA-Mamba-AVSE and for the thoughtful question regarding our methodological distinction. FA-Mamba-AVSE does not provide publicly available source code or pretrained checkpoints. We attempted to reproduce the method based on its published description; however, we could not reliably verify that our implementation faithfully reproduced the complete architecture. We also contacted the authors for clarification but received no response. Therefore, we did not report a direct comparison under our evaluation protocol to avoid potentially misleading results.
Beyond L→H coupling, the fundamental distinction lies in where and how visual information intervenes in speech recovery. FA-Mamba-AVSE employs cross-attentive gated fusion to integrate audiovisual representations. Although effective for cross-modal interaction, such representation-level integration may constrain how fully visual information contributes to speech recovery, particularly under severe acoustic degradation. In contrast, FSSCM constructs a temporally structured visual articulatory prior from continuous lip dynamics and uses it to directly govern acoustic state evolution, regulating how acoustic information is accumulated and updated during recovery. Our controlled comparison in Table 3(a) further supports the effectiveness of this design, showing 1.54–2.04 dB SDR improvements over feature-level fusion.
We sincerely appreciate the reviewer's valuable suggestion. We have further clarified this methodological distinction in Section 2.2 (Related Work) of the revised manuscript.
Response to Question 2:
We sincerely thank the reviewer for emphasizing the importance of transparent and consistent baseline comparisons. Across all five benchmarks, all external baseline methods except CogCM were independently run and evaluated by us using their publicly available implementations, rather than adopting previously reported results. Regarding CogCM, its official code and pretrained checkpoints were not publicly available, and our request to the authors received no response. We therefore independently reimplemented CogCM based on its published methodology and obtained results consistent with the original publication. Nevertheless, we retained the published CogCM values in the corresponding comparison tables to ensure faithful attribution of the reported results.
To ensure consistent and fair comparisons, all five benchmarks were constructed from the outset following the corresponding experimental protocols documented in CogCM. Within each benchmark, FSSCM and all baselines evaluated by us used the same fixed test samples, identical noise mixtures and SNR conditions, and unified metric implementations. We sincerely appreciate the reviewer's constructive suggestion and have clarified the baseline result provenance and benchmark-specific evaluation protocols in Section 4 of the revised manuscript.
We also appreciate the reviewer's careful observation regarding the numerical difference in Table 5. Our performance conclusions are based on the overall experimental evidence rather than isolated numerical differences. For instance, on TCD-TIMIT+NTCD-TIMIT, FSSCM achieves a 2.53 dB SI-SDRi improvement over CogCM's published result at −5 dB SNR. The 0.01 dB difference at 0 dB SNR is not used to substantiate our main performance conclusions.
We sincerely thank the reviewer for these valuable suggestions, which have helped us improve the transparency and rigor of our experimental evaluation.
Response to Question 3:
We sincerely thank the reviewer for carefully examining these performance variations and suggesting possible explanations. As discussed in our response to Critical Weakness 1, these variations may be influenced by dataset-specific characteristics. On MEAD, emotion-dependent articulatory variations may affect audiovisual correspondence and, consequently, the effectiveness of visual-guided recovery. On GRID, the constrained vocabulary and sentence structure may provide more predictable linguistic context, potentially benefiting CogCM's semantic contextual modeling and narrowing the performance gap. These factors may explain why the relative advantages vary across datasets and acoustic conditions. We sincerely appreciate the reviewer's constructive suggestion and have incorporated this discussion into Section 5.1 of the revised manuscript.
Response to Reviewer 3: Questions 4–5
Response to Question 4:
We sincerely thank the reviewer for highlighting the importance of computational efficiency and for this valuable suggestion. To directly address the reviewer's question, we provide the parameter counts, computational costs, inference latency, and real-time factors of FSSCM, CogCM, and AVSEMamba below.
| Method | Params (M) ↓ | GMACs ↓ | GFLOPs* ↓ | Latency (ms) ↓ | RTF ↓ |
|---|---|---|---|---|---|
| CogCM | 12.47 | 287.69 | 575.38 | 65.3 | 0.065 |
| AVSEMamba | 36.24 | 104.01 | 208.02 | 116.9 | 0.117 |
| FSSCM (Ours) | 4.06 | 45.19 | 90.38 | 58.3 | 0.058 |
GFLOPs are approximated using two FLOPs per multiply–accumulate operation.
Compared with CogCM, FSSCM reduces the parameter count by 67.4%, GMACs by 84.3%, and inference latency by 10.7%. Compared with AVSEMamba, the corresponding reductions are 88.8%, 56.6%, and 50.1%, respectively. FSSCM also achieves the lowest RTF among these three methods. These results demonstrate that FSSCM achieves strong enhancement performance with substantially fewer parameters and lower computational cost, offering a favorable balance between enhancement quality and computational efficiency.
Inspired by the reviewer's valuable suggestion, we have further extended the efficiency analysis to all compared baseline methods, covering parameter counts, GMACs, inference latency, and RTF. The complete results and measurement details are provided in Appendix A.6 of the revised manuscript.
Response to Question 5:
We sincerely thank the reviewer for carefully examining the PESQ results and raising this important reproducibility concern.
Across all five benchmarks, FSSCM and all independently evaluated baselines used identical metric implementations and evaluation configurations within each benchmark. Specifically, on LRS3+DNS (Table 1), PESQ was computed using the same wideband configuration on 16-kHz audio, with identical test utterances, DNS noise mixtures, and evaluation procedures. Thus, the reported PESQ of 2.949 at −15 dB was obtained under the same conditions as the independently evaluated baselines. We sincerely appreciate the reviewer's suggestion and have clarified these evaluation settings in Appendix A.2.
Response to Reviewer 3: Critical Weaknesses 1–2
We sincerely thank the reviewer for the careful evaluation of our work and the thoughtful, constructive feedback. We particularly appreciate the reviewer's encouraging recognition of our contributions and valuable suggestions, which have helped us strengthen the experimental validation, methodological clarity, and reproducibility of our work. We have carefully addressed each concern and provide point-by-point responses below.
Response to Weakness 1:
We sincerely thank the reviewer for the careful examination of our cross-benchmark results and the thoughtful, constructive suggestions. We greatly appreciate this observation, which helps us clarify the scope of our findings and improve the rigor of our presentation. While LRS3+DNS exhibits a clear tendency toward larger relative gains under severe acoustic interference, this trend is less pronounced across the additional benchmarks, potentially reflecting dataset-specific characteristics and evaluation conditions. One possible explanation is that acoustic SNR characterizes the degree of noise interference but does not fully determine the complementary information provided by visual speech cues. Differences in dataset scale and diversity, speech characteristics, and articulatory–acoustic correspondence may influence the relative contribution of visual guidance. For instance, emotionally expressive speech in MEAD introduces different articulatory–acoustic characteristics, while GRID features more constrained sentence structures. To provide more comprehensive cross-benchmark evidence, we have further expanded the main-text comparisons to present the results more thoroughly and in a balanced manner. Following the reviewer's valuable suggestion, we have further clarified the observed cross-benchmark performance variations in Section 5.1. We sincerely appreciate the reviewer's constructive feedback, which helps us present our findings more accurately and rigorously.
Response to Weakness 2:
We sincerely thank the reviewer for emphasizing the importance of computational efficiency and for the thoughtful, constructive suggestions. We greatly appreciate this opportunity to provide a more comprehensive assessment of FSSCM.
1. Computational efficiency.
To directly address the reviewer's main concern, we report the computational efficiency of FSSCM alongside CogCM and AVSEMamba below.
GFLOPs are approximated using two FLOPs per multiply–accumulate operation.
Compared with CogCM, FSSCM reduces the parameter count by 67.4%, GMACs by 84.3%, and inference latency by 10.7%. Compared with AVSEMamba, the corresponding reductions are 88.8%, 56.6%, and 50.1%, respectively. These results demonstrate that FSSCM achieves strong speech enhancement performance with substantially fewer parameters and lower computational cost, offering a favorable balance between enhancement quality and computational efficiency.
Inspired by the reviewer's valuable suggestion, we have further extended the computational efficiency analysis to all compared baseline methods, reporting parameter counts, GMACs, inference latency, and RTF. The complete results and measurement settings are provided in Appendix A.6. We have also included a concise discussion of current limitations and future directions for computational optimization in the Conclusion.
2. APFM depth and additional analysis.
We clarify that FSSCM employs N = 4 stacked APFM layers, each incorporating bidirectional temporal and frequency modeling. We have explicitly specified this configuration in the final paragraph of Section 3.3. We sincerely appreciate the reviewer's constructive feedback, which has helped us strengthen the completeness and rigor of our experimental evaluation.