- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a practical problem in deployed audio-visual target-speaker extractors, where the audio input is usually a single-microphone channel, leaving spatial information from the multi-microphone array unused. The paper proposes D-CRPF, a light-weight adapter with only 70.9k learnable parameters that adapts the output of a frozen AV target-speaker extractor without touching the original weights/hidden states. The adaptation follows a three-stage pipeline. First, an initial estimate is generated using the frozen extractor. Then, the estimate is used to guide two beamformers to generate target- and interference-oriented view from the multi-microphone array input. Finally, a trainable network will use the two views and the single-channel reference signal to compute a complex correction on STFT space (CRPF-R) and a signed correction based on the disagreement between the target beam and the estimate.
Experiments are performed on six-microphone reverberant mixtures from LRS2 dataset. The proposed CRPF-R is evaluated with multiple front-ends and achieves improvement in SI-SDR and reduction in WERs. With the final D-CRPF, further improvements are observed on the AV-CrossNet.
Critical strengths
-
Consistent improvement across multiple frozen extractors. Table 1 shows that CRPF-R improves Dolphin, AV-MossFormer2, and IIANet without updating their weights. This supports the practical premise that output-level spatial adaptation can work across different front ends, rather than only with one matched model.
-
Useful controlled experiment for spatial information. On matched AV-CrossNet, D-CRPF gains 1.254 dB SI-SDR while an equal-parameter single-channel control gains only 0.045 dB. This strongly supports that the additional microphone signals account for much of the improvement.
Critical weaknesses
-
The full D-CRPF's validation is insufficient. Only one training run reports the gain from CRPF-R to full D-CRPF. Further validation could enhance the usefulness of the signed correction.
-
The generality of the system remains uncertain. The experiments use simulated mixtures rather than real array recordings. The controlled circle-to-line test is helpful, but the adaptation gain falls from 1.246 to 0.441 dB.
Questions for authors
-
Does D-CRPF improve on CRPF-R for any of the three released front ends in Table 1?
-
How stable is the 0.225 dB D-CRPF advantage over CRPF-R?
-
How sensitive is the result to the fixed 0.25 reference-correction scale?
2 comments
I thank the authors for the detailed rebuttal and the additional experiments. Most of my concerns have been addressed. The consistent improvements across different front ends provide stronger evidence for the effectiveness of D-CRPF and largely alleviate my concern about training stability. The additional experiments on AMI recordings also strengthen the practical value of the proposed approach.
My remaining concern is the sensitivity to the reference-correction scale α. While the authors clarify its mathematical role, it remains unclear how this value was chosen and whether a larger α for CRPF-R could narrow the performance gap between CRPF-R and D-CRPF.
Nevertheless, I consider this a minor remaining concern. Overall, I am satisfied with the rebuttal and will maintain my positive rating.
Cross-Front-End Gains, Stability, and Real-Recording Generalization
Thank you for recognizing the frozen-extractor setting and the spatial-information control. We address your questions with full D-CRPF results on three released front ends, the role of the correction scale, and the real-recording and array-transfer evaluations.
W1 / Q1: Does full D-CRPF improve on CRPF-R for the released front ends?
Yes. Full seven-map D-CRPF improves on CRPF-R across released Dolphin, AV-MossFormer2, and IIANet in the completed runs. All three front ends remain frozen, and only the 70.9K adapter is trained. The results are:
All four metrics improve for each front end. Relative to paired frozen extractors, D-CRPF gains 1.834, 2.266, and 2.445 dB SI-SDR and reduces WER by 7.01, 3.90, and 5.87 points. Each adapter uses the full training loader and validation selection; each test covers 1,243 full-duration targets. Released audio/visual weights remain unchanged.
These results extend the evidence from the intermediate CRPF-R adapter to the final D-CRPF design and show consistent improvements across three released front-end architectures.
W1 / Q2: Stability of the 0.225-dB matched AV-CrossNet advantage
The matched comparison keeps the frozen AV-CrossNet, seven-map input, data, loss, and 30-epoch budget fixed; CRPF-R omits the signed head. D-CRPF's observed advantage is 0.225 dB. The additional results above strengthen the evidence for the complete design across front ends: all four metrics improve with Dolphin, AV-MossFormer2, and IIANet.
Q3: Sensitivity to the fixed reference-correction scale of 0.25
We use α = 0.25 and β = 1 throughout the reported configurations. Alpha controls the scale of the reference-based complex correction, while beta separately controls the signed beam-disagreement term. This gives the input-dependent bound
|Δ| ≤ √2 |α| |Y_ref| + |β| |G_target - S_stage1|.Thus, alpha sets an explicit correction budget around the frozen output; it does not scale the two branches together. The reported experiments evaluate this fixed setting.
W2: Real-recording generalization and the circle-to-line gain reduction
For real recordings, we evaluated 500 full-duration mixtures of real eight-channel AMI. Mixing different speakers across sessions at a configured 0-dB component ratio creates sustained interference while preserving recorded acoustics. Neither front ends nor adapters were trained, tuned, or selected on AMI. Without a clean acoustically matched waveform, we report corpus WER against official manual transcripts:
According to the above Table, we can find that D-CRPF reduces WER by 17.62 and 6.94 percentage points over the frozen front ends, and by 8.23 and 11.16 points over their guided MVDR outputs. For AV-CrossNet, guided MVDR worsens WER from 50.12% to 54.34%, but D-CRPF reduces it to 43.18%. These consistent aggregate gains support the practical value of residual adaptation when the frozen estimates and spatial guidance are imperfect. The adapters transfer unchanged from six-channel training to eight-channel input; this is a controlled cross-session real-recording test, not a natural-overlap AMI benchmark.
For circle-to-line transfer, the reference waveforms are sample-identical and frozen Stage 1 remains at 13.171 dB. AV-CrossNet + D-CRPF changes from 14.418 to 13.612 dB, a smaller drop than AV-SpatialNet or oracle-assigned SpatialNet. At the same time, the adapter's gain decreases from 1.246 to 0.441 dB. The result therefore shows a positive, attenuated improvement under the tested shift: retaining the frozen-output anchor helps preserve complete-system performance, while the learned correction still depends on the available spatial evidence.