acceptodds
Under review as a conference paper at ICLR 2027

Spatial Residual Adaptation of Frozen Audio-Visual Speaker Extractors

Abstract

Single-channel audio-visual target-speaker extractors leave the spatial information of microphone arrays unused. We propose D-CRPF, a 70.9K-parameter dual-direction complex residual post-filter that adds spatial evidence to a frozen extractor without accessing its hidden features or training graph. Two analytic beamformers condition complex reference correction and signed beam-disagreement correction around the frozen output. We characterize identity initialization, input-dependent correction bounds, and reference-preserving channel-permutation invariance. On simulated six-microphone mixtures covering all 1,243 full-duration official LRS2 test utterances, D-CRPF improves matched AV-CrossNet by 1.254 dB SI-SDR versus 0.045 dB from an equal-parameter single-channel control, reaching 14.433 dB and 8.14% WER. Under a fixed-reference circle-to-line change, the complete system loses 0.806 dB versus over 4 dB for direct multichannel references; its gain over Stage 1 remains positive but contracts from 1.246 to 0.441 dB. The residual-only intermediate variant, CRPF-R, isolates the first correction direction and provides complementary transfer evidence: it improves three released front ends by 0.904–1.502 dB SI-SDR and matched IIANet by 1.561 dB. Missing-video tests expose failure boundaries.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-visual target speaker extraction, Frozen-model adaptation, Spatial residual learning, Array-geometry robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_neiwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a practical problem in deployed audio-visual target-speaker extractors, where the audio input is usually a single-microphone channel, leaving spatial information from the multi-microphone array unused. The paper proposes D-CRPF, a light-weight adapter with only 70.9k learnable parameters that adapts the output of a frozen AV target-speaker extractor without touching the original weights/hidden states. The adaptation follows a three-stage pipeline. First, an initial estimate is generated using the frozen extractor. Then, the estimate is used to guide two beamformers to generate target- and interference-oriented view from the multi-microphone array input. Finally, a trainable network will use the two views and the single-channel reference signal to compute a complex correction on STFT space (CRPF-R) and a signed correction based on the disagreement between the target beam and the estimate.

Experiments are performed on six-microphone reverberant mixtures from LRS2 dataset. The proposed CRPF-R is evaluated with multiple front-ends and achieves improvement in SI-SDR and reduction in WERs. With the final D-CRPF, further improvements are observed on the AV-CrossNet.

Critical strengths
  • Consistent improvement across multiple frozen extractors. Table 1 shows that CRPF-R improves Dolphin, AV-MossFormer2, and IIANet without updating their weights. This supports the practical premise that output-level spatial adaptation can work across different front ends, rather than only with one matched model.

  • Useful controlled experiment for spatial information. On matched AV-CrossNet, D-CRPF gains 1.254 dB SI-SDR while an equal-parameter single-channel control gains only 0.045 dB. This strongly supports that the additional microphone signals account for much of the improvement.

Critical weaknesses
  • The full D-CRPF's validation is insufficient. Only one training run reports the gain from CRPF-R to full D-CRPF. Further validation could enhance the usefulness of the signed correction.

  • The generality of the system remains uncertain. The experiments use simulated mixtures rather than real array recordings. The controlled circle-to-line test is helpful, but the adaptation gain falls from 1.246 to 0.441 dB.

Questions for authors
  1. Does D-CRPF improve on CRPF-R for any of the three released front ends in Table 1?

  2. How stable is the 0.225 dB D-CRPF advantage over CRPF-R?

  3. How sensitive is the result to the fixed 0.25 reference-correction scale?

2 comments
AuthorsOpenReview ↗

Cross-Front-End Gains, Stability, and Real-Recording Generalization

Thank you for recognizing the frozen-extractor setting and the spatial-information control. We address your questions with full D-CRPF results on three released front ends, the role of the correction scale, and the real-recording and array-transfer evaluations.

W1 / Q1: Does full D-CRPF improve on CRPF-R for the released front ends?

Yes. Full seven-map D-CRPF improves on CRPF-R across released Dolphin, AV-MossFormer2, and IIANet in the completed runs. All three front ends remain frozen, and only the 70.9K adapter is trained. The results are:

SystemSI-SDR (dB) ↑PESQ ↑STOI ↑WER (%) ↓
Dolphin + CRPF-R12.1232.6330.90413.19
Dolphin + D-CRPF13.0532.8980.91810.53
AV-MossFormer2 + CRPF-R14.2382.9300.9378.35
AV-MossFormer2 + D-CRPF15.1343.1300.9457.84
IIANet + CRPF-R13.2742.7830.9269.27
IIANet + D-CRPF14.2163.0150.9368.66

All four metrics improve for each front end. Relative to paired frozen extractors, D-CRPF gains 1.834, 2.266, and 2.445 dB SI-SDR and reduces WER by 7.01, 3.90, and 5.87 points. Each adapter uses the full training loader and validation selection; each test covers 1,243 full-duration targets. Released audio/visual weights remain unchanged.

These results extend the evidence from the intermediate CRPF-R adapter to the final D-CRPF design and show consistent improvements across three released front-end architectures.

W1 / Q2: Stability of the 0.225-dB matched AV-CrossNet advantage

The matched comparison keeps the frozen AV-CrossNet, seven-map input, data, loss, and 30-epoch budget fixed; CRPF-R omits the signed head. D-CRPF's observed advantage is 0.225 dB. The additional results above strengthen the evidence for the complete design across front ends: all four metrics improve with Dolphin, AV-MossFormer2, and IIANet.

Q3: Sensitivity to the fixed reference-correction scale of 0.25

We use α = 0.25 and β = 1 throughout the reported configurations. Alpha controls the scale of the reference-based complex correction, while beta separately controls the signed beam-disagreement term. This gives the input-dependent bound

|Δ| ≤ √2 |α| |Y_ref| + |β| |G_target - S_stage1|.

Thus, alpha sets an explicit correction budget around the frozen output; it does not scale the two branches together. The reported experiments evaluate this fixed setting.

W2: Real-recording generalization and the circle-to-line gain reduction

For real recordings, we evaluated 500 full-duration mixtures of real eight-channel AMI. Mixing different speakers across sessions at a configured 0-dB component ratio creates sustained interference while preserving recorded acoustics. Neither front ends nor adapters were trained, tuned, or selected on AMI. Without a clean acoustically matched waveform, we report corpus WER against official manual transcripts:

Front endStage 1 WER (%) ↓Guided MVDR ↓D-CRPF ↓
AV-MossFormer246.6537.2629.03
AV-CrossNet50.1254.3443.18

According to the above Table, we can find that D-CRPF reduces WER by 17.62 and 6.94 percentage points over the frozen front ends, and by 8.23 and 11.16 points over their guided MVDR outputs. For AV-CrossNet, guided MVDR worsens WER from 50.12% to 54.34%, but D-CRPF reduces it to 43.18%. These consistent aggregate gains support the practical value of residual adaptation when the frozen estimates and spatial guidance are imperfect. The adapters transfer unchanged from six-channel training to eight-channel input; this is a controlled cross-session real-recording test, not a natural-overlap AMI benchmark.

For circle-to-line transfer, the reference waveforms are sample-identical and frozen Stage 1 remains at 13.171 dB. AV-CrossNet + D-CRPF changes from 14.418 to 13.612 dB, a smaller drop than AV-SpatialNet or oracle-assigned SpatialNet. At the same time, the adapter's gain decreases from 1.246 to 0.441 dB. The result therefore shows a positive, attenuated improvement under the tested shift: retaining the frozen-output anchor helps preserve complete-system performance, while the learned correction still depends on the available spatial evidence.

I thank the authors for the detailed rebuttal and the additional experiments. Most of my concerns have been addressed. The consistent improvements across different front ends provide stronger evidence for the effectiveness of D-CRPF and largely alleviate my concern about training stability. The additional experiments on AMI recordings also strengthen the practical value of the proposed approach.

My remaining concern is the sensitivity to the reference-correction scale α. While the authors clarify its mathematical role, it remains unclear how this value was chosen and whether a larger α for CRPF-R could narrow the performance gap between CRPF-R and D-CRPF.

Nevertheless, I consider this a minor remaining concern. Overall, I am satisfied with the rebuttal and will maintain my positive rating.

Reviewer_dNzGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how microphone-array observations can improve a frozen single-channel audio-visual target-speaker extractor without accessing its internal representations or training graph. The proposed D-CRPF constructs two complementary beamformed signals using covariance estimates guided by the frozen extractor. A 70.9K-parameter adapter then predicts complex reference-residual and signed beam-disagreement corrections around the frozen output. Experiments on simulated reverberant LRS2 mixtures evaluate extraction performance, capacity controls, array changes, and failure conditions. The residual-only variant, CRPF-R, provides additional evaluations across several front ends.

Critical strengths
  1. Improving an existing extractor using additional microphones while preserving its deployed mapping is a useful problem. The output-only interface avoids requiring access to backbone features or gradients and accommodates different front-end architectures.
  2. On matched AV-CrossNet, D-CRPF improves SI-SDR reduces WER. The comparison supports the conclusion that the improvement substantially depends on spatial evidence rather than additional model capacity.
Critical weaknesses
  1. The methodological novelty is limited relative to established beamforming-guided refinement. The paper combines analytic beamforming, neural post-filtering, and residual adaptation. Its distinctive contribution is the compact, output-only interface and dual-direction correction around a frozen AV estimate. However, the experiments more clearly establish the usefulness of additional spatial information than the necessity of this particular correction parameterization. A capacity-matched spatial post-filter using the same inputs but predicting an unrestricted output or additive correction would help establish the value of the proposed design.
  2. All results are on simulated 6-mic mixtures from LRS2 with simulated RIRs. No real multichannel recordings, no real array. The paper is candid about this, but for an ICLR audience the lack of any real-world validation limits impact. The "full-duration" emphasis is good, but it's still simulated. Moreover, the main tables are one training run. The 0.225 dB D-CRPF vs. CRPF-R gap is explicitly flagged as not seed-stable. This is honest but also means the headline claim (dual-direction > residual-only) is not firmly established.
  3. The geometry results support robustness of the complete system more strongly than robustness of the spatial correction. Under the fixed-reference circle-to-line change, the adaptation gain decreases from 1.246 to 0.441 dB. Retaining a strong, unchanged single-channel anchor is a useful property, but this result should not be interpreted as showing that the correction itself generalizes strongly across geometries. The authors acknowledge this distinction.
Questions for authors
  1. How does D-CRPF compare with a capacity-matched spatial post-filter that receives the same evidence but uses an unrestricted output or additive-residual parameterization? Are there existing results that isolate the benefit of the proposed two correction direction
  2. how consistent is the signed-branch improvement across training seeds? Alternatively, can a limited evaluation of complete D-CRPF on another front end establish whether this component’s benefit extends beyond AV-CrossNet?
5 comments
AuthorsOpenReview ↗

Residual Design, Cross-Front-End Transfer, and Array Robustness

Thank you for highlighting the distinction between spatial-information gains and decoder-design gains, as well as the interpretation of array robustness. We address these points through the same-input post-filter comparison, full D-CRPF results on three released front ends, and real-recording and array-transfer evaluations.

W1 / Q1: Same-evidence spatial post-filter and the parameterization

Our contribution is output-only spatial adaptation of a frozen AV extractor. To separate the benefit of this residual design from access to spatial information, we evaluated a capacity-matched spatial post-filter using exactly the same seven-map input as D-CRPF. Both systems use the same frozen AV-CrossNet and data, with 69.7K parameters for direct mapping versus 70.9K for D-CRPF.

SystemSI-SDR (dB) ↑PESQ ↑STOI ↑WER (%) ↓
Frozen Stage 113.1792.6610.92410.85
Beamforming direct mapping13.7882.8300.9359.06
D-CRPF14.4333.0910.9448.20

The direct mapper improves Stage 1 by 0.609 dB and reduces WER from 10.85% to 9.06% (16.50% relative). D-CRPF provides further gains of 0.645 dB SI-SDR, 0.261 PESQ, and 0.009 STOI and reduces WER to 8.20% (9.49% relative to direct mapping), with 1.2K additional parameters. Because the spatial inputs are identical and the capacities are closely matched, these results support the proposed anchored dual-direction residual formulation beyond simply providing spatial evidence.

The matched ablations provide complementary controls. The equal-parameter single-channel model improves SI-SDR by only 0.045 dB, versus 1.254 dB for D-CRPF, supporting the contribution of multichannel observations beyond added capacity. CRPF-R retains the same seven-map spatial input but omits the 0.481K signed head; adding this branch yields a further 0.225 dB in the matched AV-CrossNet comparison.

W2 / Q2: Cross-front-end evidence and training-seed consistency

Full seven-map D-CRPF improves on CRPF-R across released Dolphin, AV-MossFormer2, and IIANet in the completed runs. All three front ends remain frozen. The results are:

SystemSI-SDR (dB) ↑PESQ ↑STOI ↑WER (%) ↓
Dolphin + CRPF-R12.1232.6330.90413.19
Dolphin + D-CRPF13.0532.8980.91810.53
AV-MossFormer2 + CRPF-R14.2382.9300.9378.35
AV-MossFormer2 + D-CRPF15.1343.1300.9457.84
IIANet + CRPF-R13.2742.7830.9269.27
IIANet + D-CRPF14.2163.0150.9368.66

All four metrics improve for each front end. Relative to paired frozen extractors, D-CRPF gains 1.834, 2.266, and 2.445 dB SI-SDR and reduces WER by 7.01, 3.90, and 5.87 points. Each adapter uses the full training loader and validation selection; each test covers 1,243 full-duration targets. Released audio/visual weights remain unchanged.

The consistent direction of improvement across all three architectures strengthens the evidence that the complete design is useful beyond AV-CrossNet. The matched 0.225-dB comparison uses the same frozen front end, data, loss, and 30-epoch budget.

W2: Evaluation on real multichannel recordings

We evaluated 500 full-duration mixtures of real eight-channel AMI recordings. Without a clean acoustically matched waveform, we report corpus WER against official manual transcripts. MVDR is the target beam guided by the corresponding frozen front end:

Front endStage 1 WER (%) ↓Guided MVDR ↓D-CRPF ↓
AV-MossFormer246.6537.2629.03
AV-CrossNet50.1254.3443.18

We can find that D-CRPF reduces WER by 17.62 and 6.94 percentage points over the two frozen front ends, and by 8.23 and 11.16 points over their guided MVDR outputs. In particular, guided MVDR worsens AV-CrossNet's WER from 50.12% to 54.34%, whereas D-CRPF reduces it to 43.18%, improving upon both. These results support the practical value of retaining the frozen estimate as an output anchor and learning spatial residual corrections when the spatial guidance is imperfect.

W3: Complete-system retention versus correction robustness

In the controlled 8-cm circular-to-linear comparison, the reference waveform is unchanged and Stage 1 stays at 13.171 dB. AV-CrossNet + D-CRPF drops by 0.806 dB (14.418 → 13.612), versus 4.063 dB for AV-SpatialNet and 4.824 dB for oracle-assigned SpatialNet, without geometry-specific retraining. The adapter’s gain decreases from 1.246 to 0.441 dB but remains positive. This supports stronger system-level retention under the tested shift, not geometry-invariant correction. Physical array transfer is distinct from reference-preserving channel-permutation invariance.

Reviewer_dNzGOpenReview ↗

I maintain my original assessment.

The rebuttal adds more relevant experiments, but my main concern regarding limited methodological novelty remains unresolved. The direct-mapping baseline is insufficiently specified, so the comparison does not convincingly establish an advantage of the proposed correction parameterization over a well-designed spatial post-filter. A clearly specified complex-mask post-filter and an unrestricted complex additive-residual baseline, using the same frozen extractor and spatial evidence under comparable training and capacity budgets, would better isolate the benefit of the proposed dual-direction parameterization. On the other hand, although AMI contains real microphone-array recordings, the construction and overlap conditions of the 500 evaluation mixtures remain unclear. The controlled evaluation is limited to two-speaker mixtures, leaving the reliability of the extractor-derived interference covariance under multiple simultaneous interferers unestablished. Moreover, the adaptation gain falls from 1.246 to 0.441 dB under the tested array change, limiting evidence for generalization of the spatial correction itself.

AuthorsOpenReview ↗

Explicit Post-filter Controls and the Scope of Spatial Generalization (1/2)

Thank you for specifying the additional comparisons and protocol details needed. We address the four remaining issues below: output parameterization, the 500 AMI mixtures, multiple simultaneous interferers, and interpretation of the array-transfer gain.

1. Explicit complex-mask and unrestricted additive controls

Let S_0 denote the frozen AV-CrossNet output and c the reference spectral-RMS scale. Every predictor in Table 4(a) receives the same seven maps: normalized real/imaginary target-beam, interference-beam and reference spectra, plus the target proxy. S_0 is used in reconstruction, not added as two extra predictor inputs. With F=F(Z) from unrestricted linear real/imaginary heads, the controls are cF for direct mapping, (1+F)S_0 for complex masking, and S_0+cF for additive correction. The mask multiplies the frozen estimate. The additive output has no tanh bound, reference-spectrum multiplication, fixed correction-scale factor, or beam gate.

Mask and additive controls share D-CRPF's compact predictor trunk and have 70.4K parameters; zero-initialized output heads start both at S_0. D-CRPF has 70.9K parameters and instead reconstructs

S_hat = S_0 + Δ_R + Δ_B, where Δ_R = α Y_r (C_R + j C_I), Δ_B = β Q (G_s - S_0), and C_R, C_I, Q ∈ [-1, 1].

with α=0.25 and β=1. Its distinction is the separately controlled reference-based and signed beam-disagreement directions, rather than greater expressive capacity than unrestricted regression. The current full-duration Table 4(a) results are:

Post-filterOutput spectrumParams (K)↓SI-SDR (dB)↑PESQ↑STOI↑WER (%)↓
Frozen Stage 1S_0013.1792.661.923610.85
Beamforming direct mappingcF69.713.7882.830.93509.06
Complex-mask post-filter(1+F)S_070.414.0932.913.93758.86
Unrestricted additive residualS_0+cF70.414.4752.930.93708.82
D-CRPFS_0+Δ_R+Δ_B70.914.4463.092.94438.14

Compared with complex masking, D-CRPF improves SI-SDR by 0.353 dB, PESQ by 0.179 and STOI by 0.0068, while reducing WER by 0.72 percentage points. Against unrestricted additive correction, SI-SDR is 0.029 dB lower, but PESQ increases from 2.930 to 3.092, STOI from .9370 to .9443, and WER decreases from 8.82% to 8.14%. Thus, the evidence is a quality/recognition advantage of the tested correction design, rather than dominance in every metric.

2. Construction and overlap of the 500 AMI mixtures

The evaluation consists of controlled cross-session mixtures from real eight-channel recordings of ES2011a, ES2012a and ES2013a. Eligible original clips are reused for additional pairings/crops: 500 counts mixture instances, not independent target recordings or naturally occurring conversations.

Official segment/word annotations select complete target utterances longer than 5 s, with at least 70% annotated speech activity and no other participant's annotated speech within the video-aligned interval or a 200-ms guard. The interferer comes from a different meeting and person and is cropped from a sufficiently long interval without looping. The pairing rules require overlap over at least 80% of target-active frames and 60% of the full window. These are selection thresholds, not measured mean overlap rates for the expanded 500-mixture set.

All eight Array1 channels are retained, with one source gain shared across channels. The 0-dB mixing ratio equalizes the power of the two channel-average waveforms, followed by a common anti-clipping gain; it is not a clean-speech SNR because recorded noise and reverberation remain. Target video and official manual transcripts are retained. No AMI-specific training, tuning or checkpoint selection is performed. Corpus WER uses the same complete annotation-selected intervals for every system, without output-dependent VAD or fixed-duration truncation. This protocol tests sustained overlap with recorded acoustics, rather than the official AMI natural-overlap benchmark.

Reviewer_uH36OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces D-CRPF, a lightweight spatial residual adapter that incorporates multichannel information into a frozen audio-visual target-speaker extractor. The method uses two MVDR beamforming outputs to guide complex reference-spectrum correction and signed beam-disagreement correction, without modifying the pretrained backbone. On simulated six-microphone LRS2 mixtures, D-CRPF improves AV-CrossNet by 1.254 dB SI-SDR, while a parameter-matched single-channel adapter provides only a 0.045 dB gain. The paper also evaluates CRPF-R, a residual-only variant, on several pretrained extractors.

Critical strengths
  • The idea of adding spatial information to an existing AV extractor without changing the backbone is useful, particularly when retraining or accessing internal features is not feasible. The comparison with the parameter-matched single-channel control is a strong part of the evaluation. The large difference in improvement suggests that the gains come primarily from spatial information rather than additional model capacity.
  • The experimental evaluation is fairly thorough. Results are reported on all 1,243 full-duration LRS2 test utterances, and the paper includes ablations and controlled array comparisons. The improvements obtained with CRPF-R across different pretrained extractors are also encouraging, although these results do not directly validate the full D-CRPF architecture.
Critical weaknesses
  • The methodological novelty is somewhat limited. Combining neural speech estimates with MVDR beamforming and a learned refinement stage is not new, as seen in Wang et al. (2021) and Chen et al. (2022). The main distinction here is the lightweight residual adapter operating on a frozen AV extractor. However, it is not clear how much of the improvement comes from the proposed residual formulation itself, rather than from using a conventional beam-guided post-filter. A comparison with a standard neural post-filter using the same frozen AV-CrossNet backbone and a comparable parameter budget would help clarify this.
  • The benefit of the signed beam-disagreement branch needs stronger evidence. Table 2 shows a 0.225 dB SI-SDR improvement of D-CRPF over CRPF-R, but the comparison is based on a single run. Given the relatively small difference, it is difficult to tell how consistent this improvement would be across random seeds. Also, the full D-CRPF model is tested only with AV-CrossNet, whereas the cross-extractor experiments use CRPF-R. Testing the full model on another backbone would help establish whether the second correction direction is generally useful.
  • Generalization to realistic recording conditions remains uncertain. All reported experiments rely on simulated multichannel LRS2 mixtures. The array experiments are useful, but they also show sensitivity to microphone geometry. In particular, the adaptation gain drops from 1.246 dB with the circular array to 0.441 dB with the linear array. The gain remains positive, but this reduction suggests that the learned correction depends considerably on the array configuration. Results on real recordings or more varied acoustic conditions would make the practical applicability clearer.
  • The computational cost and failure behavior are not fully addressed. Although the adapter has only 70.9K parameters, the complete pipeline also requires covariance estimation, two MVDR beamformers, and recurrent processing. Parameter count alone does not indicate the actual inference overhead, and latency and memory measurements are missing. The paper also reports substantial degradation under certain missing-video and reduced-microphone conditions. Since these failure analyses mainly use the earlier AV-DPCRN setup, it remains unclear how the final D-CRPF system behaves in these cases.
Questions for authors
  • How would D-CRPF compare against a standard MVDR-guided neural post-filter trained with the same frozen AV-CrossNet checkpoint, data, and parameter budget? This seems important for separating the contribution of the residual formulation from that of beam-guided refinement more generally.
  • Is the 0.225 dB improvement from the signed correction branch consistent across multiple random seeds? It would also be useful to see whether full D-CRPF provides similar benefits on another pretrained extractor.
  • Can the authors provide end-to-end inference latency and memory measurements, including beamforming? Results on real multichannel recordings would also help assess the practical value of the approach.
6 comments
AuthorsOpenReview ↗

Additional Baselines, Real-Recording Evaluation, and Efficiency (1/2)

Thank you for distinguishing spatial-information gains from decoder-design gains. We address each concern with the completed evidence below.

W1 / Q1: Novelty and comparison with a conventional MVDR-guided post-filter

Our contribution is output-only spatial adaptation of a frozen AV extractor. No backbone updates, hidden features, or gradients are required. Two analytic spatial views feed a 70.9K predictor, whose bounded coefficients correct an explicit frozen-output anchor through a reference-based complex residual and a signed beam-disagreement branch.

We evaluated a capacity-matched spatial post-filter using exactly the same seven-map input as D-CRPF. This beamforming direct mapping baseline predicts the complete target spectrum without the frozen-output residual skip or signed correction gate. It uses the same frozen AV-CrossNet and data, with 69.7K parameters versus 70.9K for D-CRPF. On all 1,243 full-duration targets, the results are:

SystemSI-SDR (dB) ↑PESQ ↑STOI ↑WER (%) ↓
Frozen Stage 113.1792.6610.92410.85
Beamforming direct mapping13.7882.8300.9359.06
D-CRPF14.4333.0910.9448.20

The direct mapper improves Stage 1 by 0.609 dB and reduces WER from 10.85% to 9.06%, a 1.79-point reduction (16.50% relative). D-CRPF provides further gains of 0.645 dB SI-SDR, 0.261 PESQ and reduces WER from 9.06% to 8.20% (9.49% relative), with 1.2K additional parameters. Because the spatial inputs are identical, these results support a benefit of the proposed anchored dual-direction residual formulation over the tested direct predictor, beyond simply providing spatial evidence.

W2 / Q2: Full D-CRPF on other front ends and the signed-branch gain

Full seven-map D-CRPF improves on CRPF-R across released Dolphin, AV-MossFormer2, and IIANet in the completed runs. All three front ends remain frozen. The results are:

SystemSI-SDR (dB) ↑PESQ ↑STOI ↑WER (%) ↓
Dolphin + CRPF-R12.1232.6330.90413.19
Dolphin + D-CRPF13.0532.8980.91810.53
AV-MossFormer2 + CRPF-R14.2382.9300.9378.35
AV-MossFormer2 + D-CRPF15.1343.1300.9457.84
IIANet + CRPF-R13.2742.7830.9269.27
IIANet + D-CRPF14.2163.0150.9368.66

All four metrics improve for each front end. Relative to paired frozen extractors, D-CRPF gains 1.834, 2.266, and 2.445 dB SI-SDR and reduces WER by 7.01, 3.90, and 5.87 points. Each adapter uses the full training loader and validation selection; each test covers 1,243 full-duration targets. Released audio/visual weights remain unchanged.

W3 / Q3: Real recordings

We tested controlled mixtures of real eight-channel AMI recordings to demonstrate that D-CRPF can improve frozen front ends on real acoustics. The WER results reported in manuscript Table 7 cover 500 evaluated full-duration mixtures, at a configured 0-dB component ratio. MVDR is the target beam guided by the corresponding frozen front end:

Front endStage 1 WER (%) ↓Guided MVDR ↓D-CRPF ↓
AV-MossFormer246.6537.2629.03
AV-CrossNet50.1254.3443.18

D-CRPF improves both the frozen output and target beam. These are cross-session mixtures with recorded acoustics and sustained interference, not a natural-overlap AMI benchmark. Without a clean acoustically matched waveform, we evaluate WER against official manual transcripts, not intrusive waveform metrics.

According to the above Table, we can find that D-CRPF reduces WER by 17.62 and 6.94 percentage points over the two frozen front ends, and by 8.23 and 11.16 points over their guided MVDR outputs. Notably, with AV-CrossNet, guided MVDR increases WER from 50.12% to 54.34%, whereas D-CRPF reduces it to 43.18%. The adapter thus improves recognition even when beamforming alone is detrimental. This is consistent with retaining the frozen estimate as an explicit output anchor and introducing spatial observations through learned residual corrections, supporting improved robustness to unreliable spatial guidance under the tested real-recording conditions.

AuthorsOpenReview ↗

Additional Baselines, Real-Recording Evaluation, and Efficiency (2/2)

W4 / Q3: Complete-pipeline inference time and memory

We have now measured each of the three released front ends alone, with CRPF-R, and with D-CRPF. All nine systems use the same RTX 4090, batch one, eight CPU threads, and 100 complete utterances totaling 173.056 s, with three timed repetitions after warmup. Each system is loaded in a fresh subprocess; input hashes match across all nine runs, and no other process occupies the measured GPU. Timing includes visual/audio extraction and, for adapted systems, STFT/iSTFT, covariance estimation, both MVDR beams, the adapter, and synthesis.

The table reports total parameters, RTF, mean time per utterance, and peak allocated memory:

SystemParams (M) ↓RTF ↓Time (ms) ↓Peak (MiB) ↓
Dolphin11.10290.05967103.263241.95
Dolphin + CRPF-R11.17330.06176106.883241.60
Dolphin + D-CRPF11.17380.06225107.723241.68
AV-MossFormer268.51640.06375110.33659.03
AV-MossFormer2 + CRPF-R68.58680.06719116.27698.63
AV-MossFormer2 + D-CRPF68.58730.06713116.18698.63
IIANet14.22340.15626270.42243.88
IIANet + CRPF-R14.29380.15876274.74488.50
IIANet + D-CRPF14.29430.15868274.61488.50

D-CRPF adds 70.9K parameters to each front end, but we measure the complete-system cost rather than equating parameter count with compute. Its time increases over the corresponding original front ends are 4.33%, 5.30%, and 1.55%, respectively. Memory changes depend on the front end: IIANet increases by 244.62 MiB, whereas Dolphin's peaks differ by less than 1 MiB, not a meaningful memory saving. Small CRPF-R/D-CRPF timing differences are not interpreted as a speed advantage. These results further demonstrate that our proposed method achieves substantial performance gains with only marginal computational overhead compared with existing single-channel audio-visual models.

W4: Missing video and reduced microphone count with the final front end

The real-recording evaluation on W3 provides complementary evidence under realistic audiovisual conditions. The real AMI data also exhibits real-world challenges including speaker head turns and frame drops caused by body movement. D-CRPF reduces corpus WER from 46.65% to 29.03% with AV-MossFormer2 and from 50.12% to 43.18% with AV-CrossNet, without updating either front end or performing AMI-specific adaptation. Moreover, although guided MVDR degrades AV-CrossNet's WER to 54.34%, D-CRPF improves upon both the frozen extractor and its guided beam. These consistent aggregate gains across two front-end architectures support the practical value of spatial residual adaptation when the frozen estimates and spatial guidance are imperfect.

I thank the authors for the detailed rebuttal and additional experiments. Some of my concerns have been addressed especially with the additional baselines and cross-backbone experiments. But I still have concerns about the novelty of the work and the benefits of the proposed residual correction method compared to the existing approaches - so I will keep my score unchanged.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview of our response

We thank the reviewers for recognizing the frozen-extractor setting and the evidence that spatial information contributes beyond increased model capacity.

Our central idea is dual-direction spatial residual adaptation: multichannel observations provide spatial evidence, while the frozen audio-visual estimate remains an explicit output anchor. A 70.9K-parameter adapter learns a bounded complex correction relative to the reference microphone and a signed correction along the disagreement between the target beam and the frozen estimate. Unlike predicting the complete target spectrum through direct mapping, this design starts from the existing extractor's output without accessing its hidden features or training graph.

We have completed three complementary evaluations. First, full D-CRPF improves released, frozen Dolphin, AV-MossFormer2, and IIANet, with a separately trained adapter for each front end. Second, matched-data controls examine spatial information and the signed branch, alongside a beamforming direct mapping baseline. Third, recognition improves on controlled mixtures of real eight-channel AMI recordings without AMI-specific training or tuning. Paired efficiency measurements now cover each of the three released front ends alone and with either adapter, including both beamformers in the adapted pipeline.

AuthorsOpenReview ↗

Summary of Rebuttal and Clarification of Contributions

Dear Area Chair and reviewers,

We sincerely thank the Area Chair and all reviewers for their constructive feedback and careful consideration of our work.

We would like to summarize the key progress made during the rebuttal, particularly regarding the methodological novelty and the advantages of D-CRPF over generic spatial post-filtering approaches.

  1. Clarification of methodological contribution. Our central contribution is not simply incorporating spatial information into AV-TSE, but enabling lightweight multichannel adaptation of a frozen pretrained monaural AV-TSE model. The proposed dual-direction correction explicitly structures and bounds spatially guided target recovery and interference suppression, aiming to avoid unnecessary modifications when the original estimate is reliable.

  2. New controlled comparisons. To directly address the concerns about methodological novelty, we implemented both a complex-mask post-filter and an unrestricted complex additive-residual baseline, using identical spatial evidence, frozen backbones, training conditions, and closely matched parameter budgets. D-CRPF outperforms the complex-mask baseline. Compared with the unrestricted residual, it achieves comparable SI-SDR while improving PESQ from 2.930 to 3.092, STOI from 0.9370 to 0.9443, and reducing WER from 8.82% to 8.14%. These results provide direct empirical evidence that the structured correction offers benefits beyond simply introducing additional trainable spatial post-filtering capacity.

  3. Rebuttal outcome. We appreciate that one reviewer, who initially raised concerns about methodological novelty, reconsidered the new evidence and increased their rating from 2 to 3. We also sincerely thank the reviewer who consistently recognized the experimental contributions of our work. We respect the remaining concerns and believe that our additional controlled experiments provide relevant evidence for their assessment.

We respectfully ask the Area Chair to consider these new findings and the clarified methodological contributions when making the final recommendation.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.