acceptodds
Under review as a conference paper at ICLR 2027

SEPADAPTER: A SEPARATION ADAPTER FOR STREAMING AND OFFLINE MULTI-TALKER ASR BACKBONES

Abstract

Multi-talker speech recognition must transcribe a target speaker’s speech while limiting interference from overlapping voices. We propose SEPADAPTER, an activity-conditioned residual module placed after convolutional subsampling and before the automatic speech recognition (ASR) encoder. It uses a three-channel, per-frame target-relative activity signal indicating whether the target, a competing speaker, and remaining competitors are active, together with temporal context, to correct acoustic features through a residual connection. We implement the core design in FastConformer, Zipformer, and Whisper, with separately trained weights and backbone-specific context handling for streaming and offline recognition. In a parameter-matched FastConformer comparison, the 3.78M-parameter dual-path module reduces concatenated minimum-permutation word error rate (cpWER) by 0.356 percentage points (14.52% to 14.17%, 2.4% relative) over framewise conditioning. This reduction appears in all twelve seed-by-lookahead cells across three paired training runs. Diagnostics on one paired training run show gains concentrated in overlapped words and increased errors when adapter history is blocked. Insertion after encoder block 8 raises cpWER by about 7 points under the shared recipe, even though the advantage over framewise conditioning grows. The separately optimized FastConformer system achieves lower cpWER than published streaming self-speaker adaptation (SSA) results in all twelve LibriSpeechMix conditions, by 0.22–5.47 points. The mean gap grows from 0.66 points at 1120 ms ASR lookahead to 4.16 points at 80 ms, although training and diarization recipes differ. Our Whisper system, which also uses self-enrollment and encoder finetuning, improves over locally re-evaluated SE-DiCoW on three LibriMix conditions under oracle activity. With DiariZen activity, it reduces LibriSpeechMix three-speaker time-constrained cpWER (tcpWER) from the published 21.1% to 19.84%; results vary across domains. The controlled results support temporal activity-conditioned correction as a complement to the encoder’s temporal modeling.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-talker ASR, target-speaker ASR, streaming ASR, speaker activity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FYsFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SepAdapter adds an activity-conditioned residual module before an ASR encoder to help recognize a target speaker in overlapping speech. It uses target and competitor activity together with temporal context. The authors implement it in FastConformer, Zipformer, and Whisper. Paired FastConformer runs show a modest but consistent gain over framewise conditioning, mainly on overlapping words. System comparisons are promising, but use different recipes and show mixed gains.

Critical strengths
  • The parameter-matched comparison across paired training runs gives credible evidence that temporal conditioning helps

  • The design is tested in several ASR backbones, with oracle and estimated activity reported separately.

Critical weaknesses
  1. LibriSpeechMix uses 30.4 s of diarizer buffering, excluded from the advertised ASR lookaheads. The buffering preset was also selected using evaluation-set activity references. These choices leave the practical latency and held-out system comparison unresolved, even though the shared preset preserves the conditional DP-FW comparison.

  2. The reliable result is a 0.356-point cpWER reduction over framewise conditioning. A parameter-matched TCN is slightly better in the single reported run, while the larger gains over SSA use different training and diarization recipes. Thus the evidence supports temporal correction, but does not establish a compelling advantage for the particular dual-path design.

Questions for authors

Do you have results with the diarizer preset selected only on development data and the complete buffering delay included? Does the advantage over SSA remain at a comparable total latency?

What evidence supports the dual-path design over the TCN alternative, and have the dropout and positional-parameter differences in the framewise control been checked as explanations for part of the gain?

2 comments

Thank you for the careful reading.

Diarizer buffering and preset choice. The 80–1120 ms values are ASR lookaheads; the original diarizer preset buffers 30.4 s. As stated in the appendix, this preset was chosen by DER on the evaluation-set references. Our focus was the ASR module, so the preset was fixed once, shared by all models and lookaheads, and never chosen by recognition error. To test robustness with a fixed, lower-buffering preset, we re-scored the submitted seed-43 DP/FW checkpoints, without retraining, using the published 1.04-s setting of Sortformer v2.1 (model-card values, not tuned). DP/FW obtain 14.429/14.819% at 1120 ms (−0.389 pp [−0.511, −0.269]) and 16.218/16.584% at 80 ms (−0.366 pp [−0.483, −0.250]).

Comparison with SSA at the same buffering. We retrained SSA in our pipeline under a revised post-submission recipe with shared backbone initialization, data, training schedule and activity corruption. SepAdapter's history window and competitor routing were aligned with streaming inference; the recipe was developed using diagnostics on the existing evaluation benchmark. Both models use the same activity from Nemotron-3-Diarization in its published 1.04-s setting (model-card values, not tuned) and the same ASR lookahead. SepAdapter obtains 12.93% versus 13.30% for SSA (−0.371 pp [−0.494, −0.246]), and the interval excludes zero at every lookahead. SepAdapter uses past cached states and the current chunk, without future audio beyond the ASR configuration, and costs 1.4–1.9 ms more computation per step (real-time factor 0.107 vs 0.090 at 80 ms, one stream). We did not directly measure the full end-to-end output latency.

Dual-path versus TCN. The temporal operator inside SepAdapter can be swapped. On FastConformer (seed 42), TCN is 0.102 pp better than DP (CI of TCN minus DP [−0.179, −0.024]). On Whisper (one seed, development sets, different context lengths), DP is 0.934 pp better than TCN at 25k steps (CI [0.208, 1.597]). We keep DP as the common implementation and treat TCN as a tested alternative for streaming. Our module-level results do not depend on DP being the best operator.

Dropout and positional parameters. We retrained a stricter seed-43 DP/FW pair without adapter dropout and without learned positions inside chunks (3,764,416 and 3,764,432 parameters). DP/FW obtain 14.059/14.473% (−0.415 pp [−0.501, −0.328]), with the interval excluding zero at every lookahead; the original seed-43 pair gives −0.368 pp. The gain remains when both factors are removed.

Thank you again for your careful review. We have uploaded a revised PDF and abstract addressing your concerns.

Appendix B reports the dropout/position control, Appendix C includes the fixed 1.04-s diarizer evaluation, and Section 6.1 with Appendix E presents the shared-protocol SSA comparison. We also clarify the distinction between ASR lookahead, diarizer buffering, and unmeasured end-to-end latency.

We appreciate your consideration.

Reviewer_6dgiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SepAdapter, a novel activity-conditioned target speaker speech recognition for a multi-talker ASR setting. The paper positions itself as a one-time intervention sitting between the acoustic frontend and the encoder, and is designed with a zero initialization (ie., the proposed alterations are initialized in a way that the pretrained representations are preserved at the start of training), in comparison to prior works like SE-DiCow that transform representations throughout the model.

Critical strengths

The paper is very well written and the controlled experiments are quite elaborate, addressing all the immediate questions/concerns that a reader might have.

Critical weaknesses

From the architecture point of view, the authors have done an excellent job of describing what their modifications are and the reasoning behind each of their design choices. I am a little concerned about the experimental section. First, all the numbers reported in the controlled settings are relative, and the numbers comparing SepAdapter against prior works are published numbers and not actual baselines. Given that some of these prior works have made their code publicly available, testing them with the exact data and in the same training environment should provide a fairer comparison between SepAdapter and prior works. Additionally, across all the experiments, no explicit significance tests are done. Second, the experimental section lacks discussion about how much compute overhead SepAdapter adds in comparison to previous works.

Questions for authors
  1. Why is Interferer 1 (s1) given special treatment in the activity vector (equation 1)? Is there any reasoning behind this decision? If not, then couldn't a_t be simply [activity probability of target speaker, 1 - activity probability of target speaker] or something to that effect?
  2. For your zero-initialization, you could have set the gate to produce zero at the start of training, instead of W_out and b_out. Can you shed some light on this design choice?
  3. Why is your inter-attention in DP done through this mixing-matching of frames and not pooling? From what I have seen, architectures typically employ pooling for inter-attention.
3 comments

Thank you for the constructive feedback.

Absolute results. Table 5 reports absolute DP/FW results (three-pair means: 14.17/14.52% cpWER); we will move these to the main table. New revised-recipe results on LibriSpeechMix 3-mix (cpWER %):

ASR lookahead1120 ms560 ms160 ms80 msMean
SepAdapter, Sortformer (30.4-s buffer)12.68713.16614.11914.49613.617
SSA, same activity13.08613.69514.45114.82514.014
SepAdapter, Nemotron (1.04-s buffer)11.97412.52913.42313.78612.928
SSA, same activity12.36112.95113.75314.12913.299

Means/differences use unrounded counts.

Comparison with prior work. We retrained NeMo SSA (target/background kernels) with shared backbone, data, schedule, activity corruption and evaluation. For SepAdapter, the revised recipe matches training history/routing to inference and initializes the gate at 1 with a zero output projection; both models use the same activity corruption. SepAdapter improves on SSA by 0.397 pp (95% CI [−0.515, −0.281]) with Sortformer and 0.371 pp [−0.494, −0.246] with Nemotron 1.04 s. All four lookahead-specific pointwise 95% CIs exclude zero. These exploratory post-submission results were informed by evaluation-set diagnostics, not a new held-out test. The final comparison uses one 100k-step run per model and does not establish superiority for the submitted DP recipe. Table 7 re-scores public SE-DiCoW using shared references, oracle activity and scoring: SepAdapter is better on three datasets, worse on two and unresolved on five; training differs.

Statistical uncertainty. Table 1 uses 20,000 paired mixture-bootstrap draws: DP−FW is −0.356 pp (Bonferroni-adjusted 98.3% CI [−0.444, −0.273] for H1–H3). The three trained pairs give −0.363, −0.368 and −0.338 pp. New pointwise 95% CIs reflect test-mixture uncertainty, not training-run variability.

Computational overhead. Table 8 (RTX 5090, FP32, batch 1) reports 4–10% extra ASR time versus FW and 210 KiB of adapter cache per target. With one target stream, SepAdapter/SSA RTF is 0.107/0.090 at 80 ms and 0.0087/0.0072 at 1120 ms: 1.4–1.9 ms more ASR compute per streaming step. Neither diarization nor end-to-end transcript emission latency is measured.

Why a separate first-competitor channel rather than [p,1−p][p,1-p]? [p,1−p][p,1-p] contains only target activity: it conflates silence with competitor-only speech and target-only speech with overlap. [target, first competitor, rest] distinguishes these states and preserves one competitor's trajectory. Earliest arrival is a causal routing rule; the paper recipe trained with the longest-active competitor, while the revised recipe also trains with earliest arrival. We retrained same-size variants (new seed-43 runs):

Activity inputPaper recipeRevised recipe
[target, first competitor, rest]14.1413.62
[target, max(competitors), 0]14.3714.23
[target, 0, 0]15.1115.80
  • Removing competitor channels raises cpWER by 0.97 pp [0.85, 1.09] and 2.18 pp [2.03, 2.33], including on target-only words.
  • Pooling competitors raises cpWER by 0.23 pp [0.15, 0.30] and 0.61 pp [0.52, 0.70]; the largest per-word gain is on three-speaker overlap.
  • At 80 ms, masking competitor-only history hurts more than masking an equal number of symmetric-activity frames: +0.417 pp [0.278, 0.555] in the paper model, +0.184 pp [0.043, 0.325] in the revised model.

Why zero output projection rather than zero gate? For y=x+G(Woutz+bout)y=x+G(W_{\text{out}}z+b_{\text{out}}), both preserve identity at initialization. With Wout=bout=0W_{\text{out}}=b_{\text{out}}=0 and nonzero GG, the projection can receive gradients from step one; deeper blocks learn after it changes. With a zero gate, initially only the gate learns. Setting both to zero blocks branch gradients. LoRA and ControlNet use related zero-initialization strategies; we do not claim faster convergence without a direct ablation.

Why matching positions rather than chunk pooling? Intra-chunk attention mixes the KK frames; inter-chunk attention accesses matching positions across past chunks. This retains separate representations around activity changes rather than requiring one summary per chunk; intra-chunk mixing supplies context, so indices need not align phonetically. This is an architectural rationale, not an isolated empirical gain over a pooled-inter variant. At 80 ms, K=1K=1, so mean pooling changes nothing; the comparison with FW tests history access, not pooling superiority.

Thank you again for your constructive review. We have uploaded a revised PDF and abstract incorporating our rebuttal explanations and additional experiments.

Table 1 now reports absolute DP/FW results, Section 5.4 (Table 3) presents the competitor-channel ablations, and Appendices D–E provide further diagnostics and the shared-protocol SSA comparison.

We appreciate your time and consideration.

I thank the authors for addressing all my questions and concerns. Please add these details to the manuscript. I will retain my score.

Reviewer_8mVkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies adapting pretrained streaming automatic speech recognition (ASR) systems to multi-speaker scenarios conditioned on target speaker’s activity (speaking or not). A module is integrated to inform the ASR model about the activities of the multiple speakers.

Critical strengths

Experiments on multi-speaker datasets are conducted to verify the proposed approach.

The two different designs of dual-path (DP) and frame-wise (FW) are compared for better understanding of the effect of speaker activities on ASR.

DP shows advantages over FW, which suggests the positive effect of temporal information. It is further verified that in overlapping regions, DP provides lower word error rate (WER).

Critical weaknesses

The novelty of the proposed approach is limited. The paper provides a collection of empirical observations. The observations may be helpful for using the proposed DP and FW designs. However, there is no central insight.

The value of DP and FW is not well illustrated.

There is a lack of ablation studies to verify the effectiveness of DP and FW by comparing to SSA and SE-DiCoW.

Table 3 compares with previous approach of SSA (Wang et al., 2025) with different recipes, however, the numbers from their paper are directly used. It would be helpful to maintain fair comparison.

Questions for authors

see above

3 comments

Thank you. First, SepAdapter does not use target activity alone: it receives three target-relative channels (the target, the first competitor, and the remaining competitors).

Central insight. As stated in the abstract, SepAdapter combines two design choices in one residual correction before the encoder: this three-channel activity signal and temporal context. The submission tested the second choice (DP versus FW); the new experiments test the first. Both address one problem. When two people talk at once, the diarizer marks both as active, so the two target streams receive the same activity [1, 1, 0] and the same acoustic frame. Without another target cue, a correction that looks only at the current frame must then be the same for both streams. In LibriSpeechMix, among frames the diarizer marks as overlapped, the activity vectors of the active targets coincide on 99.6% (two-speaker mixtures) and 91.6% (three-speaker mixtures). The activity in surrounding frames, such as who spoke before and what the competitors were doing, can differ between the streams; SepAdapter uses this context to compute a target-specific correction. SE-DiCoW identified this ambiguity and addresses it offline using self-enrollment. We tested each design choice:

  1. Tell the module what the competitors are doing. Removing the competitor channels raises cpWER by 0.97 pp (paper recipe) and 2.18 pp (revised recipe). Merging the competitors into one channel raises it by 0.23 and 0.61 pp.
  2. Use past acoustic and activity context. SepAdapter keeps a bounded cache (5.6 s per block in FastConformer) and adds no ASR lookahead. Blocking this cache raises cpWER by 0.24/0.42 pp at 1120/80 ms. Masking only the frames where competitors speak alone hurts more than masking an equal number of frames with symmetric activity, in both the paper model (+0.42 pp) and the revised model (+0.18 pp).

So both choices give measurable gains, even though the encoder already models time. The building blocks (competitor-aware masks, zero-initialized residual branches, dual-path processing) are known. Our contribution is to combine them into one target-relative correction before the encoder, with bounded memory, and to test what each part adds.

Value of the DP/FW comparison. DP and FW receive the same three channels and have nearly the same number of parameters; the main difference is that DP looks across frames and FW does not. It therefore tests the second choice. Over three training runs, DP lowers cpWER from 14.52% to 14.17% (−0.356 pp, 98.3% CI [−0.444, −0.273]). In the seed-43 pair, the gain is 0.70–0.96 pp on overlapped words, and the intervals for non-overlapped words include zero.

Different recipes in the SSA comparison. Table 3 compares with published SSA numbers. SSA fine-tunes its diarizer and adapts its recognizer to Fisher for CH109, while we use the public Sortformer unchanged. To compare the conditioning modules under controlled conditions, we retrained SSA in our pipeline. The revised recipe makes SepAdapter's training match streaming inference (same history window and same competitor choice), starts the gate at 1 instead of 0.1 because the trained gate had stayed near 0.1 (the output projection still starts at zero), and adds the same activity corruption to both models. We developed this recipe after submission, using diagnostics on the existing evaluation set and one intermediate training experiment. The final comparison uses one 100k-step run per model. This shared-protocol comparison is exploratory post-submission evidence, not a held-out confirmation or a reproduction of the original published SSA training recipe.

Comparison with SSA and SE-DiCoW. We retrained the released NeMo SSA operator in our pipeline under a revised post-submission training recipe. SepAdapter obtains 13.62% versus 14.01% cpWER (−0.397 pp [−0.515, −0.281]), and the interval excludes zero at every lookahead. The gain comes mainly from words in two-speaker overlap (−0.355 pp) and from fewer insertions aligned to competitor-active, target-silent intervals (−0.051 pp [−0.090, −0.016]); the intervals of all other word groups include zero. This compares the two complete modules under a shared training and evaluation protocol. They differ in size, so this does not isolate temporal processing. For SE-DiCoW, Table 7 re-scores the public checkpoint in our setup: SepAdapter is better on three of ten datasets (by up to 1.68 pp), worse on two, and the intervals include zero on five.

Presentation. We will update the abstract and contributions to state both design choices with their evidence. The paper will then follow the question above: the overlap problem, shown with a two-speaker timeline; the activity channels; their use over time (DP/FW and history masking); the SSA comparison; and results across recognizers.

Thank the authors for addressing my questions. Hope the above explanation and added experiments will be integrated. I will raise my score.

Thank you for your encouraging response and for reconsidering your assessment.

We have updated the manuscript to incorporate the explanations and new experiments. The Introduction now clarifies the motivation and three design choices: competitor activity, temporal context, and placement. Section 5.4 (Table 3) presents the activity-channel ablations and history intervention, including its limitations. Section 6.1 summarizes the retrained SSA comparison, with the revised recipe and full results in Appendix E (Table 10).

We appreciate your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.