- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper evaluates whether Qwen2.5-Omni can perform speaker identification and memory retrieval natively, using its own audio features, rather than relying on an external speaker encoder.
Methodology The researchers froze the audio encoder and trained an attention pooler to compress utterances into single tokens. These tokens were averaged into a registry key and paired with text values, using a LoRA adapter to teach the model to retrieve the correct value based on the speaker.
Key Findings
- Good Identification, poor transfer: The learned pooled embedding achieved 97.3% accuracy for speaker identification, vastly outperforming mean pooling (24.8%) but not dedicated encoders (~99%). On the other hand, zero-shot transfer to other datasets dropped significantly (74.3% or lower).
- Memory retrieval failed: Using the native audio tokens for dialogue memory routing was highly ineffective. It caused a 77.5% error rate for writing to the wrong person's memory, compared to just 12.5% when using text-based identities.
- Attempted fixes failed: Additional adapters, joint training, and residual adaptation did not resolve the performance gap. A training-free workaround only succeeded in highly restricted scenarios.
- Critical vulnerabilities: The audio tokens leaked spoken transcript content (82.3% match rate) and completely failed at exclusion, falsely accepting 94–100% of unenrolled speakers.
Ultimately, while the model can identify speakers in isolation, its native audio tokens are prone to interference to reliably drive dialogue memory retrieval.
Critical strengths
- Clear separation of representation quality from downstream use. The paper shows that speaker information in a token (97.3% identification, 99.0% by cosine) does not mean the LLM can use it for decisions. This is the most original and useful finding.
Critical weaknesses
-
The central negative result is expected, and its cause is untested. An arbitrary vector from a separate pooler has no special meaning in the LLM’s embedding space, therefore the base model (which is not trained against this token in the dialogue runs) cannot be expected to use it.
-
The working solution is external, and the novelty is small. Cosine matching alone (99.0%) beats the LLM (97.3%), and the best dialogue conditions do the comparison and slot choice outside the model. The LLM contributes nothing to identification, so “omni-native” holds for the representation, not for its use. The pooler is standard (attentive statistics pooling performs the same), registry ordering is guaranteed by sorting, and dedicated encoders or Higgs-Audio perform as well or better.
-
Evidence is thin, simplified and narrow. The main task is a forced choice among five token ids with n=4, with values unrelated to speech content. Key dialogue results use N=40 or N=21 episodes (the mapping subset comes from two sessions and two speakers), and the mapping’s free-generation check failed (25/40 vs a floor of 32). Generality is limited: 55.0% at 64 speakers, 58.3% zero-shot on AliMeeting, one model family, and in-model rejection accepts 94-100% of unenrolled speakers.
Questions for authors
Given that external cosine matching outperforms the LLM for speaker identification (99.0% vs. 97.3%), what value does the LLM actually add to the whole process? Under what specific conditions would in-model processing outperform external routing? To prove the value of the omni-native representation, it would be helpful to compare your method against dedicated encoders (like CAM++ or ERes2Net) using the same external routing on dialogue tasks.
5 comments
Response to Reviewer 37rS (2/3)
Q1 (continued)
Where the in-model readout adds value.
- No external clustering. Token-Soft is above CAM++ clustering without clustering or a speaker count (.764 vs. .699 on M3-SLU, 5-fold, both unaugmented).
- Questions that span many turns. On SpeakerBind, the continuous token is above clustered tags under identical SFT (.799 vs. .788) and reaches .852 after the synthetic curriculum and GRPO. Shuffling the tokens returns accuracy to the text-only level (.148 vs. .155) (Appendix E.2, Table 17).
- Locating a changed speaker. On SpeakerSleuth, rebuilt as in the original paper because its data are not public, the in-model readout reaches .800 F1 on localization against .703 for cosine on the same tokens with synthetic voices, and .832 against .778 on LibriSpeech recordings (Appendix E.3, Table 18).
Section 6 gives the reason: an external pipeline locks in clustering errors, whereas the continuous token lets the model combine acoustics with context.
Where cosine matching or a dedicated encoder is better. These cases have one thing in common: the speaker decision is a single acoustic comparison, the other system was trained on more speakers, or neither system was trained on the target domain.
- Single acoustic comparisons. When identity is decided by one similarity comparison without dialogue context, cosine matching on our tokens matches or beats the in-model readout: .990 vs. .982 on routing (Table 1) and 1.000 vs. .867 on the SpeakerSleuth three-way choice.
- More training speakers. CAM++ and ERes2Net, trained on about 200k speakers, beat our token (26,488 speakers) on short same-gender clips (.967 vs. .938) and on verification (6.50% vs. 11.87% EER). Our token is still above Higgs-Audio on the short clips (.938 vs. .820, Table 1). The in-model interface also benefits from a stronger embedding: ResNet293 embeddings trained on about 110k speakers reach .906 on SpeakerBind through the same projector, against .833 for our token (Appendix E.2, Table 17).
- Target-domain training. With no M3-SLU training at all, CAM++ leads (.608 vs. .560). Our speaker pooler, however, can be adapted with only 800 extra training steps on the conversations of the M3-SLU training split, which has just 1,494 questions (Appendix B; CAM++ is kept as released), with the language model trained the same way for both: without this adaptation the two perform similarly (.718 vs. .712), and with it our token leads (.794 vs. .712) (Appendix E.1, Table 16).
W2. The working solution is external, and the novelty is small
The novelty is not a new pooler. It is a new source of speaker identity for the language model, and an account of when the model can use it. The simple components are a deliberate choice.
- Novelty. Prior speaker-aware language models obtain identity from a separate encoder or diarization toolkit (Section 2). We obtain it from the omni model's own frozen audio encoder, with no second speaker model, and show when it is used: the identity is present in the encoder, the language model uses it only after alignment, and once aligned it surpasses CAM++ clustering on SpeakerBind and M3-SLU (Section 1). You described this separation of representation from use as "the most original and useful finding"; the revision makes it the core contribution.
- Standard pooler and sorting. We use standard components on purpose. Attentive statistics pooling performing about the same (Table 2b; Appendix C.1, Table 7) shows that the identity comes from the encoder, not from a special architecture, and it makes the method easy to reuse: it runs on five backbones (Table 2). Sorting is a simple guarantee that removes the 2.8% of predictions that change when a registry is reordered (Appendix B.2).
- "Cosine matching alone (99.0%) beats the LLM (97.3%)." On clean closed-set routing, a pure similarity ranking, this is expected, and we report it openly (.990 vs. .982, Table 1). Our claim concerns decisions that also depend on the transcript, which cosine matching alone cannot make.
- "The best dialogue conditions do the comparison and slot choice outside the model." This held for the submitted mapping, not for the revision: the best dialogue results come from Token-Soft and Token-Hybrid, where the language model receives the token of every turn and makes the comparison itself; Token-Soft uses no external clustering (Table 3).
- "Omni-native holds for the representation, not for its use." The revision shows the use as well: the in-model readout beats cosine on the same tokens in SpeakerSleuth localization, and Token-Soft beats CAM++ clustering without any clustering (Q1 above).
- "Dedicated encoders or Higgs-Audio perform as well or better." Not Higgs-Audio: on ≤2 s same-gender clips, cosine on our token gives .938 / .905 against .820 / .753 (Table 1). Encoders trained on about 200k speakers are better on short clips and verification, and we state exactly where (Q1 above).
Response to Reviewer 37rS (3/3)
W3. Evidence is thin, simplified, and narrow
The revision examines the method end to end: how speaker identity is extracted, how the language model uses it, how it generalizes, how stable it is, and how it copes with hard settings, with three seeds and 95% paired bootstrap confidence intervals unless noted (Section 4):
- Extraction. The frozen omni audio encoder read by pooler variants (attentive statistics, mean plus MLP, one to five queries; Table 7), trained on 820 or 26,488 speakers (Table 9), tested on routing and the official VoxCeleb1-O list (Tables 1 and 10).
- Use. Before alignment, the token barely changes the candidate scores (median KL .0005; Appendix D.1); after it, identity appears in layers 16–24 (Table 14). A linear projector suffices for verification, LoRA is needed for multi-turn binding, and GRPO mainly improves aggregation questions (Tables 2b and 3b).
- Generalization. Five backbones (Table 2); cross-corpus retrieval and room-disjoint AliMeeting (Tables 9 and 12); public data (M3-SLU real meetings, AMI, VoxCeleb1) and tasks rebuilt from their papers because the data are not public (SpeakerBind, SpeakerSleuth; Tables 17 and 18).
- Stability. Means and standard deviations over seeds in the appendix tables, registry order fixed by construction (2.8% of predictions change without it; Appendix B.2), and shuffled-token and no-identity controls in the main dialogue tables (Tables 3, 16, 17, and 18).
- Hard settings. Up to 32 enrolled speakers, ≤2 s same-gender clips, unknown speaker counts, the most-overlapped windows, unenrolled speakers (Table 21), and unsegmented multi-minute audio, where the token does not help (Table 23).
Your specific points:
- Forced choice with n = 4. Retrained at each size, the router reaches .989, .982, .963, .951, and .918 for n = 2–32 (no identity: .000 from n = 8). SpeakerBind answers are generated as free text, and M3-SLU asks True/False questions about real recordings.
- N = 40 / 21 and the mapping. The main dialogue results now use M3-SLU (2,766 questions in 5-fold cross-validation, 997 held out) and SpeakerBind (7,200 test questions, rebuilt with synthetic voices as in the original paper because its data are not yet public). N = 40 / 21 are diagnostics in Appendix D; no claim rests on the mapping.
- 55.0% at 64 speakers came from a router trained at n = 4 (Appendix C.2, Table 8).
- 58.3% zero-shot on AliMeeting. Retraining only the adapter gives .701 (Appendix C.4, Table 12).
- One model family. Gemma-4, LLaMA-3.1-Omni, and MiMo-Audio are added; on LLaMA-3.1, M3-SLU is .751 against .690 for CAM++ (Table 2).
Summary: the two "critical vulnerabilities"
Your summary lists "leaked spoken transcript content (82.3% match rate)" and "falsely accepting 94–100% of unenrolled speakers". Both are reduced by a change in training.
- Rejection. Adding 10% unenrolled queries to adapter training lowers false acceptance on LibriSpeech from .987 to .146 (enrolled accuracy changes by at most 0.9 points) and to .023–.056 on unseen VoxCeleb1 and AliMeeting speakers; within one non-LibriSpeech corpus it falls only to .518–.673 (Appendix F.1, Table 21).
- Content leakage. With the speaker pooler trained on 26,488 speakers, the seven-candidate transcript test at 8,192 bits drops from 82.3% to 31.5% (random control 15.0%; Appendix F.2, Table 22).
What the method offers next to a dedicated encoder. It complements dedicated encoders, which remain better for pure acoustic ranking:
- No second model: identity comes from the omni model's own audio encoder through one 55.1M-parameter pooler, and the tokens transfer to Qwen3-Omni through a linear projector.
- No early decision: Token-Soft needs no clustering or speaker count, so the model combines acoustics with context (Section 6).
- Fast, lightweight iteration: encoder and language model stay frozen; the scaled pooler trains in about 2.5 h and an M3-SLU adapter in 0.6–1.7 h on one accelerator (Appendix B.3). CAM++ leads without M3-SLU training (.608 vs. .560); after adapting on a training split of only 1,494 questions, our token leads (.794 vs. .712, held-out).
- A diagnostic for omni models: a low speaker-benchmark score does not mean the features lack identity, so benchmarks should report encoding and use separately (Section 6).
Thank you again for your careful review and concrete guidance. Your review shaped this revision directly: your explanation of the negative result is now the paper's second finding, and the CAM++ comparison you suggested is the basis of its third. In summary, the revision tests the cause of the negative result (W1), measures what the language model adds over cosine matching and CAM++ and where it does not (Q1, W2), broadens the evidence (W3), and reduces both critical vulnerabilities. We would be glad to answer any further question.
Response to Reviewer 37rS (1/3)
Thank you for the careful and accurate reading. You identified the separation between representation quality and downstream use as "the most original and useful finding", and you asked what causes the gap and when in-model processing is better than external routing. The revision is built around these two questions, and we ran the comparison you suggested.
W1. The negative dialogue result is expected, and its cause was untested
You are right that the base model was never trained against the token. We now test this explanation directly, and the results support it.
Fixes that skip alignment either reduce accuracy or work only because cosine matching is done outside the model. We now present them as evidence for this explanation (Section 5.3; Appendix D.3, Table 15).
Q1. What does the language model add over cosine matching? When is in-model processing better than external routing?
The language model adds value when identity has to be decided together with the transcript. For pure similarity ranking, cosine matching is the right tool, as expected.
Cosine matching scores the query token against each enrolled token outside the language model. The in-model readout feeds the same tokens to the language model as input embeddings, and the model produces the answer.
We ran the comparison you suggested: CAM++ and our token on the same dialogue tasks, with the same transcripts, clustering, and label format (Section 5.3, Table 3).
Token-Label is the external route: our tokens are clustered outside the model and passed as speaker tags in text, as with CAM++. Token-Soft is the in-model readout: the language model receives only the continuous token, with no clustering or speaker count. Token-Hybrid gives both. Token-Hybrid uses training-window augmentation in all M3-SLU columns, and CAM++ only with known K and on the held-out split.
For scale, M3-SLU accuracy is not bounded by 1.0 for audio methods. With the true speaker of every 2 s Whisper window given as a label, Qwen2.5 reaches .811, and with hand-corrected transcripts and gold names it reaches .888 (Table 2). Errors from windowing and ASR therefore cap all audio methods, including CAM++, well below 1.0, and our .792–.797 is close to the .811 that true speaker labels reach on the same windows.
We report both where the method helps and where it does not; all comparisons below are new for this review (Section 5.3, Table 3; Appendix E).
Same external route, our token against CAM++ (your suggested comparison). On M3-SLU (real recordings), with the same clustering and label format, Token-Label is 3.5 points above CAM++ with the true speaker count (.784 vs. .749, CAM++ augmented), 6.6 points with an estimated count (.765 vs. .699, both unaugmented), and 4.8 points in balanced accuracy on the third of held-out windows with the most overlap (.674 vs. .626). On SpeakerBind, with identical SFT and GRPO, it is .806 against .792 (Table 3).