acceptodds
Under review as a conference paper at ICLR 2027

One Token per Speaker: An Omni-Native, Exchangeable Entity Interface for Conversational Memory Routing

Abstract

Speaker-specific memory requires linking utterances to the correct person. We investigate whether an omni model can derive and use speaker identity from its own audio features without a separate speaker encoder. A learned pooler over frozen Qwen2.5-Omni features produces one continuous token per enrolled speaker. With LoRA adaptation, speaker identification for session-memory retrieval reaches 97.3%, compared with 24.8% for mean pooling of the same features. Without target-domain adaptation, accuracy reaches 74.3% on VoxCeleb1 and 58.3% on AliMeeting. Deterministic ordering preserves predictions when unchanged enrollment entries are reordered. Reliable identification does not, however, ensure effective use in dialogue. Under controlled candidate scoring, a continuous identity cue changes only one of 160 decisions relative to no identity. Providing the same resolved identity as text reduces episodes with wrong-person writes from 77.5% to 12.5%. The tested speaker-side and model-side adaptations do not close this gap. Over 30-turn meeting sequences, cosine matching over the same tokens roughly halves wrong-person writes relative to in-model use, although its errors persist longer. Mapping cosine-matching results to existing candidate embeddings matches the gold-identity reference without additional parameters or training. This result is limited to the tested single-token candidate setting; the free-generation check ran but did not establish safety, reaching 25 of 40 valid decisions against a pre-registered floor of 32. Together, these findings demonstrate useful omni-native speaker representations while identifying their connection to downstream decisions as a remaining challenge.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
omni, speaker identity, speech-language models, multimodal dialogue, session memory, representation interfaces, exchangeability, generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_37rSOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates whether Qwen2.5-Omni can perform speaker identification and memory retrieval natively, using its own audio features, rather than relying on an external speaker encoder.

Methodology The researchers froze the audio encoder and trained an attention pooler to compress utterances into single tokens. These tokens were averaged into a registry key and paired with text values, using a LoRA adapter to teach the model to retrieve the correct value based on the speaker.

Key Findings

  • Good Identification, poor transfer: The learned pooled embedding achieved 97.3% accuracy for speaker identification, vastly outperforming mean pooling (24.8%) but not dedicated encoders (~99%). On the other hand, zero-shot transfer to other datasets dropped significantly (74.3% or lower).
  • Memory retrieval failed: Using the native audio tokens for dialogue memory routing was highly ineffective. It caused a 77.5% error rate for writing to the wrong person's memory, compared to just 12.5% when using text-based identities.
  • Attempted fixes failed: Additional adapters, joint training, and residual adaptation did not resolve the performance gap. A training-free workaround only succeeded in highly restricted scenarios.
  • Critical vulnerabilities: The audio tokens leaked spoken transcript content (82.3% match rate) and completely failed at exclusion, falsely accepting 94–100% of unenrolled speakers.

Ultimately, while the model can identify speakers in isolation, its native audio tokens are prone to interference to reliably drive dialogue memory retrieval.

Critical strengths
  • Clear separation of representation quality from downstream use. The paper shows that speaker information in a token (97.3% identification, 99.0% by cosine) does not mean the LLM can use it for decisions. This is the most original and useful finding.
Critical weaknesses
  • The central negative result is expected, and its cause is untested. An arbitrary vector from a separate pooler has no special meaning in the LLM’s embedding space, therefore the base model (which is not trained against this token in the dialogue runs) cannot be expected to use it.

  • The working solution is external, and the novelty is small. Cosine matching alone (99.0%) beats the LLM (97.3%), and the best dialogue conditions do the comparison and slot choice outside the model. The LLM contributes nothing to identification, so “omni-native” holds for the representation, not for its use. The pooler is standard (attentive statistics pooling performs the same), registry ordering is guaranteed by sorting, and dedicated encoders or Higgs-Audio perform as well or better.

  • Evidence is thin, simplified and narrow. The main task is a forced choice among five token ids with n=4, with values unrelated to speech content. Key dialogue results use N=40 or N=21 episodes (the mapping subset comes from two sessions and two speakers), and the mapping’s free-generation check failed (25/40 vs a floor of 32). Generality is limited: 55.0% at 64 speakers, 58.3% zero-shot on AliMeeting, one model family, and in-model rejection accepts 94-100% of unenrolled speakers.

Questions for authors

Given that external cosine matching outperforms the LLM for speaker identification (99.0% vs. 97.3%), what value does the LLM actually add to the whole process? Under what specific conditions would in-model processing outperform external routing? To prove the value of the omni-native representation, it would be helpful to compare your method against dedicated encoders (like CAM++ or ERes2Net) using the same external routing on dialogue tasks.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 37rS (1/3)

Thank you for the careful and accurate reading. You identified the separation between representation quality and downstream use as "the most original and useful finding", and you asked what causes the gap and when in-model processing is better than external routing. The revision is built around these two questions, and we ran the comparison you suggested.


W1. The negative dialogue result is expected, and its cause was untested

You are right that the base model was never trained against the token. We now test this explanation directly, and the results support it.

  1. The information is in the audio features. Off-the-shelf Qwen2.5-Omni recognizes gender at 99.6% but verifies speakers near chance (42.13% EER on VoxCeleb1-O). One pooled token from the same frozen features gives 18.62% EER by cosine (Section 5.1, Table 1; Appendix C.3, Table 10).
  2. The language model does not read it until it is aligned. We probe the hidden states on SpeakerBind, rebuilt as in Xu et al. (2026) because its data are not yet public, and score whether a query and a turn belong to the same speaker. Before training, every layer is at chance. After 800 steps of Stage II training, layers 16–24 separate same-speaker from different-speaker pairs. When the tokens are shuffled within a dialogue, every layer returns to chance (Appendix D.2, Table 14).
Same-speaker AUC (cosine)L16L20L21L24L28
Before alignment.498.503.508.508.485
After alignment.600.693.701.648.531
After alignment, tokens shuffled.506.512.517.525.502
  1. A small aligned interface is enough for a simple decision. A linear projector into the frozen language model lowers the VoxCeleb1-O EER to 11.87%. Multi-turn tasks need LoRA.

Fixes that skip alignment either reduce accuracy or work only because cosine matching is done outside the model. We now present them as evidence for this explanation (Section 5.3; Appendix D.3, Table 15).


Q1. What does the language model add over cosine matching? When is in-model processing better than external routing?

The language model adds value when identity has to be decided together with the transcript. For pure similarity ranking, cosine matching is the right tool, as expected.

Cosine matching scores the query token against each enrolled token outside the language model. The in-model readout feeds the same tokens to the language model as input embeddings, and the model produces the answer.

We ran the comparison you suggested: CAM++ and our token on the same dialogue tasks, with the same transcripts, clustering, and label format (Section 5.3, Table 3).

SpeakerBind (+GRPO)M3-SLU, 5-fold, known KM3-SLU, 5-fold, unknown KM3-SLU, held-out, unknown KM3-SLU, high overlap
No identity.155 (SFT).564.564.556.535
CAM++ clustering.792.749.699.712.626
Ours, Token-Label.806.784.765.739.674
Ours, Token-Soft.852.764.764.765.680
Ours, Token-Hybrid.860.792.797.794.682

Token-Label is the external route: our tokens are clustered outside the model and passed as speaker tags in text, as with CAM++. Token-Soft is the in-model readout: the language model receives only the continuous token, with no clustering or speaker count. Token-Hybrid gives both. Token-Hybrid uses training-window augmentation in all M3-SLU columns, and CAM++ only with known K and on the held-out split.

For scale, M3-SLU accuracy is not bounded by 1.0 for audio methods. With the true speaker of every 2 s Whisper window given as a label, Qwen2.5 reaches .811, and with hand-corrected transcripts and gold names it reaches .888 (Table 2). Errors from windowing and ASR therefore cap all audio methods, including CAM++, well below 1.0, and our .792–.797 is close to the .811 that true speaker labels reach on the same windows.

We report both where the method helps and where it does not; all comparisons below are new for this review (Section 5.3, Table 3; Appendix E).

Same external route, our token against CAM++ (your suggested comparison). On M3-SLU (real recordings), with the same clustering and label format, Token-Label is 3.5 points above CAM++ with the true speaker count (.784 vs. .749, CAM++ augmented), 6.6 points with an estimated count (.765 vs. .699, both unaugmented), and 4.8 points in balanced accuracy on the third of held-out windows with the most overlap (.674 vs. .626). On SpeakerBind, with identical SFT and GRPO, it is .806 against .792 (Table 3).

AuthorsOpenReview ↗

Response to Reviewer 37rS (2/3)

Q1 (continued)

Where the in-model readout adds value.

  • No external clustering. Token-Soft is above CAM++ clustering without clustering or a speaker count (.764 vs. .699 on M3-SLU, 5-fold, both unaugmented).
  • Questions that span many turns. On SpeakerBind, the continuous token is above clustered tags under identical SFT (.799 vs. .788) and reaches .852 after the synthetic curriculum and GRPO. Shuffling the tokens returns accuracy to the text-only level (.148 vs. .155) (Appendix E.2, Table 17).
  • Locating a changed speaker. On SpeakerSleuth, rebuilt as in the original paper because its data are not public, the in-model readout reaches .800 F1 on localization against .703 for cosine on the same tokens with synthetic voices, and .832 against .778 on LibriSpeech recordings (Appendix E.3, Table 18).

Section 6 gives the reason: an external pipeline locks in clustering errors, whereas the continuous token lets the model combine acoustics with context.

Where cosine matching or a dedicated encoder is better. These cases have one thing in common: the speaker decision is a single acoustic comparison, the other system was trained on more speakers, or neither system was trained on the target domain.

  • Single acoustic comparisons. When identity is decided by one similarity comparison without dialogue context, cosine matching on our tokens matches or beats the in-model readout: .990 vs. .982 on routing (Table 1) and 1.000 vs. .867 on the SpeakerSleuth three-way choice.
  • More training speakers. CAM++ and ERes2Net, trained on about 200k speakers, beat our token (26,488 speakers) on short same-gender clips (.967 vs. .938) and on verification (6.50% vs. 11.87% EER). Our token is still above Higgs-Audio on the short clips (.938 vs. .820, Table 1). The in-model interface also benefits from a stronger embedding: ResNet293 embeddings trained on about 110k speakers reach .906 on SpeakerBind through the same projector, against .833 for our token (Appendix E.2, Table 17).
  • Target-domain training. With no M3-SLU training at all, CAM++ leads (.608 vs. .560). Our speaker pooler, however, can be adapted with only 800 extra training steps on the conversations of the M3-SLU training split, which has just 1,494 questions (Appendix B; CAM++ is kept as released), with the language model trained the same way for both: without this adaptation the two perform similarly (.718 vs. .712), and with it our token leads (.794 vs. .712) (Appendix E.1, Table 16).

W2. The working solution is external, and the novelty is small

The novelty is not a new pooler. It is a new source of speaker identity for the language model, and an account of when the model can use it. The simple components are a deliberate choice.

  • Novelty. Prior speaker-aware language models obtain identity from a separate encoder or diarization toolkit (Section 2). We obtain it from the omni model's own frozen audio encoder, with no second speaker model, and show when it is used: the identity is present in the encoder, the language model uses it only after alignment, and once aligned it surpasses CAM++ clustering on SpeakerBind and M3-SLU (Section 1). You described this separation of representation from use as "the most original and useful finding"; the revision makes it the core contribution.
  • Standard pooler and sorting. We use standard components on purpose. Attentive statistics pooling performing about the same (Table 2b; Appendix C.1, Table 7) shows that the identity comes from the encoder, not from a special architecture, and it makes the method easy to reuse: it runs on five backbones (Table 2). Sorting is a simple guarantee that removes the 2.8% of predictions that change when a registry is reordered (Appendix B.2).
  • "Cosine matching alone (99.0%) beats the LLM (97.3%)." On clean closed-set routing, a pure similarity ranking, this is expected, and we report it openly (.990 vs. .982, Table 1). Our claim concerns decisions that also depend on the transcript, which cosine matching alone cannot make.
  • "The best dialogue conditions do the comparison and slot choice outside the model." This held for the submitted mapping, not for the revision: the best dialogue results come from Token-Soft and Token-Hybrid, where the language model receives the token of every turn and makes the comparison itself; Token-Soft uses no external clustering (Table 3).
  • "Omni-native holds for the representation, not for its use." The revision shows the use as well: the in-model readout beats cosine on the same tokens in SpeakerSleuth localization, and Token-Soft beats CAM++ clustering without any clustering (Q1 above).
  • "Dedicated encoders or Higgs-Audio perform as well or better." Not Higgs-Audio: on ≤2 s same-gender clips, cosine on our token gives .938 / .905 against .820 / .753 (Table 1). Encoders trained on about 200k speakers are better on short clips and verification, and we state exactly where (Q1 above).
AuthorsOpenReview ↗

Response to Reviewer 37rS (3/3)

W3. Evidence is thin, simplified, and narrow

The revision examines the method end to end: how speaker identity is extracted, how the language model uses it, how it generalizes, how stable it is, and how it copes with hard settings, with three seeds and 95% paired bootstrap confidence intervals unless noted (Section 4):

  1. Extraction. The frozen omni audio encoder read by pooler variants (attentive statistics, mean plus MLP, one to five queries; Table 7), trained on 820 or 26,488 speakers (Table 9), tested on routing and the official VoxCeleb1-O list (Tables 1 and 10).
  2. Use. Before alignment, the token barely changes the candidate scores (median KL .0005; Appendix D.1); after it, identity appears in layers 16–24 (Table 14). A linear projector suffices for verification, LoRA is needed for multi-turn binding, and GRPO mainly improves aggregation questions (Tables 2b and 3b).
  3. Generalization. Five backbones (Table 2); cross-corpus retrieval and room-disjoint AliMeeting (Tables 9 and 12); public data (M3-SLU real meetings, AMI, VoxCeleb1) and tasks rebuilt from their papers because the data are not public (SpeakerBind, SpeakerSleuth; Tables 17 and 18).
  4. Stability. Means and standard deviations over seeds in the appendix tables, registry order fixed by construction (2.8% of predictions change without it; Appendix B.2), and shuffled-token and no-identity controls in the main dialogue tables (Tables 3, 16, 17, and 18).
  5. Hard settings. Up to 32 enrolled speakers, ≤2 s same-gender clips, unknown speaker counts, the most-overlapped windows, unenrolled speakers (Table 21), and unsegmented multi-minute audio, where the token does not help (Table 23).

Your specific points:

  • Forced choice with n = 4. Retrained at each size, the router reaches .989, .982, .963, .951, and .918 for n = 2–32 (no identity: .000 from n = 8). SpeakerBind answers are generated as free text, and M3-SLU asks True/False questions about real recordings.
  • N = 40 / 21 and the mapping. The main dialogue results now use M3-SLU (2,766 questions in 5-fold cross-validation, 997 held out) and SpeakerBind (7,200 test questions, rebuilt with synthetic voices as in the original paper because its data are not yet public). N = 40 / 21 are diagnostics in Appendix D; no claim rests on the mapping.
  • 55.0% at 64 speakers came from a router trained at n = 4 (Appendix C.2, Table 8).
  • 58.3% zero-shot on AliMeeting. Retraining only the adapter gives .701 (Appendix C.4, Table 12).
  • One model family. Gemma-4, LLaMA-3.1-Omni, and MiMo-Audio are added; on LLaMA-3.1, M3-SLU is .751 against .690 for CAM++ (Table 2).

Summary: the two "critical vulnerabilities"

Your summary lists "leaked spoken transcript content (82.3% match rate)" and "falsely accepting 94–100% of unenrolled speakers". Both are reduced by a change in training.

  • Rejection. Adding 10% unenrolled queries to adapter training lowers false acceptance on LibriSpeech from .987 to .146 (enrolled accuracy changes by at most 0.9 points) and to .023–.056 on unseen VoxCeleb1 and AliMeeting speakers; within one non-LibriSpeech corpus it falls only to .518–.673 (Appendix F.1, Table 21).
  • Content leakage. With the speaker pooler trained on 26,488 speakers, the seven-candidate transcript test at 8,192 bits drops from 82.3% to 31.5% (random control 15.0%; Appendix F.2, Table 22).

What the method offers next to a dedicated encoder. It complements dedicated encoders, which remain better for pure acoustic ranking:

  1. No second model: identity comes from the omni model's own audio encoder through one 55.1M-parameter pooler, and the tokens transfer to Qwen3-Omni through a linear projector.
  2. No early decision: Token-Soft needs no clustering or speaker count, so the model combines acoustics with context (Section 6).
  3. Fast, lightweight iteration: encoder and language model stay frozen; the scaled pooler trains in about 2.5 h and an M3-SLU adapter in 0.6–1.7 h on one accelerator (Appendix B.3). CAM++ leads without M3-SLU training (.608 vs. .560); after adapting on a training split of only 1,494 questions, our token leads (.794 vs. .712, held-out).
  4. A diagnostic for omni models: a low speaker-benchmark score does not mean the features lack identity, so benchmarks should report encoding and use separately (Section 6).

Thank you again for your careful review and concrete guidance. Your review shaped this revision directly: your explanation of the negative result is now the paper's second finding, and the CAM++ comparison you suggested is the basis of its third. In summary, the revision tests the cause of the negative result (W1), measures what the language model adds over cosine matching and CAM++ and where it does not (Q1, W2), broadens the evidence (W3), and reduces both critical vulnerabilities. We would be glad to answer any further question.

Reviewer_EAM6OpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims to see whether an omni-model's audio feeatures can support speaker-specific memory retrieval by itself. However the motivation is not clear. There was no substantial method description that is well explained enough, rather than a figure and a caption. The paper denotes specific parts of a method simply as colors such as cyan, navy, green, coral. Training objectives are deferred to appendices. Main paper contains a single table and all supporting results are in appendices.

Critical strengths

None.

Critical weaknesses

This is a poorly written paper, I do not find the novelty and significance of the method nor a claim made in the paper rather than experimental logs.

  1. Presentation. Figure and main body does not contain meaningful content to review.

  2. Limited results, all deferred to appendix.

Questions for authors
  1. All primary experiments seem to be run on Qwen 2.5-Omni and Qwen families. One experiment on Kimi-Audio is only mentioned but is confirmed to fail in L281. Could the authors confirm that cross-model experiments failed?

minor questions

  • Is L 500-501's Accepted wording: “operational tolerance point under this setting.” supposed to be included in main text?
7 comments

I am disappointed with the discussion phase even more. I update my score accordingly

AuthorsOpenReview ↗

Response to Reviewer EAM6 (1/3)

Thank you for your review. Our work covers the whole chain, from extracting speaker identity out of the omni model's own audio encoder, to aligning the language model with it, to several downstream readouts and their variants. In the submission, these parts were described across the main text and the appendix. The revision brings them together in a single Method section, with the main results in three main-text tables. Numbers below refer to the revised PDF.


W1. "I do not find the novelty and significance of the method nor a claim" (and "the motivation is not clear")

The claim: omni models cannot tell speakers apart out of the box, but the cue they need is present in their own audio-encoder features. Our speaker pooler (visualized in Figure 1) turns these features into one entity token that makes the cue readable, and once the language model is aligned with this token, it surpasses an external CAM++ clustering pipeline on downstream tasks that require attributing what was said to who said it (SpeakerBind and M3-SLU), without a second encoder.

Motivation. Our goal is to give omni models a practical way to understand and use speaker identity, as a step toward letting the reasoning strength of the omni model's language model raise the ceiling on speaker attribution, for example answering who said what in a meeting with an unknown number of speakers and overlapping speech, where identity has to be decided together with the transcript. This requires an identity representation that the omni model can read natively, use directly in its decisions, and adapt from downstream feedback, uniting speech understanding and reasoning with acoustic speaker recognition in a single model. Existing speaker-aware language models instead take identity from a separate speaker encoder or diarization toolkit (Section 2). We therefore ask two questions: whether the identity cue is present in the omni model's own audio-encoder features, and whether its language model can learn to use it. Our results point toward that unification: reading and using the omni-native speaker token, and combining it with a cluster tag in Token-Hybrid, gives the best system in every dialogue column of Table 3.

Findings and significance. Section 1 states three findings:

  1. One token pooled from the frozen omni audio encoder carries speaker identity. The off-the-shelf Qwen2.5-Omni verifies speakers near chance (42.13% EER on VoxCeleb1-O), yet the token that our speaker pooler extracts from its own frozen features reaches 11.87%.
  2. The language model uses the token only after alignment. Before Stage II, no layer separates same-speaker from different-speaker turns; after it, layers 16–24 do.
  3. An aligned omni-native speaker token improves downstream identity attribution. Without a second encoder, it surpasses CAM++ clustering on SpeakerBind and M3-SLU.

So for an omni model, a low score on a standard speaker-recognition benchmark does not show that its audio features lack identity (Section 6). Reviewer 37rS describes this separation of representation from use as "the most original and useful finding".


W2. "Figure and main body does not contain meaningful content to review" (and "a figure and a caption")

The method is now in the main text: Section 3 with five equations, and a refined Figure 1.

Section 3 gives token extraction (Eq. 1), both training objectives (Eqs. 2 and 5), registry construction (Eq. 3), and cosine matching (Eq. 4). Figure 1 shows the complete, usable pipeline, from Stage I and Stage II training to an inference example. It is refined to be read at a glance: consistent colors and icons, more detail inside each module, and every module labelled by name and marked as frozen or trainable; the caption no longer refers to modules by color.


W3. "Limited results, all deferred to appendix" (and "a single table")

The main results are now in three main-text tables (setup in Section 4).

  • Table 1, "Speaker identity in one entity token": (a1) method comparison, (a2) registry-size scaling, (b) speaker verification against dedicated encoders, with published speech-aware language models compared in Section 5.1. With four enrolled speakers, the language model picks the right speaker for 98.2% of queries from our token; averaging the same features gives chance level (24.8%).
  • Table 2, "Transfer across backbones and training recipes": (a) backbones, (b) training recipe. On the meeting benchmark M3-SLU, the three non-Qwen families Gemma-4, LLaMA-3.1-Omni, and MiMo-Audio reach 77.2%, 75.1%, and 74.5%, above CAM++ clustering on each.
  • Table 3, "Identity-aware dialogue": (a) dialogue accuracy, (b) SpeakerBind by question type. On SpeakerBind, our best variant answers 86.0% of questions correctly, against 79.2% for CAM++ clustering and 15.5% without speaker information. On M3-SLU with an unknown speaker count, the continuous token alone, without clustering, reaches 76.4% against 69.9%.
AuthorsOpenReview ↗

Response to Reviewer EAM6 (2/3)

W2. "Figure and main body does not contain meaningful content to review" (and, in the summary, the method described by "colors" and "training objectives are deferred to appendices")

To make the method easier to follow, we brought all of its steps together in Section 3 with five equations, and we visually polished Figure 1.

Both training objectives are given in the main text: the contrastive loss of the first stage (Eq. 2) and the cross-entropy loss of the second stage (Eq. 5). Section 3 also explains how the token is extracted (Eq. 1), how each enrolled speaker's entry is built (Eq. 3), and how cosine matching works (Eq. 4). Figure 1 shows the complete pipeline that we use, from the two training stages to an example at inference time. It has been visually polished so that it reads more cleanly at a glance, with unified colors and icons, the inner blocks of each module drawn explicitly, and clear frozen or trainable labels. The caption refers to each module by its name.


W3. "Limited results, all deferred to appendix" (and "a single table")

We expanded the main-text results from one summary table into three dedicated tables (Tables 1 to 3), with a separate experimental setup in Section 4.

  • Table 1, "Speaker identity in one entity token". It has three parts: (a1) a comparison of methods, (a2) how accuracy changes as more speakers are enrolled, and (b) speaker verification against dedicated speaker encoders. Section 5.1 also compares with published speech-aware language models. With four enrolled speakers, the language model picks the right speaker for 98.2% of queries using our token. Simply averaging the same audio features gives 24.8%, which is chance level.
  • Table 2, "Transfer across backbones and training recipes". It has two parts: (a) different backbone models and (b) different training recipes. On M3-SLU, a benchmark of questions about real meetings, the three model families outside Qwen (Gemma-4, LLaMA-3.1-Omni, and MiMo-Audio) reach 77.2%, 75.1%, and 74.5%. Each is higher than CAM++ clustering on the same model.
  • Table 3, "Identity-aware dialogue". It has two parts: (a) accuracy on the dialogue benchmarks and (b) SpeakerBind results by question type. On SpeakerBind, our best variant answers 86.0% of the questions correctly. CAM++ clustering answers 79.2%, and a model without any speaker information answers 15.5%. On M3-SLU, when the number of speakers is not given, our continuous token alone, with no clustering at all, reaches 76.4%, against 69.9% for CAM++ clustering.

Q1. Did the cross-model experiments fail?

No. The method works on four other backbones besides Qwen2.5-Omni, three of which are outside the Qwen family. It also works on Kimi-Audio.

Kimi-Audio has two separate output heads, one for audio and one for text. The value of .190 at line 281 was read from the audio head. Read from the text head, with everything else unchanged, routing accuracy is .704 after one epoch and .866 after four (Appendix C.4, Table 11). Because Kimi-Audio uses a Qwen2.5 language model, we do not count it as evidence beyond Qwen. That evidence is in Section 5.2, Table 2:

Qwen2.5-Omni-7BQwen3-Omni-30B-A3BGemma-4-E4B-it⋆LLaMA-3.1-8B-Omni⋆MiMo-Audio-7B⋆
Routing, no identity.250‡.198.253.245.243
Routing, ours (1 epoch).973‡.835.777.852.641
M3-SLU, no identity.564.564.550.556.574
M3-SLU, CAM++ clustering.749.656.752.690.649
M3-SLU, ours.792.709.772.751.745

In the table. ⋆ marks a language model outside the Qwen family. Routing is how often the model picks the right speaker out of four enrolled speakers. It is measured on 500 test sessions after training on 1,000. ‡ For Qwen2.5 we show our main, larger setting, with 20,000 training sessions and 4,000 test sessions, so this column should not be compared directly with the others. M3-SLU is the accuracy on questions about who said what in real meetings. It uses five-fold cross-validation for Qwen2.5 and Gemma-4, and a fixed held-out split for the other models.

On every backbone, our token is well above the model without speaker information, and above CAM++ clustering on M3-SLU. Routing on MiMo varies from seed to seed (Appendix C.4, Table 11), but its average of .641 is still far above the .243 without speaker information.


Q2. Is the text at lines 500–501 supposed to be in the main text?

No. That sentence has been removed from the main text. It referred to one noise level (8,192 bits) in our test of how much spoken content the token keeps, and Appendix F.2 describes this test in full.

Reviewer_7PhWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Authors propose EntityToken, a pooled representation of speakers from adapted omnimodal models for identification and retrieval of speaker identities during LLM use sessions. Performance on the model is relatively encouraging in comparison to baselines and is rather simple for implementation with a given preexisting LLM, suggesting easy reproducibility and adaptation for future research.

Critical strengths

The proposed model is relatively straightforward and addresses a clear niche in adaptation to multi-modality models. It also provides encouraging performance in comparison to baselines. Further, from documented information, it appears clearly reproducible and easily adaptable to other models for future developments.

Critical weaknesses

The approach would have been strengthened by evaluations on other LLM backends. One of the merits that I can identify in this approach is that it can be adapted to any model compatible with LORA, so a demonstration of this would be beneficial and strengthen the paper's findings. This would also assist in the lack of developed comparative baselines for the approach (I am aware of the appendix analysis of speaker embeddings but the primary paper must serve to justify acceptance).

Further there is a dearth of significant technical detail in certain portions. Authors assume general familiarity with certain terms, but the general nature of the conference does put a responsibility on cross field dialogue for accepted papers. While it may seem trite, terms like cosine-matching do need to be technically defined to avoid confusion for potential readers and allow dissemination of the argument for disciplinary dialogue. This is compounded by the relatively sparse discussion and conclusions sections. Authors forego developing significance of their findings and simply restate results, limiting the amount of collegiate engagement for this paper to shape.

Altogether, while the approach appears interesting and potentially fruitful for future research, I'm having difficulty seeing how it positions itself for further engagement within the framework of the conference. As such, I argue for a weak rejection, with a provision that some effort in reframing results and discussion to foster research discourse will make me more comfortable assigning a weak acceptance.

On a final note, I heavily urge the authors to revisit their appendix section. While I appreciate the provision of negative results and additional information, the creation of 24 appendices (in comparison, Tolkien's Lord of the Rings just has 6) and 30 additional pages (3x the page limitation for the main paper) is a tad egregious. Given the current load of this conference, it does place an undo burden on reviewers to evaluate the results as provided and takes time away from fair assessment of your colleagues' submissions.

Questions for authors

What comparative models are preexisting in this subdomain for proper evaluation of the merits of this approach against others?

How adaptable is the approach to other omni-LLM models? Do you have results beyond Qwen?

Can you clarify cosine-matching? I have an intuition of the intended meaning but would appreciate complete clarity.

How would you situate your results amidst general findings in omni-LLM speaker retrieval research?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7PhW (1/2)

Thank you for the careful and encouraging review. You described the approach as simple, reproducible, and adaptable to other models, and you found no substantive concern about its soundness. We are also grateful that you clearly outlined what would strengthen the paper: "some effort in reframing results and discussion to foster research discourse". We have thoroughly revised the manuscript accordingly, and below we address each of your four questions as well as your comment regarding the appendix. All section and table numbers refer to the revised PDF.


Q1. Which existing models can this approach be compared with?

Existing speaker-aware language models obtain identity from a separate speaker encoder. We now compare with them in the main text.

Prior systems mainly fall into two groups. The first projects embeddings from an external speaker encoder into the language model (PAChat; Voices as Handles; SpeakerLLM; speech-aware LLMs for speaker verification, Thebaud et al., 2026). The second clusters or diarizes with an external model and passes speaker labels as text (such as the M3-SLU cascades and diarization-guided TS-ASR), whereas end-to-end models such as SpeakerLM jointly train a multimodal LLM for diarization and recognition. Our method uses no second encoder. It reads identity from the audio encoder that the omni model already has. The main text now has three comparison tables.

Routing and verification (Section 5.1, Table 1). On four-speaker LibriSpeech registries, our token is close to external speaker encoders, which were trained on far more labelled speakers. On short same-gender clips, the dedicated encoders CAM++ and ERes2Net are better. On VoxCeleb1-O, a linear projector into the frozen language model gives a lower EER than published speech-aware language models without a dedicated speaker encoder.

MethodRouting, n = 4≤2 s same-gender clips, 8 / 16 candidates
Mean pooling of the same features.248.369 / .264
Higgs-Audio.975.820 / .753
CAM++.984.967 / .951
ERes2Net.990.979 / .968
Ours, cosine matching.990.938 / .905
Ours, in-model.982.871 / .810

External encoders are given to the language model as identity prompts for routing and scored by cosine matching on the short clips.

VoxCeleb1-O (official list, 37,720 trials)EER (%)
Qwen2.5-Omni, zero-shot42.13
GPT-4o-audio (published)22.62
SA-Ministral3 (published)14.76
Frozen LLaMA2-3B + projector (published)12.08
Ours, frozen Qwen2.5 + linear projector11.87
ERes2Net (dedicated encoder)6.50
SA-TinyLLaMA, with an ECAPA-TDNN encoder (published)1.87

Multi-party dialogue (Section 5.3, Table 3). We use two benchmarks: M3-SLU Task 2 (Kwon et al., 2025), a public set of speaker-attributed questions on real meetings and conversations; and SpeakerBind, which we construct with synthetic voices following the original paper of Xu et al. (2026, Voices as Handles) because its data are not yet public: multi-turn dialogue questions that require binding facts to speakers, with names and gendered words removed. We compare with a CAM++ clustering pipeline on the same transcripts. The difference is largest when the number of speakers is unknown.

SpeakerBind (+GRPO)M3-SLU, 5-fold, known KM3-SLU, 5-fold, unknown KM3-SLU, held-out, unknown K
No identity.155 (SFT).564.564.556
CAM++ clustering.792.749.699.712
Ours, Token-Soft (continuous token only).852.764.764.765
Ours, Token-Hybrid (continuous token + cluster tag).860.792.797.794

Token-Soft gives the language model only the continuous token. It needs no clustering and no speaker count, and it is above CAM++ clustering in every column of the table above. Token-Hybrid adds the cluster tag and is the best in every column. On SpeakerBind, both of our rows use the synthetic curriculum before SFT; without the curriculum and with identical SFT, Token-Soft is still above CAM++ clustering (.799 vs. .763; Appendix E.2, Table 17). On M3-SLU, Token-Hybrid uses training-window augmentation in all columns and CAM++ only in the known-K and held-out columns; in the 5-fold unknown-K column, Token-Soft (.764, no augmentation and no clustering) is the matched comparison with CAM++ (.699). On M3-SLU, two separate steps use the training conversations: our speaker pooler is trained further on their 2 s windows (an extra Stage I step; CAM++ is used as released), and the language model is trained on the task (Stage II, the same for both systems). Without the pooler step, our token reaches .718 on the held-out split against .712 for CAM++. Without either step, CAM++, trained on about 200k speakers, transfers better than our speaker pooler trained on 26,488 speakers (.608 vs. .560; Appendix E.1, Table 16).

AuthorsOpenReview ↗

Response to Reviewer 7PhW (2/2)

Q2. Does the approach work beyond Qwen?

Yes. Besides the two Qwen backbones, we ran the method on three backbones whose language models are not from the Qwen family (Section 5.2, Table 2). On each of them, a speaker pooler is trained on its own frozen audio encoder for M3-SLU (and for routing on Gemma-4 and LLaMA-3.1); the remaining non-Qwen results read the Qwen2.5 tokens through a linear projector (Appendix C.4).

Qwen2.5-Omni-7BQwen3-Omni-30B-A3BGemma-4-E4B-it⋆LLaMA-3.1-8B-Omni⋆MiMo-Audio-7B⋆
Routing, no identity.250‡.198.253.245.243
Routing, ours (1 epoch).973‡.835.777.852.641
M3-SLU, no identity.564.564.550.556.574
M3-SLU, CAM++ clustering.749.656.752.690.649
M3-SLU, ours.792.709.772.751.745

⋆Non-Qwen language model. Routing reports n = 4 accuracy on the 500-session split (‡Qwen2.5 uses the 20k/4k protocol with the 820-speaker pooler). M3-SLU Task 2 reports five-fold cross-validation with true speaker counts on Qwen2.5 and Gemma-4, and the held-out split with development thresholds elsewhere.

On all five backbones, the token raises accuracy well above the no-identity model trained with the same adapter, and it is above CAM++ clustering on M3-SLU. Routing on MiMo varies widely across seeds (Appendix C.4, Table 11). Qwen3-Omni reads the projected Qwen2.5 tokens without a new speaker pooler and reaches 11.65% EER on VoxCeleb1-O and .840 on SpeakerBind.

The submitted paper reported a failed Kimi-Audio run at line 281. This was our evaluation error: Kimi-Audio has separate audio and text output heads, and our script read the wrong part of the output. With the correct part, routing accuracy is .704 after one epoch and .866 after four (Appendix C.4, Table 11). Kimi-Audio uses a Qwen2.5 language model, so we do not count it as a non-Qwen result.

Q3. What is cosine matching?

Cosine matching scores the query token against each enrolled speaker's token by cosine similarity, outside the language model, and returns the registered value of the most similar entry, or unknown if the highest similarity is below a threshold. It uses no language-model parameters.

Each utterance uu is mapped to one unit-norm token z(u)z(u) by the speaker pooler (Eq. 1). For enrolled speaker ii, the tokens of the three enrollment utterances are averaged and renormalized into one entry ZiZ_i (Eq. 3). For a query token ZqZ_q, cosine matching computes ⟨Zq,Zi⟩\langle Z_q, Z_i\rangle for every entry (the cosine similarity, since all tokens have unit norm) and selects i⋆=arg⁡max⁡i⟨Zq,Zi⟩i^\star = \arg\max_i \langle Z_q, Z_i\rangle. If the largest similarity is below a threshold θ\theta, the answer is unknown (Eq. 4).

The in-model readout gives the same tokens to the language model and lets it choose. Comparing the two on the same tokens shows how much the language model adds. Section 3 now defines both readouts, the training objectives of the two stages (Eqs. 2 and 5), and the symbols used in the redrawn Figure 1.

Q4. How do these results relate to other work on speaker identity in omni models?

Other work adds a speaker encoder to the language model. We show that the omni model's own audio encoder already contains the speaker information, and we measure when the language model can use it.

The revised Sections 1 and 6 make two points.

  1. When the entity token is better than an external encoder. It is better when identity has to be decided together with the transcript: an unknown number of speakers, overlapped speech, ASR errors, and questions that span many turns. An external pipeline fixes a speaker label before the language model sees the text, so its clustering errors cannot be corrected. A dedicated encoder is better when the task is only acoustic ranking, such as short same-gender clips and verification.
  2. What the interface currently requires. The audio must be segmented into turns or windows. On unsegmented multi-minute recordings, the token gives no improvement (Appendix F.3, Table 23).

Section 5.3 also explains why the submitted dialogue result was negative. Before the language model is aligned with the token, no layer distinguishes same-speaker turns from different-speaker turns. After alignment, layers 16–24 do (Appendix D.2, Table 14).

Appendix length

We agree. The 24 appendix sections are now six (A–F). They are: A, theory; B, protocols; C, Stage I and backbones; D, alignment diagnostics; E, dialogue benchmarks; F, rejection, leakage, and unsegmented audio.

Thank you again for your thoughtful and constructive review. Your suggestion to reframe the results and discussion gave the revision its structure: the three findings in Section 1 and the discussion in Section 6 of when the entity token helps both follow from it. We hope the revised paper now meets the bar you described, and we would be glad to clarify any remaining point during the discussion period.

Dear Reviewer 7PhW,

As the discussion period enters its final hours, we wanted to politely check in to make sure our response and revised PDF reached you. Following your suggestions, the revision addresses all four points you highlighted:

  • Non-Qwen backbones (Table 2): Evaluated three non-Qwen language backbones (Gemma-4-E4B-it, LLaMA-3.1-8B-Omni, and MiMo-Audio-7B).
  • Main-text baselines (Tables 1–3): Added direct comparisons against CAM++, ERes2Net, Higgs-Audio, and published speech-aware LLMs on routing, verification, and dialogue.
  • Technical clarity (Section 3, Eqs. 1–5): Added an explicit description of cosine matching, the two-stage setup, and the routing notation.
  • Reframing & appendix length (Sections 1, 5–6; Appendices A–F): Reframed the paper around the separation between speaker representation and downstream alignment to foster clearer discourse, and compressed the 24 appendix sections into six.

Best regards, The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers and AC

Dear Reviewers and Area Chair,

We thank Reviewers 7PhW, EAM6, and 37rS for their time and their reviews. We have uploaded a substantially revised PDF and answered every question in the individual replies. The main text is on pages 1–10 and the appendix has six sections (A–F). New content is marked blue at its table title, row or column header, or the first sentence of a new paragraph. Section and table numbers below refer to the revised PDF.

What the paper contributes

The method is unchanged: a speaker pooler on the frozen audio encoder of an omni model produces one token per speaker, and the language model is then trained to read it. We agree with Reviewer 37rS that the pooler and the sorting are standard. The contribution is what this setup lets us measure, now stated as three findings in Section 1:

  1. The omni model's own audio encoder already carries speaker identity. One pooled token lowers VoxCeleb1-O EER from 42.13% (off-the-shelf Qwen2.5-Omni) to 11.87% with a linear projector into the frozen language model (Table 1).
  2. The language model uses this token only after alignment. Before Stage II, no layer separates same-speaker from different-speaker turns; after it, layers 16–24 do (Section 5.3; Appendix D.2).
  3. Without a second encoder, the aligned token beats CAM++ clustering in multi-party dialogue, most clearly when identity must be resolved together with the transcript (Table 3).

Main revisions

1. Structure and presentation (EAM6, 7PhW). Section 3 now gives the full method with five equations, and Figure 1 is redrawn with labelled modules instead of color names. The experimental setup is a separate section, and the main text has three result tables instead of one. The 24 appendix sections are now six. The editing note at submitted lines 500–501 is deleted.

2. Comparison with a dedicated speaker encoder (7PhW Q1, 37rS Q1). We ran the comparison 37rS suggested: CAM++ clustering and our token with the same transcripts, label format, LoRA rank, and training schedule.

SpeakerBind (+GRPO)M3-SLU, known KM3-SLU, unknown KM3-SLU, most-overlapped windows
No identity.155 (SFT).564.564.535
CAM++ clustering.792.749.699.626
Ours, same label format (Token-Label).806.784.765.674
Ours, best (Token-Hybrid).860.792.797.682

M3-SLU is five-fold cross-validation on real meetings. The gap grows when the number of speakers is unknown.

3. Backbones outside Qwen (7PhW Q2, EAM6 Q1, 37rS W3). The same two stages run on Gemma-4, LLaMA-3.1-Omni, and MiMo-Audio. M3-SLU accuracy is .772, .751, and .745, against .550, .556, and .574 without identity, and above CAM++ clustering on every backbone (Table 2). The failed Kimi-Audio run at submitted line 281 was our evaluation error; corrected, routing reaches .866 (Appendix C.4).

4. Scale and robustness (37rS W3 and summary). With the router retrained at each registry size, accuracy goes from .989 at 2 speakers to .918 at 32 (Table 1). Adding unenrolled queries to training lowers false acceptance on LibriSpeech from .987 to .146. A speaker pooler trained on 26,488 speakers lowers transcript matching from 82.3% to 31.5% (Appendix F).

5. Limits stated in the paper (37rS, 7PhW). A dedicated encoder is still better at verification (6.50% vs. 11.87% EER) and on short same-gender clips. On M3-SLU, our gain depends on adapting the speaker pooler to the target domain: without it, the held-out accuracy is .718 against .712 for CAM++, and without any M3-SLU training CAM++ is better (.608 vs. .560; Appendix E.1). On unsegmented multi-minute audio the token gives no improvement (Section 6).

Strengths noted by the reviewers

  • Simple, reproducible, and adaptable to other models (7PhW).
  • Separating representation quality from downstream use is "the most original and useful finding" (37rS).

LLM use (EAM6)

LLM assistants were used as disclosed in the paper. The authors designed the study and checked every number against the saved result files; our reply to EAM6 gives the details.

We hope the revision addresses the reviewers' concerns, and we would be glad to answer further questions during the discussion period.

AuthorsOpenReview ↗

General Response to All Reviewers and AC (2/2)

Where each review point is answered

In this commend, "Submitted" refers to the submitted main PDF (line numbers) and its supplementary appendix (Supp.).

Reviewer & itemPoint raisedSubmitted PDFRevised PDF
7PhW — W1, Q1Comparison with existing speaker-aware models and dedicated encoders§1 L61–63; §4.1 L259–262; Supp. App. B, Q§2 (p. 2); §4.3 (p. 6); Table 1 (p. 7); Table 3 (p. 9); App. C.3 Table 10 (p. 19); App. E Tables 16–19 (pp. 23–25)
7PhW — W1, Q2Results on other backbones§4.1 L265–283; Supp. App. Q§4.1 (p. 5); Table 2 (p. 8); App. C.4 Tables 11–12 (p. 20)
7PhW — W2, Q3Definition of cosine matching and other terms§1 L62–63; §3.1 L153–161; §4.1 L246–249§3, Eqs. 1–5, Figure 1 (pp. 3–4); Table 1 (p. 7); §5.3 (pp. 8–10)
7PhW — W2, W3, Q4Sparse discussion; relation to other speaker work in omni models§2 L79–150; §5–§6 L437–475§1–§2 (pp. 1–2); §5.3 (pp. 8–10); §6 (p. 10); App. D.2 Table 14 (p. 21)
7PhW — W424 appendices, over 30 pagesSupp. App. A–XApp. A–F (pp. 15–28)
EAM6 — W1Claim and significanceAbstract; §1 L38–78§1, three findings (pp. 1–2); §6 (p. 10)
EAM6 — W2No Method section; Figure 1 described by colors§3 L151–210; Figure 1 caption L136§3, Eqs. 1–5, Figure 1 (pp. 3–4); §4 (pp. 5–6)
EAM6 — W3Results deferred to the appendixTable 1; Figures 2–3; Supp. App. A–XTables 1–3 (pp. 7–9); App. A–F (pp. 15–28)
EAM6 — Q1Did the cross-model experiments fail?L281Table 2 (p. 8); App. C.4 Table 11 (p. 20)
EAM6 — Q2Editing note at L500–501L500–501Deleted
EAM6 — EthicsLLM use; verifiability of the numbersL479–548; Supp. App. XStatements (p. 11); App. B Tables 4–6 (pp. 16–17)
37rS — W1Cause of the negative dialogue result untested§4.2–§4.3 L284–436§5.3 (pp. 8–10); App. D.1–D.3 Tables 13–15 (pp. 20–22)
37rS — W2, Q1What the LLM adds over cosine; CAM++ on dialogue; standard pooler; sorting§3.2 L171–183; §4.1 L246–264; Supp. App. Q, STable 2 (p. 8); Table 3 (p. 9); §5.3 (pp. 8–10); §6 (p. 10); App. A.1 (p. 15); App. C.1 Table 7 (p. 18); App. E (pp. 22–25)
37rS — W3Forced choice with n = 4; N = 40/21; 55.0% at 64 speakersTable 1; Figure 3; L266, L408; Supp. App. VTable 1 (p. 7); Table 3 (p. 9); App. C.2 Table 8 (p. 18); App. D.1, D.3 (pp. 20–22); App. F.3 Table 23 (p. 28)
37rS — W358.3% zero-shot on AliMeeting; one model familyAbstract L19; §4.1 L250–283; §5 L449–451Table 2 (p. 8); App. C.2 Table 9 (p. 18); App. C.4 Tables 11–12 (p. 20)
37rS — SummaryFalse acceptance of unenrolled speakers; content leakage§5 L443–445; §4.3 L414–417; Supp. App. G, K§5.3 (pp. 8–10); Ethics Statement (p. 11); App. F.1–F.2 Tables 21–22 (pp. 26–27)
AuthorsOpenReview ↗

Title: Minor corrections in the updated PDF

We uploaded a revised PDF. The changes are small and do not affect any conclusion:

  • Section 5.3: Token-Soft is above CAM++ clustering under identical SFT without the synthetic curriculum (.799 vs. .763, Appendix E.2, Table 17). The previous version said "identical GRPO training", which no table supports.
  • Section 4.2: the synthetic curriculum is also used for the SpeakerBind Token-Hybrid adapter, as Table 3 already marks.
  • Table 2 and Appendix Table 11: the Qwen2.5 gold-speaker and text-ceiling references are on the held-out split; the labels now say so.
  • LLM-use statement: wording made more precise (recorded audio is public; SpeakerBind and SpeakerSleuth voices are synthesized).

One change from the submitted version that the general response above did not mention: the title now ends with "Conversational Routing and Attribution" instead of "Conversational Memory Routing". The main results now cover two task types: routing a query to an enrolled speaker (Tables 1 and 2), and attributing statements to speakers in multi-party dialogue (M3-SLU Task 2 and SpeakerBind, Table 3), which Reviewers 37rS and 7PhW asked us to compare with dedicated encoders. The single-session memory experiments of the submission are now diagnostics in Appendix D, so "Memory" no longer describes the main evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.