acceptodds
Under review as a conference paper at ICLR 2027

CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification

Abstract

Speaker verification systems achieve high accuracy but provide little account of the acoustic evidence behind their judgments. Making these systems inspectable requires exposing interpretable evidence while retaining the richer information on which their decisions depend. We present CoLMbo-SV, a speaker language model that combines strong speaker discrimination with structured, acoustically grounded comparison reports. By connecting a pretrained speaker encoder to a language model and supplying explicit acoustic measurements, CoLMbo-SV makes voice comparisons inspectable without restricting verification to the evidence verbalized in its reports. We additionally introduce VoxReason, paired recordings with measured acoustic properties and comparison reports filtered through numerical and qualitative checks, providing supervision for this combined capability. We also develop an evaluation framework that separates what acoustic information a speaker representation encodes, what influences the verification score, and what the generated report discusses. On VoxCeleb1-O, CoLMbo-SV achieves 0.99% EER, reducing verification error by approximately 80% relative to the strongest audio-language baseline fine-tuned on VoxReason, while attaining a numerical-grounding score of 0.82. Our analysis further demonstrates that acoustic correctness and decision relevance are distinct properties of an explanation, exposing a gap that numerical-grounding metrics miss. Together, these contributions substantially advance audio-language speaker verification, bring its accuracy toward that of dedicated speaker encoders while adding checkable acoustic reporting, and establish an empirical framework for connecting natural-language explanations to the decisions they explain.

Primary area:
interpretability and explainable AI
Keywords:
Speaker Verification, Audio Language Models, Explainable AI, Evidence Grounding, Speech Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_N1TGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a way to use LLMs in speaker recognition. Rather than finetuning a speech-LLM to assess whether the pair of recordings in question comes from the same or different speakers, it uses SOTA speaker embedding extractors, their cosine similarity (and other embedding-based measures named "comparison feature"), a transformer mapper for the embeddings to map them to the LLM space, as well as 18 measurements of the two recordings and teaches the LLM to first provide an explanation of the differences between the two recordings. The decision is obtained by teacher-forcing the "same" or "different" tokens and forming the log-likelihood ratio (LLR) as their difference in the logit domain. To train the model, the authors created the VoxReason dataset, where "each example is a pair of recordings with four things attached: the two utterances, their extracted acoustic measurements, the speaker label, and a comparison report whose numerical claims have been checked against those measurements". An LLM (Gemini 2.5 Flash-Lite) writes each report (given the labels), and a numerical validator and an LLM judge filter these reports.

Critical strengths

(a) Interesting idea on using LLMs for the problem of speaker recognition. Also realistic, in the sense that the embeddings are extracted by a dedicated extractor, rather than computed by the LLM.
(b) Rich and informative ablations. The text vs soft tokens, and frozen vs. unfrozen verifier, provide useful information. (c) Honest reporting of negative results, showing that although the LLMs own reports are grounded, they are not decision-selective.

Critical weaknesses

(a) The explanations do not improve the overall performance in terms of EER, compared to using the embeddings and their cosine similarity. (b) The comparisons with the performance of other speech-LLMs are not convincing, since speech LLMs do not use a dedicated speaker embedding model. (c) The explanations are mainly verbalizations of the 18 acoustic measurements, which does not justify the use of such a pipeline, as it can be accomplished with other means (e.g. a more standard LLM). (d) In Figure 1, the explanations contain statistical tests, such as Fisher's method. I found not explanation about how are these tests are performed. Are pythonic tools called by the LLM? If not, then the LLM is performing these test using their own (inadequate) computational skills?

Questions for authors

(a) Why does the LM verdict underperforms the cosine similarity on the same encoder (0.99% vs 0.34% in EER)? Have you considered using the head for the verdict, and the LM for the report, given the verdict? (b) Are tools used to perform statistical test, such as Fisher's method?

6 comments

We thank the reviewer for the constructive feedback. We agree that an encoder-based cascade is an essential comparison and thank the reviewer for highlighting this point. Alongside verification accuracy, an important question is whether joint training produces explanations containing more speaker-discriminative information than reports generated separately by an LLM.

To investigate this, we used Gemma to obtain embeddings from the explanation text alone, after removing explicit verdicts and confidence labels. We then trained a binary classifier to predict the same-speaker/different-speaker label from these embeddings, using a five-way jackknife evaluation.

The classifier achieved an EER of 1.66% with CoLMbo-SV explanations, compared with 13.1% with Qwen explanations. These results show that CoLMbo-SV’s explanations contain substantially more information useful for distinguishing same-speaker from different-speaker pairs, even without their explicit conclusions. This supports the discriminative value of the explanations, although it does not by itself establish causal faithfulness to the verifier’s decision. We will include this analysis and its evaluation protocol in the revised manuscript.

Regarding the statistical tests, the results are computed using Python code; the LLM does not invoke external tools. We will clarify this computation procedure in the revised manuscript.

We hope these additional results and clarifications address the reviewer’s concerns and would appreciate their consideration of a higher score.

I thank the authors for their response. Some questions: (a) Regarding the new classifier experiment, which embedding model was used (if any), which text segments were kept, and how was the jackknife split performed (by speaker or by pair)? (b) Please also state how the statistical test sentence is produced (by the model itself, by Python code at inference). It is still very unclear to me.

We thank the reviewer for responding!

The embeddings were generated by Gemma as metioned above. Also, as mentioned earlier, the entire text explanation minus the confidence scores and final verdicts was used. The jackknife was by speaker.

The statistical significance numbers are not computed by the LLM, but by a separate call to a library, the same one used to generate training labels.

Reviewer_xgkBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CoLMbo-SV, an explainable speaker verification system that combines a pretrained speaker encoder with a language model to produce verification decisions and structured and detailed reports. To support this task, the authors also construct VoxReason, a dataset of paired recordings with measured acoustic properties and validated reports. Experiments show that CoLMbo-SV substantially outperforms fine-tuned audio-language models in verification performance while generating numerically grounded reports. Importantly, the authors also comprehensively analyze whether the reported features actually correspond to those influencing the model's decisions.

Critical strengths
  1. The design of the combination of LLM and speaker encoder to generate explainable speaker verification results and reports is reasonable. This is a good attempt to leverage the powerful LLM backend for explainable speaker verification.
  2. Strong experimental results. Compared with directly fine-tuning audio LLM, the system achieves quite competitive performance.
  3. Comprehensive experimental validation. The paper goes beyond numerical grounding to investigate whether reports discuss features relevant to the model's decisions. The authors clearly acknowledge the limitations of the current model's feature-selection capability rather than overstating its explainability. This analysis provides useful insights for future work on faithful natural-language explanations.
Critical weaknesses
  1. On some evaluation protocols like SITW, the performance of CoLMbo-SV still lags behind dedicated speaker encoders to a large extent. For dedicated speaker encoders, more powerful scoring methods like PLDA.
  2. Presentation: some part of the right side of Figure 1 is not shown. Figure 2 contains too many characters, making it not concise enough.
Questions for authors

Did the authors investigate how the language model attends to the input acoustic attributes when they are provided as text? For example, could attention patterns across the generated verdict and report tokens reveal which acoustic attributes the model attends to when making verification decisions, and whether these patterns differ across speaker pairs?

1 comment

We thank the reviewer for their positive and constructive feedback. Following their suggestion, we have visualized attention to the input acoustic features during both report generation and generation of the final verdict. For feature-specific clauses, attention is highest on the feature being discussed. During the generation of the final verdict, some features receive greater attention than others. Qualitative inspection suggests that these more highly attended features correspond to acoustically prominent properties of the recordings. We present these observations as qualitative attention diagnostics; they do not establish causal feature importance. We will include these visualizations and their discussion in the revised manuscript. We are unable to attach the figures here because the response interface does not support figure attachments.

We hope these additional analyses address the reviewer’s question and would appreciate their consideration of a higher score in light of these additions.

Reviewer_99o9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoLMbo-SV, a system that combines speaker verification with structured acoustic comparison reports. The architecture connects a frozen speaker encoder to a language model through a learned mapper, introduces a raw-embedding comparison bypass, and supplies explicitly extracted acoustic measurements as text. The authors also construct VoxReason, a dataset of paired recordings with acoustic measurements and synthetically generated comparison reports, filtered using numerical validation and an LLM judge. The system achieves 0.99% EER on VoxCeleb1-O, 3.39% on SITW, and a claim-level numerical-grounding score of 0.82. A further analysis separates concept decodability, decision sensitivity, and report content. It finds that the reports show little pair-specific selectivity for the concepts influencing the verification decision, despite their relatively high numerical grounding.

Critical strengths
  1. The problem formulation is clear and worthwhile. The paper distinguishes whether a report describes the recordings correctly from whether it explains the verifier’s decision. Bringing this distinction into speaker verification is useful, particularly when fluent acoustic descriptions can otherwise be mistaken for faithful explanations.

  2. The ablations provide useful engineering insight. In Table 3, supplying measurements as text improves numerical grounding substantially more than encoding the same measurements as soft tokens. Together with the failed variant in Appendix E, this illustrates that making information recoverable from an intermediate representation does not ensure that the language model uses it.

  3. The implementation and data-construction procedures are described in reasonable detail, including the training stages, token weighting, decoding controls, annotation pipeline, numerical-validation tolerance, and filtering yield. VoxReason is a potentially useful resource for studying comparative acoustic reporting, even though it does not provide supervision for decision-faithful explanations.

Critical weaknesses
  1. The key baseline for establishing the system’s added value is missing. CoLMbo-SV has access to a dedicated speaker encoder and a bypass containing the original embeddings and cosine similarity. The audio-language baselines do not receive equivalent speaker evidence, so the reported 80% error reduction over Kimi-Audio does not isolate the contribution of the proposed coupling. The frozen encoder itself achieves 0.34%/0.87% EER on VoxCeleb1-O/SITW, compared with 0.99%/3.39% for CoLMbo-SV. A direct comparison with an encoder-based verifier followed by a template or measurement-conditioned report generator is essential: it would show whether the joint architecture adds anything beyond a language interface at a cost in verification accuracy. The verification head’s standalone EER should also be reported. I would not assume the cascade’s report quality without testing it, but it is the most relevant missing comparison.

  2. The numerical-grounding comparison needs to reflect the actual information available to each system. CoLMbo-SV receives approximately 400 tokens containing the extracted measurements, whereas the main baselines are denied these values. Calling the former an internal measurement channel does not remove the language model’s access to the numbers. The measurement-in-prompt Qwen2-Audio control reaches 0.823 grounding, essentially the same as CoLMbo-SV’s 0.820, although its 71.67% unknown rate shows that adding measurements alone does not solve the joint task. The comparison should therefore include baselines trained with the same measurement access and separate accurate transcription of supplied values from inference of acoustic properties. The remaining errors also need analysis: only 0.2% of CoLMbo-SV reports are perfectly grounded despite the values being available in the input. How much of the error comes from derived statistics, attribution, or invented values?

  3. The decision-grounding analysis leaves the text-measurement channel unexamined. The verdict reads both the embedding-derived prefix and the explicit measurements, but the sensitivity analysis differentiates with respect to the speaker embeddings. Its concept rankings therefore describe the embedding path conditional on fixed measurement text, rather than all of the evidence available to the verdict. Perturbing or exchanging individual text measurements would directly test this omitted path. The concept directions may also capture correlated properties, including speaker sex, as the authors acknowledge. The analysis is a useful sensitivity diagnostic, but it does not yet establish causal feature use. Uncertainty estimates would help interpret the small DG@3 lift of 0.027; the aggregate DG of 0.835 is only slightly above its random-feature baseline of 0.812.

  4. The training data do not connect reports to the verifier’s own reasoning, which limits the explanation contribution. VoxReason reports are generated from speaker labels and measurements, not from evidence used by CoLMbo-SV’s decision function. Learning a largely fixed reporting template is therefore an understandable outcome of the supervision. I appreciate that the authors explicitly identify this problem, but the proposed system does not resolve it. The paper would be stronger if it either demonstrated a way to tie report supervision to pair-specific decision evidence or positioned its main contribution more consistently as a diagnostic study. The unevaluated statistical-strength sentences are a related concern: combining measurement-based p-values and mapping them to identity/confidence labels does not establish calibrated evidential strength, and their relationship to the actual verification decision needs clarification.

  5. The relationship between verification scores and generated conclusions is insufficiently clear. EER is computed from teacher-forced likelihoods of two candidate conclusions, whereas strict accuracy evaluates a generated conclusion. These are different protocols, so 0.99% EER and 94% strict accuracy are not inherently contradictory, but their relationship matters for a system presented as producing a decision together with a report. The conditioning context for the likelihood calculation, agreement with the generated verdict and verification head, and consistency between the report’s evidence and conclusion should be specified. The 77.1% TIMIT accuracy also needs to be related clearly to the main checkpoint and protocol. These clarifications would make the reported performance much easier to interpret.

Questions for authors
  1. What is the standalone EER of the verification head? Do you have results for a cascade using the frozen encoder’s score for the decision and the same measurements for report generation, or baselines given equivalent speaker evidence?

  2. Was the measurement-in-prompt baseline trained with that input format? Please clarify its training conditions and the types of numerical errors remaining in CoLMbo-SV when measurements are already supplied.

  3. Does the validator bind each value to its feature, utterance, unit, and comparison direction? How are differences and derived statistics treated? Please describe the voice-quality descriptor extraction and any existing validation against manually checked measurements.

  4. Is measurement text held fixed in the concept-sensitivity analysis? Have you already tested perturbing or exchanging this text channel, and are confidence intervals available for DG@3 lift? Please distinguish existing results from proposed future experiments.

  5. What precedes the teacher-forced candidate conclusions used for EER: only the input prefix, a generated report, or a reference report? What is their agreement with the freely generated verdict and the verification head? How does the TIMIT-trained checkpoint relate to the main system?

  6. Were any report targets constructed from the verifier’s pair-specific influential concepts? Please also clarify how the statistical-strength sentences relate to the verifier’s decision and what, if any, calibration has been evaluated.

1 comment

We thank the reviewer for their detailed and constructive feedback. We address each concern below.

R1.1: Missing cascade baseline and standalone verification results We agree that an encoder-based cascade is essential. To compare report content, we embedded explanations using Gemma after removing verdicts and confidence labels, then trained a same/different-speaker classifier with five-way jackknife evaluation. EER was 1.66% for CoLMbo-SV versus 13.1% for Qwen. These results show that CoLMbo-SV’s explanations contain substantially more information useful for distinguishing same-speaker from different-speaker pairs, even without their explicit conclusions. This supports the discriminative value of the explanations. We will include this analysis in the revised manuscript.

R1.2: Training conditions, measurement access, and numerical errors Table 3 shows that measurements improve grounding; audio-only CoLMbo-SV also outperforms Qwen. These ablations do not replace baselines trained with equivalent measurement access. Qwen was trained on our audio-only ablation’s manifest, with measurements added only at inference. This format mismatch limits interpretation of its 71.67% unknown rate. We moved the control to the appendix with explicit training/test conditions. The completed 600-report grounding score is 0.734, replacing the preliminary 97-report estimate mentioned earlier in the caption. Among 9,622 unmatched numbers out of 54,064 cited, 62.0% were differences or ratios of supplied values; 11.2% (2.0% of all cited numbers) lacked an identifiable input source. Other cases included omitted signs and numbering artifacts. We added this breakdown to clarify the metric’s limitations.

R1.3: Validator binding and voice-quality extraction The validator matches supplied values within 0.5 absolute or 5% relative tolerance, without enforcing feature–utterance–unit or direction binding, or recomputing derived statistics. A separate analysis found correct ordering in 73.6% of 4,543 directional claims. Voice-quality descriptors follow Dhamyal and Singh (2024), using acoustic primitives including cepstral peak prominence, HNR, jitter, and shimmer. We added the paper and implementation references.

R1.4: Supervision provided by VoxReason We agree that statistical relevance and pair-specific decision faithfulness are different requirements. Our analyses establish statistical relationships between acoustic features and match/mismatch outcomes. However, we have not counterfactually tested whether changing an individual report clause changes the verdict for that particular pair. For example, pitch may be statistically associated with match/mismatch outcomes without a pitch-related clause explaining a specific prediction. We will clarify that VoxReason provides measurement-grounded comparison supervision, rather than supervision derived from the verifier’s pair-specific influential concepts.

R1.5: Measurement-text sensitivity and uncertainty in DG Yes, TCAV differentiates with respect to embeddings while holding measurement text fixed. On 3,000 pairs, teacher-forced verdict flips were 0.6% when deleting the block, 0.4% when replacing utterance 2’s values with utterance 1’s, 0.1% with another pair’s values, 0.0% when doubling mean F0, and 0.1% when swapping columns. This indicates limited text sensitivity under these perturbations, consistent with primary reliance on embeddings. Correlated concept directions remain a limitation; TCAV is a sensitivity diagnostic, not causal proof. Bootstrap intervals for DG@3 lift are +0.027 [−0.003, +0.057] for CoLMbo-SV and +0.092 [+0.056, +0.127] for Qwen FT. CoLMbo-SV’s interval includes zero, reinforcing our negative finding on pair-specific decision selectivity.

R1.6: Context and agreement of verification outputs Teacher-forced conclusions follow the audio prefix and a fixed neutral prompt, without measurements or a generated/reference report. Agreement is 99.4% between the head and teacher-forced verdicts, 99.3% between teacher-forced verdicts with/without measurements, and 95.3–95.7% between generated verdicts and these alternatives. The 77.1% TIMIT accuracy comes from a separate TIMIT-only checkpoint used to test the earlier pipeline before scaling up. We will clarify these protocols and the checkpoint distinction.

R1.7: Report targets and statistical-strength statements No targets used the verifier’s pair-specific influential concepts. Clause-supervision experiments are ongoing and are not claimed as evidence here. The statistical-strength statements were not calibrated. The Python-computed Fisher test alone agreed with ground truth on 55.5% of TIMIT pairs and does not determine the main verdict. We will describe its outputs as descriptive statistics, rather than calibrated identity confidence or likelihood ratios.

We hope these additions address the reviewer’s concerns and would appreciate their consideration of a higher score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript uploaded

Dear AC and Reviewers,

We have uploaded the revised manuscript incorporating the additional analyses and clarifications discussed in our rebuttal.

Thank you for your constructive feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.