acceptodds
Under review as a conference paper at ICLR 2027

InterBias-SV: A Factorial Benchmark and Validity Criteria for Compound-Stressor Robustness in Speaker Verification

Abstract

Speaker verification systems encounter combinations of noise, channel distortion, and changes in speech. Evaluating each condition separately does not establish whether their effects add. InterBias-SV organises this question around a four-term comparison: joint error, two marginal errors, and a common reference. Its results artefact contains 4068 scored records across 17 experiments, 12 encoder labels, and six speech corpora, totalling trial evaluations. Three experiment families contain the same-corpus terms needed to compute additive contrasts. For labels assigned to speaker-trained encoders, their mean contrasts are , , and in equal error rate (EER), with larger variation across settings. These descriptive averages do not establish equivalence to additivity: trial matching, checkpoint identity, and parts of the condition metadata remain unverified. We also examine two interpretation problems. Near-chance EER can make additive predictions difficult to interpret, but chance performance is not a hard EER ceiling, and correlation with the prediction does not identify a saturation mechanism. Ratios of demographic gaps are unstable when their clean reference is near zero; absolute gaps provide a more direct summary. The benchmark provides condition definitions, analysis scripts, and explicit requirements for interpretable compound-condition comparisons, while separating recomputable summaries from claims that require further experimental validation.

Primary area:
datasets and benchmarks
Keywords:
speaker verification, robustness evaluation, speech benchmarks, compound conditions, interaction effects, demographic bias

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KnQgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a factorial framework for evaluating speaker verification under pairs of stressors. It compares the joint-condition EER with two single-condition EERs and a shared reference, and discusses when this contrast is interpretable. The authors provide a substantial collection of scored results across systems and conditions, while carefully distinguishing recomputable descriptive summaries from unverified trial matching, checkpoint identity, and condition metadata. They also explain why near-chance EER does not establish a hard performance ceiling and why ratios of demographic gaps can be unstable.

Critical strengths
  1. The compound-stressor question is practically important and often obscured by single-condition evaluation. The four-term contrast and its interpretation criteria give readers a useful way to inspect interactions rather than assuming that marginal degradations predict joint degradation.
  2. The authors are appropriately careful about inference. In particular, they do not equate small mean contrasts with equivalence to additivity, and they explicitly identify limitations in provenance and matching. That honesty makes the benchmark easier to assess.
Critical weaknesses
  1. The strongest claimed use of a factorial benchmark depends on matched, traceable conditions, but only three of the 17 experiment families contain the required four terms. For those families, the released aggregates do not include trial identifiers or per-trial scores, so readers cannot verify that the terms use matched speakers and trials. The E13 age stratum also overlaps its all-age reference, and the temporal condition in E17 needs clearer provenance. These are consequential validity limitations, not merely missing documentation.
  2. The resource's breadth is difficult to interpret as a reproducible benchmark when checkpoint identities and parts of the legacy scoring pipeline are unresolved. Since the factorial contrast is a standard calculation and no new verification model is proposed, the paper's contribution depends heavily on the trustworthiness and reuse value of its experimental artifacts. At present, the unresolved provenance narrows that contribution substantially.
Questions for authors
  1. Can the authors provide trial-level manifests or scores for E13, E14, and E17 that establish speaker, utterance, and trial matching across all four terms, together with a non-overlapping reference for E13 where appropriate?
  2. Which exact encoder checkpoints and scoring scripts generated each reported experiment family, and which results can be regenerated from the released artifacts today? A clear mapping would materially affect my assessment of the benchmark's validity.
Reviewer_WpbkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The "InterBias-SV" paper investigates the effect of combining two degrading conditions (like noise and phone codecs) in speaker verification (SV) systems. Using an "Interaction Effect" metric to compare actual combined errors against predicted added errors across 12 encoders, the study reveals:

  • Interaction Variability: Speaker-trained models show small average interaction effects but massive variations depending on the specific setup.

  • Model Performance: CAM++ and ECAPA-TDNN are the top-performing encoders. Self-supervised (SSL) models perform drastically worse, hitting an error ceiling that makes their combined errors lower than the predicted sum.

  • Emotion Mismatch: Testing with mismatched emotions (e.g., enrolling with neutral but testing with fearful audio) drastically degrades accuracy, increasing errors by over three times.

  • Gender Gaps: Performance discrepancies between genders widen in noisy environments, though the exact ratios are volatile due to near-zero baseline gaps in clean audio.

Critical strengths
  • The paper clearly outlines the data required to prove non-additivity by establishing a strict four-part comparison (the combined condition, both individual conditions, and a clean baseline). Additionally, its four validity rules (C1–C4)—which ensure consistent populations, trials, metrics, and measured terms—provide a practical framework for speaker recognition, and possibly other speech-related tasks.

  • The paper corrects several statistical traps that researchers typically ignore. Specifically: "chance" error rates aren't strict ceilings, correlating a metric with its own underlying variables forces a false mathematical relationship, and calculating ratios using denominators close to zero may produce unreliable data.

  • The paper evaluates an 12 encoders, including both speaker-trained models (e.g., X-Vector, ECAPA-TDNN, CAM++) and frozen self-supervised learning representations (e.g., WavLM, HuBERT). This ensures that the interaction effects found in the paper reflect broader trends rather that random behaviours related to specific architectures and training details (e.g., data augmentation).

Critical weaknesses
  • Inadequate Statistical Controls and Sampling Transparency: As the authors admit, the benchmark relies solely on aggregated measurements instead of individual, per-trial score lists, which prevents it from accurately calculating sampling uncertainty. The experiments reuse the same speakers and shared reference terms, making the test results statistically dependent on one another. Treating them as independent replicates ignores this dependence. Furthermore, because the number of evaluation trials varies across conditions and the specific trial identities are not tracked, simply using the same underlying speech database does not guarantee that the evaluated populations actually match in speaker demographics or audio content. Finally, the lack of independent control over the random seeds (used in trial and acoustic augmentation selection) prevents researchers from testing the sensitivity of the models to those specific random choices.

  • Limited Novelty and Absence of Algorithmic Mitigation: The paper's contribution relies on applying a standard factorial contrast rather than introducing a novel statistical concept. The study does not introduce new encoder architectures or newly collected datasets, and relies exclusively on existing models and corpora (e.g RAVDESS, Common Voice, and VoxCeleb). While diagnosing compound condition errors is practically useful, the paper stops at evaluation. It does not propose any novel training paradigms, loss functions, or architectural modifications to mitigate the identified non-linear degradation, which are important for a venue like ICLR.

Questions for authors

Since the paper argues that comparable trial sampling and strict condition provenance are essential for valid comparisons, do you plan to release the exact producing code and per-trial score manifests to align the benchmark artifact with your own methodological standards?

Reviewer_nZAiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a technique to evaluate the impact of domain shifts in speaker verification, such as emotions or noises.

Critical strengths

The grammar is correct and the graphs are easily read.

Critical weaknesses

The novelty of the paper is limited, as the evaluation of domain dependant biases in ASV is well studied in the litterature. The main critical weakness comes from the methodology and presentation: many formulations show hint that the paper may have been mostly written by generative AI. Non nonsensical sentences such as: "Chance EER is not a hard ceiling, and a near-zero demographic reference makes gap ratios unstable.", sentences built like a model was tasked to write an article from a github repository: "The records name..." "The artefact includes speaker counts for some records, but does not provide complete corpus-release, audio-duration, or sampling manifests." "Twelve model labels appear. Seven are assigned to speaker encoder" etc...

Questions for authors

Have you written the article yourself? Could you add dataset dependant results for the EER? it is very hard to read EER results not in percentages, especially when mixed between traditional ASV datasets and other datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.