acceptodds
Under review as a conference paper at ICLR 2027

VoiceDiff: Explaining Voice Differences with Speech Language Models

Abstract

As real-time spoken dialogue and multi-party assistants expand voice-based interaction, speech language models need to understand information about speakers beyond verbal content and use it in their responses. Because one person can sound different across utterances and different people can share similar characteristics, voice similarity must be distinguished from speaker identity. We introduce Voice Difference Explanation, a task covering speaker profiles such as gender and age, voice characteristics such as pitch and timbral brightness, and recording conditions such as noise and reverberation. It connects individual voice judgments to comparisons of shared and differing attributes, integrating them into a natural-language response alongside a separate identity verdict. Public speech metadata, acoustic measurements, and controlled recording conditions provide training targets and VoiceDiff-Bench. To study useful representations for these judgments, we introduce VoiceDiff, a reference model whose hierarchical speaker tokenizer combines a frozen speaker encoder's pooled speaker embedding with temporally resolved frame-level features. Task-specific supervision improves general Audio-LLMs on multiple voice judgments, while VoiceDiff achieves stronger pitch, timbral-brightness, and identity results than the evaluated task-adapted general models. At a matched audio-token budget, speaker embeddings favor identity judgments, frame-level features favor detailed acoustic judgments, and their combination supports both. Relation-description training integrates comparisons, summaries, and identity into one response with category-summary accuracy generally close to rule-based composition of individual outputs. Overall conclusions largely follow the comparisons stated in the response; error analysis identifies constituent comparison accuracy as a key basis for more accurate relation descriptions. The benefits of task-specific supervision also extend to selected external cases, unseen during task training, in which spoken claims conflict with vocal cues. These findings support task-specific supervision and complementary speech representations for grounding linguistic judgments and explanations in the voice itself.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
voice difference explanation, speech language models, audio-language modeling, speaker representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mLKCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Voice Difference Explanation (VoiceDiff-Bench), a benchmark built from VoxCeleb1 and LibriTTS-R/P to evaluate how Speech LLMs compare voices across speaker profile (gender, age, region), voice traits (pitch, brightness), and recording conditions (noise, reverb), while separating voice similarity from speaker identity. It also proposes VoiceDiff, which connects a frozen ReDimNet-B2 speaker encoder to Qwen2.5-1.5B-Instruct using a hierarchical tokenizer (MLP for pooled embeddings + Q-Former for frame features). Experiments show that pooled features help with speaker identity, frame features help with acoustic details, and combining them improves both while transferring well to VoxParadox.

Critical strengths
  1. Clean representation ablation: Using a fixed 48-token budget on the same frozen encoder, the paper clearly shows that pooled embeddings are better for speaker identity (97.2%), frame-level features are better for acoustic traits (pitch and brightness), and combining both (16+32 tokens) gives the best of both worlds.
  2. Good transfer to VoxParadox: Training on voice comparisons helps Audio-LLMs rely on actual vocal cues instead of being fooled by conflicting spoken text (e.g., boosting Qwen2.5-Omni gender accuracy from 0.5% to 99.0%).
Critical weaknesses
  1. Synthetic, rule-based labels instead of human perception: All training and test targets come from signal-processing thresholds (median F0, spectral centroid), country metadata (as a proxy for accent), and a fixed linear formula (Q=4g+a+r+p+bQ = 4g + a + r + p + b) filled into rigid sentence templates. Without human perceptual labels, the task mostly tests slot-filling over heuristic bins rather than true natural-language voice explanation.
  2. Stage 2 adds complexity with little gain, and joint evaluation is weak: In Table 2a, complex Stage 2 training performs no better than simply applying the deterministic rule to Stage 1 outputs (Stage 1 + Rule: 66.07% vs. 64.81% on voice traits). Also, because no single dataset has clean labels for all attributes, full 5-attribute evaluation relies on noisy VoxCeleb acoustic estimates and reaches only 25.40% exact match.
Questions for authors
  1. Human alignment: Did you run any human listening tests to check if your spectral centroid cutoffs and linear formula (Q=4g+a+r+p+bQ = 4g + a + r + p + b) match how humans perceive brightness and overall voice similarity?
  2. Backbone vs. tokenizer: VoiceDiff uses ReDimNet-B2 (pre-trained for speaker verification), while the fine-tuned baselines use general audio encoders. How much of the performance gain comes from the ReDimNet-B2 encoder itself versus your hierarchical tokenizer?
  3. Free-form prompts: Since Stage 2 is trained on rigid sentence templates, how well does the model handle free-form, open-ended user questions about voice differences?
9 comments

W1 + Q1 — Human listening supports the acoustic distinctions and relation descriptions

We thank Reviewer mLKC for recognizing the complementary representation findings and transfer to VoxParadox as strengths of the work. The reviewer raises the concern that heuristic labels and templated targets may assess slot filling rather than perceptually meaningful voice explanation (W1). The requested listening tests focus on whether spectral-centroid cutoffs and the overall-relation policy correspond to human perception (Q1). To address both points, we conducted a listening study with 14 participants completing 18 items each, yielding 252 responses per component, and evaluated acoustic distinctions, the relation policy, and actual Stage 2 descriptions separately.

Section 2.1 and Appendix A distinguish corpus metadata, acoustic measurements, and augmentation settings as target sources; templates standardize their reporting. The listening study tests the acoustic and relational components, not metadata agreement as perceptual ground truth.

Table H1. Human listening results. A/B use natural LibriTTS-R recordings. C/D use the same 18 LibriTTS-R pairs from 24 speakers and actual Stage 2 responses, with six original, six noise-added, and six reverberant pairs.

ComponentListener taskResult
A: PitchIdentify pitch direction between two voices90.48% agreement (228/252)
B: BrightnessRank five voices, one from each benchmark binMean Spearman ρ=0.7599\rho=\mathbf{0.7599}
C: Relation policySelect an overall relation given attribute comparisons84.92% agreement (214/252)
D: Generated descriptionRate how well the description fits the audio, from 1 to 54.51 ± 0.54 / 5

The SD in D is across individual ratings, not participant means. Of 252 ratings, 247 (98.02%) were 4 or 5; the remaining five were 3.

(a) [W1, Q1] Acoustic measurements track the differences listeners hear

The reviewer asks whether spectral-centroid cutoffs correspond to perceived brightness. A and B assessed distinct cues: A used clearly F0-separated, same-gender pairs with closely matched spectral centroids; B required a full ranking of five natural recordings, one from each fixed centroid bin, with gender held fixed and median F0 closely matched. A short auditory tutorial introduced pitch and brightness without teaching numerical cutoffs.

Listeners agreed with the F0-based pitch direction in 90.48% of responses. For brightness, the mean Spearman ρ=0.7599\rho=0.7599 indicates substantial correspondence between the measured and perceived orderings. Holding gender fixed and limiting pitch variation reduces the possibility that the brightness rankings merely follow those other cues. The results therefore support a perceptually meaningful ordering, not just reproducible numerical grouping. They do not establish each cutoff as a perceptual boundary or demonstrate discrimination between every adjacent bin.

(b) [W1, Q1] Attribute-informed relation choices agree with the policy

The reviewer also questions the perceptual basis of the linear aggregation policy. In C, listeners took the displayed Stage 2 attribute comparisons as given and selected the most appropriate of the three overall-relation categories; "none" and "unable" options were also available. The 18 pairs covered six cases from each policy category. The policy weights were not disclosed to participants.

Listener choices agreed with the policy in 84.92% of responses. This agreement supports the acceptability of the attribute-based relation summary within the presented comparison-and-description context. The result provides perceptual support for the policy's use in the explanation interface, rather than establishing a universal voice-similarity scale.

(c) [W1] The generated descriptions fit the audio according to listeners

The reviewer's concern extends from the targets to the explanations themselves. D therefore evaluated the actual full Stage 2 responses for the same audio pairs, with instructions to judge audio–description fit rather than writing fluency. The mean rating of 4.51/5, with 98.02% rated 4 or 5, provides direct evidence that listeners accepted the descriptions as compatible with the audio, rather than only consistent with an automatic scoring rule.

Together, these results connect the measurement-based acoustic ordering, the attribute-informed relation policy, and the model's generated descriptions to distinct listening judgments. The task is therefore supported by perceptual evidence in addition to agreement with automatically constructed targets. Reviewer mLKC's suggestion enabled us to test these connections explicitly and substantiate the task's empirical basis more clearly. We thank the reviewer for prompting this evaluation.

W2 (1/2) — Stage 2 improves the reporting of shared and differing profile attributes

We thank Reviewer mLKC for recognizing the complementary representation findings and transfer to VoxParadox. We address the concern that Stage 2 adds complexity with little gain (W2) by comparing it with continued Stage 1 training and examining which relations improve.

In submitted Table 2(a), voice-characteristic summary accuracy was 66.07% for Stage 1 + Rule and 64.81% for Stage 2. We extended this comparison to three training runs per condition and added a Stage 1 continued-training baseline, which trains on the original Stage 1 tasks for the same additional training duration as Stage 2. The original results and the additional run averages are reported separately.

(a) [W2] Profile-summary accuracy improves beyond continued Stage 1 training

Both Stage 1 baselines apply the fixed task rules to their predicted comparisons; Stage 2 generates the comparisons and summaries in one response. All models receive audio and a task question, not reference attribute values.

Table R3(a). Category-summary and identity accuracy (%), mean ± sample SD across three training runs. Profile/Identity use 1,000 pairs; Voice, 1,998; Recording, 1,500. Bold marks the highest mean, including ties.

ConditionProfileVoice char.RecordingIdentity
Stage 1 + Rule64.27 ± 1.1965.08 ± 0.9153.87 ± 0.5098.10 ± 0.44
Stage 1 continued training + Rule64.80 ± 0.4064.91 ± 0.4053.80 ± 0.4098.40 ± 0.20
Stage 268.20 ± 1.0865.73 ± 0.8053.96 ± 0.7798.53 ± 0.12

The clearest gain is in profile summaries: Stage 2 improves mean accuracy by 3.93 percentage points over Stage 1 + Rule and 3.40 points over Stage 1 continued training + Rule. Under Appendix E.4, a correct profile summary must match all three profile comparisons and identify the correct set of differing attributes, including the direction and magnitude of age differences where applicable. The improvement therefore means that more responses correctly report all three profile comparisons and identify which attributes differ.

(b) [W2] The larger gains occur when profile attributes differ

To locate the improvement, we partition all 1,000 profile pairs by their benchmark comparisons of gender, age, and region. These are reference-defined groups, not examples selected by model success. Mixed profiles contain at least one shared and one differing attribute.

Table R5. Profile-summary accuracy (%), mean ± sample SD across three training runs. Scoring is identical to Table R3(a). Shared means same/similar under the task definition. Bold marks the highest mean within each row.

Profile compositionNStage 1 + RuleStage 1 continued training + RuleStage 2
All shared56084.29 ± 1.3985.00 ± 0.3685.54 ± 0.78
Shared + differing33337.14 ± 1.0538.14 ± 0.3044.14 ± 1.80
All differing10743.93 ± 0.9342.06 ± 0.9352.34 ± 0.93
All pairs100064.27 ± 1.1964.80 ± 0.4068.20 ± 1.08

Relative to continued Stage 1 training, Stage 2 improves mixed profiles by 6.00 points and all-differing profiles by 10.28 points, compared with 0.54 points on all-shared profiles. In the mixed group, a response must preserve the shared attributes while identifying the differing ones correctly. The gain therefore concerns the specific combination of commonalities and differences that the relation-description task is designed to express.

Finding. The additional value of Stage 2 is clearest in accurately reporting profiles that contain attribute-specific differences; continuing the original Stage 1 tasks does not reproduce the same profile-summary accuracy. We thank Reviewer mLKC for prompting a more controlled and detailed evaluation of this contribution.

W2 (2/2) — Stage 2 increases the proportion of responses with at least four correct comparisons

Reviewer mLKC also highlights the 25.40% all-five exact accuracy and the use of weak VoxCeleb acoustic labels (W2). The submitted Table 3 result requires all five attribute comparisons to be correct simultaneously, including direction and magnitude where applicable; it is not the average accuracy of the five comparisons. We retain this criterion and evaluate the Rule baselines under the same conditions.

(c) [W2] All conditions are evaluated under the original five-comparison criteria

Table R4. Five-attribute comparison accuracy (%), mean ± sample SD across three training runs. All conditions use the same 996-pair subset. The attributes are gender, age, region, pitch, and brightness. Exact requires the full comparison, including direction and magnitude where applicable. Dir. requires direction or similarity; Adj. allows one signed-category step. Gender and region remain exact under all criteria. Thresholds ≥3/≥4/all 5 are cumulative. Bold marks the highest mean, including ties.

ConditionExact ≥3Exact ≥4Exact all 5Dir. ≥3Dir. ≥4Dir. all 5Adj. ≥3Adj. ≥4Adj. all 5
Stage 1 + Rule93.57 ± 0.6169.01 ± 0.4225.90 ± 0.7097.12 ± 1.0381.93 ± 0.7037.92 ± 0.5799.73 ± 0.2396.32 ± 0.6570.35 ± 3.67
Stage 1 continued training + Rule93.51 ± 0.5969.21 ± 0.2926.37 ± 1.0397.52 ± 0.6781.76 ± 0.8537.99 ± 0.4299.77 ± 0.1595.65 ± 0.9570.38 ± 2.41
Stage 294.41 ± 0.6871.79 ± 0.6027.04 ± 1.4297.29 ± 0.2783.10 ± 0.5838.76 ± 1.1699.70 ± 0.1795.65 ± 0.5570.65 ± 3.97

Stage 2 increases the proportion of responses with at least four Exact comparisons to 71.79%, compared with 69.01% for Stage 1 + Rule and 69.21% for continued Stage 1 training: gains of 2.78 and 2.58 percentage points, respectively. Mean all-five Exact accuracy is 27.04%, compared with 25.90% and 26.37%.

Comparing the three criteria reveals a common error pattern: exact agreement across all five attributes is substantially harder than correct directions or agreement within a neighboring signed category. For Stage 2, all-five agreement rises from 27.04% Exact to 38.76% Direction and 70.65% Adjacent. These complementary criteria distinguish complete correctness from the precision of the remaining errors, while retaining Exact as the strict measure.

(d) [W2] The profile-summary finding does not depend on weak acoustic labels

Pitch and brightness in the joint subset remain measurement-based weak labels from natural VoxCeleb recordings. All three conditions are scored against the same fixed references, without treating them as independent perceptual annotations.

Importantly, the profile-summary improvements in Tables R3(a) and R5 do not depend on these pitch/brightness labels. They use the metadata-supported gender, age, and regional comparisons. The benefit identified in the first response therefore remains distinct from the limitations of the five-attribute acoustic-reference evaluation.

Together, the added comparisons show where Stage 2 improves factual reporting and where fine-grained comparison errors remain. They provide a more specific account of its value than either a single category score or all-five Exact accuracy alone. We thank Reviewer mLKC for motivating this clarification and the additional evaluation.

Reviewer_rwM1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Voice Difference Explanation, a task for understanding and comparing voice attributes while distinguishing voice similarity from speaker identity. The authors propose VoiceDiff, which combines pretrained speaker embeddings and frame-level features through a hierarchical tokenizer connected to an LLM. Experiments on VoiceDiff-Bench show improvements over task-adapted Audio-LLMs in pitch, timbral brightness, and speaker identity, while ablations demonstrate the complementary benefits of the two representations.

Critical strengths
  1. Meaningful task formulation. The paper introduces a structured framework for understanding and comparing voice attributes while explicitly distinguishing voice similarity from speaker identity. This addresses a relevant problem in speech-language understanding.

  2. Useful experimental analysis. The paper provides extensive comparisons and ablation studies. In particular, the finding that speaker embeddings and frame-level features offer complementary strengths for identity recognition and acoustic attribute understanding is interesting and supported by the controlled ablations.

Critical weaknesses

Limited significance and generalizability: While the proposed method is relatively straightforward, its performance advantages are demonstrated mainly on a narrowly defined set of voice attribute tasks. Outperforming general-purpose Audio-LLMs on these specialized benchmarks is not particularly convincing, given that VoiceDiff employs a dedicated pretrained speaker encoder and task-specific supervision. Stronger performance or broader evaluations would be needed to establish the practical significance of the proposed approach.

Questions for authors
  1. Generalizability: Given the relatively straightforward architecture and specialized evaluation tasks, can the authors demonstrate VoiceDiff's effectiveness on more diverse real-world speech data or a broader range of voice attributes? Such evidence would help establish the practical significance of the proposed approach beyond the current benchmark.
  2. Fairness of comparisons: Since VoiceDiff uses a pretrained speaker encoder specifically optimized for speaker recognition, while general Audio-LLMs use their native audio encoders, how much of the performance improvement can be attributed to the hierarchical tokenizer rather than the pretrained speaker representations? Could the authors provide a more controlled comparison?
  3. Dataset reliability: How reliable are the automatically generated attribute labels and voice similarity judgments? Have the authors conducted any human perceptual evaluation to verify that the benchmark labels and generated descriptions align with human judgments?
8 comments

W1 + Q1 — Task significance and generalization beyond the source benchmark

We thank Reviewer rwM1 for recognizing the task as meaningful and the complementary roles of speaker embeddings and frame features as a useful finding. The review questions whether improvements on specialized voice-attribute tasks establish practical significance, given the dedicated speaker encoder and task-specific supervision, and requests broader evaluation. We address these connected concerns through (1) the native-backbone fine-tuning comparisons in the submitted Table 1 and (2) new evaluations on independently collected read and spontaneous speech, without further adaptation.

(a) W1 — VoiceDiff supervision benefits general Audio-LLMs with their native encoders

To assess the task without adopting our speaker-based architecture, the submitted Table 1 includes Qwen2.5-Omni-7B, Kimi-Audio-7B, and Audio Flamingo 3 fine-tuned on VoiceDiff tasks while using their own audio backbones, without our speaker encoder or hierarchical tokenizer. For Kimi-Audio, age, regional-background, and pitch accuracy increase from 27.3/50.8/24.7% to 68.7/88.1/65.5%, respectively. Its age and regional-background accuracy also exceed our reference model's 62.3/68.5%.

Thus, the task-level evidence includes gains within general Audio-LLMs, rather than relying solely on our reference model outperforming them. We will label the fine-tuned rows "FT with VoiceDiff tasks" to make this distinction explicit.

(b) W1 + Q1 — External evaluations extend the existing in-the-wild benchmark

VoiceDiff-Bench already includes recordings from VoxCeleb1 [1], a widely used in-the-wild speaker-recognition corpus collected from YouTube interviews. Its source videos span studio interviews, red-carpet events, outdoor venues, and public speeches, with ambient conversations, audience reactions, simultaneous speakers, room reverberation, and varying recording devices and channels. The submitted evaluation therefore already includes a real-world speech domain. The additional experiments broaden corpus and speaking-style coverage beyond the benchmark's sources.

We evaluated the same submission checkpoints on VCTK [2] read speech (110 speakers, 660 recordings) and EARS [3] freeform monologues (103 speakers, 309 recordings). Neither corpus was used for VoiceDiff task training. All models use the same clips and labels, with the existing prompts, decoding, and label cutoffs unchanged and no external-corpus training. Profile labels come from corpus metadata; acoustic labels use the existing F0 and spectral-centroid measurements.

External attribute-prediction accuracy (%). Bold marks the highest accuracy within each corpus and attribute, including ties.

VCTK

ModelGenderAgeRegional background
Kimi-Audio-7B, zero-shot88.3350.9145.45
Kimi-Audio-7B, FT with VoiceDiff99.3960.1582.58
VoiceDiff reference99.3955.4569.09

EARS

ModelGenderAgePitchTimbral brightness
Kimi-Audio-7B, zero-shot92.2334.2236.5731.39
Kimi-Audio-7B, FT with VoiceDiff98.3861.7858.9045.95
VoiceDiff reference97.0959.1177.6748.54

EARS age is evaluated on 75 speakers/225 recordings whose metadata intervals fit entirely within one task age category; ambiguous intervals are excluded from age scoring only.

Kimi-Audio improves in all seven evaluated corpus–attribute combinations. Regional-background accuracy rises by 37.13 percentage points on VCTK, while age and pitch rise by 27.56 and 22.33 points on EARS, without adding a speaker-specific encoder or our tokenizer. Model rankings remain attribute-dependent: adapted Kimi leads on VCTK regional background and EARS age, while our reference leads on EARS pitch and brightness.

These experiments extend the existing in-the-wild evaluation to independent read and spontaneous speech corpora. Together with the native-backbone comparisons, they show that the task's benefits are not confined to our speaker-based architecture or the source benchmark. We thank Reviewer rwM1 for prompting this broader validation of the study's significance and generality.

References

[1] A. Nagrani, J. S. Chung, and A. Zisserman. VoxCeleb: A Large-Scale Speaker Identification Dataset. Interspeech, 2017.

[2] J. Yamagishi, C. Veaux, and K. MacDonald. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92). University of Edinburgh, 2019.

[3] J. Richter et al. EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation. Interspeech, 2024.

W1 + Q2 (1/2) — Representation selection matters within each pretrained speaker encoder

We thank Reviewer rwM1 for recognizing the complementary roles of the two representations and requesting a controlled comparison. The review asks how much of the improvement comes from the pretrained speaker encoder rather than the hierarchical tokenizer. To address this question and the related concern about the specialized backbone, we extended Table 1 and Appendix F.1 (Table 27) with (1) tokenizer comparisons within three frozen encoders and (2) capacity and pooled-representation controls. This comment reports the first analysis; the companion response presents the second.

(a) W1 + Q2 — Holding each encoder fixed isolates the tokenizer comparison

Table R1 repeats the embedding-only, frame-only, and hierarchical comparisons with ReDimNet-B2, ECAPA-TDNN [1], and WavLM-Large with an ECAPA-TDNN speaker head [2]. Within each encoder, the language model and Stage 1-1 training conditions are fixed; all tokenizers use 48 tokens. Bold marks the highest accuracy within each encoder and column, including ties.

Table R1(a). Attribute prediction and speaker identity accuracy (%). Single branches use 48 tokens; the hierarchical tokenizer uses 16 embedding tokens and 32 frame tokens. Adapter parameters are trainable parameters. Bright. = timbral brightness.

EncoderRepresentation / adapterAdapter params. (M)GenderAgeRegionPitchBright.NoiseReverb.Identity
ReDimNet-B2Embedding / MLP19.0098.7051.3360.6853.4744.1039.2034.4097.20
Frames / Q-Former18.2299.1063.0067.8078.2767.5072.2066.2794.90
Both (Ours, Hierarchical)21.1699.2562.3368.4781.4770.3072.8067.7398.40
ECAPA-TDNNEmbedding / MLP19.0099.1058.6758.6453.4750.3043.4037.4794.90
Frames / Q-Former19.2199.3562.6764.7578.8069.1070.2063.0794.50
Both (Ours, Hierarchical)22.1499.3561.3366.7880.4069.9071.4066.1396.70
WavLM-Large + ECAPA-TDNNEmbedding / MLP19.0299.5060.3371.5345.7348.3037.0025.2095.90
Frames / Q-Former18.4299.6566.6776.6176.4070.1077.8068.4095.60
Both (Ours, Hierarchical)21.3799.7564.0077.9778.1372.4079.0071.3398.40

Table R1(b). Attribute comparison accuracy (%). Gender and region use categorical agreement; the remaining attributes require the correct direction and magnitude.

EncoderRepresentation / adapterGenderAgeRegionPitchBright.NoiseReverb.
ReDimNet-B2Embedding / MLP100.0040.3383.2043.8728.5325.9022.80
Frames / Q-Former99.8043.0086.2069.8750.2747.3051.70
Both (Ours, Hierarchical)100.0046.0087.4070.1350.5347.6052.90
ECAPA-TDNNEmbedding / MLP100.0039.0078.2042.9331.4726.4033.30
Frames / Q-Former99.6040.3385.4070.5343.6048.4050.00
Both (Ours, Hierarchical)100.0048.6787.0072.8042.6751.6047.60
WavLM-Large + ECAPA-TDNNEmbedding / MLP99.8046.0082.8036.5331.2024.6024.80
Frames / Q-Former100.0049.0084.0059.4748.4046.8052.50
Both (Ours, Hierarchical)100.0050.6785.6075.2051.4747.4052.60

Across all three encoders, frames outperform embeddings on pitch and brightness prediction, while embeddings outperform frames on identity. Combining them achieves the highest accuracy on these three measures within every encoder. Pairwise gains vary by attribute, as shown in R1(b).

These differences cannot be explained solely by selecting a different pretrained encoder: the encoder is unchanged within each comparison. The results identify the contribution of exposing complementary representations to the language model, beyond choosing ReDimNet-B2 itself. We thank Reviewer rwM1 for motivating this broader, controlled test of the representation finding.

References

[1] B. Desplanques, J. Thienpondt, and K. Demuynck. ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Interspeech, 2020.

[2] S. Chen et al. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing, 2022.

W1 + Q2 (2/2) — Capacity and pooled-representation controls distinguish the tokenizer contribution

To further address Reviewer rwM1's question about encoder versus tokenizer contributions, we added two controls with the same frozen ReDimNet-B2 encoder: (1) single-branch adapters expanded to approximately 24M parameters and (2) a two-branch tokenizer using a temporal frame average instead of the speaker embedding. The language model, Stage 1-1 training conditions, and total 48-token budget remain fixed.

(b) W1 + Q2 — Controlling adapter capacity and the utterance-level input

The capacity control compares larger MLP and Q-Former adapters, rather than attributing differences to representation type without testing readout size. The pooled-representation control retains the Q-Former's unpooled frame input and changes only the type of vector supplied to the MLP: a temporal frame average or the encoder's final SV-trained speaker embedding, obtained through attentive statistics pooling and projection. This tests whether a generic summary of the frames reproduces the benefit of the higher-level speaker representation.

Table R2(a). Attribute prediction and identity accuracy (%). ReDimNet-B2 is fixed. EE = SV-trained speaker embedding; FF = unpooled frame sequence; Mean⁡(F)\operatorname{Mean}(F) = temporal frame mean. In two-branch rows, the first input goes to the MLP and the second to the Q-Former.

Tokenizer adapterInputAdapter params. (M)Tokens: MLP / QFGenderAgeRegionPitchBright.NoiseReverb.Identity
MLP — originalEE19.0048 / 098.7051.3360.6853.4744.1039.2034.4097.20
MLP — expandedEE24.0048 / 099.5561.0065.4261.2057.8048.2041.3397.80
Q-Former — originalFF18.220 / 4899.1063.0067.8078.2767.5072.2066.2794.90
Q-Former — expandedFF24.000 / 4899.6566.6766.4481.3369.5071.4066.6795.50
MLP + Q-Former — mean controlMean⁡(F)\operatorname{Mean}(F); FF21.1616 / 3299.3562.0067.8076.2766.1072.2066.8095.00
Both (Ours, Hierarchical)EE; FF21.1616 / 3299.2562.3368.4781.4770.3072.8067.7398.40

Table R2(b). Attribute comparison accuracy (%) for the same conditions. Gender and region use categorical agreement; other attributes use joint direction-and-magnitude accuracy.

Tokenizer adapterAdapter params. (M)GenderAgeRegionPitchBright.NoiseReverb.
MLP — original19.00100.0040.3383.2043.8728.5325.9022.80
MLP — expanded24.00100.0044.3386.2054.1337.2031.3029.50
Q-Former — original18.2299.8043.0086.2069.8750.2747.3051.70
Q-Former — expanded24.00100.0050.3382.8067.2046.0049.2049.50
MLP + Q-Former — mean control21.16100.0044.6784.2069.6050.2747.6046.30
Both (Ours, Hierarchical)21.16100.0046.0087.4070.1350.5347.6052.90

Capacity control. At approximately 24M parameters each, the frame-based Q-Former remains stronger on pitch and brightness prediction, while the embedding-based MLP remains stronger on identity. The expanded Q-Former nearly matches the hierarchical model on pitch (81.33% vs. 81.47%), but not identity (95.50% vs. 98.40%). Thus, increasing single-branch capacity does not eliminate their distinct strengths.

Representation control. Both two-branch models already receive the unpooled frames and an utterance-level vector. Using the SV-trained embedding rather than the frame mean raises identity accuracy from 95.00% to 98.40%. The benefit therefore depends on which utterance representation is exposed, not merely on adding a second branch.

Together with R1, these controls show what the tokenizer contributes on top of a fixed pretrained encoder: access to both frame detail and an SV-trained utterance representation. They substantiate the representation-level contribution without attributing the entire gap over general Audio-LLMs to the tokenizer. We thank Reviewer rwM1 for helping us strengthen this distinction through more controlled comparisons.

Reviewer_ZPSVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes VoiceDiff, a model that analyzes the voice characteristics of speech and explains the similarities and dissimilarities between paired input utterances, along with an accompanying benchmark. The proposed task-specific supervision improves voice judgment, including the ability to predict a range of voice attributes such as gender, age, regional background, pitch, spectral brightness, noise, and reverberation. The method extends simple attribute prediction to integrated description generation using an LLM. The LLM tuning protocols are carefully designed, and speaker embeddings are analyzed by representation type (pooled vs. frame-wise features).

Critical strengths
  • Strong motivation and clear task definition. An aggregated voice judgment covering multiple acoustic characteristics would be useful to the speech research community. The benchmark and task definition are clear.
  • Useful findings supported by in-depth analyses. The authors validate several useful findings, including the need for attribute-related supervision of audio LLMs and the value of heterogeneous representation types (pooled and frame-wise) for speaker embeddings. They also conduct in-depth analyses, such as of policy consistency and conflict cases.
Critical weaknesses

Limited novelty in representation learning.

  • The paper does not sufficiently investigate representations for the voice explanation task.
  • Encoder choice. Why was ReDimNet chosen as the speaker encoder over other widely used encoders such as ECAPA-TDNN, WavLM-TDNN, or Resemblyzer?
  • What each representation encodes. Although the paper uses two representation types (pooled and frame-wise), it does not sufficiently investigate what speaker information each encodes, or whether the findings generalize to other speaker encoders.
  • Adapter confound. The adapter is not controlled: an MLP is used for pooled features and a Q-Former for frame-wise features. The effects of representation type and adapter read-out capacity therefore cannot be disentangled.

Insufficient validation of integrated generation.

  • The proposed integrated generation (Stage 2) is novel, but the validation experiments are insufficient.
  • Confounded comparison. The advantage of Stage 2 over the baseline is limited, because differences in output format and training effects are confounded, as the authors acknowledge. The performance gain is also modest (Table 2 and Appendix F).
  • Missing baseline. No rule-based baseline results are reported for overall-relation accuracy or all-five comparison correctness, which leaves the effect of integrated generation ambiguous.
  • Unexamined benefits. Integrated generation would be expected to offer advantages in latency, cost efficiency, or usability, such as reasoning beyond simple metadata-based comparison. None of these are investigated.

Lack of statistical significance testing.

  • Several key evaluations lack statistical significance tests.
  • These include the profile summary in Table 2, the identity evaluation (although the authors acknowledge its limitations), and the hierarchical tokenizer evaluation (the comparison with the Q-Former on joint accuracy).
Questions for authors
  • Disentangling representation type from adapter capacity. Could the authors evaluate both representation types using adapters with comparable read-out capacity, and check whether the distinct roles of pooled and frame-wise representations still hold?
  • Clarifying the effectiveness of Stage 2. Could the authors report overall-relation accuracy and all-five comparison correctness for a Stage 1 + rule-based baseline on the same subset, to clarify the improvement brought by Stage 2?
  • Statistical significance. Could the authors report the statistical significance of key results, such as the profile summary and the hierarchical tokenizer evaluation?
12 comments

W1-1 (1/5) — Speaker embeddings and frame features make complementary contributions to voice judgments

We thank Reviewer ZPSV for recognizing the complementary roles of speaker embeddings and frame-level features as a useful finding of this work. The reviewer's comments identify three issues that warrant further investigation: the rationale for encoder selection, the generality of the representation-specific findings, and the potential confounding effect of adapter capacity. To address these issues, we extended the analysis in Table 1 and Appendix F.1 (Table 27) through three additional studies:

  1. Cross-encoder replication: repeating the representation comparisons with ECAPA-TDNN and WavLM-based speaker encoders.
  2. Adapter-capacity controls: expanding both single-branch adapters to approximately 24M parameters while retaining 48 output tokens.
  3. Hierarchical-representation controls: testing whether the utterance branch contributes beyond a simple compression of the frames, by retaining the frame input to the Q-Former and comparing a temporal frame average against the SV-trained speaker embedding as the MLP input.

These experiments show that the representation-specific strengths recur across the tested encoders and persist under capacity matching. They further distinguish the contribution of the encoder's speaker-discriminative utterance representation from that of an additional frame-average summary. Below, we address the individual concerns and present the corresponding evidence.

(a) Combining representation levels improves voice judgments at a fixed token budget

Reviewer ZPSV notes that the representations were not investigated sufficiently for the voice explanation task. The purpose of this analysis is to establish which representations are useful to a language model when it must judge both vocal characteristics and speaker identity. The contribution is therefore the representation finding, rather than the novelty of an individual MLP, Q-Former, or pooling operator.

The submitted Table 27(A) already distinguished increasing the token count from changing the representation supplied to the language model. Increasing frame tokens improved pitch and brightness prediction, but the 48-token frame model still trailed the 48-token embedding model on identity. Combining the representations improved all three measures without increasing the total token budget.

Submitted Table 27(A), selected columns. ReDimNet-B2 is fixed. Accuracy (%); tokens are per utterance. The 48-token results are also summarized in Table 1.

RepresentationAdapterTokensPitchBrightnessIdentity
Speaker embeddingMLP3251.7350.5097.40
Speaker embeddingMLP4853.4744.1097.20
Frame sequenceQ-Former3272.2763.4094.20
Frame sequenceQ-Former4878.2767.5094.90
Both (Ours, Hierarchical)MLP + Q-Former16 + 3281.4770.3098.40

Conclusion. The original observation concerns complementary representation levels, not simply a larger audio-token budget. The following experiments test this interpretation against encoder choice and adapter capacity.

W1-2 (2/5) — ReDimNet-B2 provides a compact reference with strong speaker-verification performance

The reviewer requests a rationale for selecting ReDimNet over other widely used speaker encoders. We selected ReDimNet-B2 [1] because it combines a compact encoder with strong reported speaker-verification performance and provides both pre-pooling frames and a final speaker embedding from the same utterance.

Following the reviewer's suggestion, we added ECAPA-TDNN [2] and WavLM-Large [3] with an ECAPA-TDNN speaker head. The table reports the full frozen-extractor sizes in our implementation and the EERs reported for the released checkpoints [4–6]. These EERs characterize the reference encoders; they are not common-protocol reruns or VoiceDiff-Bench results.

Encoder characteristics. Parameter counts include the full frozen feature extractor; WavLM includes its ECAPA-TDNN speaker head. EERs are checkpoint-reported VoxCeleb1 verification results under the respective providers' protocols.

Frozen speaker encoderEncoder parametersReported EER (%)EER source
ReDimNet-B25.07M0.57Official evaluation report [4]
ECAPA-TDNN20.77M0.80Official model card [5]
WavLM-Large + ECAPA-TDNN323.86M0.431Official evaluation report [6]

Conclusion. ReDimNet-B2 was a compact, competitive reference, not an assumed optimum. Dependence on this choice was tested directly through the within-encoder comparisons in Table R1.

References

[1] I. Yakovlev et al. Reshape Dimensions Network for Speaker Recognition. Interspeech, 2024.

[2] B. Desplanques, J. Thienpondt, and K. Demuynck. ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Interspeech, 2020.

[3] S. Chen et al. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing, 2022.

[4] ID R&D. ReDimNet: Official Evaluation Results. Checkpoint: b2-vox2-ft_lm.

[5] SpeechBrain. Speaker Verification with ECAPA-TDNN Embeddings on VoxCeleb. Official model card: spkrec-ecapa-voxceleb.

[6] Microsoft. UniSpeech/WavLM: Speaker Verification. Official evaluation report for the WavLM-Large speaker-verification checkpoint.

W1-3 (3/5) — The complementary roles recur across three speaker encoders

The reviewer questions what speaker information each representation supports and whether the findings generalize to other encoders. Table R1 compares embedding-only, frame-only, and hierarchical tokenizers within each frozen encoder, using 48 audio tokens per utterance throughout. We report both attribute prediction and pairwise comparison results to distinguish the recurring representation pattern from attribute-specific improvements.

Table R1(a). Attribute prediction and speaker identity accuracy (%). Single branches use 48 tokens; the hierarchical tokenizer uses 16 embedding tokens and 32 frame tokens. Adapter parameters are trainable parameters. Bright. = timbral brightness.

EncoderRepresentation / adapterAdapter params. (M)GenderAgeRegionPitchBright.NoiseReverb.Identity
ReDimNet-B2Embedding / MLP19.0098.7051.3360.6853.4744.1039.2034.4097.20
Frames / Q-Former18.2299.1063.0067.8078.2767.5072.2066.2794.90
Both (Ours, Hierarchical)21.1699.2562.3368.4781.4770.3072.8067.7398.40
ECAPA-TDNNEmbedding / MLP19.0099.1058.6758.6453.4750.3043.4037.4794.90
Frames / Q-Former19.2199.3562.6764.7578.8069.1070.2063.0794.50
Both (Ours, Hierarchical)22.1499.3561.3366.7880.4069.9071.4066.1396.70
WavLM-Large + ECAPA-TDNNEmbedding / MLP19.0299.5060.3371.5345.7348.3037.0025.2095.90
Frames / Q-Former18.4299.6566.6776.6176.4070.1077.8068.4095.60
Both (Ours, Hierarchical)21.3799.7564.0077.9778.1372.4079.0071.3398.40

Table R1(b). Attribute comparison accuracy (%). Gender and region use categorical agreement; the remaining attributes require the correct direction and magnitude.

EncoderRepresentation / adapterGenderAgeRegionPitchBright.NoiseReverb.
ReDimNet-B2Embedding / MLP100.0040.3383.2043.8728.5325.9022.80
Frames / Q-Former99.8043.0086.2069.8750.2747.3051.70
Both (Ours, Hierarchical)100.0046.0087.4070.1350.5347.6052.90
ECAPA-TDNNEmbedding / MLP100.0039.0078.2042.9331.4726.4033.30
Frames / Q-Former99.6040.3385.4070.5343.6048.4050.00
Both (Ours, Hierarchical)100.0048.6787.0072.8042.6751.6047.60
WavLM-Large + ECAPA-TDNNEmbedding / MLP99.8046.0082.8036.5331.2024.6024.80
Frames / Q-Former100.0049.0084.0059.4748.4046.8052.50
Both (Ours, Hierarchical)100.0050.6785.6075.2051.4747.4052.60

Conclusion. All three encoders reproduce the same role distinction in attribute prediction and identity: frames outperform embeddings on pitch and brightness, whereas embeddings outperform frames on identity. The hierarchical tokenizer achieves the highest accuracy on these three measures within every encoder. The pairwise comparison results are more attribute-dependent and do not establish uniform superiority. The recurring finding is that the complementary value of the two representation levels is not specific to ReDimNet-B2.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for the careful reviews and discussion. We have revised the manuscript to incorporate the additional experiments and clarify the claims they support. To make the changes easy to locate, revised passages and newly added results are marked in blue throughout the main text, tables, and appendices.

The reviews identified value in several aspects of the original submission. Reviewer rwM1 highlighted the importance of distinguishing voice similarity from speaker identity. Reviewer ZPSV noted the usefulness of bringing several vocal characteristics into one judgment and described integrated generation as novel, while asking for stronger evidence of its benefit. Reviewer mLKC highlighted the controlled representation comparison and the transfer to VoxParadox when spoken claims conflict with vocal cues. The questions that followed helped us establish which findings persist beyond the original setting and where the benefit of describing voice differences is concentrated.

1. Which representation is most useful depends on the judgment being made (ZPSV, rwM1, mLKC; Tables 2 and 31). At a fixed token budget, speaker embeddings favor identity, whereas frame features favor pitch and timbral brightness. Combining them improves all three judgments over either representation alone across three speaker encoders. The additional controls sharpen this finding: increasing the frame adapter’s capacity nearly closes the pitch gap, but replacing the speaker embedding with a temporal mean of the same frames lowers identity accuracy from 98.4% to 95.0%. What matters is not simply that the language model receives an utterance summary, but which speaker information that summary preserves.

2. Relation-description training helps most when the answer must specify what differs (ZPSV, mLKC; Tables 3–4 and 33). The revision compares Stage 2 with both Stage 1 + Rule and continued Stage 1 + Rule over three training runs. The gain on all-five exact match is modest. Yet pairs with at least one differing profile attribute constitute 44% of trials and account for about 3.1 of the 3.4 percentage points gained in profile-summary accuracy over continued training. This was not apparent from the aggregate score: even when a rule already supplies the sentence structure, learning to give a joint account helps the model identify the particular characteristics on which two voices differ.

3. Listeners provide evidence beyond agreement with the task rules (rwM1, mLKC; Table 5 and Appendix F.3). Fourteen listeners each completed the same 18 items. With pitch closely matched, their brightness rankings tracked spectral-centroid order (mean ρ = 0.76). Given the reported attribute comparisons, listeners chose the policy-defined overall relation in 84.9% of judgments. Separately, they rated the generated descriptions’ fit to the recordings at 4.51/5. These results support the perceptual relevance of the ordering and the descriptions without treating the numerical cutoffs as universal perceptual thresholds.

4. The supervision benefits models beyond our reference architecture (rwM1, mLKC; Table 6). On the same VoxParadox recordings with conflicting spoken claims, Qwen2.5-Omni’s accuracy against the acoustic gender target rose from 0.5% to 99.0% after task training. On VCTK read speech and EARS freeform monologues, accuracy improved in 16 of 21 comparisons across three general Audio-LLMs, without adaptation to either corpus. A competitive general model is therefore an informative outcome: it shows that these judgments can be learned with an existing audio encoder, rather than requiring ours.

We hope the Area Chair and reviewers will weigh the task together with the findings it makes possible. An identity-only evaluation would favor the representation that is substantially less accurate on pitch. A single perfect-response score would obscure where training helps a model explain what two voices share and how they differ. Making those distinctions visible is the contribution we hope the paper will be recognized for. We thank the reviewers for the questions that allowed us to test them more rigorously, and the Area Chair for considering the revised evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.