- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Voice Difference Explanation (VoiceDiff-Bench), a benchmark built from VoxCeleb1 and LibriTTS-R/P to evaluate how Speech LLMs compare voices across speaker profile (gender, age, region), voice traits (pitch, brightness), and recording conditions (noise, reverb), while separating voice similarity from speaker identity. It also proposes VoiceDiff, which connects a frozen ReDimNet-B2 speaker encoder to Qwen2.5-1.5B-Instruct using a hierarchical tokenizer (MLP for pooled embeddings + Q-Former for frame features). Experiments show that pooled features help with speaker identity, frame features help with acoustic details, and combining them improves both while transferring well to VoxParadox.
Critical strengths
- Clean representation ablation: Using a fixed 48-token budget on the same frozen encoder, the paper clearly shows that pooled embeddings are better for speaker identity (97.2%), frame-level features are better for acoustic traits (pitch and brightness), and combining both (
16+32tokens) gives the best of both worlds. - Good transfer to
VoxParadox: Training on voice comparisons helps Audio-LLMs rely on actual vocal cues instead of being fooled by conflicting spoken text (e.g., boostingQwen2.5-Omnigender accuracy from 0.5% to 99.0%).
Critical weaknesses
- Synthetic, rule-based labels instead of human perception: All training and test targets come from signal-processing thresholds (median F0, spectral centroid), country metadata (as a proxy for accent), and a fixed linear formula () filled into rigid sentence templates. Without human perceptual labels, the task mostly tests slot-filling over heuristic bins rather than true natural-language voice explanation.
- Stage 2 adds complexity with little gain, and joint evaluation is weak: In Table 2a, complex Stage 2 training performs no better than simply applying the deterministic rule to Stage 1 outputs (
Stage 1 + Rule: 66.07% vs. 64.81% on voice traits). Also, because no single dataset has clean labels for all attributes, full 5-attribute evaluation relies on noisy VoxCeleb acoustic estimates and reaches only 25.40% exact match.
Questions for authors
- Human alignment: Did you run any human listening tests to check if your spectral centroid cutoffs and linear formula () match how humans perceive brightness and overall voice similarity?
- Backbone vs. tokenizer:
VoiceDiffusesReDimNet-B2(pre-trained for speaker verification), while the fine-tuned baselines use general audio encoders. How much of the performance gain comes from theReDimNet-B2encoder itself versus your hierarchical tokenizer? - Free-form prompts: Since Stage 2 is trained on rigid sentence templates, how well does the model handle free-form, open-ended user questions about voice differences?
9 comments
W2 (1/2) — Stage 2 improves the reporting of shared and differing profile attributes
We thank Reviewer mLKC for recognizing the complementary representation findings and transfer to VoxParadox. We address the concern that Stage 2 adds complexity with little gain (W2) by comparing it with continued Stage 1 training and examining which relations improve.
In submitted Table 2(a), voice-characteristic summary accuracy was 66.07% for Stage 1 + Rule and 64.81% for Stage 2. We extended this comparison to three training runs per condition and added a Stage 1 continued-training baseline, which trains on the original Stage 1 tasks for the same additional training duration as Stage 2. The original results and the additional run averages are reported separately.
(a) [W2] Profile-summary accuracy improves beyond continued Stage 1 training
Both Stage 1 baselines apply the fixed task rules to their predicted comparisons; Stage 2 generates the comparisons and summaries in one response. All models receive audio and a task question, not reference attribute values.
Table R3(a). Category-summary and identity accuracy (%), mean ± sample SD across three training runs. Profile/Identity use 1,000 pairs; Voice, 1,998; Recording, 1,500. Bold marks the highest mean, including ties.
| Condition | Profile | Voice char. | Recording | Identity |
|---|---|---|---|---|
| Stage 1 + Rule | 64.27 ± 1.19 | 65.08 ± 0.91 | 53.87 ± 0.50 | 98.10 ± 0.44 |
| Stage 1 continued training + Rule | 64.80 ± 0.40 | 64.91 ± 0.40 | 53.80 ± 0.40 | 98.40 ± 0.20 |
| Stage 2 | 68.20 ± 1.08 | 65.73 ± 0.80 | 53.96 ± 0.77 | 98.53 ± 0.12 |
The clearest gain is in profile summaries: Stage 2 improves mean accuracy by 3.93 percentage points over Stage 1 + Rule and 3.40 points over Stage 1 continued training + Rule. Under Appendix E.4, a correct profile summary must match all three profile comparisons and identify the correct set of differing attributes, including the direction and magnitude of age differences where applicable. The improvement therefore means that more responses correctly report all three profile comparisons and identify which attributes differ.
(b) [W2] The larger gains occur when profile attributes differ
To locate the improvement, we partition all 1,000 profile pairs by their benchmark comparisons of gender, age, and region. These are reference-defined groups, not examples selected by model success. Mixed profiles contain at least one shared and one differing attribute.
Table R5. Profile-summary accuracy (%), mean ± sample SD across three training runs. Scoring is identical to Table R3(a). Shared means same/similar under the task definition. Bold marks the highest mean within each row.
| Profile composition | N | Stage 1 + Rule | Stage 1 continued training + Rule | Stage 2 |
|---|---|---|---|---|
| All shared | 560 | 84.29 ± 1.39 | 85.00 ± 0.36 | 85.54 ± 0.78 |
| Shared + differing | 333 | 37.14 ± 1.05 | 38.14 ± 0.30 | 44.14 ± 1.80 |
| All differing | 107 | 43.93 ± 0.93 | 42.06 ± 0.93 | 52.34 ± 0.93 |
| All pairs | 1000 | 64.27 ± 1.19 | 64.80 ± 0.40 | 68.20 ± 1.08 |
Relative to continued Stage 1 training, Stage 2 improves mixed profiles by 6.00 points and all-differing profiles by 10.28 points, compared with 0.54 points on all-shared profiles. In the mixed group, a response must preserve the shared attributes while identifying the differing ones correctly. The gain therefore concerns the specific combination of commonalities and differences that the relation-description task is designed to express.
Finding. The additional value of Stage 2 is clearest in accurately reporting profiles that contain attribute-specific differences; continuing the original Stage 1 tasks does not reproduce the same profile-summary accuracy. We thank Reviewer mLKC for prompting a more controlled and detailed evaluation of this contribution.
W2 (2/2) — Stage 2 increases the proportion of responses with at least four correct comparisons
Reviewer mLKC also highlights the 25.40% all-five exact accuracy and the use of weak VoxCeleb acoustic labels (W2). The submitted Table 3 result requires all five attribute comparisons to be correct simultaneously, including direction and magnitude where applicable; it is not the average accuracy of the five comparisons. We retain this criterion and evaluate the Rule baselines under the same conditions.
(c) [W2] All conditions are evaluated under the original five-comparison criteria
Table R4. Five-attribute comparison accuracy (%), mean ± sample SD across three training runs. All conditions use the same 996-pair subset. The attributes are gender, age, region, pitch, and brightness. Exact requires the full comparison, including direction and magnitude where applicable. Dir. requires direction or similarity; Adj. allows one signed-category step. Gender and region remain exact under all criteria. Thresholds ≥3/≥4/all 5 are cumulative. Bold marks the highest mean, including ties.
| Condition | Exact ≥3 | Exact ≥4 | Exact all 5 | Dir. ≥3 | Dir. ≥4 | Dir. all 5 | Adj. ≥3 | Adj. ≥4 | Adj. all 5 |
|---|---|---|---|---|---|---|---|---|---|
| Stage 1 + Rule | 93.57 ± 0.61 | 69.01 ± 0.42 | 25.90 ± 0.70 | 97.12 ± 1.03 | 81.93 ± 0.70 | 37.92 ± 0.57 | 99.73 ± 0.23 | 96.32 ± 0.65 | 70.35 ± 3.67 |
| Stage 1 continued training + Rule | 93.51 ± 0.59 | 69.21 ± 0.29 | 26.37 ± 1.03 | 97.52 ± 0.67 | 81.76 ± 0.85 | 37.99 ± 0.42 | 99.77 ± 0.15 | 95.65 ± 0.95 | 70.38 ± 2.41 |
| Stage 2 | 94.41 ± 0.68 | 71.79 ± 0.60 | 27.04 ± 1.42 | 97.29 ± 0.27 | 83.10 ± 0.58 | 38.76 ± 1.16 | 99.70 ± 0.17 | 95.65 ± 0.55 | 70.65 ± 3.97 |
Stage 2 increases the proportion of responses with at least four Exact comparisons to 71.79%, compared with 69.01% for Stage 1 + Rule and 69.21% for continued Stage 1 training: gains of 2.78 and 2.58 percentage points, respectively. Mean all-five Exact accuracy is 27.04%, compared with 25.90% and 26.37%.
Comparing the three criteria reveals a common error pattern: exact agreement across all five attributes is substantially harder than correct directions or agreement within a neighboring signed category. For Stage 2, all-five agreement rises from 27.04% Exact to 38.76% Direction and 70.65% Adjacent. These complementary criteria distinguish complete correctness from the precision of the remaining errors, while retaining Exact as the strict measure.
(d) [W2] The profile-summary finding does not depend on weak acoustic labels
Pitch and brightness in the joint subset remain measurement-based weak labels from natural VoxCeleb recordings. All three conditions are scored against the same fixed references, without treating them as independent perceptual annotations.
Importantly, the profile-summary improvements in Tables R3(a) and R5 do not depend on these pitch/brightness labels. They use the metadata-supported gender, age, and regional comparisons. The benefit identified in the first response therefore remains distinct from the limitations of the five-attribute acoustic-reference evaluation.
Together, the added comparisons show where Stage 2 improves factual reporting and where fine-grained comparison errors remain. They provide a more specific account of its value than either a single category score or all-five Exact accuracy alone. We thank Reviewer mLKC for motivating this clarification and the additional evaluation.
W1 + Q1 — Human listening supports the acoustic distinctions and relation descriptions
We thank Reviewer mLKC for recognizing the complementary representation findings and transfer to VoxParadox as strengths of the work. The reviewer raises the concern that heuristic labels and templated targets may assess slot filling rather than perceptually meaningful voice explanation (W1). The requested listening tests focus on whether spectral-centroid cutoffs and the overall-relation policy correspond to human perception (Q1). To address both points, we conducted a listening study with 14 participants completing 18 items each, yielding 252 responses per component, and evaluated acoustic distinctions, the relation policy, and actual Stage 2 descriptions separately.
Section 2.1 and Appendix A distinguish corpus metadata, acoustic measurements, and augmentation settings as target sources; templates standardize their reporting. The listening study tests the acoustic and relational components, not metadata agreement as perceptual ground truth.
Table H1. Human listening results. A/B use natural LibriTTS-R recordings. C/D use the same 18 LibriTTS-R pairs from 24 speakers and actual Stage 2 responses, with six original, six noise-added, and six reverberant pairs.
The SD in D is across individual ratings, not participant means. Of 252 ratings, 247 (98.02%) were 4 or 5; the remaining five were 3.
(a) [W1, Q1] Acoustic measurements track the differences listeners hear
The reviewer asks whether spectral-centroid cutoffs correspond to perceived brightness. A and B assessed distinct cues: A used clearly F0-separated, same-gender pairs with closely matched spectral centroids; B required a full ranking of five natural recordings, one from each fixed centroid bin, with gender held fixed and median F0 closely matched. A short auditory tutorial introduced pitch and brightness without teaching numerical cutoffs.
Listeners agreed with the F0-based pitch direction in 90.48% of responses. For brightness, the mean Spearman ρ=0.7599 indicates substantial correspondence between the measured and perceived orderings. Holding gender fixed and limiting pitch variation reduces the possibility that the brightness rankings merely follow those other cues. The results therefore support a perceptually meaningful ordering, not just reproducible numerical grouping. They do not establish each cutoff as a perceptual boundary or demonstrate discrimination between every adjacent bin.
(b) [W1, Q1] Attribute-informed relation choices agree with the policy
The reviewer also questions the perceptual basis of the linear aggregation policy. In C, listeners took the displayed Stage 2 attribute comparisons as given and selected the most appropriate of the three overall-relation categories; "none" and "unable" options were also available. The 18 pairs covered six cases from each policy category. The policy weights were not disclosed to participants.
Listener choices agreed with the policy in 84.92% of responses. This agreement supports the acceptability of the attribute-based relation summary within the presented comparison-and-description context. The result provides perceptual support for the policy's use in the explanation interface, rather than establishing a universal voice-similarity scale.
(c) [W1] The generated descriptions fit the audio according to listeners
The reviewer's concern extends from the targets to the explanations themselves. D therefore evaluated the actual full Stage 2 responses for the same audio pairs, with instructions to judge audio–description fit rather than writing fluency. The mean rating of 4.51/5, with 98.02% rated 4 or 5, provides direct evidence that listeners accepted the descriptions as compatible with the audio, rather than only consistent with an automatic scoring rule.
Together, these results connect the measurement-based acoustic ordering, the attribute-informed relation policy, and the model's generated descriptions to distinct listening judgments. The task is therefore supported by perceptual evidence in addition to agreement with automatically constructed targets. Reviewer mLKC's suggestion enabled us to test these connections explicitly and substantiate the task's empirical basis more clearly. We thank the reviewer for prompting this evaluation.