- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the TIMBRETRACE system that addresses the problem of identifying whether an AI generated singing voice is similar to the target singer. It adapts pretrained speaker verification models to real and synthetically generated singing data. Experiments show substantial improvements on a controlled benchmark and smaller improvements on an external collection covering ten unseen singers. The paper also investigates whether pretrained models retain source-singer identity despite humans perceiving the intended target.
Critical strengths
-
The main contribution of this paper is to address the problem of determining whose voice AI-generated music resembles, not just whether the music was generated by AI. This is an interesting problem that is practical as musicians want to know whether their vocal identity was imitated by AI.
-
The results show that this paper substantially improves the pretrained models in terms of Equal Error Rate: from 45.28% to 9.08% using ERes2Net in real–synthetic experiments. This paper also shows improvements over real-only adaptation and an AAM-Softmax baseline.
Critical weaknesses
-
The explanation in the source–target setting is not clear. Section 4.2 states that pretrained models frequently favor the source singer, while Figure 2 shows higher average target similarity for all three pretrained models; these findings could be consistent, but we still do not have the proportions at trial level. In the human evaluation, the models’ disagreement with the listeners on the same examples is not reported, so the main explanation remains weak.
-
The evaluation on the external datasets does not isolate the benefit of the synthetic supervision. Table.2 compares full-adapted models vs. "pretrained models", while Table.4 ablates on the training datasets based on the pooled benchmark scores. The authors’ constructed benchmark shares singer identities and conversion checkpoints between training and testing; an evaluation of the “real-only” models externally is thus needed to distinguish the benefit of synthetic supervision from adaptation to singing.
Questions for authors
-
What proportion of examples favor the source before and after adaptation, particularly among the human-validated songs? Do source references use different recordings from the conversion inputs?
-
Can the existing real-only and full-adaptation models be compared separately on controlled real–synthetic trials and the external collection? Are all verification trials in Table 2 real–synthetic pairs, or does the evaluation also include real–real pairs?
2 comments
Response to W2 & Q2
W2 & Q2. Real-only versus Full adaptation on controlled and external RS trials
We agree that comparisons against pretrained models alone do not isolate the benefit of synthetic supervision beyond adaptation to real singing. We therefore compared Real-only and Full adaptation separately on controlled RS trials and external zero-day RS trials, using models retrained under the revised singer-disjoint protocol described in our general response.
Both evaluations below contain RS pairs only, with a real reference and a synthetic candidate. EER and TPR@1%FAR are reported in percentages.
| Backbone | Training setting | Controlled RS EER ↓ | Controlled RS AUC ↑ | Controlled RS TPR@1%FAR ↑ | Zero-day RS EER ↓ | Zero-day RS AUC ↑ | Zero-day RS TPR@1%FAR ↑ |
|---|---|---|---|---|---|---|---|
| ERes2Net | Real-only | 14.29 | 0.9355 | 36.83 | 14.15 | 0.9410 | 45.12 |
| ERes2Net | Full | 11.96 | 0.9476 | 43.29 | 13.57 | 0.9399 | 44.69 |
| CAM++ | Real-only | 15.71 | 0.9208 | 32.79 | 14.08 | 0.9376 | 40.62 |
| CAM++ | Full | 11.00 | 0.9550 | 50.50 | 13.04 | 0.9466 | 45.98 |
| ECAPA-TDNN | Real-only | 13.67 | 0.9357 | 41.33 | 15.33 | 0.9186 | 38.26 |
| ECAPA-TDNN | Full | 11.88 | 0.9506 | 44.67 | 14.82 | 0.9295 | 44.27 |
On the controlled RS subset, Full adaptation improves over Real-only adaptation across all three metrics for every backbone. These comparisons support a benefit from including synthetic supervision beyond adaptation to real singing under the revised protocol.
On the external zero-day RS subset, Full adaptation achieves lower EER for all three backbones and improves all three metrics for CAM++ and ECAPA-TDNN. For ERes2Net, EER decreases from 14.15% to 13.57%, while AUC and TPR@1%FAR are slightly lower. Thus, the external benefits depend on the backbone and metric rather than being uniform.
Because these evaluations contain only RS pairs, the observed improvements cannot be attributed to better performance on RR trials. We will include these comparisons and their qualifications in the revised manuscript.
Q2. Composition of the zero-day evaluation in Table 2
The overall zero-day results in Table 2 include both RR and RS pairs, rather than exclusively RS pairs. The evaluation contains 29,856 verification pairs:
| Pair type | Positive pairs | Negative pairs | Total |
|---|---|---|---|
| Real–real (RR) | 7,464 | 7,464 | 14,928 |
| Real–synthetic (RS) | 7,464 | 7,464 | 14,928 |
| Total | 14,928 | 14,928 | 29,856 |
Positive pairs share the same singer identity, whereas negative pairs have different singer identities. The reference is always a real recording; the candidate is real for RR pairs and synthetic for RS pairs.
All 29,856 pairs contribute to the pooled evaluation in Table 2. The separate zero-day RS results above use only the 14,928 RS pairs. These verification pairs are distinct from the source–target comparison trials reported in W1 & Q1. We will clarify the evaluation procedures and their respective counts in the revised manuscript.
Response to W1 & Q1
We thank the reviewer for requesting clearer evidence about source–target preference and comparisons that isolate the benefit of synthetic supervision. We have added trial-level statistics, a larger sample of model judgments complementing the human validation, and separate controlled and external real–synthetic evaluations.
We agree that average similarities alone do not establish how frequently individual examples favor the source singer. We therefore report source-preference proportions over 11,984 segment-level source–target comparison trials.
For each trial, we compare a synthetic segment's mean cosine similarity to source-singer references with its mean cosine similarity to target-singer references. A trial is source-favored when the former exceeds the latter. Each segment contributes one comparison trial; 11,984 is the number of trials, not the number of songs. We use the same trials and references before and after adaptation.
The corresponding mean cosine similarities are:
These results clarify why higher average target similarity can coexist with source-favored individual trials. Before adaptation, 10.94%–14.14% of the analyzed trials favor the source, despite higher target similarity on average. After adaptation, this proportion decreases to 0.17%–3.65% across the three backbones. Meanwhile, mean source similarity decreases and mean target similarity increases, widening the target–source similarity gap.
The original human validation covered 50 songs with three listeners. We extended it to 500 newly sampled segments, each judged by 3 blinded listeners as closer to the source or the target singer. Listeners judged 486/500 (97.2%) of the synthetic samples as closer to the intended target singer.
On the same 500 samples, we compare each model's source–target preference (defined above) with the listener majority:
We will replace the imprecise wording "frequently favor the source" with these measured proportions. These results characterize the change in source–target preference on the analyzed samples; they do not by themselves establish that this change explains all verification improvements.
Regarding reference selection, all source-singer references come from recordings different from the conversion input. We exclude the entire recording used as the conversion input, not merely the corresponding segment.