- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aims to identify which features of source speech are utilized by zero-shot TTS models for voice cloning, analyzing them through feature extraction methods such as SAE and PCA (Probing + Steering).
The paper proposes a new evaluation pipeline called Auto-Interpretability, which utilizes an Audio LLM (Gemini 3.5 Flash) to evaluate features based on three metrics: Coherence, Label Fit, and Label Stability.
Critical strengths
- The paper conducts various feature analysis experiments using existing methods such as SAEs, PCA, and Random Directions through sparse probing and steering in zero-shot voice-cloning TTS models. It also proposes an Auto-Interpretability evaluation method using an Audio LLM.
Critical weaknesses
- The feature extraction analysis relies on previously established methods rather than introducing new feature extraction techniques.
- Due to API costs, the Auto-Interpretability evaluation is conducted on only one model, Qwen-TTS. Furthermore, the proposed method is essentially a metric-based evaluation relying on the inference results of an existing Audio LLM (Gemini 3.5 Flash).
- Although the layers used for activation extraction are aligned as closely as possible, they still differ across models. Additionally, a fair comparison would require consistent data splits, but the full feature-fitting split is used for some models, while only a subset is used for others. This raises concerns about the consistency of the experimental setup.
Questions for authors
- Can the proposed Auto-Interpretability method be reproduced and generalized to other model families beyond Qwen3-TTS?
- The proposed method uses Gemini 3.5 Flash, a closed-source Audio LLM. Can the evaluation results be reproduced using open-source Audio LLMs, such as Qwen3-Audio, in environments where API access is unavailable?
4 comments
Follow-up on [W2, Q1]: auto-interpretability results on other model families
[W2, Q1] Here, we follow up on our earlier response with the results of applying our auto-interpretability pipeline to other model families. We ran it on all 5 models (Higgs TTS 3, Higgs TTS 2, Fish S2 Pro, Fish S1-mini and CosyVoice2) with the same methodology as for Qwen3-TTS. Here are the results:
| Model | Feature set | Coherence | Label Fit | Label Stability | Validated (of 500) |
|---|---|---|---|---|---|
| Qwen3-TTS (paper) | SAE | 0.43 | 0.63 | 0.41 | 92 |
| Random directions | 0.25 | 0.60 | 0.32 | 17 | |
| PCA | 0.08 | 0.50 | 0.22 | 1 | |
| Higgs TTS 3 | SAE | 0.45 | 0.64 | 0.43 | 106 |
| Random directions | 0.22 | 0.57 | 0.29 | 12 | |
| PCA | 0.08 | 0.49 | 0.24 | 0 | |
| Higgs TTS 2 | SAE | 0.36 | 0.62 | 0.38 | 61 |
| Random directions | 0.21 | 0.57 | 0.34 | 10 | |
| PCA | 0.06 | 0.51 | 0.27 | 1 | |
| Fish S2 Pro | SAE | 0.34 | 0.60 | 0.36 | 63 |
| Random directions | 0.22 | 0.60 | 0.34 | 11 | |
| PCA | 0.06 | 0.51 | 0.27 | 0 | |
| Fish S1-mini | SAE | 0.27 | 0.58 | 0.28 | 21 |
| Random directions | 0.25 | 0.60 | 0.36 | 23 | |
| PCA | 0.08 | 0.51 | 0.26 | 1 | |
| CosyVoice2 | SAE | 0.27 | 0.59 | 0.34 | 27 |
| Random directions | 0.24 | 0.61 | 0.37 | 22 | |
| PCA | 0.08 | 0.52 | 0.28 | 0 |
Across all six models, PCA scores the lowest, and the shuffled-label and random-audio controls stay at chance (Label Fit 0.49 to 0.51), replicating our Qwen3-TTS findings. However, what differs is how SAEs compare with random directions. SAEs yield far more validated features in four of the six models, Qwen3-TTS, Higgs TTS 3, Higgs TTS 2 and Fish S2 Pro, but are not significantly better than random directions in Fish S1-mini and CosyVoice2.
For CosyVoice2, this is expected given its architecture. Its backbone predicts supervised semantic tokens, which mostly carry what is said and retain only part of how it is said. The final flow-matching decoder, after the backbone, is conditioned again on the reference speech and a speaker embedding, so the backbone is under little optimization pressure to keep detailed acoustic information, such as the speaker's voice. This likely explains why its SAE finds no more interpretable acoustic concepts than random directions. For Fish S1-mini, the cause is less clear, and we leave it to future work.
The kinds of validated features also differ across models. As in Qwen3-TTS, most validated SAE features in Higgs TTS 3 and Higgs TTS 2 are speaker cluster features, which describe voice profiles (86 of 106 and 42 of 61). In Fish S2 Pro, many instead describe laughter (19), specific words such as "probably" and "awesome" (10), and accents (6). Fish S1-mini also has 6 features for specific words or sounds, which are nearly absent in Qwen3-TTS.
Thus, the auto-interpretability method can be applied to other model families, and its main result for Qwen3-TTS, i.e. the SAE's advantage over random directions, holds in Higgs TTS 3, Higgs TTS 2 and Fish S2 Pro i.e. 4/6 models, and does not hold in Fish S1-mini or CosyVoice2. We will add these results to the paper.
Follow-up to previous response
Thank you again for your review and feedback! In response to your review, we extended auto-interpretability from 1 to 6 TTS models, added 2 audio LLM judges (including the open-source Qwen3-Omni) and refit 2 models on matched data. The feature-set ranking holds under all 3 judges and our pipeline works across all model families, with SAEs leading in 4 of 6 models.
We hope these clarifications and results address your concerns, and we would be grateful if you would consider raising your score.
Thank you for the authors' cautious answers. I appreciate the detailed experiments and hard work on the requested experiments. Most of my concerns are resolved by the extra experiments. I have raised my score. Good luck.
Thank you for your review. We are glad you found our probing and steering analysis across TTS models and our auto-interpretability evaluation worthwhile! Please see our response to each of your comments below.
[W1] About the use of previously established feature extraction methods, we would like to clarify that this is an interpretability analysis paper: we ask what concepts voice-cloning TTS models represent and how they have control over those concepts, rather than aiming to build a new, better feature extractor. Established methods like SAE and PCA are the right choices to answer that question. Most practitioners use them and their behavior on text models is well understood, so our findings reveal facts about TTS models rather than confounding with an untested new method. Thus, we contribute (a) the framework that neatly brings these techniques together for voice-cloning TTS and (b) the findings it yields, which we report in the paper.
[W2, Q1] Yes, the auto-interpretability method can be extended to other model families beyond Qwen3-TTS since the pipeline only needs per-utterance feature activations and selected audio clips, which all models have. We are now running it on CosyVoice2, Fish S1-mini and Higgs TTS 2 using the same setup we used for Qwen3-TTS and will post the results in a follow-up comment once it completes.
[W2, Q2] Yes, we were able to successfully reproduce the auto-interpretability results using an open-source audio LLM judge, Qwen3-Omni! Thank you very much for this suggestion. We reran the Qwen3-TTS layer 19 auto-interpretability evaluation using Qwen3-Omni-30B-A3B-Instruct as labeler/judge (replacing Gemini 3.5 Flash) with the same prompts, feature set, clips and thresholds.
The ordering between SAE, random directions and PCA remains consistent under both judges, and SAE has the highest Coherence, Label Fit and number of validated features. The controls (shuffled labels and random audio) also stay at chance for both judges. 21 of the 46 SAE features validated by Qwen3-Omni are also passed by Gemini 3.5 Flash, showing agreement. The two judges differ mainly in how many features pass. When we ran the evaluation with an older Gemini model, Gemini 2.5 Flash (not shown in the table above), it validated 55, 4 and 1 SAE, random-direction and PCA features, which roughly matches with Qwen3-Omni. We will add the full results to the paper. Our auto-interpretability results are therefore robust across judges, including an open-source audio LLM judge!
[W3] Regarding the layers question: fundamentally, the models have different depths (24, 28 or 36 backbone layers) so using the same layer index across them would be unwise. Therefore, we match relative depth, using the layer about two-thirds into each backbone. This is the closest alignment possible across models of different depths.
Regarding consistency of data splits, the probe splits and steering examples are already identical across models, but yes, the feature-fitting data differed, with the full split for Qwen3-TTS and CosyVoice2 and a 20% subset for Fish and Higgs (to save on compute costs). This is a valid concern that the reviewer raised. Therefore, we ran experiments that retrained the Qwen3-TTS and CosyVoice2 SAEs and PCAs on the same 20% subset used for Fish and Higgs, and reran the probing and steering experiments. Here are the probing results over the 55 extractor concepts:
(probing results for human-labelled concepts and steering results are also available, just not shown here)
None of the conclusions changed due to this refit; the best method at every k is the same, no score changes by more than 0.031, and steering conclusions are also unchanged. So the different fitting data does not affect our conclusions, and we will add these refits to the paper.