- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies sufficient conditions for reliable speech attribute conversion using disentangled latent representations. The authors introduce an Independence Conditional Autoencoder (ICAE), which jointly optimizes reconstruction and independence between the learned latent representation and controllable attributes. Under assumptions including perfect reconstruction, statistical independence, a finite discrete latent variable, and an asymmetric latent prior, the paper establishes exact conversion guarantees. It further derives an expected conversion-error bound under approximate reconstruction and independence conditions. To demonstrate practical applicability, the authors propose Independence-based Voice Conversion (IVC), which uses a reconstruction objective and an HSIC-based independence penalty. Experiments on voice and pitch conversion tasks show competitive intelligibility, speaker similarity, and automatic naturalness scores.
Critical strengths
- The paper establishes sufficient conditions under which reconstruction and latent independence enable exact attribute conversion. The extension to approximate conditions through an explicit conversion-error bound provides a useful theoretical perspective on disentangled conditional generation.
- The proposed method avoids adversarial training and achieves competitive intelligibility, strong speaker similarity under the reported protocol, and promising automatic naturalness scores. Its relatively lightweight implementation is also practically appealing.
Critical weaknesses
- The independence constraint is a central component of the proposed method. However, the ablation results in Appendix B (Table 5) show relatively small differences between IVC and its variant without HSIC. Specifically, removing HSIC changes WER from 5.18 to 5.29, CER from 1.58 to 1.59, and Target EER from 23.00 to 22.67. While these results suggest some benefit from the independence objective, its empirical contribution appears limited under the reported evaluation. Additional evidence would help establish the practical importance of this component.
- There are several inconsistencies that require clarification. First, Table 1 reports WER/CER of 5.29/1.59 for IVC, whereas Appendix B (Table 5) reports 5.18/1.58 for the same method, without clearly explaining the difference in evaluation settings or model checkpoints. Second, the conclusion states that IVC outperforms all baselines on the pitch conversion task. However, Table 3 reports a higher average PCC for GENVC (0.754) than for IVC (0.736). In fact, Section 6.3 correctly describes IVC as achieving the second-best performance. These inconsistencies should be addressed to ensure accurate interpretation of the experimental findings.
Questions for authors
- Appendix B (Table 5) shows relatively small performance differences between IVC and the variant without HSIC. Could the authors further clarify the empirical contribution of the independence constraint?
- Table 1 and Appendix B (Table 5) report different WER/CER values for IVC (5.29/1.59 vs. 5.18/1.58). Could the authors clarify whether these results correspond to different checkpoints or evaluation settings?
- The conclusion states that IVC outperforms all baselines on pitch conversion, whereas Table 3 shows that GENVC achieves a higher average PCC (0.754 vs. 0.736). Could the authors clarify this inconsistency?
3 comments
Dear Reviewer,
Thank you for your thoughtful review. We have submitted our rebuttal and would be grateful if you could review our responses. We would be happy to address any remaining questions or concerns.
Thank you again for your time and consideration!
Official Comment by Reviewer
Thank you for clarifying the results and adding the bottleneck analysis. I’m keeping my rating at 2 because adding HSIC to the quantized model yields only a small improvement in the ablation, so its practical contribution remains unclear to me.
Rebuttal
We thank the reviewer for the encouraging assessment of the theoretical perspective and of the method's practical simplicity. We also appreciate the careful comparison of the ablation and main-result tables.
W1. The measured contribution of the HSIC objective appears small
We thank the reviewer for this observation. We believe that an additional factor encouraging latent independence in our model is the quantization of the learned representation, which acts as an implicit regularizer by restricting the information that can be encoded in the latent space. Consequently, even without an explicit HSIC penalty, the discrete bottleneck may already suppress some speaker-dependent information, potentially explaining the relatively small improvements obtained by adding HSIC.
Supporting this interpretation, Table 5 shows that the continuous variant, where the encoder predicts S^∈R1×T, achieves worse results in speaker conversion metric (EER 20.50%) than the discrete variant trained without HSIC (EER 22.67%). This suggests that quantization itself contributes to the model's performance, potentially reducing the additional benefit of explicit independence regularization.
To further support our interpretation, we conducted an additional experiment that sweeps continuous bottleneck size from 1 to 1024 as presented in the general comment, exp (2).
Importantly, WER and CER primarily evaluate linguistic content preservation rather than the effectiveness of speaker conversion. Since HSIC encourages the removal of speaker-dependent information from the latent representation, this regularization may introduce a trade-off with reconstruction and potentially worsen WER/CER, while improving the speaker conversion metric (Target EER). Therefore, improvements in WER/CER are not necessarily the primary expected benefit of HSIC; its contribution should also be evaluated through speaker similarity. In our results, adding HSIC and latent quantization improves Target EER from 20.50% to 23.00%, while maintaining comparable intelligibility.
W2. Inconsistent WER/CER values and incorrect pitch-ranking statement
We thank the reviewer for carefully identifying these inconsistencies. Regarding the WER/CER results, the values reported in Appendix B (Table 5), 5.18%/1.58%, are correct, while Table 1 contains a typographical error. We have corrected these values in the revised manuscript.
Regarding pitch conversion, the statement in the conclusion was inadvertently retained from an earlier version of the manuscript, before the GENVC results were included in the comparison. As the reviewer correctly noted, Section 6.3 already reports that our method achieves the second-best performance, with a PCC of 0.736 compared to GENVC's 0.754. We replace the conclusion's incorrect “outperforms all baselines” wording with “achieves the second-best average pitch correlation.” The result remains relevant when considering model scale: Section 6.3 reports approximately 122M parameters for the full IVC pipeline versus more than 500M for GENVC.
We are grateful for this careful, constructive feedback and would be happy to engage further on the ablation evidence and the corrected presentation of the results.