acceptodds
Under review as a conference paper at ICLR 2027

Provable Speech Attribute Control via Latent Independence

Abstract

Conditional generation and disentangled representation learning are central to controlled generation across audio, vision, and multimodal domains. However, despite strong empirical progress, particularly in speech style transfer, most existing approaches rely on heuristic objectives and architectural choices, offering limited theoretical understanding of when and why reliable attribute control is achievable. In this work, we develop a formal framework for speech attribute conversion and provide a theoretical analysis of sufficient conditions for exact and consistent transfer. Our analysis focuses on a deterministic autoencoder setting augmented with an independence constraint between the learned latent representation and the controllable attribute. Under explicit population-level assumptions about the data-generating process, we establish guarantees linking reconstruction, independence, and the feasibility of attribute manipulation while preserving task-relevant content. We further show how the theoretical framework translates into practice by proposing a practical voice conversion method that directly implements its core principles. Experimental evaluations on voice and pitch conversion tasks demonstrate the applicability of the theoretical analysis to real-world speech conversion settings and show that the resulting method achieves competitive performance against existing approaches.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
voice conversion, conditional generation, autoencoders

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NHRtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies sufficient conditions for reliable speech attribute conversion using disentangled latent representations. The authors introduce an Independence Conditional Autoencoder (ICAE), which jointly optimizes reconstruction and independence between the learned latent representation and controllable attributes. Under assumptions including perfect reconstruction, statistical independence, a finite discrete latent variable, and an asymmetric latent prior, the paper establishes exact conversion guarantees. It further derives an expected conversion-error bound under approximate reconstruction and independence conditions. To demonstrate practical applicability, the authors propose Independence-based Voice Conversion (IVC), which uses a reconstruction objective and an HSIC-based independence penalty. Experiments on voice and pitch conversion tasks show competitive intelligibility, speaker similarity, and automatic naturalness scores.

Critical strengths
  1. The paper establishes sufficient conditions under which reconstruction and latent independence enable exact attribute conversion. The extension to approximate conditions through an explicit conversion-error bound provides a useful theoretical perspective on disentangled conditional generation.
  2. The proposed method avoids adversarial training and achieves competitive intelligibility, strong speaker similarity under the reported protocol, and promising automatic naturalness scores. Its relatively lightweight implementation is also practically appealing.
Critical weaknesses
  1. The independence constraint is a central component of the proposed method. However, the ablation results in Appendix B (Table 5) show relatively small differences between IVC and its variant without HSIC. Specifically, removing HSIC changes WER from 5.18 to 5.29, CER from 1.58 to 1.59, and Target EER from 23.00 to 22.67. While these results suggest some benefit from the independence objective, its empirical contribution appears limited under the reported evaluation. Additional evidence would help establish the practical importance of this component.
  2. There are several inconsistencies that require clarification. First, Table 1 reports WER/CER of 5.29/1.59 for IVC, whereas Appendix B (Table 5) reports 5.18/1.58 for the same method, without clearly explaining the difference in evaluation settings or model checkpoints. Second, the conclusion states that IVC outperforms all baselines on the pitch conversion task. However, Table 3 reports a higher average PCC for GENVC (0.754) than for IVC (0.736). In fact, Section 6.3 correctly describes IVC as achieving the second-best performance. These inconsistencies should be addressed to ensure accurate interpretation of the experimental findings.
Questions for authors
  1. Appendix B (Table 5) shows relatively small performance differences between IVC and the variant without HSIC. Could the authors further clarify the empirical contribution of the independence constraint?
  2. Table 1 and Appendix B (Table 5) report different WER/CER values for IVC (5.29/1.59 vs. 5.18/1.58). Could the authors clarify whether these results correspond to different checkpoints or evaluation settings?
  3. The conclusion states that IVC outperforms all baselines on pitch conversion, whereas Table 3 shows that GENVC achieves a higher average PCC (0.754 vs. 0.736). Could the authors clarify this inconsistency?
3 comments

We thank the reviewer for the encouraging assessment of the theoretical perspective and of the method's practical simplicity. We also appreciate the careful comparison of the ablation and main-result tables.

W1. The measured contribution of the HSIC objective appears small

We thank the reviewer for this observation. We believe that an additional factor encouraging latent independence in our model is the quantization of the learned representation, which acts as an implicit regularizer by restricting the information that can be encoded in the latent space. Consequently, even without an explicit HSIC penalty, the discrete bottleneck may already suppress some speaker-dependent information, potentially explaining the relatively small improvements obtained by adding HSIC.

Supporting this interpretation, Table 5 shows that the continuous variant, where the encoder predicts S^∈R1×T\hat S\in\mathbb{R}^{1\times T}, achieves worse results in speaker conversion metric (EER 20.50%) than the discrete variant trained without HSIC (EER 22.67%). This suggests that quantization itself contributes to the model's performance, potentially reducing the additional benefit of explicit independence regularization.

To further support our interpretation, we conducted an additional experiment that sweeps continuous bottleneck size from 1 to 1024 as presented in the general comment, exp (2).

Importantly, WER and CER primarily evaluate linguistic content preservation rather than the effectiveness of speaker conversion. Since HSIC encourages the removal of speaker-dependent information from the latent representation, this regularization may introduce a trade-off with reconstruction and potentially worsen WER/CER, while improving the speaker conversion metric (Target EER). Therefore, improvements in WER/CER are not necessarily the primary expected benefit of HSIC; its contribution should also be evaluated through speaker similarity. In our results, adding HSIC and latent quantization improves Target EER from 20.50% to 23.00%, while maintaining comparable intelligibility.

W2. Inconsistent WER/CER values and incorrect pitch-ranking statement

We thank the reviewer for carefully identifying these inconsistencies. Regarding the WER/CER results, the values reported in Appendix B (Table 5), 5.18%/1.58%, are correct, while Table 1 contains a typographical error. We have corrected these values in the revised manuscript.

Regarding pitch conversion, the statement in the conclusion was inadvertently retained from an earlier version of the manuscript, before the GENVC results were included in the comparison. As the reviewer correctly noted, Section 6.3 already reports that our method achieves the second-best performance, with a PCC of 0.736 compared to GENVC's 0.754. We replace the conclusion's incorrect “outperforms all baselines” wording with “achieves the second-best average pitch correlation.” The result remains relevant when considering model scale: Section 6.3 reports approximately 122M parameters for the full IVC pipeline versus more than 500M for GENVC.

We are grateful for this careful, constructive feedback and would be happy to engage further on the ablation evidence and the corrected presentation of the results.

Dear Reviewer,

Thank you for your thoughtful review. We have submitted our rebuttal and would be grateful if you could review our responses. We would be happy to address any remaining questions or concerns.

Thank you again for your time and consideration!

Reviewer_NHRtOpenReview ↗

Official Comment by Reviewer

Thank you for clarifying the results and adding the bottleneck analysis. I’m keeping my rating at 2 because adding HSIC to the quantized model yields only a small improvement in the ablation, so its practical contribution remains unclear to me.

Reviewer_XD84OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies controllable speech generation through a latent representation intended to separate speaker identity from other speech attributes. It motivates the separation through a sequence of formal assumptions, a lemma, propositions, and corollaries, and implements an independence-promoting training objective. The experiments suggest that the proposed system can improve attribute control while retaining useful speech quality and identity information. I find the motivation and the empirical direction worthwhile, but I am less convinced by the scope of the theoretical guarantee.

Critical strengths
  1. The problem is well motivated. Attribute control without unwanted changes to speaker identity is a meaningful speech-generation challenge, and the proposed formulation addresses that motivation directly.
  2. The paper goes beyond a purely formal construction: the implementation is evaluated empirically, and the reported results support the practical usefulness of the method to some extent.
Critical weaknesses
  1. The central theoretical statement appears stronger than its proof. In Appendix D (p. 15, Lemma 1 proof), the argument moves from an almost-sure statement to a claim over all conditioning values without an additional regularity or support assumption. The pointwise versions of Proposition 1 and Corollary 1 are vulnerable to changes on a probability-zero conditioning event when the conditioning variable is continuous. This matters because the paper presents these results as general guarantees rather than almost-everywhere statements.
  2. The assumptions used in the proofs are not established for the implemented model. In particular, a frame-level empirical HSIC penalty does not imply independence of full latent sequences, nor does it provide the uniform total-variation control invoked by the theory. Quantizing a learned latent also does not by itself establish the assumed finite, identifiable latent variable. Thus, even after repairing local proof details, the stated theory does not yet justify the practical controllability claim for the trained system.
Questions for authors
  1. Can the authors state and prove Lemma 1, Proposition 1, and Corollary 1 with the correct quantifiers (for example, almost everywhere), or add explicit assumptions that justify their pointwise forms? Please address the step in Appendix D, p. 15, where the quantifier changes.
  2. What result connects the empirical frame-level HSIC objective to the sequence-level independence and distributional bounds assumed in the propositions? If no such result is intended, what narrower claim should readers take from the theory?
3 comments

We thank the reviewer for their careful evaluation of our theoretical results and for recognizing the value of our problem formulation and practical approach. We address both concerns in detail below.

W1. The almost-sure argument in Lemma 1

We thank the reviewer for highlighting this technical distinction. We agree that for continuous CC, statistical independence implies

$ P_{\hat S\mid C=c}=P_{\hat S} \quad\text{for }P_C\text{-a.e. }c, $ rather than for every c∈supp⁡(C)c\in\operatorname{supp}(C). However, this does not affect the central identifiability argument. Since SS has finite support with distinct probabilities pip_i, the proof establishes a common set C0⊆supp⁡(C)\mathcal C_0\subseteq\operatorname{supp}(C), with PC(C0)=1P_C(\mathcal C_0)=1, such that $ e(f(s,c))=T(s), \quad \forall s\in\operatorname{supp}(S), \forall c\in\mathcal C_0, $ where TT is invertible. Consequently, $ d(e(f(s,c)),c')=f(s,c'), \quad \forall s,\ \forall c,c'\in\mathcal C_0. $

Thus, exact conversion holds with probability one for independently sampled conditioning values. When CC is discrete with positive mass at every supported value, C0=supp⁡(C)\mathcal C_0=\operatorname{supp}(C), recovering the original pointwise guarantee.

We clarify the quantifiers in Appendix D and the corresponding theoretical statements, without changing the underlying assumptions or identifiability argument.

W2. The hypotheses of the theorems are not verified for the trained IVC system

We agree that the theoretical results and the particular training loss must not be combined. Their intended relationship is principle versus instantiation, not finite-sample certificate. The results in Section 4 say what can be guaranteed if reconstruction, latent independence, latent identifiability, and (for Proposition 2) the required uniform error, variation, and regularity properties hold. Section 5 then implements two corresponding optimization objectives: feature reconstruction and an empirical independence-promoting penalty. As Section 4.1 explicitly notes, ideal population-level convergence is an assumption, not an observed property of trained neural networks.

AuthorsOpenReview ↗

Rebuttal Part 2

Frame-level HSIC

Our underlying modeling assumption is that the speaker-dependent acoustic information targeted by voice conversion, such as timbre and vocal characteristics, is sufficiently represented in these contextualized frame-level embeddings. While speakers may also be identifiable through longer-range linguistic patterns, such as characteristic vocabulary or expressions, these are not the attributes we aim to manipulate. Our objective is to modify the acoustic realization of speech while preserving its linguistic content. Specifically, WavLM's seven-layer CNN feature encoder has a receptive field of 25 ms, while its convolutional positional embedding spans 128 frames (approximately 2.56 seconds). Moreover, we extract representations from the sixth Transformer layer of WavLM-Large, where global self-attention allows each frame representation to integrate information from the entire input utterance

Regarding latent quantization, our theoretical framework does not require exact recovery of the underlying latent variable SS, but rather its recovery up to an unknown invertible transformation TT, i.e., S^=T(S)\hat S=T(S). Moreover, to investigate the plausibility of Assumption 3 (Asymmetric Latent Prior), we provide an empirical analysis in Appendix J (Figure 2), using a proxy variable S~\tilde S derived from real-world speech data. Specifically, we estimate the empirical prior probabilities of content-related speech units and demonstrate that they are pairwise distinct, providing empirical support for the asymmetric-prior assumption in a meaningful speech representation.

Our theory provides sufficient conditions for correct attribute conversion in an idealized generative setting. Specifically, it establishes that exact reconstruction and independence of the learned representation from the conditioning variable, together with the identifiability assumptions, guarantee correct conversion. Proposition 2 further characterizes conversion error under approximate reconstruction, bounded conditional distributional variation, and additional regularity assumptions.

Our practical IVC model follows these principles by jointly optimizing reconstruction and a frame-level HSIC penalty. The latter encourages the removal of speaker-dependent information from local acoustic representations, but does not certify independence of complete latent sequences or the uniform total-variation condition.

Therefore, the theoretical contribution should be understood as a characterization of sufficient conditions for reliable conversion, rather than a finite-sample guarantee for the implemented IVC model. The practical results independently demonstrate the usefulness of the resulting design principles. We clarify this distinction in the revised manuscript and explicitly.

Thank you again for identifying these foundational points; we value constructive feedback and would be pleased to discuss the issues further.

Dear Reviewer,

Thank you for your thoughtful review. We have submitted our rebuttal and would be grateful if you could review our responses. We would be happy to address any remaining questions or concerns.

Thank you again for your time and consideration!

Reviewer_Nk36OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies conditional speech conversion using a deterministic autoencoder whose latent representation is encouraged to be independent of the attribute being changed. Under an invertible generative model, independence between content and condition, a finite content variable with distinct probability masses, and perfect reconstruction, it proves recovery of content up to a condition-independent invertible mapping and hence exact conversion. It also derives a conversion-error bound under uniform reconstruction, total-variation dependence, separation, and decoder-regularity assumptions. The practical IVC system reconstructs WavLM features using a discretized WaveNet-style autoencoder, speaker or pitch conditioning, and a frame-level HSIC penalty. Experiments report strong target-speaker EER and automatic naturalness scores, competitive intelligibility, and pitch-transfer results.

Critical strengths

The paper clearly states several restrictive assumptions and distinguishes its discrete guarantee from the continuous empirical variant. The conditional probability-mass argument provides an interpretable sufficient condition for resolving condition-dependent latent relabeling. The experiments include contemporary baselines, an HSIC-removal ablation, a continuous bottleneck, and an acoustic-domain variant. The reported target-speaker EER and UTMOSv2 improvements are promising, and the paper appropriately acknowledges that automatic naturalness prediction does not replace human listening tests.

Critical weaknesses

-- The implemented independence objective does not establish the independence required by the theory. Section 5.3 pools all frames and penalizes dependence between a single latent frame and its condition. The encoder and decoder, however, process sequences using temporal convolutions. Independence of pooled frame marginals does not imply independence of the latent sequence: a speaker-dependent reversal of token order can preserve every pooled token frequency while revealing speaker identity through temporal structure. Thus even ideal minimization of the stated frame-level objective need not satisfy the theorem for sequence conversion.

-- The specified kernel-bandwidth rule degenerates for the stated speaker-conditioning construction. The same speaker embedding is broadcast across all T frames before computing HSIC.

-- The identifiability assumptions have not been connected to a useful speech-conversion regime. Assumptions 2–3 concern the unobserved true content variable S, whereas Section 5.1 motivates their applicability by discretizing the learned representation.

Questions for authors

How is the RBF bandwidth actually computed after broadcasting speaker embeddings, and what prevents zero bandwidths caused by duplicated frames? Please provide the exact estimator, sampling procedure, and numerical safeguards.

What additional assumption or measurement connects pooled frame-level HSIC to independence of the complete latent sequence and to the uniform total-variation condition in Proposition 2? Can a temporal speaker probe test residual leakage?

Can you demonstrate a non-vacuous robustness regime using controlled data with known content probabilities, including near-symmetric priors, and report how the empirical conversion error changes with the identifiability margin?

Do the gains over the no-HSIC variant persist across seeds and regularization weights? Please also reconcile the inconsistent full-model scores and supplement pitch correlation with absolute pitch and content-preservation measurements.

3 comments

We thank the reviewer for the thorough technical assessment and for recognizing the value of the probability-mass argument, the explicit assumptions, and the experimental comparisons. We address each concern below.

W1+Q2 Frame-level HSIC does not establish independence of the full latent sequence

We thank the reviewer for highlighting this important distinction. First, we would like to point the reviewer to the new experiments we added and posted for all reviewers, including frame vs chunk level comparison. Second, we would like to clarify the modeling assumption underlying our practical implementation.

Our approach focuses on acoustic speaker characteristics represented at the local frame level (e.g. timbre), rather than all speaker-identifying information that may be present across an entire utterance (e.g. repeating words). Importantly, although WavLM produces embeddings at 20 ms intervals, each embedding incorporates substantially more temporal context than its frame resolution. Specifically, WavLM's seven-layer CNN feature encoder has a receptive field of 25 ms, while its convolutional positional embedding spans 128 frames (approximately 2.56 seconds). Moreover, we extract representations from the sixth Transformer layer of WavLM-Large, where global self-attention allows each frame representation to integrate information from the entire input utterance [1]. Thus, although our independence objective operates on individual frame embeddings, these embeddings are contextualized representations rather than isolated acoustic frames, combining local acoustic information with long-range dependencies. The theoretical receptive field at layer 6 therefore covers the entire utterance, although the effective contribution of different temporal regions is determined by the learned attention patterns [2].

Our underlying modeling assumption is that the speaker-dependent acoustic information targeted by voice conversion, such as timbre and vocal characteristics, is sufficiently represented in these contextualized frame-level embeddings. While speakers may also be identifiable through longer-range linguistic patterns, such as characteristic vocabulary or expressions, these are not the attributes we aim to manipulate. Our objective is to modify the acoustic realization of speech while preserving its linguistic content. This motivates applying the independence objective at the frame level, leveraging WavLM's contextualized representations, rather than enforcing independence from all speaker-identifying information at the utterance level.

We agree that frame-level independence does not mathematically imply independence of the entire latent sequence, as illustrated by the reviewer's counterexample. Nevertheless, our objective is intentionally designed to address local acoustic attribute dependence rather than arbitrary sequence-level dependencies. We make this modeling assumption explicit and clarify that the practical HSIC objective encourages independence at the local representation level, without claiming that it establishes the stronger sequence-level independence condition.

[1] Chen, Sanyuan, et al. "Wavlm: Large-scale self-supervised pre-training for full stack speech processing." IEEE Journal of Selected Topics in Signal Processing 16.6 (2022): 1505-1518.

[2] Luo, Wenjie, et al. "Understanding the effective receptive field in deep convolutional neural networks." Advances in neural information processing systems 29 (2016).

W2+Q1 RBF bandwidth selection with repeated speaker embeddings

We thank the reviewer for pointing out this potential numerical issue. In our implementation we avoid it since the HSIC computation includes random subsampling of at most 512 frame-level observations, and the estimated bandwidth is lower-bounded by $10^{-7}$, preventing division by zero.

In the released version, we will address this issue by estimating the speaker-condition kernel bandwidths from the original utterance-level speaker embeddings, prior to temporal broadcasting, while retaining the frame-level HSIC objective. This will prevent repeated embeddings from affecting bandwidth estimation and ensure that the kernel scales reflect meaningful variation between speaker-condition embeddings.

AuthorsOpenReview ↗

Rebuttal Part 2

W3. Applicability of the identifiability assumptions to speech content.**

We thank the reviewer for catching this imprecise formulation in Sec 5.1. Our intention was to explain that using a discrete learned representation is consistent with the discrete setting considered in our theoretical analysis. We will rephrase the sentence in Section 5.1: "We adopt a discrete latent representation S^\hat S, consistent with the discrete setting considered in our theoretical analysis."

Additionally, we connect the theoretical identifiability conditions to practical speech conversion by proposing a method that jointly optimizes reconstruction (MSE) and independence (HSIC), reflecting the sufficient conditions established by our theoretical analysis. Furthermore, we provide empirical support for the plausibility of Assumption 3 by approximating the unobserved content variable SS with S~\tilde{S} and estimating the empirical prior probabilities of content-related speech units extracted from real-world speech data. These results illustrate a practical setting in which the asymmetric-prior assumption is approximately satisfied.

Q3 Demonstration of a non-vacuous robustness regime using controlled data

We thank the reviewer for this valuable suggestion. We added a controlled experiment as requested, please see experiment (4) in the general comment for all reviewers.

no HSIC variants

Following standard voice conversion evaluation protocols, we report corpus-level WER, CER, and EER, computed over the full test set using fixed pretrained evaluation models. These metrics are aggregated across utterances or verification trials rather than averaged over independent runs; therefore, per-utterance standard deviations would not meaningfully reflect their statistical uncertainty. Moreover, retraining large VC models across multiple random seeds is computationally expensive. We therefore follow the reporting conventions adopted in VC literature, e.g. kNN-VC (Baas et al., 2023), FreeVC (Li et al., 2023), GENVC (Cai et al, 2025), which report objective evaluation metrics without standard deviations.

Typo in Table 1.

We thank the reviewer for detecting the typo, Appendix B, Table 5 contains the right results with WER=5.18% and CER=1.58% and we fix Table 1 in the revised manuscript.

Additional absolute pitch and content-preservation measurements

We thank the reviewer for this valuable suggestion. To further strengthen our empirical evaluation and directly address this concern, we conducted additional experiments comparing our method with GenVC on absolute pitch accuracy (Pitch MAE) and linguistic content preservation (WER and CER). Given the limited rebuttal period, we focused on GenVC as a strong and competitive baseline, enabling a direct comparison across these complementary evaluation metrics.

SystemWER (%) ↓CER (%) ↓Pitch PCC ↑Pitch MAE (Hz) ↓
Target10.995.42--
GENVC15.817.600.75419.93
IVC16.928.430.73620.06

The additional evaluation confirms that IVC achieves competitive pitch conversion and linguistic content preservation compared to GENVC. Despite its substantially smaller model size, IVC achieves comparable pitch accuracy (PCC: 0.736 vs. 0.754; MAE: 20.06 vs. 19.93 Hz), with only modest differences in WER and CER. These results further demonstrate the effectiveness of our lightweight approach.

We appreciate the detailed and constructive nature of this review and would welcome further discussions.

Dear Reviewer,

Thank you for your valuable feedback. We have posted our rebuttal and would greatly appreciate your thoughts. Please let us know if you have any remaining concerns or questions.

Thank you for your time!

Comments on the paper

2 comments

Dear Reviewers, First of all, we sincerely thank you for your thoughtful feedback and constructive suggestions, which have helped us strengthen our work. Following your suggestions, we have conducted and added four new experiments to the revised manuscript:

  1. Frame-level vs. chunk-level HSIC on speech data.
  2. Bottleneck dimensionality analysis on speech data.
  3. Speaker identity leakage analysis in the learned encoder representations.
  4. Identifiability margin analysis on a controlled synthetic dataset.

These additional experiments provide further empirical evidence supporting our approach and address important questions raised during the review process.

We appreciate your time and consideration and welcome any further questions or feedback.

1. Frame-level vs. chunk-level HSIC

We analyze frame level HSIC objective vs chunk level one by training two identical voice-conversion models on 10 LibriSpeech speakers (25 utterances per speaker, 82,356 total frame-level embeddings). Frame HSIC was computed across utterances at individual frames, whereas chunk HSIC operated on 2-sec segments, allowing it to penalize speaker information encoded in temporal structure. The models were trained for 500 epochs and evaluated on 50 nonparallel conversions using wav2vec2 WER/CER and x-vector target-speaker EER on seen speakers but non-seen converted samples.

ObjectiveChunk durationWER ↓CER ↓Target EER ↑
Frame HSICOne frame3.016%1.026%34%
Chunk HSIC2.0 s3.016%0.981%32%

The results show that frame-level and chunk-level HSIC achieve nearly identical performance, with the same WER (3.016%) and only minor differences in CER (1.026% vs. 0.981%) and target-speaker EER (34% vs. 32%). Importantly, chunk-level HSIC, which additionally targets speaker information encoded in temporal structure, does not provide a consistent improvement over frame-level HSIC. These findings suggest that frame-level independence is a reasonable and computationally simpler surrogate for our voice-conversion setting, although they do not establish sequence-level independence or eliminate the possibility of temporal speaker leakage.

Bottleneck dimensionality analysis

By using the same setup as in (1) we evaluated how bottleneck capacity affects frame-level HSIC voice conversion. Frozen WavLM features were encoded by our model into bottlenecks of 1-1024 dimensions and reconstructed by a speaker-conditioned decoder.

DimWER ↓CER ↓Target EER ↑
10243.480%1.070%24%
5123.364%1.048%24%
2563.364%1.048%22%
1283.364%1.093%24%
643.480%1.026%24%
323.596%1.048%26%
163.016%1.026%34%
83.248%1.026%38%
43.712%1.338%42%
28.121%3.233%46%
126.334%12.330%42%

The results provide empirical evidence that bottleneck dimensionality plays an important role in speaker-information suppression, suggesting that disentanglement is encouraged not only by the explicit HSIC objective but also by the architectural information bottleneck. Reducing the dimension from 512 to 8 increases Target EER from 24% to 38%, while WER remains comparable (3.364% vs. 3.248%). This suggests that a more restrictive representation can suppress speaker-specific information without sacrificing linguistic content.

Speaker identity leakage analysis

We evaluated speaker leakage using 50 randomly selected VoxCeleb1 speakers, with 10 three-second training utterances and 10 session-disjoint test utterances per speaker. Two ReDimNet2-B0 classifiers were trained from scratch for 100 epochs: one on raw waveforms and one on latent sequences produced by pre-trained IVC encoder. WavLM and the VC model were never updated. The raw-audio model achieved 19.2% accuracy, while the latent-based model achieved 6.4%, compared with a 2% chance baseline. These results show that the IVC encoder representation substantially suppresses speaker identity relative to the original waveform, while retaining only limited residual speaker information

Input TypeValidation AccuracyMacro-F1Random Chance
Raw waveform19.20%17.75%2.00%
Frozen WavLM → VC latent6.40%5.81%2.00%

Identifiability margin analysis

We design a controlled experiment in which IVC is trained end-to-end with a known identifiability margin Δp\Delta_p. Specifically, we consider binary content SS with probabilities ((1−Δp)/2,(1+Δp)/2)((1-\Delta_p)/2,(1+\Delta_p)/2), an independent binary condition CC, and an invertible generative function defined by condition-dependent permutations. The model is trained using only reconstruction and independence objectives, without access to content labels or paired counterfactuals. To satisfy the regularity conditions of Proposition 2, we employ an injective categorical encoder based on Sinkhorn relaxation and a norm-constrained decoder.

We evaluate conversion error across nine values of Δp∈[0,0.3]\Delta_p\in[0,0.3], using 100 random initializations per setting. As predicted by our analysis, conversion error remains a near chance for symmetric or nearly symmetric priors but decreases substantially as Δp\Delta_p increases. In particular, at Δp=0.08\Delta_p=0.08, ICAE achieves 4.77% conversion error, compared to 53% without the independence objective, and achieves zero conversion error across all 100 runs at Δp=0.16\Delta_p=0.16 and 0.300.30. These results demonstrate a non-vacuous conversion regime and support the qualitative dependence on the identifiability margin predicted by Proposition 2. Additional experimental details, including the protocol and and plot, have been added to the appendix.

We add all results to the appendix in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.