Q1. Validity of the minimality assumption and necessity result
In Section 5.3.2, Assumption 3 defines the minimum-entropy encoder subject to the reconstruction constraint
d~(e~(x),c)=x∀x,c.
As written, this appears to require exact reconstruction under every possible conditioning value, rather than only under the original condition associated with each observation. Under this literal interpretation, every feasible encoder must be injective on the input support, and all feasible discrete representations have the same entropy as the input.
Consider independent discrete content and attribute variables, where the content variable has distinct atom masses and the attribute has at least two possible values. Let
f(s,c)=(s,c),e(x)=x,d((s,c),c′)=(s,c′).
This model achieves exact attribute conversion, and the identity encoder satisfies the minimum-entropy requirement under the stated all-pairs reconstruction constraint. Nevertheless, the learned representation explicitly contains the controllable attribute and is therefore not independent of it, contradicting Proposition 3. The construction also satisfies Assumptions 1 and 2.
Appendix C appears to use a different reconstruction constraint, under which the candidate representation and decoder based on the original content variable are feasible only for correctly paired input-attribute samples.
Could the authors clarify the intended quantification in Assumption 3? Should reconstruction be required only for pairs belonging to the joint data support? If so, could the authors provide a corrected formal statement and verify that Proposition 3 and the corresponding necessity claim in Theorem 1 remain valid?
This clarification is important because the necessity result is a central component of the paper's claimed theoretical characterization.
Q2. Relationship between statistical independence and the approximate conversion guarantee
The paper motivates reconstruction- and independence-based learning through Property 2 and the objective in Section 7. However, Lemma 2 relies on Assumption 6, which requires a uniform, sample-wise invariance bound across different attributes for the same content.
These conditions are not equivalent. In particular, arbitrarily small statistical dependence does not necessarily imply small sample-wise representation differences, even when Assumption 2 holds.
For example, consider independent binary variables
S∼Bernoulli(1/2+δ),C∼Bernoulli(1/2),0<∣δ∣<1/2,
with the injective generator and encoder
f(s,c)=(s,c),e(f(s,c))=s⊕c.
The corresponding inverse decoder achieves perfect reconstruction. For nonzero delta, Assumption 2 holds because the content values have distinct probabilities. As delta approaches zero, the statistical dependence between the learned representation and the attribute approaches zero, while the sample-wise representation difference remains one.
Could the authors establish additional conditions under which minimizing the dependence objective in Section 7 controls the invariance error required by Assumption 6? For example, would a quantitative lower bound on the asymmetry of the content distribution be necessary?
Furthermore, Appendix H directly optimizes a paired invariance loss using different attribute realizations of the same underlying content, rather than the dependence-based objective in Section 7. Such counterfactual pairs are generally unavailable in unpaired real-world style transfer settings.
Could the authors clarify how the numerical validation in Appendix H supports the proposed practical learning objective?
Alternatively, if no direct connection is established, could the authors clarify that Lemma 2 provides a conditional stability guarantee under a separate, stronger invariance assumption, rather than a guarantee resulting from minimizing the independence-based objective?
Q3. Applicability of the distributional asymmetry assumption
Assumption 2 excludes all nontrivial measure-preserving automorphisms of the latent content distribution. This assumption plays an essential role in Lemma 1 and Proposition 1 by ensuring consistent alignment of learned representations across different attribute conditions.
However, non-atomic continuous distributions admit nontrivial measure-preserving transformations. Consequently, Assumption 2, as currently formulated, cannot hold for such distributions, substantially restricting the applicability of the sufficient condition to continuous content representations commonly used in practical style transfer systems.
Could the authors clarify the intended scope of Assumption 2? In particular, is Proposition 1 intended to apply primarily to discrete latent variables, or is there an alternative formulation that would permit meaningful sufficient conditions for continuous content representations?
Although footnote 1 discusses extending entropy-based analysis to continuous variables, this does not appear to resolve the restriction imposed by Assumption 2.
Additionally, Appendix H employs a continuous content distribution. While this experiment evaluates Lemma 2 rather than Proposition 1 and is therefore not formally inconsistent with the theory, it illustrates that even the authors' own synthetic setting falls outside the scope of the exact-conversion sufficient condition.
Appendix E also reports distinct empirical frequencies of WavLM-based speech-unit clusters. However, these clusters are proxies for the unobserved content variable rather than direct observations of the underlying generative factors.
Could the authors explain to what extent these empirical observations support Assumption 2, and identify realistic settings in which the assumption is expected to hold?
A clearer characterization of the applicable distributions would materially strengthen the generality and significance of Proposition 1.
Q4. Distinct scientific contribution beyond prior latent-representation theory
The paper emphasizes a shift from latent recovery to synthesis-oriented guarantees. However, Shaham et al. (2022) already investigate reconstruction- and independence-based representation learning and demonstrate synthesis applications, including image synthesis and voice cloning.
I recognize that the present paper provides additional theoretical results, particularly the distributional asymmetry-based sufficient condition, necessity characterizations, and approximate conversion error bounds.
Nevertheless, the distinct scientific insights enabled by these results could be articulated more clearly.
Could the authors explain precisely which limitations or ambiguities in the theoretical guarantees of Shaham et al. (2022) are resolved by Proposition 1? In particular, does Assumption 2 eliminate an identifiability ambiguity that remains unresolved under the previous assumptions, and can the authors provide a concrete comparison illustrating this distinction?
Furthermore, what new conclusions about conditional autoencoders become possible through the necessity results and approximate conversion bounds that could not be obtained from the existing reconstruction- and independence-based framework?
Finally, could the authors identify concrete settings in which the proposed characterization would lead to different model design choices, training objectives, or evaluation procedures?
The paper need not introduce a new architecture or demonstrate state-of-the-art empirical performance to make a meaningful theoretical contribution. However, a clearer explanation of what the new characterization enables researchers to understand or do differently would substantially strengthen its significance.
Rebuttal
We thank the reviewer for the detailed review and address the raised concerns below.
W1: Positioning relative to Shaham et al. (2022) and AutoVC (also Q2)
We thank the reviewer for highlighting the importance of distinguishing our contributions from prior synthesis-oriented work. We revised the relevant sentence in Section 1 to clarify that our contribution shifts the theoretical focus from analysis to synthesis, rather than introducing synthesis as a new application. We also revised the "Non-probabilistic Autoencoders" paragraph in Section 8 to explicitly acknowledge and compare the synthesis contributions and theoretical guarantees of Shaham et al. (2022) and AutoVC. While Shaham et al. (2022) provide theoretical guarantees for latent variable recovery up to an entropy-preserving transformation, alongside a methodology for conditional synthesis, our work extends the theoretical characterization to pointwise conversion correctness. Specifically, we show that perfect reconstruction and independence, together with a distributional asymmetry assumption that eliminates nontrivial measure-preserving transformations, are sufficient to guarantee exact conversion (Proposition 1). Unlike AutoVC, which provides asymptotic distribution-level conversion guarantees under speech-specific assumptions and bottleneck constraints, our framework establishes pointwise exact conversion guarantees under explicit structural conditions, without requiring an asymptotic utterance-length regime. We further establish qualified necessity results: Proposition 2 shows that every exact converter admits an equivalent representation satisfying reconstruction and independence, while Proposition 3 establishes necessity for the original encoder under an additional minimality condition. Beyond exact conversion, Lemma 2 provides an explicit non-asymptotic conversion error bound under bounded reconstruction error, sample-wise latent invariance, and decoder Lipschitz continuity.
Together, these results distinguish our theoretical contribution from prior synthesis methodologies by identifying conditions for pointwise correctness, characterizing the necessity of these properties, and quantifying the stability of approximate conversion under explicit assumptions. We hope these clarifications better position our contribution relative to the existing literature.
W2: Interpretation and reproducibility of the experiments
Appendix E. The purpose of Appendix E is to provide an example of distributional asymmetry in speech-related content representations derived from the real-world speech data. We extract WavLM embeddings from LibriSpeech-clean-100 and cluster them into K=100 categories to construct the discrete speech-content units by capturing recurring phonetic patterns. These units serve as a proxy for the unobserved variable S. Their empirical frequencies are consistent with the asymmetric prior in Example 1. The analysis illustrates the plausibility of asymmetric content distributions in real speech data, not as a statistical verification of Assumption 2.
Regarding Appendix H, we consider a controlled synthetic setting with paired examples to investigate the informativeness and tightness of the conversion-error bound in Lemma 2. The experiment is not intended to validate the independence-based objective, but rather to demonstrate how deviations from ideal reconstruction and latent invariance affect conversion accuracy. Please note that, following the suggestion of Reviewer 2mdc, we added an additional synthetic experiment that satisfies Assumption 2 of our theoretical framework. In addition, we have introduced a new experiment on the Colored-MNIST dataset using an HSIC-based independence objective, providing further empirical evidence of the applicability of our theoretical analysis. In addition, we narrow the empirical claims to accurately reflect the scope of our experiments. Finally, we will release all code we used for these experiments including hyperparameters to improve reproducibility.
Q1: Finite-entropy scope of Proposition 3 and the continuous-extension sentence
We thank the reviewer for raising this point. To maintain simplicity and clarity of presentation, we retain the restriction of Proposition 3 to discrete random variables with finite entropy and remove the statement regarding its extension to continuous variables. Developing a rigorous continuous counterpart is nontrivial and requires additional theoretical analysis beyond the scope of the present work.
Thank you once more for the constructive and careful feedback. We welcome any further questions and would be glad to engage in additional discussion.