- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies "split-latent generators"; models that requires two codes during inference to capture the content and the style. They posit that when you use two codes that were unseen in training time, it can lead to undesirable results. They formulate 4 error categories for why these codes might be incompatible: a style-prior mismatch, a leakage that allows you to guess one code given another, a content-prior mismatch, and independence between the content and style codes. They also propose ways to fix it. First, they change the sampler so that it samples on seen codes, rather than a fitted distribution of possible codes. They show good performance for this. Second, they refit the sampler to better reflect the distribution of seen codes.
Critical strengths
This paper works on an interesting problem and is a well-contained paper. It has not only a problem definition, but also an error-analysis, proposed fixes for the errors, and analysis on the results.
Critical weaknesses
To preface, I am not an expert in this field. However, I am finding this paper quite difficult to read. I have a suspicion that it was written solely with AI because "split latent generators" is not a common term in machine learning, yet this paper has no clear definition or citation.
Questions for authors
I'm curious whether a stronger backbone model for split-latent generation would also face this problem: like a diffusion model? This will not affect my assessment, but I am curious.
2 comments
Thank you, authors, for the clarifications. I still think the readability concern is too significant for me to raise my score. I've alerted ACs again that this is not in my expertise.
We thank the reviewer for recognizing the problem’s interest and the connection between the diagnosis, repairs, and evaluation. We take the readability concern seriously. In particular, “split-latent generator” should have been defined through concrete examples before being used as shorthand.
Terminology and the problem being studied. By “split-latent generator,” we mean a generator whose decoder takes separately designated content and style codes. We do not intend this as an established architecture name. For example, Huang et al., MUNIT, ECCV 2018 explicitly recombine encoded content with sampled style. We will replace our initial shorthand with this definition and introduce the connection to existing models immediately.
The issue is not that individual code values were unseen during training—new values are expected in continuous latent generation. It is that independently combining codes can change their joint distribution relative to the paired codes used for reconstruction. Even individually plausible codes may form combinations that the decoder handles poorly. Whether this distributional difference affects the outputs is an empirical question, which is why we evaluate latent mismatch and decoded consequences separately.
We will also clarify the four terms. “Condition leakage” means that the designated style code carries information about the requested label; it does not mean predicting one code from the other. The fourth term measures dependence between content and style within a condition. Independence is the sampling assumption being tested, rather than an error category.
What the sampler repair does. The principal repair in Table 3 fits class-conditional mixture distributions to training-split posterior samples and draws from those distributions; it does not simply select stored code pairs. S3 fits a joint mixture, while S4 draws independently from fitted content and style mixtures within each class. The encoder and decoder remain fixed, but fitting the replacement sampler requires additional computation. This follows the established idea of ex-post density estimation, including Ghosh et al., From Variational to Deterministic Autoencoders, ICLR 2020. Our experiments examine which sampling requirements need to be addressed and whether doing so improves generation.
Would a stronger backbone, such as a diffusion model, face the same problem? Potentially, but our experiments do not establish this for diffusion models. A stronger decoder does not by itself ensure that independently recombined inputs follow the paired input distribution. However, it may be less sensitive to that difference. The relevant questions are therefore how a particular system constructs its content/style inputs and whether changing their joint distribution affects its outputs.
There is evidence that related content–style separation difficulties persist in diffusion systems. Zhu et al., Less is More, ICLR 2025 document unwanted content entering style-transfer diffusion models through style-reference features. This motivates investigating our question in those systems, but does not demonstrate the same sampling mismatch or validate our repair there. A system that directly learns and samples the appropriate joint code distribution would not make the independent substitution studied in our paper.
Regarding the concern about AI-generated writing, our AI Use Statement discloses assistance with literature search and code and states the authors’ responsibility for the mathematical claims and experimental measurements. We recognize that this disclosure does not resolve the readability concern. We will address that concern directly by defining the terminology at first use, explaining the distributional comparison in plain language, and distinguishing the audit from the repair experiments.