acceptodds
Under review as a conference paper at ICLR 2027

From Marginal Matching to Sampler Compatibility: Auditing Independent Latent Recombination

Abstract

Split-latent generators reconstruct an observation from content and style codes encoded from that same observation, yet generate by sampling the two codes independently. We formalize this train–generation shift as sampler compatibility between the encoded conditional joint and the declared sampling law . An exact expected-conditional KL identity decomposes incompatibility into style-prior mismatch, condition information in style, content-prior mismatch, and within-condition code dependence, so aggregate prior matching certifies only the first clause. We turn the identity into an audit: a direct joint test, clause-specific diagnostics, and decoder and task checks that separate latent compatibility, decoded transport, and generation competence. An exact-marginal counterexample, image and known-factor audits, and a released-checkpoint boundary case show that aggregate matching can coexist with strong leakage that the decoder actually uses. The audit also guides the repair. On retrained MNIST models, a sampler fitted to all four clauses raises generation accuracy from 33.9% to 96.4% and closes 97% of the Fréchet gap to the paired-code ceiling with higher recall and a memorization ratio near one, whereas a style-only repair recovers part of the gap and a shrunk class-style shortcut collapses diversity. A capacity-matched ablation shows that fitting the class-conditional marginals suffices, so content and style stay independent within a class but style must depend on the class. Our encoder-side contract training instead lowers the joint discrepancy by 27–38% yet worsens decoded transport in every matched pair and makes the model harder to repair.

Primary area:
generative models
Keywords:
Sampler Compatibility, Generative Models, Latent Variable Models, Disentangled Representations, Distribution Matching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pFhgOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies "split-latent generators"; models that requires two codes during inference to capture the content and the style. They posit that when you use two codes that were unseen in training time, it can lead to undesirable results. They formulate 4 error categories for why these codes might be incompatible: a style-prior mismatch, a leakage that allows you to guess one code given another, a content-prior mismatch, and independence between the content and style codes. They also propose ways to fix it. First, they change the sampler so that it samples on seen codes, rather than a fitted distribution of possible codes. They show good performance for this. Second, they refit the sampler to better reflect the distribution of seen codes.

Critical strengths

This paper works on an interesting problem and is a well-contained paper. It has not only a problem definition, but also an error-analysis, proposed fixes for the errors, and analysis on the results.

Critical weaknesses

To preface, I am not an expert in this field. However, I am finding this paper quite difficult to read. I have a suspicion that it was written solely with AI because "split latent generators" is not a common term in machine learning, yet this paper has no clear definition or citation.

Questions for authors

I'm curious whether a stronger backbone model for split-latent generation would also face this problem: like a diffusion model? This will not affect my assessment, but I am curious.

2 comments

We thank the reviewer for recognizing the problem’s interest and the connection between the diagnosis, repairs, and evaluation. We take the readability concern seriously. In particular, “split-latent generator” should have been defined through concrete examples before being used as shorthand.

Terminology and the problem being studied. By “split-latent generator,” we mean a generator whose decoder takes separately designated content and style codes. We do not intend this as an established architecture name. For example, Huang et al., MUNIT, ECCV 2018 explicitly recombine encoded content with sampled style. We will replace our initial shorthand with this definition and introduce the connection to existing models immediately.

The issue is not that individual code values were unseen during training—new values are expected in continuous latent generation. It is that independently combining codes can change their joint distribution relative to the paired codes used for reconstruction. Even individually plausible codes may form combinations that the decoder handles poorly. Whether this distributional difference affects the outputs is an empirical question, which is why we evaluate latent mismatch and decoded consequences separately.

We will also clarify the four terms. “Condition leakage” means that the designated style code carries information about the requested label; it does not mean predicting one code from the other. The fourth term measures dependence between content and style within a condition. Independence is the sampling assumption being tested, rather than an error category.

What the sampler repair does. The principal repair in Table 3 fits class-conditional mixture distributions to training-split posterior samples and draws from those distributions; it does not simply select stored code pairs. S3 fits a joint mixture, while S4 draws independently from fitted content and style mixtures within each class. The encoder and decoder remain fixed, but fitting the replacement sampler requires additional computation. This follows the established idea of ex-post density estimation, including Ghosh et al., From Variational to Deterministic Autoencoders, ICLR 2020. Our experiments examine which sampling requirements need to be addressed and whether doing so improves generation.

Would a stronger backbone, such as a diffusion model, face the same problem? Potentially, but our experiments do not establish this for diffusion models. A stronger decoder does not by itself ensure that independently recombined inputs follow the paired input distribution. However, it may be less sensitive to that difference. The relevant questions are therefore how a particular system constructs its content/style inputs and whether changing their joint distribution affects its outputs.

There is evidence that related content–style separation difficulties persist in diffusion systems. Zhu et al., Less is More, ICLR 2025 document unwanted content entering style-transfer diffusion models through style-reference features. This motivates investigating our question in those systems, but does not demonstrate the same sampling mismatch or validate our repair there. A system that directly learns and samples the appropriate joint code distribution would not make the independent substitution studied in our paper.

Regarding the concern about AI-generated writing, our AI Use Statement discloses assistance with literature search and code and states the authors’ responsibility for the mathematical claims and experimental measurements. We recognize that this disclosure does not resolve the readability concern. We will address that concern directly by defining the terminology at first use, explaining the distributional comparison in plain language, and distinguishing the audit from the repair experiments.

Thank you, authors, for the clarifications. I still think the readability concern is too significant for me to raise my score. I've alerted ACs again that this is not in my expertise.

Reviewer_Uj7gOpenReview ↗
Rating
1(−2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this work, the authors study generative models that are conditioned on separate content and style codes during training (typically from the same training data item). Then at inference, the codes can be sampled independently and can then cause train-test mismatch. The authors itemize the sources of mismatch and propose tests to study the sources of mismatch. Evaluation is done in the image-domain via MNIST, Fashion-MNIST, and CIFAR-10. The authors show that fitting new class-conditional sampling distributions improves generation accuracy while mostly keeping diversity without retraining the encoder or decoder (still requires fitting the new sampling distribution).

Critical strengths

I generally really like the framing of auditing the different sources of issues. The work focuses on diagnosing code issues and their sources rather than typical algorithmic novelty, which I am fine with. This framing is a strength because it organizes the different sources of train-test mismatch. The related work section is nice, concise, and useful, and sentence structure is fine.

Critical weaknesses
  1. Relevance to real applications and current models. The paper is heavily motivated by applications involving the separation of content and style, but there is relatively little discussion of applications, qualitative evaluation, and how real-world the approach could be on state-of-the-art, large-data models and the application frontier. Much of the evaluation is a controlled stress test. To me, this is likely the largest issue, but could be managable. Clearer qualitative examples and failure inputs/outputs on recent models would improve the work, as would discussion of more application papers.

A core issue in the application domain is how to define content versus style, as this is typically not clear-cut. The discussion somewhat glosses over these real application issues. It would be useful to expand the discussion and explain how the proposed approach fits into the broader issues of content-by-style disentanglement. The comparison of generated versus data content/style codes could also be clearer.

  1. Scientific insight and a more actionable practitioner protocol. The conclusions are more or less what we expect: matching the style distribution does not ensure good independent sampling, and detectable latent mismatch does not necessarily mean bad generation. There is also less direct prescription for fixing issues when they arise. This can be fine, but it would be useful to expand this aspect. The paper explicitly has a section on 'A protocol for practitioners,' but it is relatively short and I would not say it is immediately easy for a practitioner to read and apply. I would love for this to be expanded and become more central and woven throughout. This would make the diagnostic framing more useful in practice.
Questions for authors

What current evidence supports applying this approach to state-of-the-art, large-data models beyond the controlled stress tests? Are qualitative examples or failure inputs/outputs available that make this connection clearer?

How do the paper's definitions of content and style relate to applications where the separation is ambiguous, and where does the proposed approach fit within the broader disentanglement problem?

Do you disagree that the main conclusions are largely what we would expect? What additional insight should a practitioner take away, and what concrete action should follow when each diagnostic identifies an issue?

2 comments

We thank the reviewer for recognizing the diagnostic contribution. We agree that the application scope and practical guidance need to be clearer.

1. Evidence beyond controlled stress tests and qualitative examples

Our experiments establish the mechanism and its consequences in controlled settings; they do not establish effectiveness on large-scale, high-resolution generators. The cross-family comparison nevertheless distinguishes different regimes: in Table 2, F-CS-WAE has 91.9% style-probe accuracy and 12.4% prior-style digit retention, whereas DIVA has 10.7% and 89.5%, respectively. These are common-backbone adaptations, which limits the generalization claim.

The application connection is concrete: Huang et al., MUNIT, ECCV 2018 recombine encoded content with sampled style. More recently, Zhu et al., ICLR 2025 document content leakage through style-reference features in diffusion models. This supports investigating the issue in current systems, without validating our particular audit or repair there.

Figure 2 provides qualitative comparisons using four samplers with the same CIFAR-10 encoder and decoder. Section 5.4 further reports donor-swap interventions: decoded identity follows the style donor in 99.28% of MNIST and 63.92% of CIFAR-10 cases in that experiment. These connect latent leakage to decoder behavior, but do not replace evaluation on recent large-scale models. We will clarify this scope and explain the qualitative evidence more directly.

2. Ambiguous content/style definitions and disentanglement

Our definitions are task-dependent: content is the code intended to carry the requested condition; style is the designated residual code. The intended condition must therefore be specified before interpreting the audit. We do not claim a unique semantic decomposition or identifiability. Locatello et al., ICML 2019 establish why assumptions and inductive biases matter for unsupervised disentanglement. Our narrower question is whether the declared sampling procedure matches the encoded distribution for a designated split.

We will clarify the three comparisons: paired codes from held-out observations versus sampled codes; their outputs through the same fixed decoder; and generated outputs versus real images. Distributional compatibility alone does not establish semantic disentanglement or generation quality.

3. Additional insight and concrete practitioner actions

We agree that marginal matching does not imply joint matching. The empirical contribution concerns which violations require intervention. Table 3 reports similar performance for independent and joint class-conditional mixtures despite detected within-class dependence. Conversely, Table 4 shows FACT reducing the joint latent statistic by 27–38% on MNIST while increasing decoded discrepancy by 7–20%. Thus, modelling detected dependence or reducing latent proxies need not improve the reported output metrics.

We will expand the protocol around four actions: refit the style sampler for style-prior mismatch; test class-conditional style sampling for condition leakage when the goal is class-conditional generation; refit the conditional content sampler for content-prior mismatch; and compare independent with joint conditional samplers when within-class dependence is detected. Each intervention must be checked for decoded fidelity, condition retention, diversity, and preservation of the intended capability.

Post-hoc density fitting has precedent in Ghosh et al., ICLR 2020. Our contribution connects such interventions to diagnosed sampling requirements and evaluates their consequences with the encoder and decoder fixed.

The tradeoffs matter: conditional style sampling accommodates leakage rather than removing it and does not repair cross-class style transfer. On CIFAR-10, S3 improves accuracy from 26.6% to 45.8% and FD from 18.5 to 12.4, while recall falls from 0.87 to 0.78. We will make these limitations central to the practical guidance.

Thanks for the clear and candid response. The task-dependent definition of content and style is reasonable, and the four-action protocol is what I was hoping for. I'd put it in one table: diagnostic, test, action, and what to check after the fix.

My main concern still stands and are reinforced by the other reviewers.

Given that, I've lowered my score to 1 The response concedes the first and only promises the second. I do think the audit framing is valuable, and a version with the protocol up front and a test on at least one modern generator would be much stronger.

Reviewer_X4fcOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that latent generative models with style and content latent variables are trained where the latent variables are drawn jointly and hence can have dependence, while during generation they are drawn independently. This creates a potential mismatch between the joint distributions during training and generation. The authors show that the KL between the training-time and generation-time joint distributions can be decomposed into four terms capturing style-prior mismatch, content-prior mismatch, and dependence between class and style, and dependence of content and style with a class. The authors argue that current work is concerned only with constraining the style prior term during training and thus overlooks the other parts. The authors then quantify the magnitudes of the different terms in real-world examples and propose methods to address the overlooked terms.

Critical strengths

None

Critical weaknesses

Hard to Understand/ Readable

  • In general, it has heavy tail-tale signs of pure AI-generated text (Many small paragraphs, technical terms used before being introduced).
  • This makes it significantly hard to read overall, and after multiple re-readings, I still found the experimental narrative and justifications difficult to follow.

Theorem 1 is applying a standard result

The Theorem applies standard KL and MI identities and does not seem theorem-worthy (I cannot evaluate whether they have already been applied in this way in the literature / close prior work). Consequently, the contribution must rest on the empirical findings. Due to the presentation, I could unfortunately not examine this properly.

I want to emphasize that I cannot evaluate the paper's technical usefulness, as this is completely outside my area of research. However, I strongly believe that every paper should be accessible to readers with a general ML background, so they can understand the basic ideas behind the goal and experiments. This is unfortunately clearly not the case.

Questions for authors

None

2 comments

We thank the reviewer for reading the paper carefully despite the difficulties with its presentation. We recognize that the terminology and experimental narrative made the contribution harder to assess than it should have been. We will simplify the introduction, define the content–style setup before introducing our terminology, and explain what each experiment is intended to establish.

The basic problem is straightforward: during reconstruction, the decoder receives content and style codes extracted from the same observation; during generation, these codes are sampled independently. The decoder may therefore encounter combinations unlike those used for reconstruction. Content–style recombination is an established operation—for example, Huang et al.’s MUNIT (ECCV 2018) combines encoded content with sampled style. Our study asks whether the change in input distribution affects the decoder and which changes to the sampler are actually needed. We will explain this motivation before introducing “sampler compatibility,” and make clear that “split-latent generator” is simply our shorthand for a decoder with separately designated content and style inputs.

We agree with the reviewer that Theorem 1 uses standard KL and mutual-information identities. Relevant precedents include Hoffman and Johnson’s ELBO surgery (2016) and Chen et al.’s Isolating Sources of Disentanglement in Variational Autoencoders (NeurIPS 2018). We do not claim new information-theoretic identities. The decomposition organizes the comparison against the model’s actual sampling law, Py=p(zc∣y)p(zs)P_y=p(z_c\mid y)p(z_s), and identifies the distributional requirements that this sampling operation imposes. We will make this limited role explicit beside the result. We also agree that the paper’s value must be assessed primarily through the empirical findings.

Those findings distinguish three situations that call for different responses. In Section 5.1, an independence test reaches 80% power while a held-out linear probe achieves only 51.2% accuracy in a balanced binary setting: a rejection alone gives little indication of practical severity. In Table 3, independent class-conditional mixtures perform similarly to the joint mixture despite detected within-class dependence. Thus, modelling that dependence provides no clear additional benefit on the reported metrics for these checkpoints. Table 4 shows a different difficulty: on MNIST, FACT reduces the joint latent statistic by 27–38%, yet increases the decoded discrepancy by 7–20% across matched pairs. These results explain why we evaluate what happens after decoding rather than treating every detected latent mismatch as something that stronger regularization will necessarily fix.

One further clarification is important for interpreting the evidence: we do not numerically estimate the four KL terms. As stated in Section 3 and Table 1, MMD and HSIC test the associated equalities; their values cannot be added or interpreted as estimates of the KL decomposition. Corollary 1 concerns KL itself, not the finite-sample proxies used in training. We will bring this distinction into the experimental explanation and organize the results around the three decisions above. We take the concern about the prose seriously and will address the specific obstacles to understanding the setup, evidence, and scope of the conclusions.

Thank you for your explanations. Unfortunately, I could not find any changes in the manuscript; as such, my main concerns regarding readability and presentation persist, and it seems that reviewer pFhg shares these concerns. Moreover, as you state yourself, the decomposition uses basic information-theoretic identities, and the core contribution of your work seems to apply these identities to end up with a form where single terms are useful for analyzing your problem (i.e., the different distribution mismatches regarding content & style). While this is totally fine, I would therefore recommend writing it as a proposition. At this point, I want to reiterate that I cannot judge the technical usefulness of this approach/analysis and therefore have to stick to my rating of the overall presentation of the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.