acceptodds
Under review as a conference paper at ICLR 2027

Disentangling Latent Representations by Groups, not Dimensions, via Group-wise Total Correlation

Abstract

Many unsupervised representation learning methods enforce statistical independence at the level of individual latent dimensions, typically through total-correlation (TC) penalties or ICA-style priors. Natural factors, however, are often block structured: several latent dimensions may need to co-vary on a constrained manifold while remaining independent of other groups. We introduce Group-wise Independent Latent Disentanglement (GILD), which learns such group-wise independent representations with a group-wise total correlation (GTC) objective in latent space. Experiments on synthetic grouped latents show that component-wise objectives and prior-only grouped models miss the intended group structure, demonstrating the correctness and practicality of GILD. When applied to real-world datasets, GILD further reveals interpretable multi-component groups that fully disentangled training tends to suppress, implying its versatility and effectiveness.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, partial disentanglement, group-wise independence, variational auto-encoders, total correlation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_yHQ1OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper introduces Group-wise Independent Latent Disentanglement (GILD), a VAE-based framework that relaxes component-wise independence to group-wise (block) independence. By penalizing Group-wise Total Correlation (GTC), GILD allows arbitrary statistical dependence within predefined latent blocks specified apriori while enforcing independence across blocks. To estimate GTC per minibatch, the authors adopt an Importance Sampling (IS) estimator and prove it achieves lower variance than the Minibatch Stratified Sampling (MSS) estimator from β\beta-TCVAE. GILD is evaluated on synthetic grouped datasets (including a proposed pdSprites dataset) and real-world datasets (mouse cortex imaging and CelebA).

Critical strengths
  • Problem Formulation: Relaxing the strict 1D marginal independence assumption to block-wise independence is a practical direction for preserving interpretability when human semantic concepts are known to be statistically coupled.
  • Estimator Variance Analysis: The formal proof in Appendix A.3 showing that the IS minibatch estimator strictly dominates MSS in variance is a clean theoretical contribution.
  • Handling Over-Specified Group Ranks: Penalizing GTC on the aggregated posterior allows over-specifying the within-group dimension. In theory, this can help excess dimensions collapse to Gaussian noise ("dummy dimensions"), but needs to be verified in practice.
Critical weaknesses
  • Double Circularity of the Unsupervised GTC Evaluation: The authors propose using test-set GTC to evaluate group-wise disentanglement when ground-truth labels are unavailable. This evaluation is completely circular in two ways: (1) GTC is the exact regularization penalty GILD minimizes during training, and (2) GTC is evaluated directly using the encoder's own internal variational parameters via the Gaussian KDE in Algorithm 1. The model that produces the representation is used to parameterize its own density estimator for evaluation. Validating block independence requires an external, independently trained metric (e.g., a post-hoc predictability score like LPS, or a separate mutual information / HSIC estimator on a held-out validation split).
  • Framing of Statistical Independence vs. Interpretability & Identifiability: The paper argues that factors like planar locations on an irregular support cannot satisfy p(x,y)=p(x)p(y) and must be represented jointly. However, under nonlinear transformations (e.g., nonlinear ICA), statistically independent 1D marginals can always be constructed, though doing so warps the coordinate chart and harms human interpretability. The issue is therefore semantic interpretability and coordinate parameterization, not a failure of statistical independence to exist. Furthermore, while citing Locatello et al. (2019), the paper ignores its core identifiability result: unsupervised group independence still admits infinitely many entangled/rotated solutions without structural or weak supervision, and requires a priori knowledge of the group partition. Therefore, I have doubts that the method can be competitive against other similar methods such as Locatello et al. 2020’s “Weakly-supervised disentanglement without compromises” and Fruytier et al. 2025’s “Sculpting Latent Spaces With MMD: Disentanglement With Programmable Priors”; both enable weak supervision that allow for a structure statistical dependence structure with maximum alignment to human interpretable features.
  • Missing Baselines & Prior-Matching Literature: The paper overlooks foundational work on latent space structuring, specifically Sculpting the Latent Space with MMD (WAE-MMD), Orthogonality-enforced latent space in autoencoders, and weakly supervised methods (Weakly-Supervised Disentanglement Without Compromises, Locatello et al., 2020). Prior work on WAE-MMD showed that the failure of -VAE/FactorVAE stems from the per-sample ELBO KL conflicting with aggregated posterior alignment, whereas matching q(z) to a structured prior via MMD successfully recovers independent marginals and allows encoding known block-dependence structures directly into the prior.
  • Unclear Metric Attribution & Evaluation Protocols: The supervised R2R^2 alignment matrix (fitting least-squares between estimated and true groups) closely mirrors existing linear disentanglement/completeness metrics (e.g., DCI / SAP) without proper citation or comparison. Moreover, pdSprites contains only 805 images, and it is unclear whether a strict train/test split was used when fitting the post-hoc linear R2R^2 regression, or how many seeds are averaged in the main figures. I would personally suggest using DCI scores and report the standard deviation here since that is the closest metric to what you are already trying to evaluate, and additionally allows for non-linearity..
Questions for authors
  • A Priori Group Specification: GILD requires specifying the group partition prior to training (or grid-searching over them on real datasets). How can GILD be applied in truly unsupervised settings where the number of underlying coupled blocks is unknown, given that setting the hyperparameters arbitrarily large degrades training efficiency and model quality (as noted in Sec. 5)?
  • Coordinate Parameterization vs. Intrinsic Entanglement: In pdSprites (Fig. 4a), any continuous 2D distribution on a non-rectangular support can be bijectively mapped to two independent 1D uniform variables. Are the authors claiming that independent 1D representations do not exist, or simply that Cartesian coordinates are a preferred canonical representation for human interpretability? How does this reconcile with recent literature arguing that canonical "ground-truth" coordinate choices are arbitrary?
  • External Unsupervised Validation: Since test-set GTC is computed directly from the encoder's own Gaussian KDE parameters and mirrors the training loss, will the authors evaluate their representations using an independent model/metric (e.g., training a separate classifier/regressor across groups to measure cross-group predictability such as the LPS score)? Prior Adherence & WAE-MMD Comparison: In Eq. (3), when G=1, GILD reduces to a standard VAE. Given documented failures of the VAE ELBO to actually enforce their assumed prior, have you verified whether the learned group marginals adhere to the assumed prior distribution? How does GILD compare against WAE-MMD with a block-structured prior?
  • Statistical Reporting & Splits: Are the values reported in Fig. 2, Fig. 3a, and Fig. 4c computed on held-out test splits, and over how many random seeds? Please provide standard deviations alongside all reported metrics in the main figures (especially Fig. 2 and Fig. 3b/c) and clarify the exact train/val/test split sizes for the 805-image pdSprites dataset.
5 comments

Dear Reviewer yHQ1,

We thank you for the careful reading and for the constructive comments on our work. We highly appreciate your recognition of the strengths of our paper. We have uploaded a revised PDF, and several of your points led us to run new experiments that are reported here and tabulated in the appendix. Hopefully, the following responses address your concerns and answer your questions.

W1/Q3

  • Why GTC and not another dependence measure: because GTC is the one that matches the ground truth. Taking the aligned R2R^2 as the reference, GTC orders the methods in the same way as R2R^2 on both pdSprites and the grouped synthetic data, whereas the kernel criterion the review suggested does not. The comparison is in Q3 of cLtK. That is what decides which metric to use, and it favors GTC.
  • GTC is the metric, by definition: group-wise independence is q(z)=∏gq(zg)q(\boldsymbol z)=\prod_g q(\boldsymbol z_g). GTC is the KL divergence to that factorization, so it is non-negative and zero exactly when the groups are independent. It is not a proxy or a heuristic that correlates with the property; it is the property itself. This is similar to the test log-likelihood for an MLE-trained model, or test ELBO. It is also what β\beta-TCVAE and FactorVAE do when they report total correlation on held-out data. What would be circular is refitting the model on the evaluation set; we do not. The encoder is fit on the training split and applied unchanged to held-out data, where only the density estimate is computed. Reporting GTC there measures whether the group structure generalizes, which is the question.
  • "Computed from the encoder's own parameters" is natural practice: the encoder defines the representation, so its density is what any evaluation has to work with, and every disentanglement metric we are aware of is computed from the learned latents in the same way, e.g. β\beta-TCVAE, FactorVAE. The one real caveat is that GTC is not comparable across different GG, because the group dimensionality changes with GG.

W2/Q2

  • Coordinate parameterization vs. intrinsic independence: You are right that a nonlinear map can make the constrained 2D patch in Fig. 4a look like independent 1D variables. In the limit the whole patch can even be squeezed into one dimension, because it is only a subset of the plane. We do not dispute that, and we do not claim identifiability either: without supervision it is impossible, as Locatello et al. show. Our setting is unsupervised, with the encoder and decoder fixed. The question we ask is which latent structure such a model finds useful. An unsupervised objective has no reason to find such a map. What it produces is not useful either: the coordinates become unreadable, and the empty triangular region disappears. That region is exactly what tells us that xx and yy are coupled. This also costs real accuracy: forcing full independence loses 0.40 to 0.47 in aligned R2R^2 on the grouped benchmark. So the construction the reviewer mentioned exists, but it is not a meaningful answer, not what we want.

W3/Q4

  • WAE-MMD with a block-structured prior is missing: We deliberately do not add it. Such a prior requires encoding the within-block dependence structure by hand, which is exactly the information GILD is designed not to need. For pdSprites that means specifying the non-rectangular support; for CelebA, knowing which semantic factors are coupled. The comparison would measure our ability to guess the structure, not the value of the group-wise penalty. We are glad to take this up as future work and have stated it as such in the limitations.
  • Do the learned group marginals adhere to the prior: We checked, and they do not, which does not matter here. The KL term in the ELBO is only a normalizer: it keeps the magnitude of each q(z∣x)q(z|x) near N(0,I)\mathcal N(0,I) and has never been strong enough to make the aggregated posterior match the prior, e.g., in MNIST VAE. Our measurement confirms this and shows it is not specific to GILD. Mean ∣std−1∣|\mathrm{std}-1| per group is 1.581.58 for VAE, 1.691.69 for β\beta-TCVAE and 1.671.67 for GILD, all far from the prior and indistinguishable from each other. This does not affect the GTC term, which is KL(q(z)∏gq(zg))\mathrm{KL}(q(\boldsymbol z)\prod_g q(\boldsymbol z_g)) and depends on qq only: whether q(zg)q(\boldsymbol z_g) matches N(0,I)\mathcal N(\boldsymbol 0,\boldsymbol I) has no bearing on whether the groups are independent.

W4/Q5

  • R2R^2 mirrors DCI/SAP without citation: We agree the lineage needs stating, and we now also report the closest of the two metrics. SAP uses the same linear-R2R^2 matrix as our alignment but aggregates it as a top-2 gap, (Rtop12−Rtop22)/Rtop12(R^2_{\text{top1}}-R^2_{\text{top2}})/R^2_{\text{top1}}; DCI instead fits a nonlinear predictor and measures the entropy of the normalized feature importance, which is a different question. We compute the group-level version of both metrics (vector groups rather than single dimensions):

    methodR2R^2 in-sample ↑\uparrowR2R^2 held-out ↑\uparrowSAP in-sample ↑\uparrowSAP held-out ↑\uparrow
    VAE0.2240.215 (0.017)0.3010.388 (0.108)
    ICA0.2190.208 (0.025)0.5410.700 (0.194)
    ISA-VAE0.3460.340 (0.022)0.8940.943 (0.046)
    β\beta-TCVAE0.3340.328 (0.019)0.7880.810 (0.087)
    GILD0.5400.541 (0.016)0.9891.025 (0.022)

    Held-out values are means over 20 random 80/20 splits, with standard deviations in parentheses. GILD is first under both metrics. (Its SAP marginally exceeds 1 because the second-best R2R^2 of a vector-valued target can be slightly negative.) We do not add full DCI: it needs discretization and a downstream regressor, whose own choices then drive the number.

  • Are the pdSprites numbers on held-out data: Yes. pdSprites is an exhaustive enumeration of a 17×17×517\times17\times5 grid filtered by ∣zx∣⩾∣zy∣|z_x|\geqslant|z_y|, so the 805 images are the whole support and no further points exist to sample. A random split is therefore a complete held-out evaluation rather than a subsample, and the table above reports it: every held-out R2R^2 is within 0.02 of its in-sample value, so the numbers are not inflated by scoring on the fitting points. The synthetic benchmarks use a separate 2000/1000 split, and the main text states that every method is run 10 times with different random seeds.

Q1

  • How are the group rank HH and the number of groups GG chosen: Both can be over-specified, and the redundancy is then removed automatically. Set HH larger than needed and the surplus components collapse to approximately Gaussian noise, detectable by a normality test, the explained-variance plateau, and the within-group TC falling to zero; the normality-test pp-values are now a table rather than an arrow in a figure. The ultimate difficulty is indeed, and is acknowledged in the limitation section, and it is a challenge shared by unsupervised methods in general rather than one specific to GILD. We are happy to keep working on automatic selection, but it is beyond the scope of this paper.

We thank you again for your time; the changes to the evaluation protocol came directly from your comments.

Thank you for the responses and the additional experiments. Because it is pretty late in the reviewing process, I do not expect the authors to make any additional experiments going forward. Having read your replies (including across all reviews), I find that several of my core concerns were side-stepped, and the new results introduced in the rebuttal actually reinforce the limitations I raised regarding evaluation circularity and prior enforcement.

[W1/Q3]:

The rebuttal argues that test-set GTC is not circular because it is evaluated on held-out data, and points to another response as claim that GTC is preferable to other external independence metrics because GTC orders the methods in the same way as supervised ground-truth R2R^2 (which is is not unique).

I still see two fundamental issues with this argument:

  • Parametric Circularity: In Algorithm 1, test-set GTC is not estimated against a known ground-truth distribution or via an external density/independence model. Rather, it is approximated via a Gaussian KDE parameterized directly by the encoder’s own learned means and variances q^ϕ(z∣x)\hat{q}_{\phi}(z|x). What’s more, the metric is the exact quantity the model was trained to minimize. Just as LLMs are not benchmarked solely by their own training-objective perplexity and diffusion models are not evaluated by their own noise-prediction MSE (using external feature spaces like FID instead), evaluating disentanglement requires an external metric independent of the model's own variational density parameters.

  • Independence vs Alignment: In the table reported to Reviewer cLtK (Q3), both Standard VAE and β\beta-TCVAE achieves substantially better (lower) cross-group dependence than GILD which was my exact concern. Dismissing the independence metrics (HSIC and dCor) as "incorrect measures of group-wise independence" simply because they do not correlate with supervised linear R2R^2 conflates statistical independence with canonical coordinate alignment (identifiability). A baseline can learn statistically independent subspaces that are rotated or nonlinearly warped relative to the ground-truth axes (resulting in low HSIC/dCor and low linear R2R^2).

[W2/Q2]:

The rebuttal concedes that a nonlinear map can transform the constrained 2D support in pdSprites into statistically independent 1D variables, and agrees that unsupervised identifiability is impossible per Locatello et al. (2019). However, the rebuttal then claims that "an unsupervised objective has no reason to find such a map" and that forcing 1D independence loses accuracy in linear R2R^2.

If an unsupervised objective has no reason to prefer a specific nonlinear coordinate chart, GILD likewise has no guarantee of preserving the canonical Cartesian coordinate chart within a 2D block outside of architectural inductive biases. No unsupervised method, as a result, is capable of enforcing independence?

Moreover, penalizing 1D-disentangled baselines for applying a nonlinear coordinate transformation by evaluating them with a linear least-squares R2R^2 fit (while declining to run a nonlinear metric like DCI) only measures linearity of the parameterization, not whether the underlying factors were captured.

[W3/Q4]:

The rebuttal states that the learned marginals do not match the prior, and claims that "this does not matter here" because "the KL term in the ELBO is only a normalizer."

If the ELBO KL term is merely a scale normalizer, could it simply be replaced by Batch Normalization? The authors seem to contradict themselves in their response to Reviewer cLtK (W1/Q1) where they suggest the ELBO KL term actively supplies the index-code MI penalty, the within-group TC penalty, and the marginal prior regularizer.

Moreover, the authors claim that when a group's rank (H) is over-specified, the surplus dimensions collapse into dummy dimensions and can be automatically detected via a hypothesis test. However, the GTC penalty acts strictly between groups and exerts zero pressure within a group. The only objective pushing surplus within-group dimensions toward a Gaussian distribution is the ELBO KL divergence to the Standard Gaussian prior (which you stated was purely for normalization). If the model fails to enforce the prior, the theoretical basis for detecting surplus dimensions via normality tests breaks down. This is precisely why prior literature introduced stricter distributional regularizers such as Wasserstein distance and Maximum Mean Discrepancy (WAE-MMD).

Furthermore, since GILD still requires specifying the block structure apriori, other prior matching baselines that operate under the same block specification remain directly relevant, and the paper makes no effort to offer such a comparison.

[Q1]:

I acnkowledge the authors response to Q1. Appendix (Table 4) suggest this does not work at scale.


I thank the authors for engaging with my concerns and questions, and will revise my score accordingly.

Reviewer_7CVyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Group-wise Independent Latent Disentanglement (GILD), which learns group-wise independent representations with a group-wise total correlation (GTC) objective in the correspoding latent space. Specifically, the paper extends the total correlation (TC) of individual latent factors to group-wise TC, resulting in a generalised loss as in Eq. (3). Since the GTC term is computationally intractable, the paper proposes to use an approximation in Eq. (4) instead. The core theoretical contribution of the work is in Theorem 3.1, which shows that the importance sampling (IS) batch approximation is the optimal that is unbiased and has the lowest variance. Extensive experiments including two synthetic datasets and two real datasets are conducted for demonstrating the performance of the GILD framework.

Critical strengths

The paper is well articulated to extend the single Independent Latent Disentanglement to a group-wise setting. Particularly, it constructs the loss in Eq. (3) which bridges the standard VAE and TC. The main theoretical result in Theorem 3.1 is interesting, by showing that IS batch approximation is the optimal that is unbiased and has the lowest variance. And results from extensive experiments are quite convincing.

Critical weaknesses

The paper is slightly unbalanced. Particularly, when Theorem 3.1 is one core contribution of the work, it only occupies less than one page in the paper. This makes the theoretical part of the paper underrepresented with a loss of details. For example, how is the approximation in Eq. (4) derived? Is this approximation biased, unbiased, or unknown? To a certain extent, the paper over-emphasises its experimental efforts. The reviewer recommended giving a bit more space for illustrating its main theorem and its indication. Although two synthetic datasets in the main body of the paper are totally justified, the ablation analysis provides more important information about the model than that the GILD works.

Questions for authors

The dimension of the two synthetic datasets is quite low. It is necessary to conduct a stress test of the GILD framework. By increasing the number of groups, can GILD still disentange? Will the advantage of GILD still be maintained? Or when will such an advantage be diminished or lost?

2 comments

Dear Reviewer 7CVy,

We thank you for the careful reading and for the constructive comments on our work. We highly appreciate your recognition of the strengths of our paper. We have uploaded a revised PDF, in which the theorem and its derivation now appear in the main text as you suggested. Hopefully, the following responses address your concerns and answer your questions.

W1

  • Theory is under-represented: Agreed. The theorem and its derivation move into the main text.
  • What Eq. (4) actually is: It is a plain Monte Carlo estimate. Since GTC=∫q(z)[ln⁡q(z)−∑gln⁡q(zg)]dz\mathrm{GTC} = \int q(\boldsymbol z)\big[\ln q(\boldsymbol z) - \sum_g \ln q(\boldsymbol z_g)\big]\mathrm{d}\boldsymbol z, the batch version simply draws samples from q(z)q(\boldsymbol z) and averages the integrand, and hence it is an unbiased estimation. We have revised the paper on this part.

Q1

  • Stress test with more groups: Good suggestion. We ran H=2H=2 fixed with G∈{4,8,12,16,20}G\in\{4,8,12,16,20\}, 3 seeds. Test-split latent R2↑R^2\uparrow, mean (std):
    GGVAEISA-VAEβ\beta-TCVAEGILDmargin over best baseline
    40.415 (0.067)0.539 (0.074)0.619 (0.082)0.812 (0.019)+0.193
    80.292 (0.036)0.346 (0.006)0.425 (0.036)0.573 (0.032)+0.148
    120.223 (0.010)0.245 (0.022)0.325 (0.028)0.455 (0.049)+0.130
    160.194 (0.011)0.225 (0.003)0.308 (0.056)0.397 (0.074)+0.089
    200.175 (0.014)0.189 (0.009)0.245 (0.011)0.258 (0.030)+0.014
  • When does the advantage diminish: Gradually, and essentially gone at G=20G=20: the margin falls monotonically, +0.193→+0.148→+0.130→+0.089→+0.014+0.193 \to +0.148 \to +0.130 \to +0.089 \to +0.014 from G=4G=4 to 2020, while GILD stays the best method throughout. This is expected: as G→KG\to K the group structure approaches component-wise independence, so the two objectives coincide in the limit.

We thank you again for your time and for the suggestions on presentation and for the pressure test, which we have now included.

Thanks for the authors' response and the stress test makes the message of the work stronger. I recommend acceptance of the paper.

Comments on the paper

4 comments

Dear Reviewer cLtK,

We thank you for the careful reading and for the constructive comments on our work. We highly appreciate your recognition of the strengths of our paper, in particular the XNOR counterexample and the pdSprites benchmark. We have uploaded a revised PDF. Hopefully, the following responses address your concerns and answer your questions.

W1/Q1

  • Relationship with HFVAE: We agree that GTC is HFVAE's between-block term, and we now say so directly instead of claiming novelty. Under a standard Gaussian prior, GILD(β)=HFVAE(α=1,γ=1,βH=1+β)\text{GILD}(\beta)=\text{HFVAE}(\alpha{=}1,\gamma{=}1,\beta_H{=}1{+}\beta). HFVAE is now included as a baseline.

  • Why we keep this simplification, and why it is useful: HFVAE turns two knobs at once. For group-wise dependent data that is the wrong choice: extra within-group pressure is harmful, and a block-structured prior requires the within-group geometry to be known in advance. GILD fixes the within-group weight at the ELBO default and keeps a plain Gaussian prior, so the only free knob is the between-group penalty. That is what makes it usable when the within-group structure is unknown, and the measurement supports it (linear grouped benchmark, 3 seeds, test split):

    modellatent R2R^2 ↑\uparrowGTC ↓\downarrow
    GILD / HFVAE(γ=1\gamma{=}1)0.9640 / 0.96400.647 / 0.647
    HFVAE(γ=0\gamma{=}0)0.96110.614
    HFVAE(γ=3\gamma{=}3)0.66221.122

    γ=1\gamma=1 matches GILD to three decimals, while γ=3\gamma=3 drops to β\beta-TCVAE level, so the within-group weight is a real modelling choice rather than a detail we skipped.

  • What this work contributes: A clean framework (one between-group penalty on top of a standard VAE, which makes the assumption explicit and easy to state), the benchmarks that make group-wise disentanglement testable (linear grouped, fully independent, pdSprites), and, as the experiments in Q3 show, the identification of which dependence measure actually reflects group-wise independence: GTC agrees with the ground-truth R2R^2 while the kernel criterion does not. HFVAE was evaluated on fully disentangled latents, so none of this was tested before.

  • Classical ISA is missing: We added Cardoso 1998, Hyvärinen–Hoyer 2000, Szabó 2012, and ran classical ISA (whitening, orthogonal parameterization, energy non-Gaussianity, restarts) on the linear benchmark. It does not solve it either: R2=0.567±0.010R^2=0.567\pm0.010, comparable to ICA (0.580) and below β\beta-TCVAE (0.668), far below GILD (0.964). Its group-energy criterion does separate subspaces, but not the true ones.

  • "ICA followed by grouping of dependent components" cannot be the standard solution: Linear ICA returns components that are independent by construction, so there is nothing left to group, even when those components are only spuriously independent.

W2/Q2

  • Thm 3.1 conflates three different variances: Conceded, and "optimal" is retracted. App. A.3.2 measured the spread of the inverse weights, which is not an estimator variance; the theorem concerns the density at a fixed z∗z^*; what matters in practice is the variance of GTC^\widehat{\mathrm{GTC}}. Fig. 6 is also too small: the IS/MSS difference is of relative size 1/N1/N, so it is visible at n=10n=10 and not at a large NN.
  • Is IS still the right choice then: Yes, but not because of our theorem. HFVAE adopted this estimator and distinguished it from Chen et al.'s approximations, and MWS is biased for q(z)q(z) by their own statement. Our addition is the quantitative account of the density-level variance, not new evidence that IS is better. In practice IS and MSS are identical (retraining GILD gives R2R^2 = 0.9639 for both) and only the biased MWS is slightly worse (0.9603).
  • Real-data GG and HH chosen by eye: We ran the check the review asks for. On the mouse data, over G∈{2,3,4,6,12}G\in\{2,3,4,6,12\} with K=12K=12 and 3 seeds each, reconstruction is flat (RMSE 0.0447 to 0.0472, a 5 spread across a six-fold change in GG) and seed stability is high: the mean canonical correlation between latent spaces learned with different seeds is 0.918, 0.968, 0.968, 0.987 and 0.9998 for G=4,2,3,6,12G=4,2,3,6,12. So the representation is stable; what is missing is a criterion, and we now state why none exists. An independence criterion is not comparable across GG because the group dimensionality changes, and at G=1G=1 there are no group pairs at all, so no such criterion can rank the full range. Reconstruction cannot either, since it improves monotonically toward G=KG=K, the degenerate full-disentanglement solution. Choosing GG therefore requires domain knowledge or ground truth; we state this in the conclusion instead of presenting brain-map inspection as a criterion, and the G∈{4,8,12,16,20}G\in\{4,8,12,16,20\} stress test quantifies the cost of guessing it wrong.
  • Why does the true latent score 0.332: It is a finite-sample floor, not residual dependence. The population GTC is exactly zero under group independence, but the KDE estimate of it is not, and the bias grows with GG and shrinks with NN. Reporting the ground-truth latent's value calibrates this floor, which is what makes the comparison across methods meaningful: GILD reaches 0.647 against a floor of 0.332, while the fully disentangled baselines sit at 0.91 to 1.51.

Q3

  • Which grouping is used for the baselines: For a fully disentangled method there is no group structure to read off, so we report the best achievable grouping under the same Hungarian alignment, and for the small cases also the best over all partitions, which for pdSprites (G=2G{=}2, H=2H{=}2) is 0.334.

  • An external independence test: We ran one, and the result is itself a finding. Normalized HSIC and distance correlation over all group pairs:

    methodGTC ↓\downarrowR2R^2 ↑\uparrownHSIC ↓\downarrowdCor ↓\downarrow
    VAE1.010.220.0110.116
    ICA1.760.220.0310.171
    ISA-VAE0.700.230.0250.154
    β\beta-TCVAE0.910.330.0120.117
    GILD0.680.540.0220.142

    HSIC is not a correct measure of group-wise independence: against the ground-truth R2R^2 it ranks VAE, whose representation is the least aligned with the true groups, as the most independent, and distance correlation gives the same ordering. GTC does not have this problem: it places GILD and ISA-VAE at the top and VAE and ICA at the bottom, exactly as R2R^2 does. Many dependence measures have existed for a long time, but they do not identify the one that reflects group-wise independence, and our experiments emphasize that GTC is the correct measure by its definition.

We thank you again for your time and for the comparison with HFVAE, which made the positioning of this work much clearer.

Reviewer_cLtKOpenReview ↗

Response to Rebuttal

Hi Authors,

Thank you for the response. I think most of my concerns have been addressed. I will raise my score to weak acceptance.

Thank you very much for your valuable suggestions and acknowledgement of our rebuttal!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.