- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
The paper introduces Group-wise Independent Latent Disentanglement (GILD), a VAE-based framework that relaxes component-wise independence to group-wise (block) independence. By penalizing Group-wise Total Correlation (GTC), GILD allows arbitrary statistical dependence within predefined latent blocks specified apriori while enforcing independence across blocks. To estimate GTC per minibatch, the authors adopt an Importance Sampling (IS) estimator and prove it achieves lower variance than the Minibatch Stratified Sampling (MSS) estimator from -TCVAE. GILD is evaluated on synthetic grouped datasets (including a proposed pdSprites dataset) and real-world datasets (mouse cortex imaging and CelebA).
Critical strengths
- Problem Formulation: Relaxing the strict 1D marginal independence assumption to block-wise independence is a practical direction for preserving interpretability when human semantic concepts are known to be statistically coupled.
- Estimator Variance Analysis: The formal proof in Appendix A.3 showing that the IS minibatch estimator strictly dominates MSS in variance is a clean theoretical contribution.
- Handling Over-Specified Group Ranks: Penalizing GTC on the aggregated posterior allows over-specifying the within-group dimension. In theory, this can help excess dimensions collapse to Gaussian noise ("dummy dimensions"), but needs to be verified in practice.
Critical weaknesses
- Double Circularity of the Unsupervised GTC Evaluation: The authors propose using test-set GTC to evaluate group-wise disentanglement when ground-truth labels are unavailable. This evaluation is completely circular in two ways: (1) GTC is the exact regularization penalty GILD minimizes during training, and (2) GTC is evaluated directly using the encoder's own internal variational parameters via the Gaussian KDE in Algorithm 1. The model that produces the representation is used to parameterize its own density estimator for evaluation. Validating block independence requires an external, independently trained metric (e.g., a post-hoc predictability score like LPS, or a separate mutual information / HSIC estimator on a held-out validation split).
- Framing of Statistical Independence vs. Interpretability & Identifiability: The paper argues that factors like planar locations on an irregular support cannot satisfy p(x,y)=p(x)p(y) and must be represented jointly. However, under nonlinear transformations (e.g., nonlinear ICA), statistically independent 1D marginals can always be constructed, though doing so warps the coordinate chart and harms human interpretability. The issue is therefore semantic interpretability and coordinate parameterization, not a failure of statistical independence to exist. Furthermore, while citing Locatello et al. (2019), the paper ignores its core identifiability result: unsupervised group independence still admits infinitely many entangled/rotated solutions without structural or weak supervision, and requires a priori knowledge of the group partition. Therefore, I have doubts that the method can be competitive against other similar methods such as Locatello et al. 2020’s “Weakly-supervised disentanglement without compromises” and Fruytier et al. 2025’s “Sculpting Latent Spaces With MMD: Disentanglement With Programmable Priors”; both enable weak supervision that allow for a structure statistical dependence structure with maximum alignment to human interpretable features.
- Missing Baselines & Prior-Matching Literature: The paper overlooks foundational work on latent space structuring, specifically Sculpting the Latent Space with MMD (WAE-MMD), Orthogonality-enforced latent space in autoencoders, and weakly supervised methods (Weakly-Supervised Disentanglement Without Compromises, Locatello et al., 2020). Prior work on WAE-MMD showed that the failure of -VAE/FactorVAE stems from the per-sample ELBO KL conflicting with aggregated posterior alignment, whereas matching q(z) to a structured prior via MMD successfully recovers independent marginals and allows encoding known block-dependence structures directly into the prior.
- Unclear Metric Attribution & Evaluation Protocols: The supervised alignment matrix (fitting least-squares between estimated and true groups) closely mirrors existing linear disentanglement/completeness metrics (e.g., DCI / SAP) without proper citation or comparison. Moreover, pdSprites contains only 805 images, and it is unclear whether a strict train/test split was used when fitting the post-hoc linear regression, or how many seeds are averaged in the main figures. I would personally suggest using DCI scores and report the standard deviation here since that is the closest metric to what you are already trying to evaluate, and additionally allows for non-linearity..
Questions for authors
- A Priori Group Specification: GILD requires specifying the group partition prior to training (or grid-searching over them on real datasets). How can GILD be applied in truly unsupervised settings where the number of underlying coupled blocks is unknown, given that setting the hyperparameters arbitrarily large degrades training efficiency and model quality (as noted in Sec. 5)?
- Coordinate Parameterization vs. Intrinsic Entanglement: In pdSprites (Fig. 4a), any continuous 2D distribution on a non-rectangular support can be bijectively mapped to two independent 1D uniform variables. Are the authors claiming that independent 1D representations do not exist, or simply that Cartesian coordinates are a preferred canonical representation for human interpretability? How does this reconcile with recent literature arguing that canonical "ground-truth" coordinate choices are arbitrary?
- External Unsupervised Validation: Since test-set GTC is computed directly from the encoder's own Gaussian KDE parameters and mirrors the training loss, will the authors evaluate their representations using an independent model/metric (e.g., training a separate classifier/regressor across groups to measure cross-group predictability such as the LPS score)? Prior Adherence & WAE-MMD Comparison: In Eq. (3), when G=1, GILD reduces to a standard VAE. Given documented failures of the VAE ELBO to actually enforce their assumed prior, have you verified whether the learned group marginals adhere to the assumed prior distribution? How does GILD compare against WAE-MMD with a block-structured prior?
- Statistical Reporting & Splits: Are the values reported in Fig. 2, Fig. 3a, and Fig. 4c computed on held-out test splits, and over how many random seeds? Please provide standard deviations alongside all reported metrics in the main figures (especially Fig. 2 and Fig. 3b/c) and clarify the exact train/val/test split sizes for the 805-image pdSprites dataset.
5 comments
W4/Q5
-
mirrors DCI/SAP without citation: We agree the lineage needs stating, and we now also report the closest of the two metrics. SAP uses the same linear- matrix as our alignment but aggregates it as a top-2 gap, ; DCI instead fits a nonlinear predictor and measures the entropy of the normalized feature importance, which is a different question. We compute the group-level version of both metrics (vector groups rather than single dimensions):
method in-sample held-out SAP in-sample SAP held-out VAE 0.224 0.215 (0.017) 0.301 0.388 (0.108) ICA 0.219 0.208 (0.025) 0.541 0.700 (0.194) ISA-VAE 0.346 0.340 (0.022) 0.894 0.943 (0.046) -TCVAE 0.334 0.328 (0.019) 0.788 0.810 (0.087) GILD 0.540 0.541 (0.016) 0.989 1.025 (0.022) Held-out values are means over 20 random 80/20 splits, with standard deviations in parentheses. GILD is first under both metrics. (Its SAP marginally exceeds 1 because the second-best of a vector-valued target can be slightly negative.) We do not add full DCI: it needs discretization and a downstream regressor, whose own choices then drive the number.
-
Are the pdSprites numbers on held-out data: Yes. pdSprites is an exhaustive enumeration of a grid filtered by , so the 805 images are the whole support and no further points exist to sample. A random split is therefore a complete held-out evaluation rather than a subsample, and the table above reports it: every held-out is within 0.02 of its in-sample value, so the numbers are not inflated by scoring on the fitting points. The synthetic benchmarks use a separate 2000/1000 split, and the main text states that every method is run 10 times with different random seeds.
Q1
- How are the group rank and the number of groups chosen: Both can be over-specified, and the redundancy is then removed automatically. Set larger than needed and the surplus components collapse to approximately Gaussian noise, detectable by a normality test, the explained-variance plateau, and the within-group TC falling to zero; the normality-test -values are now a table rather than an arrow in a figure. The ultimate difficulty is indeed, and is acknowledged in the limitation section, and it is a challenge shared by unsupervised methods in general rather than one specific to GILD. We are happy to keep working on automatic selection, but it is beyond the scope of this paper.
We thank you again for your time; the changes to the evaluation protocol came directly from your comments.
Thank you for the responses and the additional experiments. Because it is pretty late in the reviewing process, I do not expect the authors to make any additional experiments going forward. Having read your replies (including across all reviews), I find that several of my core concerns were side-stepped, and the new results introduced in the rebuttal actually reinforce the limitations I raised regarding evaluation circularity and prior enforcement.
[W1/Q3]:
The rebuttal argues that test-set GTC is not circular because it is evaluated on held-out data, and points to another response as claim that GTC is preferable to other external independence metrics because GTC orders the methods in the same way as supervised ground-truth (which is is not unique).
I still see two fundamental issues with this argument:
-
Parametric Circularity: In Algorithm 1, test-set GTC is not estimated against a known ground-truth distribution or via an external density/independence model. Rather, it is approximated via a Gaussian KDE parameterized directly by the encoder’s own learned means and variances . What’s more, the metric is the exact quantity the model was trained to minimize. Just as LLMs are not benchmarked solely by their own training-objective perplexity and diffusion models are not evaluated by their own noise-prediction MSE (using external feature spaces like FID instead), evaluating disentanglement requires an external metric independent of the model's own variational density parameters.
-
Independence vs Alignment: In the table reported to Reviewer cLtK (Q3), both Standard VAE and -TCVAE achieves substantially better (lower) cross-group dependence than GILD which was my exact concern. Dismissing the independence metrics (HSIC and dCor) as "incorrect measures of group-wise independence" simply because they do not correlate with supervised linear conflates statistical independence with canonical coordinate alignment (identifiability). A baseline can learn statistically independent subspaces that are rotated or nonlinearly warped relative to the ground-truth axes (resulting in low HSIC/dCor and low linear ).
[W2/Q2]:
The rebuttal concedes that a nonlinear map can transform the constrained 2D support in pdSprites into statistically independent 1D variables, and agrees that unsupervised identifiability is impossible per Locatello et al. (2019). However, the rebuttal then claims that "an unsupervised objective has no reason to find such a map" and that forcing 1D independence loses accuracy in linear .
If an unsupervised objective has no reason to prefer a specific nonlinear coordinate chart, GILD likewise has no guarantee of preserving the canonical Cartesian coordinate chart within a 2D block outside of architectural inductive biases. No unsupervised method, as a result, is capable of enforcing independence?
Moreover, penalizing 1D-disentangled baselines for applying a nonlinear coordinate transformation by evaluating them with a linear least-squares fit (while declining to run a nonlinear metric like DCI) only measures linearity of the parameterization, not whether the underlying factors were captured.
[W3/Q4]:
The rebuttal states that the learned marginals do not match the prior, and claims that "this does not matter here" because "the KL term in the ELBO is only a normalizer."
If the ELBO KL term is merely a scale normalizer, could it simply be replaced by Batch Normalization? The authors seem to contradict themselves in their response to Reviewer cLtK (W1/Q1) where they suggest the ELBO KL term actively supplies the index-code MI penalty, the within-group TC penalty, and the marginal prior regularizer.
Moreover, the authors claim that when a group's rank (H) is over-specified, the surplus dimensions collapse into dummy dimensions and can be automatically detected via a hypothesis test. However, the GTC penalty acts strictly between groups and exerts zero pressure within a group. The only objective pushing surplus within-group dimensions toward a Gaussian distribution is the ELBO KL divergence to the Standard Gaussian prior (which you stated was purely for normalization). If the model fails to enforce the prior, the theoretical basis for detecting surplus dimensions via normality tests breaks down. This is precisely why prior literature introduced stricter distributional regularizers such as Wasserstein distance and Maximum Mean Discrepancy (WAE-MMD).
Furthermore, since GILD still requires specifying the block structure apriori, other prior matching baselines that operate under the same block specification remain directly relevant, and the paper makes no effort to offer such a comparison.
[Q1]:
I acnkowledge the authors response to Q1. Appendix (Table 4) suggest this does not work at scale.
I thank the authors for engaging with my concerns and questions, and will revise my score accordingly.
Dear Reviewer yHQ1,
We thank you for the careful reading and for the constructive comments on our work. We highly appreciate your recognition of the strengths of our paper. We have uploaded a revised PDF, and several of your points led us to run new experiments that are reported here and tabulated in the appendix. Hopefully, the following responses address your concerns and answer your questions.
W1/Q3
W2/Q2
W3/Q4