acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Identifiability in Multilayer Representation Learning

Abstract

Despite the hierarchical structure exhibited by many real-world generative processes, identifiability theory in representation learning has primarily focused on flat generative models, leaving intermediate representations in hierarchical models largely unexplored. We show that hierarchical structure fundamentally changes the identifiability problem: intermediate representations are governed by admissible inter-layer reparameterizations, which must be controlled across successive layers. We study a class of hierarchical stochastic models with additive nonlinear feature maps, overdetermined linear mixing, and layer-wise innovations. Under explicit nondegeneracy and innovation assumptions, we establish hierarchical identifiability from observational data alone: nonlinear ambiguities at intermediate layers reduce to permutation and component-wise affine transformations, while the root representation is identifiable up to permutation and component-wise local diffeomorphisms. We further show that identifiability can propagate even when only a sufficiently informative subset of intermediate coordinates is affinely identifiable. The proposed structural constraints can be incorporated into simple reconstruction-based estimation. Experiments on synthetic hierarchical data support the qualitative predictions of the theory, while experiments on high-dimensional image data suggest that the resulting identifiability-inspired architectural constraints can provide useful inductive biases beyond the idealized theoretical setting.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
identifiable representation learning, hierarchical representation learning, latent variable models, nonlinear generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dBNkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a particular multi-layer neural network architecture, and show theoretically that it yields identifiable representations at each layer, up to some trivial, well-defined indeterminacies. The architecture hinges on numerous assumptions, perhaps most onerous being a rank assumption on the joint first and second derivatives, which together with expansivity requirements rules out typical architectures. It also hinges on DGP assumptions, namely, a rank condition on the relationship between the true DGP's mixing and the model's linear parameter.

Critical strengths
  • The paper's structure is excellent. I am very familiar with this area of the literature, and these results immediately made sense to me.
  • The assumptions are clearly defined in the text. Furthermore, each assumption's heritage in the literature is well-explained and contextualized, which is very helpful to the reader.
  • The framing of hierarchical identifiability is genuinely new to me, and potentially useful.
Critical weaknesses

Designing hyper-specific neural network architectures which have identifiable representations is now a fairly accessible research task, as evidenced by the ample related work cited in the submitted draft. My opinion is that for a new identifiable architecture to be a meaningful contribution, it should have some combination of: (1) a novel theoretical approach which could conceivably influence later identifiable architectures, (2) convincing empirics which show that it is more identifiable than strong baselines at matched levels of performance, or (3) some connection to existing architectures to explain emergent cases of approximate identifiability that have been documented already empirically.

Re: (1) - theory

In my opinion, this work's theoretical contributions are moderate. Assumptions 6/7 in particular are of a very unique flavour for this genre of work; they mix the functional specification of the data-generating process with the functional specification of the model. In general, this will not be even intuitively assessable a priori by a practitioner, which renders the applicability of the theory in the real-world somewhat questionable. Furthermore, the combined assumptions suggest significant expansivity requirements on the network (a layer must have at least ~twice the dimension of its parent). In my opinion, the difficulty of assessing these assumptions means that the main theoretical contribution is perhaps the proof strategy; as best I can tell, the main novelty there is the usage of Gaussian innovations in Proposition 1 to construct the smooth map and prove its rigidity. While interesting, it relies on a kind of ICA-like assumption of non-Gaussianity in the signals to separate it from the Gaussianity in the noise. It is difficult, in my opinion, for the reader to understand how onerous this assumption is in practice.

Hierarchical representations are not, in my opinion, a super novel thing here - although there do not seem to be many recent identifiability papers that consider them, all modern neural networks have multiple layers and e.g. the representation alignment literature has considered their empirical identifiability extensively (see also works which select the optimal layer for a downstream task, etc.). In other words: it is already often the case in modern applications of NNs that there is no "privileged hidden layer" like in the era of vanilla autoencoders, so the hierarchical representation identifiability framing is intriguing but not substantially novel, in my opinion.

Re: (2) - empirics

In my opinion, the main weak spot of the paper is the empirics. The authors compare to two baselines from 2016/2017 (presumably selected because they were explicitly sold as hierarchical probabilistic-style models) in a synthetic setting, investigating how well these models recover their true data-generating latents, when data is generated according to the authors' proposed model. It is perhaps unsurprising that these two baselines do not apparently identify the true latents given this bias. Perhaps a more expanded synthetic setting here is warranted, with synthetic parameters that are more easily understood (rather than violations of particular assumptions of the authors' construction) [see follow up question].

In the real setting on CIFAR-10, it is clear to me that self-supervised baselines and/or pretrained ResNets or similar at least need a mention. Their empirical identifiability is already extremely well-studied in the literature on representation alignment, and they are good at linear probing in settings like this. These results would serve to better anchor the numbers in this table.

(3) relevance to existing architectures is presumably out of scope for this work, which is fine.

My concerns are very addressable and in particular if the empirics are strengthened I will almost certainly recommend acceptance.

Questions for authors
  1. Can you expand the synthetic settings with more misspecification in the DGP and (perhaps) more baselines? For example, it would even be interesting to see if so-called "flatly identifiable" networks have coincidentally identifiable representations at other layers?
  2. Can you add other baselines (SSL, pretrained supervised models) to the real settings? Can you explore beyond CIFAR-10 as far as dataset goes?
  3. Can you talk more about the practicality of your assumptions? In particular, why do Assumptions 6/7 take this form of combining assumptions on the DGP and the model? And how restrictive is the role of Gaussianity of the innovations here?
  4. Are there connections to conventional networks already used in practice? For example, skip connections are known to help invertibility? What about normalizing flow-type models/invertible neural networks?
6 comments
AuthorsOpenReview ↗

(1/4): Theoretical contribution

Thank you for your insightful comments and constructive suggestions. They motivated additional experiments and helped us clarify the theoretical contribution and scope of our guarantees.

(W1) Theoretical contribution

Contribution. We agree that hierarchical representations themselves are not new. Our theoretical contribution concerns when intermediate representations recover the generating states of a stochastic hierarchy, up to specified ambiguities. Empirical alignment across trained networks addresses a related but different question: agreement between learned representations does not by itself establish recovery of the underlying generating states.

Proof strategy. Our argument combines two mechanisms.

First, Gaussian innovations, together with the Gaussian-flat subspace condition, enable identification of innovation directions. Conditional expectation introduces a Gaussian-smoothed reparameterization, and deconvolution transfers the resulting rigidity back to the original map. We discuss the role and restrictiveness of the innovation assumptions in our response to Q3.

Second, additive structure and overdetermined mixing yield cross-coordinate derivative constraints that, under Assumptions 6–7 and the remaining conditions, restrict component-wise nonlinear reparameterizations at intermediate layers to affine transformations. This restriction is essential for propagation: when an identified layer serves as the output of the next upstream transition, a general component-wise nonlinear transformation of that output need not preserve its additive signal–innovation decomposition. A layer-wise identifiability result allowing arbitrary component-wise transformations therefore cannot simply be applied recursively.

Together, these mechanisms control ambiguities across successive stochastic transitions and provide a basis for developing further hierarchically identifiable model classes.

AuthorsOpenReview ↗

(2/4): Additional synthetic experiments and baselines

(Q1, W2) Additional misspecification experiments and baselines

The original experiments examine violations of sufficient nonlinearity (Assumption 2; Figs. 2a,c), overdetermination (Assumption 3; Fig. 2b), and latent-dimension misspecification (Fig. 3b). We further conducted the following experiments.

Controlled cross-variable interactions. To examine a gradual departure from the additive structure in Assumption 1, we introduced pairwise interactions into each generating transition: zℓ+1=Aℓrℓ(zℓ)+αh~ℓ(zℓ)+Bℓϵℓ,z^{\ell+1} = A^\ell r^\ell(z^\ell) +\alpha\widetilde h^\ell(z^\ell) +B^\ell\epsilon^\ell, where hℓ(z)=Hℓ[tanh⁡(zi)tanh⁡(zj)](i,j)∈Pℓh^\ell(z)=H^\ell[\tanh(z_i)\tanh(z_j)]_{(i,j)\in{{\mathcal P}\ell}}. Each output coordinate of hℓh^\ell is centered and rescaled to match the standard deviation of the corresponding additive signal coordinate, yielding h~ℓ\widetilde h^\ell. Thus, α\alpha controls the relative interaction amplitude. The HIRL estimator is unchanged.

HIRL retains high latent recovery at α=0.1\alpha=0.1, but recovery deteriorates as interactions strengthen, indicating robustness to weak departures from additivity (Figure 4b).

Non-Gaussian and absent innovations. To assess sensitivity to Assumption 4, we replaced Gaussian innovations with independent Laplace innovations having the same mean and variance, leaving the remaining generator settings unchanged. Latent recovery and innovation alignment remain comparable to the Gaussian setting (Table 4).

Innovation distributionRecovery z⁰ ↑Recovery z¹ ↑Recovery z² ↑Alignment ε⁰ ↑Alignment ε¹ ↑Alignment ε² ↑MSE ↓
Gaussian (default)0.94 ± 0.030.95 ± 0.030.98 ± 0.010.97 ± 0.010.88 ± 0.010.88 ± 0.000.0056 ± 0.0008
Laplace0.92 ± 0.040.96 ± 0.030.98 ± 0.010.96 ± 0.010.87 ± 0.020.88 ± 0.000.0059 ± 0.0009

Recovery also remains high without innovations (Figure 4a). In contrast, increasing the innovation dimension degrades recovery, including at the dimensional equality required by Assumption 1 (see our response to Reviewer AzW6). These results characterize the estimator’s empirical robustness and limitations, without establishing identifiability beyond the theorem’s assumptions.

MLP decoder baseline. We added a decoder with standard element-wise activations, retaining the same encoder, latent dimensions, and reconstruction objective as HIRL. This provides a controlled comparison of decoder structure.

On the original synthetic generator, the MLP decoder achieves latent recovery scores of 0.52/0.76/0.81, compared with 0.94/0.95/0.98 for HIRL (Table 1).

On CIFAR-100, reconstruction errors are similar (0.267 for HIRL versus 0.262 for the MLP decoder), while HIRL achieves higher linear-probe accuracy at z¹, z², and z³: 14.6/17.7/20.4% versus 14.3/16.7/18.4%. This supports the usefulness of the decoder constraints for downstream representation learning at comparable reconstruction quality.

Additional baselines targeting root-level identification

To address your suggestion, we compared HIRL with iVAE (Khemakhem et al., 2020) and TCL augmented with reconstruction (TCL-AE; Hyvarinen and Morioka, 2016) on deterministic hierarchies with nonstationary roots. We used reduced and expanded nonlinear features (Kr=1 and Kr=2), with matched latent widths and decoder parameterizations across methods within each setting. iVAE and TCL-AE use segment labels, whereas HIRL uses only observations; our theorem does not require root nonstationarity.

SettingMethodMCC z⁰ ↑MCC z¹ ↑MCC z² ↑MSE ↓
Kr = 1HIRL (ours)0.437 ± 0.0190.714 ± 0.0500.923 ± 0.0200.005 ± 0.001
Kr = 1iVAE0.480 ± 0.0400.812 ± 0.0540.943 ± 0.0220.013 ± 0.001
Kr = 1TCL-AE0.777 ± 0.0400.880 ± 0.0320.968 ± 0.0160.010 ± 0.002
Kr = 2HIRL (ours)0.979 ± 0.0170.980 ± 0.0310.975 ± 0.0150.006 ± 0.002
Kr = 2iVAE0.991 ± 0.0020.967 ± 0.0450.983 ± 0.0210.003 ± 0.003
Kr = 2TCL-AE0.897 ± 0.0110.842 ± 0.0230.888 ± 0.0160.077 ± 0.003

Values are mean ± standard deviation over 10 seeds. Best mean scores within each setting are shown in bold.

TCL-AE achieves the highest mean recovery at every layer with Kr=1, where Assumption 2 is violated. With Kr=2, both HIRL and iVAE achieve mean recovery above 0.95 throughout the hierarchy with low reconstruction error.

Thus, methods targeting root-level identification can also recover intermediate states empirically. iVAE's strong recovery with the expanded decoder suggests that the proposed structural constraints can be useful across estimation objectives. These results neither establish intermediate-layer identifiability nor validate our full theorem, since innovations are omitted.

We have incorporated these comparisons and sensitivity analyses into Section 5 and Appendix D.2–D.4.

AuthorsOpenReview ↗

(3/4): Additional image experiments

(Q2)

Additional self-supervised and supervised baselines

Thank you for suggesting these comparisons. We evaluated publicly available ResNet-34 models trained on CIFAR-100 using SimCLR (Chen et al., 2020) and supervised learning, with frozen representations and the same linear-probe protocol used for HIRL, after applying spatial global average pooling. The following table reports test accuracy for 100-class classification across backbone layers.

Modellayer4 (512)layer3 (256)layer2 (128)layer1 (64)stem (64)
SimCLR ResNet-3457.1%63.0%45.7%32.6%25.2%
Supervised ResNet-3479.6%72.0%50.4%33.3%20.8%

SimCLR checkpoint

Supervised checkpoint

Feature dimensions are shown in parentheses. Layers are ordered from deeper to shallower for consistency with HIRL’s presentation, without implying a correspondence between their layers.

SimCLR achieves substantially higher linear-probe accuracy than HIRL. Its use of image augmentations in contrastive learning may contribute to this advantage, but architecture and training protocols also differ, so the comparison does not isolate the effect of the objective. The supervised model additionally uses class labels during representation learning. These results contextualize HIRL's downstream performance; our experiments assess structural constraints within hierarchical generative models, rather than establish competitiveness with specialized image-classification methods.

Additional high-dimensional dataset

To complement the classification probes on CIFAR-100, we added experiments on 3DIdent (Zimmermann et al., 2021), a rendered RGB image dataset with ten known continuous generating factors. We evaluate their linear predictability from each latent layer using test R2R^2, and coordinate-wise alignment using MCC at the ten-dimensional root representation z0z^0.

MethodR² z⁰ ↑R² z¹ ↑R² z² ↑R² z³ ↑MCC z⁰ ↑Test MSE ↓
HIRL (ours)0.585 ± 0.0260.621 ± 0.0220.676 ± 0.0100.687 ± 0.0030.598 ± 0.0410.212 ± 0.001
HIRL (reduced feat.)0.573 ± 0.0270.572 ± 0.0280.642 ± 0.0170.661 ± 0.0030.591 ± 0.0390.233 ± 0.001
MLP decoder0.542 ± 0.0130.541 ± 0.0130.622 ± 0.0150.660 ± 0.0040.514 ± 0.0340.246 ± 0.001
LVAE0.435 ± 0.0200.323 ± 0.0550.467 ± 0.0200.486 ± 0.0270.394 ± 0.0180.850 ± 0.007
VLAE0.150 ± 0.0190.072 ± 0.0110.176 ± 0.0280.346 ± 0.0250.198 ± 0.0280.150 ± 0.003

Values are mean ± standard deviation over 10 seeds on the held-out test set. R² is averaged over the ten factors. Best mean scores in each column are bold.

HIRL achieves higher mean root MCC and higher mean linear-probe R2R^2 at every latent layer than the evaluated baselines. Notably, VLAE achieves lower reconstruction error but substantially lower factor-prediction scores. These results support the usefulness of the proposed constraints for learning representations aligned with known generating factors beyond our synthetic generator.

Since 3DIdent does not provide ground-truth intermediate states corresponding to our learned hierarchy, these results assess factor alignment and linear predictability, rather than recovery of the image-generating hierarchy. We have added these experiments and clarified their interpretation in Section 5, with experimental details in Appendix D.5–D.6.

Reviewer_AzW6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies identifiability of intermediate representations in hierarchical latent-variable models. The authors argue that flat identifiability results don't transfer to hierarchies because invertible maps can be inserted between layers without changing the observed distribution. Under explicit nondegeneracy and innovation assumptions, the authors establish hierarchical identifiability from observational data alone. Experiments on synthetic hierarchies support the theory, and results on CIFAR-100 test the architecture as an inductive bias.

Critical strengths
  1. The result needs no independence among coordinates within a latent state, which is a meaningful relaxation compared with nonlinear-ICA-style results.
  2. The paper states clearly why hierarchical identifiability is not a layer-wise application of flat results. The framing is clear and useful.
Critical weaknesses
  1. Assumption 1 requires dϵl=dzl+1−dzld_\epsilon^l=d_z^{l+1}-d_z^l. However, the experiments use (dϵ0,dϵ1,dϵ2)=(3,9,27)(d^0_\epsilon,d^1_\epsilon,d^2_\epsilon)=(3,9,27). The experiments therefore do not test the theorem.
  2. The dimensional requirements apply restrictions at every layer. It may limit applicability greatly. It is unclear whether common systems satisfy these requirements.
Questions for authors
  1. Assumption 1 requires dϵl=dzl+1−dzld_\epsilon^l=d_z^{l+1}-d_z^l. However, the experiments use (dϵ0,dϵ1,dϵ2)=(3,9,27)(d^0_\epsilon,d^1_\epsilon,d^2_\epsilon)=(3,9,27). The experiments therefore do not test the theorem. Please check whether synthetic experiments violate assumptions that are needed for the theorem.
  2. The dimensional requirements apply restrictions at every layer. It may limit applicability greatly. It is unclear whether common systems satisfy these requirements.
1 comment

(W1, Q1) Violation of the dimensional requirement and additional sensitivity analysis

We agree that the default experiments do not satisfy the dimensional condition for transition-level local invertibility in Assumption 1. Specifically, the theorem requires dϵℓ=dzℓ+1−dzℓd_\epsilon^\ell=d_{\bf z}^{\ell+1}-d_{\bf z}^\ell, whereas our default configuration uses dϵℓ=0.3dzℓd_\epsilon^\ell=0.3d_{\bf z}^\ell.

As noted in Appendix D, the reduced innovation dimension was adopted because larger innovation spaces degraded estimation and latent recovery. The default experiments therefore assess the qualitative structural predictions of the theory under a relaxed configuration, rather than directly validate the full theorem.

We varied $d_\epsilon^\ell/d_{\bf z}^\ell$ from 0 to 2, with 2 corresponding to the dimensional equality required for our threefold layer expansion. Figure 4a shows high latent recovery without innovations and at the tested ratios of 0.1–0.3, but substantial deterioration at larger ratios, including the dimensionally matched setting. Increasing the ratio from 0.3 to 2 raises the total number of root-state and innovation components from 49 to 270, which may contribute to estimation difficulty, although the experiment does not isolate its cause.

These results suggest that empirical usefulness is not confined to the default ratio of 0.3, while revealing limitations of the current estimator at larger ratios. Dimensional matching alone does not ensure local invertibility, and empirical recovery does not establish that the fitted models satisfy Assumptions 6–7. Thus, these experiments do not constitute validation under the full theorem assumptions or establish identifiability outside their scope.

We have added the sensitivity analysis to Appendix D.2 and Figure 4, and summarized its findings in Section 5, clarifying the distinction between empirical recovery and the scope of the theoretical guarantee. Developing estimation procedures better suited to larger innovation spaces remains an important direction for future work.

(W2, Q2) Dimension expansion

We agree that the dimensional requirements restrict the hierarchies covered by our theorem.

Dimensional requirements compared with prior work

In HIRL, Assumptions 2–3 imply dzℓ+1>2dzℓd_{\bf z}^{\ell+1}>2d_{\bf z}^\ell. Related expansion requirements arise in prior works from rank conditions on decoder derivatives. The following table compares their dimensional implications for an input latent dimension of d=10d=10, with singleton blocks. These dimensional inequalities are not sufficient conditions for identifiability by themselves.

FrameworkDimensional requirementOutput dimension for d=10d=10
Lachapelle et al. (2023b), Assumption 2dx≥2dd_{\bf x} \ge 2d≥20\ge 20
Brady et al. (2025), first-order interactions, Appendix H.1dx≥2dd_{\bf x} \ge 2d≥20\ge 20
Brady et al. (2025), second-order interactions, Appendix H.1dx≥d(d+5)/2d_{\bf x} \ge d(d+5)/2≥75\ge 75
HIRL, Assumptions 2–3, per transitiondzℓ+1>2dzℓd_{\bf z}^{\ell+1}>2d_{\bf z}^\ell≥21\ge 21

The per-transition dimensional lower bound implied by Assumptions 2–3 is thus comparable in scale to those in existing decoder-based results, although the frameworks address different model classes and impose additional conditions, including HIRL's null-space nondegeneracy condition.

HIRL’s additional restriction is that expansion must hold at every transition, so its effect compounds with depth. We acknowledge that this is a substantive limitation: our current guarantee excludes constant-width and contracting generative hierarchies.

Applicability to real-world systems

We agree that whether common real-world systems satisfy these assumptions requires application-specific justification, which our experiments do not establish. Our theoretical contribution is to identify explicit sufficient conditions under which intermediate generating states can be recovered.

Theorem 2 allows propagation through an informative, affinely identifiable subset of next-layer variables, under the stated restricted conditions. This relaxes full next-layer affine identifiability but does not remove the dimensional restrictions.

Separately, our synthetic ablations provide empirical evidence of recovery under certain departures from the assumptions. The CIFAR-100 experiments evaluate downstream classification utility, while the newly added 3DIdent experiments (Section 5) evaluate alignment with and linear predictability of known generating factors (also see our reply to Reviewers GmEd and dBNk). These results support the usefulness of the proposed constraints as inductive biases beyond the theoretical setting, without establishing recovery of a true generating hierarchy.

We have clarified the cumulative dimensional restriction and the need for application-specific justification in Section 6.

Reviewer_GmEdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether intermediate latent representations in hierarchical generative models can be identified from observational data.

Critical strengths
  1. The paper addresses an important theoretical problem.
  2. The recursive theoretical formulation is interesting. This paper develops a backward induction argument that connects the identifiability of adjacent layers and distinguishes the ambiguity of intermediate representations from that of the root representation.
Critical weaknesses

I am not an expert on the identifiability problem, but I am familiar with the hierarchical latent variables. The problem itself is interesting to ask if the assumed generative model is correct, whether its latent variables be uniquely recovered. However, I find it hard to understand the practical meaning of the identifiability guarantee. Specifically, does the assumed generative model actually describe the real data, and do its latent variables correspond to meaningful underlying factors?

Secondly, the paper verifies the learned latent variables based on synthetic data, where the authors construct a known hierarchical generative process, so they have access to the true latent variables. While, on real data CIFAR100, the authors evaluate learned representations through linear classification probes. Better classification accuracy indicates that the learned features contain more linearly accessible class information, but it does not demonstrate that these features recover the true latent variables or the true hierarchical structure of image formation. More critically, in real-world data, we generally do not know whether a unique hierarchical latent structure exists. So, even if the authors prove that this decomposition is identifiable, identifiability within a restricted model class does not imply that the identified variables are the true underlying factors of the data.

Questions for authors

See Weaknesses.

3 comments

Thank you for raising this concern. We agree that identifiability within an assumed model class must be distinguished from evidence that the model describes a particular dataset.

1. Relation to hierarchical representation learning

Learning a hierarchy that explains the observations does not by itself establish that its intermediate states recover those of the generating process. Even under correct model specification, intermediate representations may admit reparameterizations that leave the observations unchanged.

Our contribution is to establish conditions that control this ambiguity across a stochastic hierarchy, restricting the remaining ambiguities to permutation and component-wise affine transformations at intermediate levels, and permutation and component-wise local diffeomorphisms at the root. Identifying the root alone does not, in general, provide this guarantee for intermediate states. We have clarified in the Introduction how hierarchical identifiability complements hierarchical representation learning. For further details on the proof strategy and why flat identifiability results cannot simply be applied recursively, please see our response to Reviewer dBNk (W1).

2. Practical meaning and scope of the guarantee

We agree that our theorem does not establish that a particular real-world dataset follows the assumed model or that its latent states correspond to human-interpretable categories. Model validity and identifiability are separate questions: application-specific evidence is needed to justify the former, while our theory addresses ambiguities that can remain even when the model is valid.

The guarantee is therefore relevant when intermediate representations are interpreted as underlying states: under the stated assumptions, it specifies which aspects of those states are determined by the observations. We have explicitly clarified this conditional scope in the Introduction.

3. Additional evaluation using known generating factors

To complement the classification probes on CIFAR-100, we added experiments on 3DIdent (Zimmermann et al., 2021), a rendered RGB image dataset with ten known continuous generating factors. We evaluate their linear predictability from each latent layer using test R2R^2, and coordinate-wise alignment using MCC at the ten-dimensional root representation z0\bf z^0.

Values are mean ± standard deviation over 10 seeds, evaluated on the held-out test set. Linear-probe R2R^2 is averaged over the ten factors.

MethodR² z⁰ ↑R² z¹ ↑R² z² ↑R² z³ ↑MCC z⁰ ↑Test MSE ↓
HIRL (ours)0.585 ± 0.0260.621 ± 0.0220.676 ± 0.0100.687 ± 0.0030.598 ± 0.0410.212 ± 0.001
HIRL (reduced feat.)0.573 ± 0.0270.572 ± 0.0280.642 ± 0.0170.661 ± 0.0030.591 ± 0.0390.233 ± 0.001
MLP decoder0.542 ± 0.0130.541 ± 0.0130.622 ± 0.0150.660 ± 0.0040.514 ± 0.0340.246 ± 0.001
LVAE0.435 ± 0.0200.323 ± 0.0550.467 ± 0.0200.486 ± 0.0270.394 ± 0.0180.850 ± 0.007
VLAE0.150 ± 0.0190.072 ± 0.0110.176 ± 0.0280.346 ± 0.0250.198 ± 0.0280.150 ± 0.003

HIRL achieves the highest mean root MCC and linear-probe R2R^2 at every latent layer among the evaluated methods. Notably, VLAE achieves lower reconstruction error but substantially lower factor-prediction scores.

Root MCC measures coordinate-wise alignment with the ten known factors, while linear probes measure their accessibility from each latent layer. We report MCC at the root because its dimension matches the number of factors. However, all layers are evaluated against the same factors, and 3DIdent does not provide ground-truth intermediate states corresponding to our learned hierarchy. These results therefore support factor recovery and predictability, but do not establish recovery of the image-generating hierarchy or verify our model assumptions. Likewise, the CIFAR-100 probes assess downstream representation utility. We have clarified these distinctions in Section 5.

Nevertheless, these experiments provide evidence that the proposed architectural constraints can serve as useful inductive biases for learning representations aligned with known generating factors, even beyond the setting covered by our theoretical guarantees.

Thanks for the responses. The relationship to hierarchical representation learning is clear, while I do still have concerns about the limitations of the proposed model when tested on real-world data. It seems that the evaluation of the proposed method requires synthetic data for which we know the underlying hidden factors. I feel this limitation is quite important, and a way to demonstrate the practical capability of the learned representations is needed. As such, I tend to maintain my score.

AuthorsOpenReview ↗

Follow-up Clarifications for Reviewer GmEd

Thank you for clarifying your concern.

Known factors are not required for training or application.

HIRL learns from observations alone. Known factors are needed only to directly assess whether the learned representations recover them. This evaluation requirement is not specific to HIRL and should be distinguished from a limitation on where the method can be applied.

Evidence of representation usefulness.

The added 3DIdent experiment evaluates factor recovery on an independently developed image-rendering benchmark beyond our own synthetic generator. HIRL learns from images alone and achieves a root MCC of 0.598 ± 0.041. For context, Zimmermann et al. (2021, Table 4, last row) report a test MCC of 0.4634 ± 0.0159 for contrastive learning using SimCLR-style image augmentations. Because protocols differ, this provides context for our recovery scores rather than a controlled comparison with self-supervised learning (SSL).

These results support HIRL's usefulness for learning factor-aligned image representations beyond our proposed generating model. We acknowledge that 3DIdent is a rendered benchmark, while CIFAR-100 assesses downstream classification utility rather than latent-factor recovery. Broader practical usefulness on real-world tasks remains to be established.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.