- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a particular multi-layer neural network architecture, and show theoretically that it yields identifiable representations at each layer, up to some trivial, well-defined indeterminacies. The architecture hinges on numerous assumptions, perhaps most onerous being a rank assumption on the joint first and second derivatives, which together with expansivity requirements rules out typical architectures. It also hinges on DGP assumptions, namely, a rank condition on the relationship between the true DGP's mixing and the model's linear parameter.
Critical strengths
- The paper's structure is excellent. I am very familiar with this area of the literature, and these results immediately made sense to me.
- The assumptions are clearly defined in the text. Furthermore, each assumption's heritage in the literature is well-explained and contextualized, which is very helpful to the reader.
- The framing of hierarchical identifiability is genuinely new to me, and potentially useful.
Critical weaknesses
Designing hyper-specific neural network architectures which have identifiable representations is now a fairly accessible research task, as evidenced by the ample related work cited in the submitted draft. My opinion is that for a new identifiable architecture to be a meaningful contribution, it should have some combination of: (1) a novel theoretical approach which could conceivably influence later identifiable architectures, (2) convincing empirics which show that it is more identifiable than strong baselines at matched levels of performance, or (3) some connection to existing architectures to explain emergent cases of approximate identifiability that have been documented already empirically.
Re: (1) - theory
In my opinion, this work's theoretical contributions are moderate. Assumptions 6/7 in particular are of a very unique flavour for this genre of work; they mix the functional specification of the data-generating process with the functional specification of the model. In general, this will not be even intuitively assessable a priori by a practitioner, which renders the applicability of the theory in the real-world somewhat questionable. Furthermore, the combined assumptions suggest significant expansivity requirements on the network (a layer must have at least ~twice the dimension of its parent). In my opinion, the difficulty of assessing these assumptions means that the main theoretical contribution is perhaps the proof strategy; as best I can tell, the main novelty there is the usage of Gaussian innovations in Proposition 1 to construct the smooth map and prove its rigidity. While interesting, it relies on a kind of ICA-like assumption of non-Gaussianity in the signals to separate it from the Gaussianity in the noise. It is difficult, in my opinion, for the reader to understand how onerous this assumption is in practice.
Hierarchical representations are not, in my opinion, a super novel thing here - although there do not seem to be many recent identifiability papers that consider them, all modern neural networks have multiple layers and e.g. the representation alignment literature has considered their empirical identifiability extensively (see also works which select the optimal layer for a downstream task, etc.). In other words: it is already often the case in modern applications of NNs that there is no "privileged hidden layer" like in the era of vanilla autoencoders, so the hierarchical representation identifiability framing is intriguing but not substantially novel, in my opinion.
Re: (2) - empirics
In my opinion, the main weak spot of the paper is the empirics. The authors compare to two baselines from 2016/2017 (presumably selected because they were explicitly sold as hierarchical probabilistic-style models) in a synthetic setting, investigating how well these models recover their true data-generating latents, when data is generated according to the authors' proposed model. It is perhaps unsurprising that these two baselines do not apparently identify the true latents given this bias. Perhaps a more expanded synthetic setting here is warranted, with synthetic parameters that are more easily understood (rather than violations of particular assumptions of the authors' construction) [see follow up question].
In the real setting on CIFAR-10, it is clear to me that self-supervised baselines and/or pretrained ResNets or similar at least need a mention. Their empirical identifiability is already extremely well-studied in the literature on representation alignment, and they are good at linear probing in settings like this. These results would serve to better anchor the numbers in this table.
(3) relevance to existing architectures is presumably out of scope for this work, which is fine.
My concerns are very addressable and in particular if the empirics are strengthened I will almost certainly recommend acceptance.
Questions for authors
- Can you expand the synthetic settings with more misspecification in the DGP and (perhaps) more baselines? For example, it would even be interesting to see if so-called "flatly identifiable" networks have coincidentally identifiable representations at other layers?
- Can you add other baselines (SSL, pretrained supervised models) to the real settings? Can you explore beyond CIFAR-10 as far as dataset goes?
- Can you talk more about the practicality of your assumptions? In particular, why do Assumptions 6/7 take this form of combining assumptions on the DGP and the model? And how restrictive is the role of Gaussianity of the innovations here?
- Are there connections to conventional networks already used in practice? For example, skip connections are known to help invertibility? What about normalizing flow-type models/invertible neural networks?
6 comments
(2/4): Additional synthetic experiments and baselines
(Q1, W2) Additional misspecification experiments and baselines
The original experiments examine violations of sufficient nonlinearity (Assumption 2; Figs. 2a,c), overdetermination (Assumption 3; Fig. 2b), and latent-dimension misspecification (Fig. 3b). We further conducted the following experiments.
Controlled cross-variable interactions. To examine a gradual departure from the additive structure in Assumption 1, we introduced pairwise interactions into each generating transition: where . Each output coordinate of is centered and rescaled to match the standard deviation of the corresponding additive signal coordinate, yielding . Thus, controls the relative interaction amplitude. The HIRL estimator is unchanged.
HIRL retains high latent recovery at , but recovery deteriorates as interactions strengthen, indicating robustness to weak departures from additivity (Figure 4b).
Non-Gaussian and absent innovations. To assess sensitivity to Assumption 4, we replaced Gaussian innovations with independent Laplace innovations having the same mean and variance, leaving the remaining generator settings unchanged. Latent recovery and innovation alignment remain comparable to the Gaussian setting (Table 4).
| Innovation distribution | Recovery z⁰ ↑ | Recovery z¹ ↑ | Recovery z² ↑ | Alignment ε⁰ ↑ | Alignment ε¹ ↑ | Alignment ε² ↑ | MSE ↓ |
|---|---|---|---|---|---|---|---|
| Gaussian (default) | 0.94 ± 0.03 | 0.95 ± 0.03 | 0.98 ± 0.01 | 0.97 ± 0.01 | 0.88 ± 0.01 | 0.88 ± 0.00 | 0.0056 ± 0.0008 |
| Laplace | 0.92 ± 0.04 | 0.96 ± 0.03 | 0.98 ± 0.01 | 0.96 ± 0.01 | 0.87 ± 0.02 | 0.88 ± 0.00 | 0.0059 ± 0.0009 |
Recovery also remains high without innovations (Figure 4a). In contrast, increasing the innovation dimension degrades recovery, including at the dimensional equality required by Assumption 1 (see our response to Reviewer AzW6). These results characterize the estimator’s empirical robustness and limitations, without establishing identifiability beyond the theorem’s assumptions.
MLP decoder baseline. We added a decoder with standard element-wise activations, retaining the same encoder, latent dimensions, and reconstruction objective as HIRL. This provides a controlled comparison of decoder structure.
On the original synthetic generator, the MLP decoder achieves latent recovery scores of 0.52/0.76/0.81, compared with 0.94/0.95/0.98 for HIRL (Table 1).
On CIFAR-100, reconstruction errors are similar (0.267 for HIRL versus 0.262 for the MLP decoder), while HIRL achieves higher linear-probe accuracy at z¹, z², and z³: 14.6/17.7/20.4% versus 14.3/16.7/18.4%. This supports the usefulness of the decoder constraints for downstream representation learning at comparable reconstruction quality.
Additional baselines targeting root-level identification
To address your suggestion, we compared HIRL with iVAE (Khemakhem et al., 2020) and TCL augmented with reconstruction (TCL-AE; Hyvarinen and Morioka, 2016) on deterministic hierarchies with nonstationary roots. We used reduced and expanded nonlinear features (Kr=1 and Kr=2), with matched latent widths and decoder parameterizations across methods within each setting. iVAE and TCL-AE use segment labels, whereas HIRL uses only observations; our theorem does not require root nonstationarity.
| Setting | Method | MCC z⁰ ↑ | MCC z¹ ↑ | MCC z² ↑ | MSE ↓ |
|---|---|---|---|---|---|
| Kr = 1 | HIRL (ours) | 0.437 ± 0.019 | 0.714 ± 0.050 | 0.923 ± 0.020 | 0.005 ± 0.001 |
| Kr = 1 | iVAE | 0.480 ± 0.040 | 0.812 ± 0.054 | 0.943 ± 0.022 | 0.013 ± 0.001 |
| Kr = 1 | TCL-AE | 0.777 ± 0.040 | 0.880 ± 0.032 | 0.968 ± 0.016 | 0.010 ± 0.002 |
| Kr = 2 | HIRL (ours) | 0.979 ± 0.017 | 0.980 ± 0.031 | 0.975 ± 0.015 | 0.006 ± 0.002 |
| Kr = 2 | iVAE | 0.991 ± 0.002 | 0.967 ± 0.045 | 0.983 ± 0.021 | 0.003 ± 0.003 |
| Kr = 2 | TCL-AE | 0.897 ± 0.011 | 0.842 ± 0.023 | 0.888 ± 0.016 | 0.077 ± 0.003 |
Values are mean ± standard deviation over 10 seeds. Best mean scores within each setting are shown in bold.
TCL-AE achieves the highest mean recovery at every layer with Kr=1, where Assumption 2 is violated. With Kr=2, both HIRL and iVAE achieve mean recovery above 0.95 throughout the hierarchy with low reconstruction error.
Thus, methods targeting root-level identification can also recover intermediate states empirically. iVAE's strong recovery with the expanded decoder suggests that the proposed structural constraints can be useful across estimation objectives. These results neither establish intermediate-layer identifiability nor validate our full theorem, since innovations are omitted.
We have incorporated these comparisons and sensitivity analyses into Section 5 and Appendix D.2–D.4.
(3/4): Additional image experiments
(Q2)
Additional self-supervised and supervised baselines
Thank you for suggesting these comparisons. We evaluated publicly available ResNet-34 models trained on CIFAR-100 using SimCLR (Chen et al., 2020) and supervised learning, with frozen representations and the same linear-probe protocol used for HIRL, after applying spatial global average pooling. The following table reports test accuracy for 100-class classification across backbone layers.
| Model | layer4 (512) | layer3 (256) | layer2 (128) | layer1 (64) | stem (64) |
|---|---|---|---|---|---|
| SimCLR ResNet-34 | 57.1% | 63.0% | 45.7% | 32.6% | 25.2% |
| Supervised ResNet-34 | 79.6% | 72.0% | 50.4% | 33.3% | 20.8% |
Feature dimensions are shown in parentheses. Layers are ordered from deeper to shallower for consistency with HIRL’s presentation, without implying a correspondence between their layers.
SimCLR achieves substantially higher linear-probe accuracy than HIRL. Its use of image augmentations in contrastive learning may contribute to this advantage, but architecture and training protocols also differ, so the comparison does not isolate the effect of the objective. The supervised model additionally uses class labels during representation learning. These results contextualize HIRL's downstream performance; our experiments assess structural constraints within hierarchical generative models, rather than establish competitiveness with specialized image-classification methods.
Additional high-dimensional dataset
To complement the classification probes on CIFAR-100, we added experiments on 3DIdent (Zimmermann et al., 2021), a rendered RGB image dataset with ten known continuous generating factors. We evaluate their linear predictability from each latent layer using test , and coordinate-wise alignment using MCC at the ten-dimensional root representation .
| Method | R² z⁰ ↑ | R² z¹ ↑ | R² z² ↑ | R² z³ ↑ | MCC z⁰ ↑ | Test MSE ↓ |
|---|---|---|---|---|---|---|
| HIRL (ours) | 0.585 ± 0.026 | 0.621 ± 0.022 | 0.676 ± 0.010 | 0.687 ± 0.003 | 0.598 ± 0.041 | 0.212 ± 0.001 |
| HIRL (reduced feat.) | 0.573 ± 0.027 | 0.572 ± 0.028 | 0.642 ± 0.017 | 0.661 ± 0.003 | 0.591 ± 0.039 | 0.233 ± 0.001 |
| MLP decoder | 0.542 ± 0.013 | 0.541 ± 0.013 | 0.622 ± 0.015 | 0.660 ± 0.004 | 0.514 ± 0.034 | 0.246 ± 0.001 |
| LVAE | 0.435 ± 0.020 | 0.323 ± 0.055 | 0.467 ± 0.020 | 0.486 ± 0.027 | 0.394 ± 0.018 | 0.850 ± 0.007 |
| VLAE | 0.150 ± 0.019 | 0.072 ± 0.011 | 0.176 ± 0.028 | 0.346 ± 0.025 | 0.198 ± 0.028 | 0.150 ± 0.003 |
Values are mean ± standard deviation over 10 seeds on the held-out test set. R² is averaged over the ten factors. Best mean scores in each column are bold.
HIRL achieves higher mean root MCC and higher mean linear-probe at every latent layer than the evaluated baselines. Notably, VLAE achieves lower reconstruction error but substantially lower factor-prediction scores. These results support the usefulness of the proposed constraints for learning representations aligned with known generating factors beyond our synthetic generator.
Since 3DIdent does not provide ground-truth intermediate states corresponding to our learned hierarchy, these results assess factor alignment and linear predictability, rather than recovery of the image-generating hierarchy. We have added these experiments and clarified their interpretation in Section 5, with experimental details in Appendix D.5–D.6.
(1/4): Theoretical contribution
Thank you for your insightful comments and constructive suggestions. They motivated additional experiments and helped us clarify the theoretical contribution and scope of our guarantees.
(W1) Theoretical contribution
Contribution. We agree that hierarchical representations themselves are not new. Our theoretical contribution concerns when intermediate representations recover the generating states of a stochastic hierarchy, up to specified ambiguities. Empirical alignment across trained networks addresses a related but different question: agreement between learned representations does not by itself establish recovery of the underlying generating states.
Proof strategy. Our argument combines two mechanisms.
First, Gaussian innovations, together with the Gaussian-flat subspace condition, enable identification of innovation directions. Conditional expectation introduces a Gaussian-smoothed reparameterization, and deconvolution transfers the resulting rigidity back to the original map. We discuss the role and restrictiveness of the innovation assumptions in our response to Q3.
Second, additive structure and overdetermined mixing yield cross-coordinate derivative constraints that, under Assumptions 6–7 and the remaining conditions, restrict component-wise nonlinear reparameterizations at intermediate layers to affine transformations. This restriction is essential for propagation: when an identified layer serves as the output of the next upstream transition, a general component-wise nonlinear transformation of that output need not preserve its additive signal–innovation decomposition. A layer-wise identifiability result allowing arbitrary component-wise transformations therefore cannot simply be applied recursively.
Together, these mechanisms control ambiguities across successive stochastic transitions and provide a basis for developing further hierarchically identifiable model classes.