- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors start with the interesting premise to relax model assumptions in identifiable representation learning. The term misspecified structure is somewhat odd in my view, since the central issue appears more specifically to be dependence between nuisance/noise variables across observation blocks, such that the observations are no longer conditionally independent given the latent variables. Nevertheless, the general idea makes sense: they start from an identification result based on conditional independence, characterize violations of this assumption as an operator perturbation, and ask whether identification remains possible when this perturbation is sufficiently controlled. The paper then proposes a latent-variable model with explicit latent-dependent noise and a sparsity regularizer. However, I currently see several major problems that speak against publication in the present form. Most importantly, the paper is unusually difficult to parse, the relationship between the theoretical results and the implemented estimator is unclear, and several central proof steps do not appear to establish what is claimed.
Critical strengths
The premise of the paper is its biggest strength. Given that both noise and signal variables can affect observations, under what conditions can we still identify the signal? Under the premise that noise and signal might be correlated this seems intractable at first glance, since nothing appears to stop us to exchange these variables in the argument. Establishing identification in this condition would indeed be an unexpected and interesting result. The perturbation approach the authors propose seems a interesting first step into this direction.
Critical weaknesses
Major concerns
-
The presentation makes the technical contribution very difficult to evaluate. The writing and mathematical exposition need substantial improvement. Definitions and assumptions are introduced with little motivation, and several important steps are barely explained. In particular, the relation to the previous identification result is mostly implicit. The central starting point appears to be the previous factorization by Hu & Schennach (2008); Zheng et al. (2025), but it is never explained clearly enough in the main text. E.g., how does the equality on the linear operators constitute an identification result? etc. Instead, much of the main paper consists of dense operator notation and short proof sketches, while important experimental details and additional empirical results are delegated to the appendix. For publication more care would be needed to make the main text understandable and complete on its own, without having to reference other papers or the appendix.
-
There appears to be a substantial disconnect between the formal identification results and the implemented model. The theoretical analysis treats the violation of conditional independence as an unknown perturbation. The key argument is that identification is possible if this perturbation is sufficiently small relative to an appropriate spectral separation. The implemented model, however, does not appear to rely on this result in any way. Instead the authors propose a model with separate encoders for z and noise epsilon, and fit a "correctly specified" latent model. I could not find a result establishing that optimization of this variational objective recovers the object characterized by the operator-identification theorem, or even that a population optimum of the stated objective satisfies the conditions required by Theorem 1/2. Thus, the theory seems to establish robustness of a population identification argument, whereas the algorithm explicitly models the nuisance process. It is unclear in what sense the proposed estimator is an implementation of the theory rather than simply a different latent model motivated by the same setting. This also complicates interpretation of the empirical results. Is improved recovery due to the richer model or the sparsity regularizer, and how is this related to the identification results?
-
Several central proof steps do not appear to support the claimed identification results. In Theorem 1 the authors move from “the spectrum changes only a little” to “the latent representation is still exactly identified.” I am unsure if this generally follows. Small perturbations can leave eigenvalues well separated while still changing the corresponding eigenvectors or spectral subspaces. So the argument may support approximate stability, but not exact recovery as claimed in Theorem 1.
There is a similar issue in Theorem 2. The proof uses closeness of spectral values to conclude that the conditional distributions p(x_b| z) and p(x_b | hat z) are close. But the spectrum of a multiplication operator only says which values the function takes, not which latent state z corresponds to which value. Therefore, closeness of the spectra does not by itself imply pointwise closeness of the conditional distributions.
I would therefore need a clearer and more rigorous argument connecting the perturbation of the observable operator to either exact or approximate recovery of the latent representation.
Additional concerns
The synthetic experiment is quite restricted relative to the generality of the motivating discussion. In particular, the noise epsilon is generated in a way that make the relation to z linear. Given that the paper allows an arbitrary nonlinear relation this setting would be important to test.
At present, I find the premise interesting, but I am not confident that the main theoretical claims follow from the provided proofs, and the relationship between those claims and the empirical method remains too unclear.
Questions for authors
What is the formal connection between Theorems 1–2 and the proposed variational estimator, and is there any guarantee that optimizing the stated objective recovers the theoretically identified representation?
How exactly do the operator perturbation results in Theorems 1–2 imply exact or approximate recovery of the latent representation, rather than only stability of spectral values?
5 comments
Rebuttal 2
W2 \\& Q1: Connecting theory and estimator:
A: Given the similarity between those concerns, we would like to address them together.
- Purpose of estimator:
We would like to first clarify our Theorem 1 and 2 are population-level and estimator-agnostic: given the observational distribution, they establish that the latent variable can be uniquely determined up to the equivalence class stated in Def. 2.1 and 2.2.
The first paragraph of Section 4 states that the learning objective seeks to learn by fitting the data-generating process in Eq.1. Eq.10 gives its probabilistic factorization, and Eq.16 provides the corresponding variational objective.
- Enforcing Assumptions:
Rather than quantities that the estimator is expected to recover, we instantiate the assumptions in both Theorem 1 and 2 by constructing the synthetic data, as detailed in Appendix C.1 and C.2.
- What is experimentally verified:
We verify whether the estimator recovers the known latent variables generated under those assumptions. Fig.3(a) directly visualizes component-wise alignment between and , and MCC quantitatively measures latent recovery up to permutation. Fig.4 shows the scores, which indicates the recovery of at a subspace level. Thus, the experiment validates recovery of the identifiable latent representation.
Notably, enforcing assumptions for data synthesizing and the test the identifiability on the synthetic data is the standard protocol adopted in the literature of identifying latent representations, such as [1-14], for testing identifiability results when the true latent variables and data-generating process are available.
- "Is improved recovery due to the richer model or the sparsity regularizer, and how is this related to the identification results?"
We have explained the connections between experimental results and our identifiability results on point 3.
Our full model achieves the best recovery in both synthetic experiments because it captures the data-generating process better than other methods. Meanwhile, MCRL [2] uses a sparsity penalty but scores below even W/O e on both metrics. W/O e retains our sparsity regularizer lowering both MCC in Fig. 3(b) and aligned in Fig. 4. These comparisons show that sparsity alone does not account for the full model’s gains.
Rebuttal 3
W3 \\& Q2: Clarify our identifiability results:
A: We address these concerns together.
Could the reviewer identify the specific assumption or inference in our proof that they consider insufficient for this conclusion? In particular, which step in the proof leads to the assessment that the argument supports only “approximate stability”?
- Regarding Theorem 1, our Proposition 1 establishes subspace identifiability through spectral separation.
The passage from spectral separation to precise subspace identification is addressed in Step 3 of Appendix B.3. That step uses injectivity of to distinguish latent states by their full conditional-density profiles and invokes Assumption 5 for the differentiability required by Definition 1. The conclusion is for an invertible, differentiable .
- By " But the spectrum of a multiplication operator only says which values the function takes, not which latent state z corresponds to which value", could you clarify if you are suggesting how to derive Eq.44 from Eq.43?
Equation (43) applies the perturbation bound in Eq. (5) to the corresponding spectral values described in lines 190–193. For each fixed , the spectrum of is the essential range of . This identifies the density values involved in the comparison. To obtain Eq. (44), the matched values must refer to the same pair for almost every . Under this correspondence and a bound uniform in , the essential supremum gives Eq. (45), and Assumption 6 yields the latent error bound in Eq. (9).
- Nonlinear dependence of the noise on the latent variables
Evaluating a nonlinear estimator on synthetic data with a linear dependence mechanism is an established protocol in identifiable representation learning [7, 9]. In our original benchmark, the dependence of on is linear in Eq. (73), while both the observation map in Eq. (74) and the fitted model are nonlinear. Equation (1) allows nonlinear dependence of the noise on , and our additional experiment tests that case directly.
To test the nonlinear case directly, we ran an additional experiment that applies the coordinate-wise nonlinearity from Eq. (74) to the latent-dependent term in Eq. (73), while retaining the shared and block-specific noise terms. Our full method achieved an mean MCC of 0.86, compared with 0.79 for the next-best variant, W/O s. It also achieved an with mean of 0.70, compared with 0.61 for the next-best variant, W/O e. The MCC result reproduces the advantage over W/O s seen in Fig. 3(b). The result shows that modeling the dependence of the noise on continues to improve empirical subspace recovery in the nonlinear setting.
References
-
Fu, et al., Learning General Causal Structures with Hidden Dynamic Process for Climate Analysis, ICML 2026;
-
Sun, et al., Causal representation learning from multimodal biomedical observations, ICLR 2025;
-
Brady, et al., Interaction asymmetry: A general principle for learning composable abstractions, ICLR 2025;
-
Lachapelle, et al., Additive decoders for latent variables identification and cartesian-product extrapolation, NeurIPS 2023;
-
Kong, et al., Partial disentanglement for domain adaptation, ICML 2022;
-
Von Kügelgen, et al., Self-supervised learning with data augmentations provably isolates content from style, NeurIPS 2021;
-
Kong, et al., Identification 420 of nonlinear latent hierarchical models, NeurIPS 2023;
-
Li, et al., Online time series forecasting with theoretical guarantees, NeurIPS 2025;
-
Liang, et al., Causal component analysis, NeurIPS 2023;
-
Zhang, et al., Causal representation learning from multiple distributions: A general setting, ICML 2024;
-
Hu, et al., Instrumental variable treatment of nonclassical measurement error models. Econometrica, 2008.
-
Zheng, et al., Nonparametric factor analysis and beyond. AISTATS25;
-
Li, et al., Identification of intermittent temporal latent process. ICLR25;
-
Lachapelle, et al., Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning, ICML 2023;
Rebuttal 1
A: We would appreciate it if you can specify which part of "The writing and mathematical exposition" need substantial improvement, and which part of "several important steps" are barely explained.
Regarding ""Definitions and assumptions are introduced with little motivation", we have explicitly and clearly introduce the motivation of all definitons and assumptions in the main text, and we summarize them below:
Definitions 1 and 2 (Section 2, p. 2): Immediately before Definition 1, we introduce an “invertible and differentiable” transformation. The two definitions formalizes our objective.
Definition 3 (Section 2, p. 2): The preceding remark identifies “relaxing the conditional-independence structure” as the main challenge. The paragraph after Definition 3 explains which factorization can fail and points to Fig. 1 for an illustration.
Definitions 4 and 5 (Section 3, p. 3): The opening paragraph introduces the integral and multiplication operators to characterize “structural misspecification.” Eq. (3) uses both operators to express the discrepancy from the prior factorization.
Definition 6 (Section 3.2, p. 5): The preceding paragraph explains why a quantitative recovery criterion is needed when the sufficient condition for precise identification is unavailable. It introduces the deviation from the “admissible equivalence class.”
Assumptions 1–3 (Section 3, pp. 3–4): The intuition box after Eq. (5) describes Assumption 1 as a “regularity condition,” explains that Assumption 2 prevents the operators from “collapsing distinct functions” to the same output, and links injectivity of Θ to distinguishing “distinct latent states.” On p. 3, the text following Assumption 2 shows where the bounded inverses enter Eq. (4), and the paragraph preceding Assumption 3 motivates the spectral condition by discussing repeated spectral values.
Assumptions 4 and 5 (Sections 3.1–3.2, pp. 5–6): The passages immediately before Assumption 4 and after Theorem 1 explain why the latent transformation must satisfy the differentiability requirement in Definition 1. The paragraph and intuition box after Eq. (9) explain how the inverse-Lipschitz condition in Assumption 5 connects a conditional-density discrepancy to latent recovery error. The intuition box after the Proposition 2 proof sketch also explains the roles of latent independence, structural sparsity, and the support-size condition.
Also, could the reviewer identify the passages they consider unclear, including the particular definitions, assumptions, or additional proof transitions they believe need further explanation?
On p. 3, immediately after Assumption 1, we state the factorization of [11,12] and cite Section B.1 in the same sentence. The next sentence introduces the discrepancy d(xb) in Eq. (3), which leads to our perturbation formulation in Eq. (4). The identification step raised by the reviewer is explained in Appendix B.1, Eqs. (21)–(23). Eq. (23) connects an observable operator to a latent multiplication operator. Under the assumptions of the prior result, the spectral decomposition identifies the conditional-density components up to an invertible relabeling of the latent states.
Given the main-text page limit, we use the available space to explain our contribution and analyze identification when the exact factorization fails. We therefore introduce the prior factorization briefly where our analysis uses it and direct readers to Appendix B.1 for the detailed proof. If the reviewer has an alternative organization in mind that would make this connection easier to follow within the same page limit, we would welcome a specific suggestion.
The submission page limit determines what we include in the main text. We use that space to explain the key steps of the new analysis and present the experiments that directly evaluate our recovery claims. Sections 3–3.3 state the results and give the assumption motivations and proof steps cited above. Section 5.1 specifies the experimental settings, metrics, and baselines. Section 5.2 presents and discusses the main component-wise and subspace recovery results in Figs. 3 and 4. To make room for these central parts within the page limit, we must place the remaining material in the appendix. Appendix B contains the complete proofs, Appendix C documents the synthetic data generation and assumption checks, and Appendix D presents the real-world experiments and additional ablations. This is the standard structure in the literature of identifiable representation learning. For instance, [1-13] place full proofs and supplementary experimental details in their appendices as well.