acceptodds
Under review as a conference paper at ICLR 2027

Towards Identifiable Representations under Misspecified Structure

Abstract

The presence of noise that depends on the latent variables poses a fundamental challenge to identifiability. Existing results rely on conditional independence among the observations given the latent variables. We study a more general misspecified structure, where this conditional factorization does not hold, and establish both precise and approximate identifiability guarantees. We characterize structural misspecification as a perturbed factor analysis problem. For precise identifiability, we establish subspace identifiability under spectral separation and controlled perturbation, followed by component-wise identifiability under structural sparsity. When the precise condition is not guaranteed, we derive an approximate subspace-identifiability theorem. Based on these results, we develop an unsupervised variational estimator for recovering latent variables. Experiments demonstrate the effectiveness of the proposed framework.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Causality, Identifiable Representation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SJpoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors start with the interesting premise to relax model assumptions in identifiable representation learning. The term misspecified structure is somewhat odd in my view, since the central issue appears more specifically to be dependence between nuisance/noise variables across observation blocks, such that the observations are no longer conditionally independent given the latent variables. Nevertheless, the general idea makes sense: they start from an identification result based on conditional independence, characterize violations of this assumption as an operator perturbation, and ask whether identification remains possible when this perturbation is sufficiently controlled. The paper then proposes a latent-variable model with explicit latent-dependent noise and a sparsity regularizer. However, I currently see several major problems that speak against publication in the present form. Most importantly, the paper is unusually difficult to parse, the relationship between the theoretical results and the implemented estimator is unclear, and several central proof steps do not appear to establish what is claimed.

Critical strengths

The premise of the paper is its biggest strength. Given that both noise and signal variables can affect observations, under what conditions can we still identify the signal? Under the premise that noise and signal might be correlated this seems intractable at first glance, since nothing appears to stop us to exchange these variables in the argument. Establishing identification in this condition would indeed be an unexpected and interesting result. The perturbation approach the authors propose seems a interesting first step into this direction.

Critical weaknesses

Major concerns

  1. The presentation makes the technical contribution very difficult to evaluate. The writing and mathematical exposition need substantial improvement. Definitions and assumptions are introduced with little motivation, and several important steps are barely explained. In particular, the relation to the previous identification result is mostly implicit. The central starting point appears to be the previous factorization by Hu & Schennach (2008); Zheng et al. (2025), but it is never explained clearly enough in the main text. E.g., how does the equality on the linear operators constitute an identification result? etc. Instead, much of the main paper consists of dense operator notation and short proof sketches, while important experimental details and additional empirical results are delegated to the appendix. For publication more care would be needed to make the main text understandable and complete on its own, without having to reference other papers or the appendix.

  2. There appears to be a substantial disconnect between the formal identification results and the implemented model. The theoretical analysis treats the violation of conditional independence as an unknown perturbation. The key argument is that identification is possible if this perturbation is sufficiently small relative to an appropriate spectral separation. The implemented model, however, does not appear to rely on this result in any way. Instead the authors propose a model with separate encoders for z and noise epsilon, and fit a "correctly specified" latent model. I could not find a result establishing that optimization of this variational objective recovers the object characterized by the operator-identification theorem, or even that a population optimum of the stated objective satisfies the conditions required by Theorem 1/2. Thus, the theory seems to establish robustness of a population identification argument, whereas the algorithm explicitly models the nuisance process. It is unclear in what sense the proposed estimator is an implementation of the theory rather than simply a different latent model motivated by the same setting. This also complicates interpretation of the empirical results. Is improved recovery due to the richer model or the sparsity regularizer, and how is this related to the identification results?

  3. Several central proof steps do not appear to support the claimed identification results. In Theorem 1 the authors move from “the spectrum changes only a little” to “the latent representation is still exactly identified.” I am unsure if this generally follows. Small perturbations can leave eigenvalues well separated while still changing the corresponding eigenvectors or spectral subspaces. So the argument may support approximate stability, but not exact recovery as claimed in Theorem 1.
    There is a similar issue in Theorem 2. The proof uses closeness of spectral values to conclude that the conditional distributions p(x_b| z) and p(x_b | hat z) are close. But the spectrum of a multiplication operator only says which values the function takes, not which latent state z corresponds to which value. Therefore, closeness of the spectra does not by itself imply pointwise closeness of the conditional distributions.
    I would therefore need a clearer and more rigorous argument connecting the perturbation of the observable operator to either exact or approximate recovery of the latent representation.

Additional concerns

The synthetic experiment is quite restricted relative to the generality of the motivating discussion. In particular, the noise epsilon is generated in a way that make the relation to z linear. Given that the paper allows an arbitrary nonlinear relation this setting would be important to test.

At present, I find the premise interesting, but I am not confident that the main theoretical claims follow from the provided proofs, and the relationship between those claims and the empirical method remains too unclear.

Questions for authors

What is the formal connection between Theorems 1–2 and the proposed variational estimator, and is there any guarantee that optimizing the stated objective recovers the theoretically identified representation?

How exactly do the operator perturbation results in Theorems 1–2 imply exact or approximate recovery of the latent representation, rather than only stability of spectral values?

5 comments
AuthorsOpenReview ↗

Rebuttal 1

W1: writing

A: We would appreciate it if you can specify which part of "The writing and mathematical exposition" need substantial improvement, and which part of "several important steps" are barely explained.

  1. Motivation for the definitions and assumptions :

Regarding ""Definitions and assumptions are introduced with little motivation", we have explicitly and clearly introduce the motivation of all definitons and assumptions in the main text, and we summarize them below:

  • Definitions 1 and 2 (Section 2, p. 2): Immediately before Definition 1, we introduce an “invertible and differentiable” transformation. The two definitions formalizes our objective.

  • Definition 3 (Section 2, p. 2): The preceding remark identifies “relaxing the conditional-independence structure” as the main challenge. The paragraph after Definition 3 explains which factorization can fail and points to Fig. 1 for an illustration.

  • Definitions 4 and 5 (Section 3, p. 3): The opening paragraph introduces the integral and multiplication operators to characterize “structural misspecification.” Eq. (3) uses both operators to express the discrepancy from the prior factorization.

  • Definition 6 (Section 3.2, p. 5): The preceding paragraph explains why a quantitative recovery criterion is needed when the sufficient condition for precise identification is unavailable. It introduces the deviation from the “admissible equivalence class.”

  • Assumptions 1–3 (Section 3, pp. 3–4): The intuition box after Eq. (5) describes Assumption 1 as a “regularity condition,” explains that Assumption 2 prevents the operators from “collapsing distinct functions” to the same output, and links injectivity of Θ\Theta to distinguishing “distinct latent states.” On p. 3, the text following Assumption 2 shows where the bounded inverses enter Eq. (4), and the paragraph preceding Assumption 3 motivates the spectral condition by discussing repeated spectral values.

  • Assumptions 4 and 5 (Sections 3.1–3.2, pp. 5–6): The passages immediately before Assumption 4 and after Theorem 1 explain why the latent transformation must satisfy the differentiability requirement in Definition 1. The paragraph and intuition box after Eq. (9) explain how the inverse-Lipschitz condition in Assumption 5 connects a conditional-density discrepancy to latent recovery error. The intuition box after the Proposition 2 proof sketch also explains the roles of latent independence, structural sparsity, and the support-size condition.

Also, could the reviewer identify the passages they consider unclear, including the particular definitions, assumptions, or additional proof transitions they believe need further explanation?

  1. Role of the prior operator factorization:

On p. 3, immediately after Assumption 1, we state the factorization of [11,12] and cite Section B.1 in the same sentence. The next sentence introduces the discrepancy d(xb)d(x_b) in Eq. (3), which leads to our perturbation formulation in Eq. (4). The identification step raised by the reviewer is explained in Appendix B.1, Eqs. (21)–(23). Eq. (23) connects an observable operator to a latent multiplication operator. Under the assumptions of the prior result, the spectral decomposition identifies the conditional-density components up to an invertible relabeling of the latent states.

Given the main-text page limit, we use the available space to explain our contribution and analyze identification when the exact factorization fails. We therefore introduce the prior factorization briefly where our analysis uses it and direct readers to Appendix B.1 for the detailed proof. If the reviewer has an alternative organization in mind that would make this connection easier to follow within the same page limit, we would welcome a specific suggestion.

  1. Placement of proof and experimental details:

The submission page limit determines what we include in the main text. We use that space to explain the key steps of the new analysis and present the experiments that directly evaluate our recovery claims. Sections 3–3.3 state the results and give the assumption motivations and proof steps cited above. Section 5.1 specifies the experimental settings, metrics, and baselines. Section 5.2 presents and discusses the main component-wise and subspace recovery results in Figs. 3 and 4. To make room for these central parts within the page limit, we must place the remaining material in the appendix. Appendix B contains the complete proofs, Appendix C documents the synthetic data generation and assumption checks, and Appendix D presents the real-world experiments and additional ablations. This is the standard structure in the literature of identifiable representation learning. For instance, [1-13] place full proofs and supplementary experimental details in their appendices as well.

AuthorsOpenReview ↗

Rebuttal 2

W2 \\& Q1: Connecting theory and estimator:

A: Given the similarity between those concerns, we would like to address them together.

  1. Purpose of estimator:

We would like to first clarify our Theorem 1 and 2 are population-level and estimator-agnostic: given the observational distribution, they establish that the latent variable can be uniquely determined up to the equivalence class stated in Def. 2.1 and 2.2.

The first paragraph of Section 4 states that the learning objective seeks to learn p(x∣z)p(x|z) by fitting the data-generating process in Eq.1. Eq.10 gives its probabilistic factorization, and Eq.16 provides the corresponding variational objective.

  1. Enforcing Assumptions:

Rather than quantities that the estimator is expected to recover, we instantiate the assumptions in both Theorem 1 and 2 by constructing the synthetic data, as detailed in Appendix C.1 and C.2.

  1. What is experimentally verified:

We verify whether the estimator recovers the known latent variables generated under those assumptions. Fig.3(a) directly visualizes component-wise alignment between zz and z^\hat{z}, and MCC quantitatively measures latent recovery up to permutation. Fig.4 shows the R2R^2 scores, which indicates the recovery of zz at a subspace level. Thus, the experiment validates recovery of the identifiable latent representation.

Notably, enforcing assumptions for data synthesizing and the test the identifiability on the synthetic data is the standard protocol adopted in the literature of identifying latent representations, such as [1-14], for testing identifiability results when the true latent variables and data-generating process are available.

  1. "Is improved recovery due to the richer model or the sparsity regularizer, and how is this related to the identification results?"

We have explained the connections between experimental results and our identifiability results on point 3.

Our full model achieves the best recovery in both synthetic experiments because it captures the data-generating process better than other methods. Meanwhile, MCRL [2] uses a sparsity penalty but scores below even W/O e on both metrics. W/O e retains our sparsity regularizer lowering both MCC in Fig. 3(b) and aligned R2R^2 in Fig. 4. These comparisons show that sparsity alone does not account for the full model’s gains.

AuthorsOpenReview ↗

Rebuttal 3

W3 \\& Q2: Clarify our identifiability results:

A: We address these concerns together.

Could the reviewer identify the specific assumption or inference in our proof that they consider insufficient for this conclusion? In particular, which step in the proof leads to the assessment that the argument supports only “approximate stability”?

  1. Regarding Theorem 1, our Proposition 1 establishes subspace identifiability through spectral separation.

The passage from spectral separation to precise subspace identification is addressed in Step 3 of Appendix B.3. That step uses injectivity of Θ\Theta to distinguish latent states by their full conditional-density profiles and invokes Assumption 5 for the differentiability required by Definition 1. The conclusion is z^=h(z)\hat z=h(z) for an invertible, differentiable hh.

  1. By " But the spectrum of a multiplication operator only says which values the function takes, not which latent state z corresponds to which value", could you clarify if you are suggesting how to derive Eq.44 from Eq.43?

Equation (43) applies the perturbation bound in Eq. (5) to the corresponding spectral values described in lines 190–193. For each fixed xbx_b, the spectrum of Λxb∣z\Lambda_{x_b\mid z} is the essential range of z↦p(xb∣z)z\mapsto p(x_b\mid z). This identifies the density values involved in the comparison. To obtain Eq. (44), the matched values must refer to the same pair (z,z^)(z,\hat z) for almost every xbx_b. Under this correspondence and a bound uniform in xbx_b, the essential supremum gives Eq. (45), and Assumption 6 yields the latent error bound in Eq. (9).

  1. Nonlinear dependence of the noise on the latent variables

Evaluating a nonlinear estimator on synthetic data with a linear dependence mechanism is an established protocol in identifiable representation learning [7, 9]. In our original benchmark, the dependence of ϵv\epsilon^v on zz is linear in Eq. (73), while both the observation map in Eq. (74) and the fitted model are nonlinear. Equation (1) allows nonlinear dependence of the noise on zz, and our additional experiment tests that case directly.

To test the nonlinear case directly, we ran an additional experiment that applies the coordinate-wise nonlinearity Ψ\Psi from Eq. (74) to the latent-dependent term CvzC^v z in Eq. (73), while retaining the shared and block-specific noise terms. Our full method achieved an mean MCC of 0.86, compared with 0.79 for the next-best variant, W/O s. It also achieved an R2R^2 with mean of 0.70, compared with 0.61 for the next-best variant, W/O e. The MCC result reproduces the advantage over W/O s seen in Fig. 3(b). The R2R^2 result shows that modeling the dependence of the noise on zz continues to improve empirical subspace recovery in the nonlinear setting.

References

  1. Fu, et al., Learning General Causal Structures with Hidden Dynamic Process for Climate Analysis, ICML 2026;

  2. Sun, et al., Causal representation learning from multimodal biomedical observations, ICLR 2025;

  3. Brady, et al., Interaction asymmetry: A general principle for learning composable abstractions, ICLR 2025;

  4. Lachapelle, et al., Additive decoders for latent variables identification and cartesian-product extrapolation, NeurIPS 2023;

  5. Kong, et al., Partial disentanglement for domain adaptation, ICML 2022;

  6. Von Kügelgen, et al., Self-supervised learning with data augmentations provably isolates content from style, NeurIPS 2021;

  7. Kong, et al., Identification 420 of nonlinear latent hierarchical models, NeurIPS 2023;

  8. Li, et al., Online time series forecasting with theoretical guarantees, NeurIPS 2025;

  9. Liang, et al., Causal component analysis, NeurIPS 2023;

  10. Zhang, et al., Causal representation learning from multiple distributions: A general setting, ICML 2024;

  11. Hu, et al., Instrumental variable treatment of nonclassical measurement error models. Econometrica, 2008.

  12. Zheng, et al., Nonparametric factor analysis and beyond. AISTATS25;

  13. Li, et al., Identification of intermittent temporal latent process. ICLR25;

  14. Lachapelle, et al., Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning, ICML 2023;

Reviewer_vjRSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a fundamental limitation in causal representation learning, where noise that depends on the latent variables and affects several parts of the observation breaks the conditional independence assumption among observations used in prior identifiability work. To address this structural misspecification, the authors model the task as a perturbed factor analysis problem and establish two theoretical regimes: precise subspace identifiability under spectral separation (∥Per∥op≤Per‾<η\Vert Per \Vert_{op} \le \overline{Per} < \eta), extended to component-wise identifiability under structural sparsity, and approximate subspace identifiability with a bounded error when this condition does not hold. Based on these guarantees, they propose an unsupervised VAE architecture combining dual encoders, normalizing flows to model the dependent noise pγ(ϵ∣z)p_{\gamma}(\epsilon \mid z), and a differentiable Jacobian sparsity loss (LsparseL_{sparse}), and report improved results over the baselines on synthetic data (Section 5) and on real-world tasks (Appendix D).

Critical strengths
  1. Previous research on identifiability usually assumes that the observations are conditionally independent given the latent variables, i.e., p(x∣z)=p(xa∣z) p(xb∣z) p(xc∣z)p(x \mid z) = p(x_a \mid z)\, p(x_b \mid z)\, p(x_c \mid z). In real-world data, however, noise may depend on the latent variables and affect several parts of the observation at once, which breaks this assumption. For example, in the paper's chest X-ray example, the disease state and inspiration level can affect different image regions, so these regions may remain statistically dependent even after conditioning on the disease state (Figure 1). The paper frames this issue as a perturbed factor analysis problem, in which the violation of conditional independence appears as a perturbation operator (Eq. 4). This provides a theoretical way to recover the latent variables even when the conditional-independence assumption does not hold. The paper offers two guarantees: precise subspace identifiability when the perturbation is sufficiently small relative to the spectral separation (Theorem 1), and approximate subspace identifiability with a bounded error when the condition for the sufficient precise guarantee is not met (Theorem 2). Proposition 2 further shows component-wise identifiability under structural sparsity and independence assumptions on the latent variables.

  2. Beyond the theoretical results, the paper proposes a variational estimator that models the dependence between the latent variables and the noise. In the synthetic experiments, the method achieves better results than the evaluated baselines for both component-wise recovery (MCC, Figure 3) and approximate subspace recovery (R2R^2, Figure 4). The ablation studies also show that removing either the dependent-noise model or the structural-sparsity regularizer reduces performance in these settings.

Critical weaknesses
  1. The theoretical guarantees rely on conditions that are difficult to verify in practice. Specifically, Theorem 1 requires the upper bound on the operator norm of the perturbation to be smaller than a threshold determined by the spectral separation (∥Per∥op≤Per‾<η\|Per\|_{op} \le \overline{Per} < \eta), which is difficult to check on real datasets, since PerPer depends on operators involving the true latent variables zz (Eq. 3-4), that are unknown in a real application such as chest X-ray diagnosis. The paper acknowledges that this condition is not always guaranteed and offers approximate identifiability (Theorem 2) as a fallback. However, the error bound of Theorem 2 (Eq. 9) depends on the same unknown quantities (Per‾\overline{Per} and κ\kappa), so it cannot be evaluated either. Therefore, although the paper provides theoretical guarantees, it remains unclear how real users can determine whether these guarantees apply to a real dataset or estimate the resulting identification error.

  2. The connection between the theoretical guarantees and the proposed estimator is unclear. Section 3 shows identifiability under specific assumptions, including a bound on the operator norm of the perturbation operator as in Theorem 1 (∥Per∥op≤Per‾<η\|Per\|_{op} \le \overline{Per} < \eta), and structural-sparsity conditions for component-wise recovery. Section 4 introduces a variational objective with a sparsity regularizer, but the paper does not show whether optimizing this objective is sufficient to recover a representation satisfying the conditions of the theoretical results. The experiments in Section 5 show that the estimator performs well on the evaluated datasets, but they do not directly establish that the learned representations satisfy the assumptions required by the identifiability theorems. Therefore, it remains unclear whether the theoretical guarantees actually apply to the empirical results.

  3. Presentation and structural issues: The paper has a few organizational drawbacks that should be addressed to improve readability. First, there is no dedicated Related Work section; the literature discussion is scattered across the introduction and theory sections, which makes it harder to clearly assess the novelty against existing identifiability methods. Second, the Conclusion and Limitations are located only in the appendix, leaving the main text without summarizing the findings, scope, or future directions. Finally, there are some minor formatting problems, such as Figure 1 being low-resolution and referenced inconsistently in the text ("Fig. 1'' vs. "Figure 1''). Please add a dedicated Related Work section and move the Conclusion and Limitations into the main text. Addressing these issues would substantially improve readability of the paper!

Questions for authors
  1. The paper proves identifiability under certain conditions (Section 3), but it is not clear whether the proposed estimator in Section 4 is theoretically guaranteed to recover a representation satisfying these conditions. Is there a theoretical connection between optimizing the ELBO (Eq. 16) and the identifiability results in Section 3?
  2. Since the perturbation bound (Per‾\overline{Per}), the spectral gap threshold (η\eta), and the quantities in the approximate error bound depend on unknown aspects of the data generation process, how can we tell in practice whether the precise or approximate identifiability guarantees are likely to hold on a real dataset?
  3. How does the method perform when the structural sparsity assumption in Proposition 2 does not hold, for example, when each latent component affects many observed variables?
5 comments
AuthorsOpenReview ↗

Rebuttal 1

W1 \\& Q2: Applicability of theoretical guarantees:

A: Due the similarity between those questions, we would like to address them together.

  1. Determining identification regimes:

Our work formalizes the sufficient criterion for determining the precise identification regime ∣∣Per∣∣op≤Per‾<η||Per||_{op}\leq \overline{Per}<\eta from L.193~194 in the original paper. This spectral criterion is defined in terms of the true latent variable zz and requires no estimate z^\hat z. When Per‾≥η\overline{Per}\geq\eta introduces the approximate identification regime On the contrary, $ \overline{Per}\geq\eta$ failing introduces the approximate identification regime. **Both criteria are defined in terms of the true latent variable zz and requires NO estimate z^\hat z. Accordingly, for our experiments, identification regimes are determined upon data synthesizing. **

Following prior work on identifiable representations [1–13], we construct two synthetic datasets. Revised Section C.2 specifies the precise identification regime by Per‾4(xb)<0.0055<0.00600<η4(xb;0.02)\overline{Per}_4(x_b)<0.0055<0.00600<\eta_4(x_b;0.02) . Also, Eq. (100) documents the approximate identification regime by Per‾4(xb)≥0.017\overline{Per}_4(x_b)\geq 0.017. Notably, we use the same learning objective across both the identification regimes.

  1. Real-world experiment scope:

For the real-world experiment, since no true latent variables are observed, we do not justify identifications. In line with previous work [1-13], we conduct experiment on both classification and domain adaptation task in Section D. These evaluations score predictions against observed task labels and require no assignment of an identification regime. We report the results from Tab.2-8 in Section D. Our method outperforms the comparisons on both tasks, supporting the suitability of the inductive bias in Eq.10 (probabilistic factorization of Eq.1) and its domain-conditioned extension in Eq.110.

W2 \\& Q1: Connection between the training objective and the identifiability results

Given the similarity between those concerns, we would like to address them together.

  1. Purpose of estimator: We would like to first clarify our Theorem 1 and 2 are population-level and estimator-agnostic: given the observational distribution, they establish that the latent variable can be uniquely determined up to the equivalence class stated in Def. 2.1 and 2.2.

The first paragraph of Section 4 states that the learning objective seeks to learn p(x∣z)p(x|z) by fitting the data-generating process in Eq.1. Eq.10 gives its probabilistic factorization, and Eq.16 provides the corresponding variational objective.

  1. Enforcing Assumptions: Rather than quantities that the estimator is expected to recover, we instantiate the assumptions in both Theorem 1 and 2 by constructing the synthetic data, as detailed in Appendix C.1 and C.2.

  2. What is experimentally verified: We verify whether the estimator recovers the known latent variables generated under those assumptions. Fig.3(a) directly visualizes component-wise alignment between zz and z^\hat{z}, and MCC quantitatively measures latent recovery up to permutation. Fig.4 shows the R2R^2 scores, which indicates the recovery of zz at a subspace level. Thus, the experiment validates recovery of the identifiable latent representation.

Notably, enforcing assumptions for data synthesizing and the test the identifiability on the synthetic data is the standard protocol adopted in the literature of identifying latent representations, such as [1-13], for testing identifiability results when the true latent variables and data-generating process are available.

AuthorsOpenReview ↗

Rebuttal 2

W3: Paper structure and wording:

A:

  1. Related work:

Our paper structures in the following: Introduction positions the paper because the closest prior result directly motivates the problem we study. Section 2 formalizes that problem, Section 3 develops the identifiability results, Section 4 presents the estimator, and Section 5 evaluates latent recovery.

Under such a structure, our introduction brings together the comparison needed to assess our novelty and contributions. Lines 26–35 review how prior work handles structured noise, including approaches that use auxiliary information, distinguish content from style, or impose restrictions on the latent structure [2, 6, 7, 9, 10]. Additionally, L.36 -- 46 discussed [1,8,11,12] might fail under misspecified structure. This yields a novel problem: identifying the latent representations with the observation blocks being dependent. Paragraph 3 then explains our contribution.

This structure gives readers the comparison needed to assess the novelty in one place: the closest prior result, the assumption we relax, and the guarantees we derive under that relaxation. This is why we present the literature review in the Introduction, where it motivates the problem before the formal development.

  1. Conclusion and wording:

We placed the conclusion in the appendix of the original submission to meet the page limit. In light of your suggestion, we have move it back to the main paper in the revision.

Per your request, we have also standardized figure references throughout the text to use “Fig.”

Q3: W/O s basline:

A: We have introduced an “W/O s” ablation, which removes the structural-sparsity regularizer on Φ\Phi in Eq. (18).

Removing the regularizer lowers MCC by about 0.1 points in the component-wise recovery experiment (Fig. 3(b)) and lowers aligned R2R^2 in the subspace recovery experiment (Fig. 4). These results show that the regularizer improves empirical latent recovery in both settings.

References

  1. Fu, et al., Learning General Causal Structures with Hidden Dynamic Process for Climate Analysis, ICML 2026;

  2. Sun, et al., Causal representation learning from multimodal biomedical observations, ICLR 2025;

  3. Brady, et al., Interaction asymmetry: A general principle for learning composable abstractions, ICLR 2025;

  4. Lachapelle, et al., Additive decoders for latent variables identification and cartesian-product extrapolation, NeurIPS 2023;

  5. Kong, et al., Partial disentanglement for domain adaptation, ICML 2022;

  6. Von Kügelgen, et al., Self-supervised learning with data augmentations provably isolates content from style, NeurIPS 2021;

  7. Kong, et al., Identification 420 of nonlinear latent hierarchical models, NeurIPS 2023;

  8. Li, et al., Online time series forecasting with theoretical guarantees, NeurIPS 2025;

  9. Liang, et al., Causal component analysis, NeurIPS 2023;

  10. Zhang, et al., Causal representation learning from multiple distributions: A general setting, ICML 2024;

  11. Hu, et al., Instrumental variable treatment of nonclassical measurement error models. Econometrica, 2008.

  12. Zheng, et al., Nonparametric factor analysis and beyond. AISTATS25;

  13. Li, et al., Identification of intermittent temporal latent process. ICLR25;

AuthorsOpenReview ↗

Thanks for your feedback

Dear Reviewer vjRS,

We are grateful for your positive assessment. With the discussion period ending on November 18, we would like to check in on our response. Should any questions remain, we would be glad to address them.

Best regards,

The Authors

Reviewer_vpHXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper studies latent identifiability under violations of three-view conditional independence, proposes spectral-perturbation conditions for exact and approximate recovery, and evaluates a VAE estimator on synthetic and downstream tasks.

Critical strengths

The problem is important, and the operator-perturbation formulation remains a promising tool for quantitative stability analysis.

Critical weaknesses
  1. I’m not convinced by the exact-identifiability argument. Your proof establishes spectral uniqueness for a fixed model, but does not rule out different models that produce the same observations with different unknown perturbations and no equivalent classes are explicitly given. This leave me a major concern in Theorem 1, namely partial absence of the proof.

  2. I also don’t see how the assumptions cover the continuous experiments. On standard L1/L2L^1/L^2 spaces, Gaussian smoothing has no bounded inverse, and the relevant conditional-density multiplication operator has no positive uniform spectral gap. It is therefore unclear how the synthetic setup satisfies the assumptions of the claimed ''precise`` regime.

  3. The connection between the theory and the training loss is unclear to me. Proposition 2 concerns the sparsity of p(xi∣z)p(x_i\mid z), whereas Eq. (18) uses likelihood and prior scores under variational expectations. The explanation about log densities having the same zero derivatives does not establish why this loss enforces the required sparsity.

  4. The two experimental regimes are hard to verify. I could not find the actual noise strengths or numerical checks of Per‾<η\overline{\mathrm{Per}}<\eta. When this is missing, it is hard to say whether the experiments actually test the two theoretical regimes.

  5. The recovery metric needs a clearer definition. The paper reports R2R^2 “after aligning” the learned and true latents, but the author does not specific what exactly the alignment is here or how the learned latent is fitted and evaluated. Since the theory allows nonlinear invertible transformations, this choice matters for interpreting the results.

  6. The notation is unnecessarily complicated and makes the paper even harder to follow. For example, η\eta denotes both exogenous noise and the spectral threshold, while several versions of hh appear in the identification argument. I generally have the feeling that the theoretical part can be much better organized and polished and better presentation will make the paper significant easier to read.

Questions for authors

see weakness

9 comments
AuthorsOpenReview ↗

Rebuttal 1

Q1: Cross-Model Ambiguity for identifiability results

A:

  1. Clarification of the fixed-model setting:

Our work follows the fixed model setting adopted in previous studies [1–3], including Zheng et al. [1], which focus on identifying the true latent variable zz through its estimate z^\hat z, rather than explicitly addressing the additional ambiguity involving a distinct model with latent variable z~\tilde z. Importantly, even when Per=0Per=0 under their setup, the cross-model ambiguity might still exist.

  1. Our approach to solve Cross-model ambiguity:

In light of your suggestion, if we would like to solve the cross-model ambiguity for precise identifiability, that is totally possible. We previously omit our approach to solve this to be in line with the settings of [1-3].

Let us first formalize the cross-model ambiguity in the following: Two distinct data-generating models with latent variables zz and z~\tilde z may induce the same observed distribution p(x)=p~(x)p(x)=\tilde p(x) while admitting different unknown perturbations Per≠Per~Per\neq\tilde{Per}. In other word

Lxa∣zΛxb∣zLxa∣z−1−PerL_{x_a|z}\Lambda_{x_b|z}L^{-1}_{x_a|z}-Per

=Lxa∣z~Λxb∣z~Lxa∣z~−1−Per~=L_{x_a|\tilde{z}}\Lambda_{x_b|\tilde{z}}L^{-1}_{x_a|\tilde{z}}-\tilde{Per}

=Lxa,xb∣xcLxa∣xc−1= L_{x_a,x_b|x_c}L^{-1}_{x_a|x_c}

To the end of proving the precise identifiability, in addition to the original injectivity of Θ\Theta in Assumption 3, we propose the following assumption: let Θ~:Z~→D~,s.t.,Θ(z~)=p(xb∣z~)\tilde{\Theta}:\tilde{\mathcal{Z}}\to \tilde{\mathcal{D}}, s.t., \Theta(\tilde{z})=p(x_b|\tilde{z}), we assume Θ~\tilde{\Theta} is injective, and p(xb∣z)=p(xb∣z~)p(x_b|z)=p(x_b|\tilde{z}), which suggests Θ(z)=Θ~(z~)\Theta(z)=\tilde{\Theta}(\tilde{z}). Given the injectivity of Θ\Theta and Θ~\tilde{\Theta}: z=Θ−1Θ~(z~)=t(z~)z = \Theta^{-1}\tilde{\Theta}(\tilde{z})=t(\tilde{z}) where t=Θ−1∘Θ~t=\Theta^{-1}\circ\tilde{\Theta} denotes an invertible transformation. At this point, we can focus on identifying zz.

The proposed additional assumption does NOT alter our synthetic data-generating process, which evaluates latent recovery under a specified model rather than comparing distinct models that induce the same observations. Establishing cross-model identifiability under arbitrary unknown perturbations without the compatibility condition is a broader question left for future work.

We have added the above assumption in Assumption 4 in the paper revision, and the above proof has been added at the end of Section B.3 in the paper revison as well.

AuthorsOpenReview ↗

Rebuttal 2

Q2: Bouded below assumption Gaussian smoothing in data synthesizing:

A:

In line with the compact operator formulation used in previous identification studies [1–3], we omitted explicitly specifying the finite-dimensional working spaces Hu,m\mathcal H_{u,m} and the dependence of the spectral gap on xbx_b in the original manuscript, not on a restricted L2L^2 space. Our Gaussian-based synthetic construction also follows the experimental setting of related work.

  1. Clarification on Bounded invertibility:

We explicitly define the mm-dimensional space for the domain of our operators in the revision:

$ \mathcal H_{u,m}= span\{\psi_{u,1},\ldots,\psi_{u,m}\} \subset L^2(\mathcal U). $ where ψu,1,…,ψu,m\\{\psi_{u,1},\ldots,\psi_{u,m}\\} is the orthonormal basis of Hu,m\mathcal H_{u,m}, and u={xa,xc,z}u=\{x_a,x_c,z\}.

Thus, under Assumption 2 (injectivity), these operators have full rank, yielding

∣∣Lm−1∣∣op||L_m^{-1}||_{op}

=1σmin⁡(Lm)<∞=\frac{1}{\sigma_{\min}(L_m)}<\infty. when σmin⁡(L)>0\sigma_{\min}(L_)>0.

At this point, bounded-blow assumption in Assumption 3 is imposed and justified for the finite-dimensional operators, not for Gaussian smoothing on unrestricted L2L^2.

To facilitate this clarification, in the paper revision from L.121--125, we introduce Hu,m⊂L2(U)\mathcal H_{u,m}\subset L^2(\mathcal U) and their orthonormal bases. Section B.2, Eqs.~(24)--(27), defines the orthogonal projectors and the projected conditional and multiplication operators. This specifies the domain and codomain used in the finite-resolution calculation.

Section C.2 now verifies the finite-dimensional version of Assumption 2 for the Gaussian construction. It first demonstrates the full-space obstruction by constructing unit vectors fnf_n for which ∣Kfn∣2→0\\|Kf_n\\|_2\to 0. It then derives the four-dimensional matrices from Gaussian integrals. Their diagonal entries are products of

\qquad t_1(\beta,v)=\frac{2\beta}{1+\beta^2+v}\\,t_0(\beta,v),$$ which are strictly positive for $\beta,v > 0$. Consequently, both projected conditional operators have strictly positive smallest singular values and bounded inverses. Section C.2 also records the projection contributions to the factorization and includes them in the perturbation bound. 2. Spectral separation: Given the definition of $H_{u,m}$, we further clarify that the resolvent perturbation bound in Eq.5, as well as Proposition 1's spectral separation is evaluated **for each fixed $x_b$**, rather than assuming a uniform positive gap over the entire continuous observation space. At L.181–184 of paper revision, we explicitly state that the observable operator is indexed by $x_b$ through $p(x_a,x_b\| x_c)$, while the multiplication operator is indexed by $x_b$ through $p(x_b\| z)$. Accordingly, we additionally evaluates the following in the revised Section C.2: $$\delta_m(x_b)=\min_{i\ne j}|\rho_m^i(x_b)-\rho_m^j(x_b)|, \qquad \eta_m(x_b)=\frac{\delta_m(x_b)}{2\kappa_m}-\alpha,$$ and compares the perturbation bound with the threshold at the same observation.
AuthorsOpenReview ↗

Rebuttal 3

Q3: How Eq.18 enforce sparsity in Proposition 2?

A:

  1. Connection between the theory and the training objective:

Section 4 aims to learn the conditional distribution of the observations given the latent representation. Eq. (10) factorizes the data-generating model in Eq. (1), and Eq. (16) provides a variational objective for fitting that model. The fitted model induces pθ,γ(x∣z^)p_{\theta,\gamma}(x|\hat z) by integrating out the latent noise.

Our identifiability results characterize the relationship between zz and z^\hat z at the population level and are estimator agonostic under the stated assumptions. Rather than quantities that the estimator is expected to recover, we instantiate these assumptions in both Theorem 1 and 2 by constructing the synthetic data, as detailed in Appendix C.1 and C.2. This is the standard protocol adopted in [1-7] for testing identifiability results when the true latent variables and data-generating process are available. Specifically, we construct the structrual sparsity as Eq.107 in Section C.2 suggested.

  1. Role of the sparsity penalty in Eq. (18):

We would like to first clarify that Eq.18 enforces the sparsity regularization ∣G^∣0≤∣G∣0|\hat{G}|_0\leq |G|_0 (item 3 in Proposition 2).

Here GG and G^\hat G are binary support matrices, which we define in Proposition 2. In particular, G^ij=1\hat G_{ij}=1 when the derivative of pθ,γ(xi∣z^)p_{\theta,\gamma}(x_i|\hat z) with respect to z^j\hat z_j has nonzero L2L^2 norm. Where the marginal density is positive, its derivative and the derivative of its log have the same zeros.

The connection to the score terms in Eq. (18) comes from marginalizing the latent noise:

pθ,γ(xi∣z^)p_{\theta,\gamma}(x_i|\hat z)

$= \int p_\theta(x_i|\hat z,\hat\epsilon) p_\gamma(\hat\epsilon|\hat z)\,d\hat\epsilon. $

Therefore, we obtain the following using the score of pθ,γ(xi∣z^)p_{\theta,\gamma}(x_i|\hat z) by:

$ \partial_{\hat z_j}\log p_{\theta,\gamma}(x_i|\hat z)$

$= \mathbb E_{\epsilon}[ \partial_{\hat z_j}\log p_\theta(x_i|\hat z,\hat\epsilon) + \partial_{\hat z_j}\log p_\gamma(\hat\epsilon|\hat z) ] $

Accordingly, Eq. (18) is a sparsity-promoting surrogate. Its ℓ1\ell_1 penalty encourages smaller scores, and the exact inequality ∣G^∣0≤∣G∣0|\hat G|_0\leq |G|_0 remains a separate premise of Proposition 2. We have added the above discussion before Eq.18 in our paper revision.

The final part of Section C.2 verifies the true-model structural sparsity condition w.r.t. GG for our synthetic construction.

Q4: Numerical check for identification regime

A: In the synthetic experiments, we fit the model in Eq. (10) and learn z^\hat z using the objective in Eq. (16) at either settings. The training procedure receives no regime label and does not estimate τ\tau or Per\mathrm{Per} to select an objective. The regime check is a separate calculation on the known synthetic model.

Section C.2 in paper revision Eq.98 in the paper revision gives the noise strengths and evaluates the perturbation bound and threshold at the same xbx_b for the four-dimensional projected operators. For every xbx_b: $\overline{Per}_4(x_b)<0.0055<0.00600 <\eta_4(x_b;0.02)<0.00630$ In other words, our precise identification regime is defined.

However, when Per‾4(xb)>η4(xb;0.02)\overline{Per}_4(x_b)>\eta_4(x_b;0.02), we consider the Theorem 1 for precise identifiability no longer holds. Therefore, we synthesize the data using settings presented in Eq.100 in the paper revision for approximately identifying zz. In practice, we synthesize the approximate identifiability regime when Per‾4(xb)≥0.017\overline{Per}_4(x_b)\geq 0.017, as suggested by Eq. 100.

Q5: Calculating alignment using R2R^2 metric

A: Our work opts in the similar procedure of [2] (Theorem 3.2 and Appendix C.1.2), to freeze each trained encoder and fit a Gaussian-kernel ridge regressor gg from the learned representations z^i\hat z_i to the known synthetic latents ziz_i with a fixed kernel bandwidth. For each latent coordinate jj, the R2R^2 score is

1- \frac{\sum_{i\in I_{\mathrm{test}}}(z_{ij}-g_j(\hat z_i))^2} {\sum_{i\in I_{\mathrm{test}}}(z_{ij}-\bar z_{\mathrm{test},j})^2},$$ where $\bar z_{\mathrm{test},j}$ is the held-out mean of coordinate $j$. We average these scores over the true latent coordinates. The resulting $R^2$ measures how well the learned representation predicts the ground-truth latents on held-out observations within the chosen nonlinear regression class. The regressor is used only for evaluation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General author response

Dear Reviewers, AC, and SAC,

Thank you for your time and efforts on our paper. We summarize our responses and revisions below. Our paper addresses a gap in prior works by characterizing what can be recovered under misspecified structure. Below we summarize the comments and our responses:

-Reviewer vpHX

The reviewer asked about cross-model identification, operator assumptions, and experimental verification. We addressed this concern by explicitly explaining, in line with previous works thus previously omitted: 1. assumptions underlying the operator formulation ; 2. the assumption for cross-model identification. Also, We revise Appendix C.2 now reports numerical checks at both noise strengths. Our response also explains how Eq.18 promotes structural sparsity, what aligned R2R^2 measures, and how the identification maps differ.

-Reviewer vjRS

The reviewer asked how to assess identification on real data, how the ELBO relates to the theory, and what happens when structural sparsity fails. We clarified that the synthetic experiments use the known data-generating process to check the identification conditions and assess latent recovery, while the real-data experiments evaluate classification and domain adaptation. We explained how the ELBO fits the model studied in the theory, what Proposition 2 requires, and what the sparsity ablation tests. We also pointed to the novelty comparison in the Introduction and moved the Conclusion and Limitations into the main text.

-Reviewer SJpo

The reviewer’s first major concern was that the writing and mathematical exposition make the contribution difficult to evaluate. We pointed to where the original submission motivates the definitions in Section 2, explains the assumptions in Section 3, refers the reviewer to Appendix B.1 for the prior derivation within the main-text page limit. Our detailed response addresses the proof questions, uses two ablations to assess the sources of the recovery gains, and reports that the full model leads on both recovery metrics in the additional nonlinear-noise experiment.

Thanks for considering the paper revision and our point-by-point responses

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.