acceptodds
Under review as a conference paper at ICLR 2027

Hessian Rank Constraint for Learning Structure of Nonlinear Latent Variable Models

10 views

Abstract

Uncovering latent variables and their causal relations from observed data is a fundamental yet challenging problem. Existing methods often rely on restrictive assumptions such as linear relations or invertible mixing functions. To better address this problem under general nonlinear mixing procedures, we propose a condition called Hessian Rank Constraint (HRC) as a primitive rank-based tool for nonlinear latent causal discovery. In particular, we show that, interestingly, a particular rank-based property arises from the cross-Hessian of the observed-data log-density in the nonlinear case that reveals information about the latent variables and is reduced to the Tetrad constraints in the linear Gaussian case. When two groups of observed variables are d-separated by a set of lower-dimensional latent variables, the rank of this cross-Hessian is equal to the latent dimension, under a mild local linearity assumption on the conditional log-density derivatives. This assumption can be naturally satisfied when the noise level is low or the relevant nonlinearity is not very strong. Based on HRC, we identify the locations of latent variables and recover causal relations among latent variables up to the Markov equivalence class by using observed variables as surrogates. Experimental results on synthetic and real-world datasets support the theoretical claims.

Primary area:
causal reasoning
Keywords:
Causal Discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_y6t8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Hessian Rank Constraint (HRC) for nonlinear latent causal discovery. The authors provide theoretical results showing how the rank of the cross-Hessian of the observed log-density can identify the dimension of latent sets separating disjoint groups of observed variables, under specific assumptions. They then integrate HRC into FOFC for latent variable identification under a pure measurement model and into a PC-based algorithm to recover causal relations among latent variables. The method is evaluated on synthetic datasets with different nonlinear functions and latent dimensions. An additional experiment on a real-world macroeconomic dataset provides an illustration of the approach.

Critical strengths

I see two main strengths in this paper that support my positive assessment of the proposed approach:

  1. I find this work interesting, well written, and clearly presented. The paper addresses a timely and important research topic, as nonlinear causal discovery in the presence of latent variables remains a challenging and active research area. I particularly appreciate the theoretical contribution, which extends rank-based constraints to nonlinear settings, as well as the pedagogical presentation of the assumptions and theoretical results. I consider this a valuable contribution to the field.
  2. I appreciate the extensive theoretical developments. The experiments on synthetic data cover different nonlinear functions, latent dimensions, and sample sizes, with promising results compared to existing methods. This provides encouraging evidence for the proposed approach within the considered setting.
Critical weaknesses
  1. The scope (of the nonlinear extension & of the algorithmic and experimental validation) : Although the proposed HRC provides an interesting theoretical contribution, I have some concerns about the generality of the nonlinear setting considered. In particular, the rank characterization relies on an approximation whose accuracy depends on the conditional-score residuals being sufficiently small relative to the spectral gap. The nonlinear constructions provided by the authors are interesting, but they mainly illustrate particular mechanisms for which the conditional scores are exactly affine, or settings where the approximation error can be controlled under additional conditions. I would have appreciated a clearer characterization of the classes of nonlinear models covered by these assumptions, as well as their limitations. In particular, experiments involving stronger nonlinearities, such as polynomial functions, highly oscillatory functions, or interactions between several latent factors, would help better understand the boundaries of the proposed approach. Although HRC is introduced as a more general theoretical criterion, its algorithmic application relies on the pure one-factor measurement model, where each observed variable has at most one latent parent. I acknowledge that the authors explicitly present FOFC+PC as an application of HRC rather than a restriction of the theoretical contribution. Nevertheless, the experimental evaluation is conducted within this specific measurement setting. I would have appreciated a discussion of how HRC could be extended to more general measurement structures, particularly when observed variables depend on multiple latent factors. This is especially relevant given the recent developments in causal representation learning involving more complex latent measurement structures.

  2. Experiments and analysis: I have several concerns regarding the experimental evaluation and the interpretation of the results.

    • Dong et al. (2024) is discussed alongside Silva et al. (2006) and Kummerfeld and Ramsey (2016), but is not included in the experimental comparisons. Given its relevance to latent variable identification and causal structure discovery, I would have appreciated either its inclusion or an explanation of its exclusion.
    • The performance gap between HRC and Prashant et al. (2025) is substantial, but its origin is not sufficiently discussed. It remains unclear to what extent the improvement comes from the proposed Hessian rank criterion rather than differences in the overall causal discovery procedures.
    • I would also appreciate clarification of the experimental protocol used for Kummerfeld and Ramsey (2016), since FOFC primarily addresses latent variable clustering rather than the complete recovery of a latent causal graph. How were the graphs evaluated in Tables 1 and 2 obtained for this baseline? PC-algorithm?
    • Figure 4 provides an interesting comparison between Hessian-based and covariance-based constraints within FOFC. However, it does not establish an advantage over other nonlinear rank-based criteria, such as the Jacobian-based approach of Prashant et al. (2025). A comparison on the latent-variable identification task, if technically feasible, would be informative. I also noticed some non-monotonic variations in clustering performance with sample size, which deserve further discussion.
    • More generally, I find that the analysis of the experimental results could be more developed, particularly to explain the differences observed between the baselines and the behavior of the proposed method across the considered settings.
    • I am not fully convinced by the contribution of Section 5.2 to the experimental validation. The groups of observed variables are predefined using economic knowledge, rather than identified by the proposed clustering procedure. Consequently, this experiment mainly provides an illustration of the recovered causal structure, without evaluating the complete framework or independently validating the inferred causal relations. I would have preferred a more informative experimental analysis, particularly given the space devoted to this section. Or, I would have preferred the detailed related-work discussion to be included in the main paper rather than in the appendix, as positioning the contribution with respect to existing approaches is particularly important for this work. I also believe that Montagna et al. (2025), “Score Matching Through the Roof: Linear, Nonlinear, and Latent Variables Causal Discovery” (CLeaR 2025), is a relevant missing reference. Although the objectives and theoretical guarantees differ, this work also exploits derivatives of the observed log-density for causal discovery in the presence of latent variables. Discussing it would help clarify the novelty and positioning of HRC. Finally, the discussion of Kong et al. (2023) could be more nuanced. The authors suggest that assuming latent variables are deterministic functions of observed variables is often violated when each observed variable has its own noise term. However, the presence of observation noise alone does not necessarily imply that latent variables cannot be recovered deterministically from the observations. This depends on the observation mechanism and the noise structure. I would suggest clarifying this statement.
Questions for authors
  1. Could the authors clarify the scope of the nonlinear models covered by HRC? In particular, beyond the examples with exactly affine conditional scores, for which classes of nonlinear measurement mechanisms can the approximation error be controlled relative to the required spectral gap? How restrictive are these conditions in practice?
  2. To what extent could HRC be applied beyond the pure one-factor measurement setting? In particular, could it be used when observed variables have multiple latent parents? I understand that FOFC+PC is presented as an illustrative application, but I would appreciate clarification of which restrictions come from HRC itself and which are specific to the proposed algorithmic implementation.
  3. Could the authors clarify why Dong et al. (2024) was not included among the experimental baselines, despite being discussed alongside Silva et al. (2006) and Kummerfeld and Ramsey (2016)? Is there a methodological reason that would make this comparison inappropriate?
  4. Could the authors provide more details on the experimental protocol used for Kummerfeld and Ramsey (2016)? Since FOFC primarily addresses latent variable clustering, how were the latent causal graphs evaluated in Tables 1 and 2 obtained for this baseline? Was an additional causal discovery procedure applied?
  5. How do the authors explain the substantial performance gap between HRC and Prashant et al. (2025)? To what extent can this improvement be attributed to the Hessian rank criterion itself, rather than differences in the downstream algorithms or the experimental setting?
  6. Would it be possible to evaluate HRC under stronger nonlinearities, such as polynomial or highly oscillatory measurement functions, where the conditional scores may deviate more substantially from affine behavior? Such experiments would help assess the practical scope of the theoretical assumptions and could materially strengthen the empirical conclusions.
20 comments
AuthorsOpenReview ↗

Response to Reviewer y6t8 (1/17): W1.1

Response to Reviewer y6t8 (1/18): W1.1

We sincerely thank you for your detailed and thoughtful review, and for finding our work interesting, well written and a valuable contribution that extends rank-based constraints to nonlinear settings. Your comments on the scope of the nonlinear extension, the experimental analysis and the positioning of HRC have helped us substantially improve the paper. We address your comments one by one below.

W1.1 The scope (of the nonlinear extension & of the algorithmic and experimental validation) : Although the proposed HRC provides an interesting theoretical contribution, I have some concerns about the generality of the nonlinear setting considered. In particular, the rank characterization relies on an approximation whose accuracy depends on the conditional-score residuals being sufficiently small relative to the spectral gap.

Thank you for this insightful comment. We would like to kindly note that the paper characterizes when this residual is small: Theorem 3 (Section 4.2) gives explicit sufficient conditions through posterior concentration and the curvature of the conditional scores, and the sensitivity study in Appendix G.3 (Tables A6 and A7) examines them by varying the noise level and the latent range. We hope this addresses your concern. Prompted by your comment, we have further detailed what Theorem 3 implies for the residual relative to the spectral gap and measured this residual in practice.

If the posterior of the separating latents lies in a ball of radius δ\delta, Theorem 3 bounds the residual by terms of order δ3\delta^3 (Equation 10), while the spectral gap σd∗(HA,Baff)\sigma_{d^{\ast}}(\mathbf H^{\mathrm{aff}}_{A,B}) grows with the score slopes times the posterior variance, of order δ2\delta^2. The residual relative to the gap is therefore of order δ\delta times the ratio of the curvature to the slope of the conditional scores, so it is small when the posterior is narrow, that is, when the observation noise is small and the measurement is steep. The requirement is thus local rather than global: it depends on how noisy and how nonlinear the measurement is over the posterior neighbourhood, not on how nonlinear it is over the whole range of the latent. Without noise and with an invertible measurement, the residual vanishes however strong the nonlinearity; it becomes large only where the slope of the measurement approaches zero within the range of the latent.

To see how small this residual is in practice, we measured it directly on the conditional score in the generator of the paper. Around each latent value, the valid neighbourhood is the largest interval over which the score is affine in the latent up to 1% of its variation; the needed range is the range covered by the posterior (±2\pm 2 posterior standard deviations), over which Theorem 3 requires the score to be close to affine (medians over latent values and evaluation points):

Measurement functionValid neighbourhood (% of latent range)Needed range (% of latent range)Evaluation points with the needed range inside the valid neighbourhood (%)HRC skeleton F1
Four functions of the paper63 to 1005 to 1292 to 1000.80 to 1.00
x+x3x + x^3, input scaled by 4120.11000.93
tanh⁡\tanh, input scaled by 4514350.29

In the settings of the paper, the needed range lies inside the valid neighbourhood at almost every point, so the residual is small relative to the affine signal that carries the rank. A stronger but steep nonlinearity, such as x+x3x + x^3 with its input scaled by 4, keeps it inside and HRC accurate, whereas a measurement whose slope approaches zero, such as the saturating tanh⁡\tanh with its input scaled by 4, widens the needed range beyond the valid neighbourhood and HRC degrades.

In light of your suggestions, we have provided this analysis of the residual relative to the spectral gap and its measurement (Appendix G.6, Table A14) in the updated version.

AuthorsOpenReview ↗

Response to Reviewer y6t8 (2/17): W1.2

W1.2 The nonlinear constructions provided by the authors are interesting, but they mainly illustrate particular mechanisms for which the conditional scores are exactly affine, or settings where the approximation error can be controlled under additional conditions. I would have appreciated a clearer characterization of the classes of nonlinear models covered by these assumptions, as well as their limitations.

Thank you for this constructive suggestion. We would like to kindly note that the paper delineates the covered models at two levels and states their limitations: Assumption 2 and its discussion (Section 4.1) place the condition on the conditional scores rather than on the measurement functions, with exactly affine nonlinear measurements in Figure 2 and Appendix E.2; Theorem 3 (Section 4.2) gives sufficient conditions under which approximately affine scores keep the error small; and Appendix I states that these conditions are not claimed for arbitrary nonlinear models. We hope this addresses your concern. Prompted by your comment, we have further characterized the two classes and their limitations explicitly.

Measurements with exactly affine conditional scores. The constructions of the paper are instances of a complete description: the conditional score of a group of observed variables is exactly affine in the latent vector if and only if, given the latent vector, the group follows an exponential family whose natural parameter is the latent vector (Proposition 2, Appendix E.3). This restricts only the measurement: the latent causal model may be arbitrarily nonlinear, subject to the regularity of Assumption 1 and the non-degeneracy of Assumption 3, and the separator may have any dimension. It covers every post-nonlinear measurement A=g(ΛZ+ε)A = g(\Lambda Z + \varepsilon) with a smooth invertible gg and Gaussian noise ε\varepsilon, as well as the constructions that stack several such layers.

Measurements with approximately affine conditional scores. Beyond these models, Theorem 3 only asks the conditional scores to be close to affine over the latent values that the posterior covers, which depends on how nonlinear and how noisy the measurement is over that neighbourhood. This covers every smooth measurement whose curvature stays small relative to its slope at the given noise level, for example tanh, softplus and sinh over moderate input ranges, polynomials such as x+x3x + x^3, whose curvature relative to the slope decreases as the input grows, and periodic functions such as sin while the data stay within a range where the function is invertible. Without noise and with an invertible measurement, the scores are affine over the posterior however strong the nonlinearity.

Limitations. The assumptions can fail when the measurement is nearly flat, non-invertible or not smooth over the range of the latent, as for a saturating, periodic or clipped sensor, or when the observation noise is large relative to the loading, since the conditional scores are then far from affine over the posterior.

The experiments illustrate this characterization. We scale the input of each measurement function by a factor α\alpha (α=1\alpha = 1 is the reference scale), which moves the data along ff from the covered class towards these limitations.

Measurement functionSkeleton F1 of HRC
x+x3x + x^3, α=1\alpha = 10.98
x+x3x + x^3, α=4\alpha = 40.93
softplus, α=1\alpha = 10.98
softplus, α=4\alpha = 40.62
sinh, α=1\alpha = 11.00
sin⁡\sin, α=1\alpha = 10.88
sin⁡\sin, α=4\alpha = 40.12

HRC stays accurate while the measurement remains steep relative to its curvature over the data, even for the strongly curved x+x3x + x^3 at α=4\alpha = 4, degrades as a nearly flat region enters the data (softplus at α=4\alpha = 4), and fails, as the characterization predicts, when different latent values give the same measurement (sin⁡\sin at α=4\alpha = 4).

In light of your suggestions, we have provided this characterization of the covered models and their limitations (Appendix E.2) and the corresponding experiments (Appendix G.4, Table A8) in the updated version.

AuthorsOpenReview ↗

Response to Reviewer y6t8 (3/17): W2

W2. In particular, experiments involving stronger nonlinearities, such as polynomial functions, highly oscillatory functions, or interactions between several latent factors, would help better understand the boundaries of the proposed approach.

Thank you for this valuable suggestion. Following it, we have added experiments on the three kinds of stronger nonlinearity you mention: polynomial measurement functions, highly oscillatory measurement functions, and interactions between latent factors, both inside a measurement function and in the latent causal mechanisms.

Polynomial measurements. We add terms with unit coefficients up to degree five.

Measurement functionSkeleton F1 of HRC
x+x3x + x^30.98
x+x2+x3x + x^2 + x^30.96
x+x2+⋯+x5x + x^2 + \dots + x^50.82

HRC remains accurate as the polynomial grows to degree five.

Oscillating measurements. We increase the frequency of sin⁡\sin and add a fast oscillation to a linear trend.

Measurement functionSkeleton F1 of HRC
sin⁡(0.5x)\sin(0.5x)0.97
sin⁡(x)\sin(x)0.88
x+0.1125sin⁡(8x)x + 0.1125\sin(8x)0.98
sin⁡(4x)\sin(4x)0.12

HRC remains accurate under slow oscillations and under fast oscillations around a trend, and degrades only when the measurement becomes non-invertible within the range of the latent, as for sin⁡(4x)\sin(4x).

Interactions inside a measurement function. A second latent enters the nonlinearity of one indicator with weight cc.

InteractionSkeleton F1 of HRC
None0.91
c=0.25c = 0.250.76
c=0.5c = 0.50.76
c=1c = 10.62

HRC remains accurate for a weak interaction and degrades gradually as the second latent gains weight, since the indicator then has two latent parents, which lies outside the pure measurement model of the FOFC + PC application rather than outside HRC itself.

Interactions in the latent causal mechanisms. In latent graphs where some latent has two or more parents, we add pairwise products of the parents with strength γ\gamma to its mechanism.

Latent mechanismSkeleton F1 of HRC
Linear0.67
Pairwise products, γ=1\gamma = 10.62
Pairwise products, γ=3\gamma = 30.31

HRC handles moderate interactions among the latent causes and degrades as they become strong.

In light of your suggestions, we have provided these experiments (Appendices G.4 and G.5, Tables A9, A10, A12 and A13) in the updated version.

Reviewer_joWXOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of causal structure discovering. The so-called cross-Hessian Rank Constraint is developed to overcome the limitation of covariance-based methods that only handle linear models. The work provides a systematic theoretical framework. Several experiments demonstrate better results on both synthetic benchmarks and a small-scale real-world dataset.

Critical strengths

The paper proposes a new modeling for the nonlinear setting of latent causal discovering. It pushes the domain from Tetrad constraints or cross-covariance approaches to cross-Hessian rank. The core part is Theorem 1 (Approximate Cross-Hessian Rank under d-Separation), which also shows the link between the cross-Hessian and the latent posterior covariance. The mathematical derivation is rigorous and thoughtfully bridges multiple gaps from different perspectives. The self-contained logical chain is verified by the experiments. Consistent empirical gains on both well-controlled synthetic data and real-world data.

Critical weaknesses

It's well known that the Hessian is only a local indicator. And the affine assumption only holds true around a reference point. How large is this valid neighbourhood in practice, especially under strong nonlinearities? The transition from local rank to the global conclusion seems to jump too quickly. Relying on "measure zero" might not suffice the statement of the paper. Is there any concrete condition, such as continuity of mapping, that may help explain the phenomenon? Finally, adding heuristic voting in practice makes this part handled a bit roughly.

Questions for authors

Theoretical works need practical workarounds, However, more elaborations on boundary conditions can always be helpful. How severe must the nonlinearity be to break the model, or when might the measure-zero assumption fail in practice?

10 comments
AuthorsOpenReview ↗

Response to Reviewer joWX (1/5): W1

Response to Reviewer joWX (1/4): W1

We sincerely thank you for your careful review and for recognizing that HRC extends rank-based causal discovery from Tetrad and cross-covariance constraints to the nonlinear setting, with a rigorous derivation that links the cross-Hessian to the latent posterior covariance. Your questions on the local nature of the Hessian, the non-degeneracy condition and the boundary of the method have helped us make these points much clearer. We address your comments one by one below.

W1. It's well known that the Hessian is only a local indicator. And the affine assumption only holds true around a reference point. How large is this valid neighbourhood in practice, especially under strong nonlinearities?

Thank you for this insightful question, which helps us make precise where our affine approximation is needed. The size of the valid neighbourhood depends on the properties of the measurement function. For a linear measurement with Gaussian noise, the conditional score is affine in the latent everywhere, so the valid neighbourhood is the whole latent domain, however large the noise; for a nonlinear measurement it is local, and its size depends on how strongly the function bends.

Assumption 2 concerns the conditional score s(x,z)=∇xlog⁡p(x∣z)s(x, z) = \nabla_x \log p(x \mid z) as a function of the latent value zz, so we measured the valid neighbourhood directly on this score: around each latent value, it is the largest interval over which the score is affine in zz up to 1% of its variation. This neighbourhood does not need to cover the whole domain. At each evaluation point, Theorem 3 bounds the remainder by how far the score departs from affine over the latent values compatible with the observation, that is, the range covered by the posterior p(z∣x)p(z \mid x) (±2\pm 2 posterior standard deviations), so HRC is accurate when this needed range lies inside the valid neighbourhood. The needed range narrows as the noise decreases and shrinks to a point without noise when the measurement is invertible. In the generator of the paper (medians over latent values and evaluation points):

Measurement functionValid neighbourhood (% of latent range)Needed range (% of latent range)Evaluation points with the needed range inside the valid neighbourhood (%)HRC skeleton F1
Four functions of the paper63 to 1005 to 1292 to 1000.80 to 1.00
x+x3x + x^3, α=4\alpha = 4120.11000.93
tanh⁡\tanh, α=4\alpha = 4514350.29

As you can see, the size of the valid neighbourhood depends on the property of the nonlinear function: in the settings of the paper, the needed range (5 to 12% of the latent range) lies well inside it, and HRC remains accurate whenever it covers the needed range, even under a strong nonlinearity such as x+x3x + x^3 at α=4\alpha = 4; only when the needed range exceeds it, as for the saturating tanh⁡\tanh at α=4\alpha = 4, does the accuracy drop. Moreover, each rank decision of HRC takes the most frequent rank over many evaluation points, so a minority of points outside the valid neighbourhood does not change the decision much.

In light of your suggestions, we have added this measurement of the valid neighbourhood (Appendix G.6, Table A14) in the updated version.

AuthorsOpenReview ↗

Response to Reviewer joWX (2/5): W2

**W2.**The transition from local rank to the global conclusion seems to jump too quickly.

Thank you for this feedback, which helps us clarify our theory and improve the readability of the paper. We would like to highlight that the key to the transition from the local rank to the global conclusion is that the quantity revealed by a local cross-Hessian is itself global: at any point yy where the assumptions hold, the rank of HA,B(y)\mathbf H_{A,B}(y) equals the size d∗d^{\ast} of a minimum set of latent variables that d-separates AA and BB, which is a property of the causal graph and does not depend on yy. The transition proceeds in four steps.

  1. Pointwise factorization (Theorem 1). At every point yy and for every set of latent variables ZZ that d-separates AA and BB, HA,B(y)=VACov⁡(Z∣y),VB⊤+EA,B(y)\mathbf H_{A,B}(y) = \mathbf V_A \operatorname{Cov}(Z \mid y)\\, \mathbf V_B^\top + \mathbf E_{A,B}(y) with ∣EA,B(y)∣2≤εA,B(y)\\|\mathbf E_{A,B}(y)\\|_2 \le \varepsilon_{A,B}(y), so rank⁡εHA,B(y)≤∣Z∣\operatorname{rank}_{\varepsilon}\mathbf H_{A,B}(y) \le |Z| at every point.
  2. Pointwise equality (Theorem 2). Non-degeneracy (Assumption 3) holds at almost every point; where the signal gap also holds, rank⁡εHA,B(y)=d∗\operatorname{rank}_{\varepsilon}\mathbf H_{A,B}(y) = d^{\ast}. The value is the same at all such points, because d∗d^{\ast} is fixed by the graph.
  3. From rank to conditional independence (Proposition 1 and Theorem 4). With groups built from pure indicators, Li⊥!!!⊥Lj∣QL_i \perp\\!\\!\\!\perp L_j \mid Q holds if and only if the rank equals ∣Q∣|Q|, and the same rank constraints locate the clusters in FOFC.
  4. From conditional independence to the graph (Theorem 5). Under the Markov and faithfulness conditions on the latent variables, the PC search over these conditional independences returns the Markov equivalence class of the latent graph.

Only the first two steps involve an evaluation point; the last two use the point-independent value d∗d^{\ast}. In estimation, the decision does not rest on a single point either: HRC computes the cross-Hessian at many points in the high-density region of the data and reports the most frequent rank (Appendix D.7), so a few points with a larger remainder do not change the decision.

In light of your suggestions, we have provided this step-by-step explanation from the local rank to the global structure (Appendix D.11) in the updated version.

AuthorsOpenReview ↗

Response to Reviewer joWX (3/5): W3

W3 Relying on "measure zero" might not suffice the statement of the paper. Is there any concrete condition, such as continuity of mapping, that may help explain the phenomenon?

Thank you for this insightful question, which helps us make the non-degeneracy condition concrete. Yes: beyond the measure-zero argument, which concerns the loading parameters of the linear Gaussian case, Assumption 3 can be stated through concrete conditions on the latent noise and on the measurement functions. At a high level, it requires that the dependence between the two groups does not cancel out into a simpler pattern that fewer latent dimensions could explain. Its two parts translate as follows.

  1. Condition (i), positive definite posterior covariance: noisy latent relations. Cov⁡(Z∗∣a,b)\operatorname{Cov}(Z^{\ast} \mid a, b) is singular if and only if some combination v⊤Z∗v^\top Z^{\ast} is fixed given (a,b)(a, b), that is, the posterior lies on a proper affine subspace (Appendix E.1). This cannot happen when the latent variables have a joint density, for example when every latent has its own exogenous noise with a density, as in Lk=gk(Pa(Lk))+εkL_k = g_k(\mathrm{Pa}(L_k)) + \varepsilon_k; condition (i) then holds at every point. It fails when a latent relation is noise-free: with L2=λL1+cL_2 = \lambda L_1 + c, the two latents act as one, and the rank drops from 2 to 1.
  2. Condition (ii), full-rank slope matrices: smooth and informative measurements. With additive Gaussian observation noise, an indicator Xj=fj(Lk)+εjX_j = f_j(L_k) + \varepsilon_j of a latent Lk∈Z∗L_k \in Z^{\ast} contributes to VA\mathbf V_A a row whose only nonzero entry, in the column of LkL_k, is the slope of fjf_j at the reference point of the affine expansion, divided by the noise variance. Hence condition (ii) holds at a point whenever every latent in Z∗Z^{\ast} has, in each group, an indicator with a nonzero slope there. For such groups, it holds at every point when the measurement functions are continuously differentiable with a nonvanishing derivative (tanh⁡\tanh, softplus, sinh⁡\sinh, x+x3x + x^3), and at almost every point when the derivative vanishes only at isolated points (sin⁡\sin). Continuity alone is not sufficient: a continuous measurement can be flat on an interval, as for a clipped sensor, and then carries no information about the latent there, and at a kink it has no derivative, so the local affine approximation breaks down.

The noise level also determines how close a model comes to a violation. When we shrink the noise on every latent edge of the generator of the paper, HRC stays accurate until an edge explains about 90% of the variance of the child (skeleton F1 0.91 at about 68% and 0.86 at about 90%), and degrades only as the relation becomes nearly deterministic (0.55 at about 97%).

In light of your suggestions, we have provided these concrete conditions (Appendix E.1) in the updated version.

Reviewer_V5W2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a cross-Hessian rank criterion for nonlinear latent-variable models. Under smoothness and controlled approximately affine conditional scores, the observed marginal log-density cross-Hessian admits an approximate factorization through a latent separator. Non-degeneracy and a singular-value gap permit thresholded-rank identification. The criterion is used within FOFC clustering and PC-style latent graph discovery under a pure measurement model. Synthetic experiments cover multiple nonlinear functions, latent dimensions, and sample sizes; the FRED-MD application is an exploratory illustration with prespecified groups.

Critical strengths
  • A useful theoretical formulation with explicit conditions. The paper separates the approximate factorization from the additional non-degeneracy and signal-gap conditions needed for rank identification. Its nonlinear zero-remainder examples show that affine conditional scores need not imply linear measurement functions.

  • Encouraging synthetic evidence and candid scope statements. The evaluation spans several nonlinear functions, latent dimensions, and sample sizes.

Critical weaknesses
  • The sufficient conditions are difficult to assess in an actual application.

  • The empirical evidence is largely confined to synthetic data, and the real-data validation is inadequate.

Questions for authors
  • How can the applicability conditions be assessed on real data?

  • What is the scope of applicability to more complex data?

  • What evidence establishes generalization beyond the chosen synthetic settings?

10 comments
AuthorsOpenReview ↗

Response to Reviewer V5W2 (1/6): W1

We sincerely thank you for your time and constructive review, and for recognizing the explicit conditions of our theoretical formulation, the nonlinear examples and scope. We address your comments one by one below.

W1. The sufficient conditions are difficult to assess in an actual application.

Thank you for this constructive comment, which helps us make the practical side of our assumptions much clearer. Our result rests on two conditions with different roles, so we discuss them separately: approximately affine conditional scores (Assumption 2), which imply a structure that can be checked on observed data, and non-degeneracy (Assumption 3), a standard condition shared with classical rank-based methods [1, 2, 3], which fails only through an exact coincidence whose risk we examine by simulation.

Assumption 2 concerns two observed groups AA and BB separated by latent variables Z∈RdZ \in \mathbb{R}^d. It requires that the conditional score of each group, viewed as a function of zz, is close to affine:

sA(a,z):=∇alog⁡p(a∣z)=uA(a)+VA(a) z+rA(a,z),s_A(a,z) := \nabla_a \log p(a \mid z) = u_A(a) + V_A(a)\,z + r_A(a,z),

where the remainder rAr_A is small over the latent values that the posterior p(z∣a,b)p(z \mid a,b) covers, and likewise for BB. It relaxes exact local linearity: the first-order term in zz only needs to dominate, and Theorems 1 and 2 keep the rank result up to an error controlled by the remainder.

This holds when the posterior of the latents is concentrated, as at low observation noise, and when the conditional scores have low local curvature in the latents (Theorem 3). The measurement itself may be strongly nonlinear: X=exp⁡(Z+ε)X = \exp(Z + \varepsilon), for instance, has an exactly affine conditional score.

Assumption 2 describes how a latent drives its indicators, but it also leaves a trace in how the indicators relate to each other, and this trace can be checked on observed data alone. We derive it in three steps for one latent zz and its indicators, which are conditionally independent given zz.

Step 1: a change of variables makes the latent enter linearly. If the score of XiX_i is affine in zz, ∂xilog⁡p(xi∣z)=ui(xi)+vi(xi) z\partial_{x_i} \log p(x_i \mid z) = u_i(x_i) + v_i(x_i)\,z, integrating in xix_i gives

log⁡p(xi∣z)=Ui(xi)+Ti(xi) z−ψi(z),Ti′(xi)=vi(xi).\log p(x_i \mid z) = U_i(x_i) + T_i(x_i)\,z - \psi_i(z), \qquad T_i'(x_i) = v_i(x_i).

So zz and xix_i interact only through the product Ti(xi) zT_i(x_i)\,z: whatever the shape of the map from zz to XiX_i, everything XiX_i tells about zz is contained in the single variable Yi=Ti(Xi)Y_i = T_i(X_i), on which zz acts linearly.

Step 2: after this change of variables, indicators of the same latent combine additively. For two indicators, integrating out zz gives

log⁡p(xi,xj)=Ui(xi)+Uj(xj)+Φ(Ti(xi)+Tj(xj)),Φ(s)=log⁡∫p(z) e sz−ψi(z)−ψj(z) dz.\log p(x_i, x_j) = U_i(x_i) + U_j(x_j) + \Phi\big(T_i(x_i) + T_j(x_j)\big), \qquad \Phi(s) = \log \int p(z)\, e^{\,s z - \psi_i(z) - \psi_j(z)}\, dz.

The two indicators may be related nonlinearly in the original variables, but they depend on each other only through the sum Yi+YjY_i + Y_j. Differentiating this density recovers Theorem 1 for the pair, ∂xi∂xjlog⁡p(xi,xj)=vi(xi) vj(xj) Var(z∣xi,xj)\partial_{x_i}\partial_{x_j} \log p(x_i, x_j) = v_i(x_i)\, v_j(x_j)\, \mathrm{Var}(z \mid x_i, x_j), since Φ′′(s)=Var(z∣xi,xj)\Phi''(s) = \mathrm{Var}(z \mid x_i, x_j).

Step 3: a testable form. The functions TiT_i are unknown, so we test a consequence that does not need them. The posterior of zz given (xi,xj)(x_i, x_j) depends on them only through Yi+YjY_i + Y_j, and therefore so does the regression of a third indicator XkX_k of the same latent:

E [ Xk∣Xi,Xj ]=φ(Ti(Xi)+Tj(Xj)).\mathbb{E}\ [\ X_k \mid X_i, X_j\ ] = \varphi\big(T_i(X_i) + T_j(X_j)\big).

We fit E [ Xk∣Xi,Xj ]\mathbb{E}\ [\ X_k \mid X_i, X_j\ ] with cross-validated neural networks twice: once without restriction, and once in the form φ(hi(Xi)+hj(Xj))\varphi\big(h_i(X_i) + h_j(X_j)\big), where the networks hih_i and hjh_j learn the change of variables. The departure is the loss of held-out explained variance,

Δ=Runrestricted2−Rtransformed2.\Delta = R^2_{\text{unrestricted}} - R^2_{\text{transformed}}.

If Assumption 2 holds, the learned change of variables fits as well as the unrestricted model and Δ\Delta is close to zero; if no change of variables makes the indicators combine additively, Δ\Delta is large. The check is a necessary consequence of the assumption, so a small Δ\Delta suggests, rather than proves, that the assumption holds. In simulation with 5000 samples, Δ\Delta stays below 0.1 percentage points for the four measurement functions of the paper, where HRC is accurate, and grows to 1.29 and 3.26 points when sin⁡\sin becomes non-invertible within the range of the data, where HRC degrades:

Measurement functionΔ\Delta (percentage points)Skeleton F1 of HRC
sin⁡(x)\sin(x)0.080.88
tanh⁡(x)\tanh(x)0.000.80
softplus(x)\mathrm{softplus}(x)0.000.98
sinh⁡(x)\sinh(x)0.031.00
sin⁡(2x)\sin(2x)1.290.35
sin⁡(4x)\sin(4x)3.260.12

In light of your suggestions, we have provided this observable consequence of Assumption 2 (Section 4.1) and its proof, a worked example and the simulation above (Appendix E.3, Proposition 2, Example 1 and Table A5) in the updated version.

AuthorsOpenReview ↗

Response to Reviewer V5W2 (2/6): W1 (continued)

In light of your suggestions, we have provided this observable consequence of Assumption 2 (Section 4.1) and its proof, a worked example and the simulation above (Appendix E.3, Proposition 2, Example 1 and Table A5) in the updated version.

Assumption 3 requires that no rank is lost in the affine benchmark

HA,Baff=VA(a) Cov(Z∗∣a,b) VB(b)⊤,H^{\mathrm{aff}}_{A,B} = V_A(a)\,\mathrm{Cov}(Z^* \mid a,b)\,V_B(b)^\top,

where Z∗∈Rd∗Z^* \in \mathbb{R} ^{d^* } is a minimum set of latents that d-separates AA and BB. Specifically, (i) Cov(Z∗∣a,b)≻0\mathrm{Cov}(Z^* \mid a,b) \succ 0, so the posterior keeps variation along every latent direction, and (ii) rank⁡VA(a)=rank⁡VB(b)=d∗\operatorname{rank} V_A(a) = \operatorname{rank} V_B(b) = d^*, so each group is informative about every such direction.

It plays the role of rank faithfulness in linear rank-based methods [1,2,3] and, like faithfulness, cannot be verified from data alone. It fails only through an exact coincidence, for example when one latent is a noise-free affine function of another (L2=λL1+cL_2 = \lambda L_1 + c) or when the loading columns of a group are proportional. In linear Gaussian models such loadings form a set of Lebesgue measure zero; for nonlinear models we check the assumption by simulation.

When we shrink the noise of every latent edge toward zero, which brings the model toward the coincidence above, HRC remains accurate until the latent edges become nearly noise-free:

Noise of each latent edgeVariance of the child explained by the edgeSkeleton F1 of HRC
0.2, referenceabout 68%0.91
0.1about 90%0.86
0.05about 97%0.55

HRC remains accurate while each latent edge leaves about 10% or more of the variance of the child to its own noise, and degrades only as the latent edges approach a noise-free relation, the coincidence that Assumption 3 excludes.

References

[1] Sullivant et al. Trek separation for Gaussian graphical models. The Annals of Statistics, 2010.

[2] Huang et al. Latent hierarchical causal structure discovery with rank constraints. NeurIPS, 2022.

[3] Dong et al. A versatile causal discovery framework to allow causally-related hidden variables. ICLR, 2024.

AuthorsOpenReview ↗

Response to Reviewer V5W2 (3/6): W2

W2. The empirical evidence is largely confined to synthetic data, and the real-data validation is inadequate.

Thank you for raising this point, which helps us strengthen the empirical support of our paper beyond simulation. We extended the real-data evaluation in two ways: four additional real datasets whose measurement structure and latent graph are given by the literature, and a check of the applicability condition of HRC on the FRED-MD data used in the paper.

Real datasets with a reference structure.

  • Teacher burnout [1]: 30 survey indicators of 10 latent factors, 599 teachers; reference model of Byrne (2016).
  • Stock market indices [2]: 13 indices of three regional factors (Asia, Europe, United States), 1078 trading days.
  • Self-esteem, depression and impulsivity [3, 4]: 12 questionnaire items of three factors, 204 respondents.
  • Multitasking [5]: 9 task scores of three factors under a higher-order multitasking factor, 202 participants.
DatasetHRC (cluster F1)Cov. rank (cluster F1)Prashant (cluster F1)BPC (cluster F1)RLCD (cluster F1)HRC (skeleton F1)Cov. rank (skeleton F1)Prashant (skeleton F1)BPC (skeleton F1)RLCD (skeleton F1)
Teacher burnout [1]0.850.770.110.790.810.610.400.280.390.30
Stock market indices [2]1.000.400.390.510.491.001.000.800.500.80
Self-esteem, depression and impulsivity [3, 4]1.000.720.360.910.501.000.500.800.500.00
Multitasking [5]1.001.000.331.000.881.000.800.801.000.80

HRC attains the best cluster F1 and the best skeleton F1 on all four datasets.

Applicability on FRED-MD. Assumption 2 implies that, within a cluster, a third indicator depends on two others only through a sum of their transforms. The departure Δ\Delta from this form, the loss of held-out R2R^2 when the regression is restricted to it, is near zero in simulation wherever HRC is accurate. For the exchange-rate group of FRED-MD, Δ\Delta is within one standard error of this range and far below the settings where the assumption is violated, so the condition is plausible for these data.

DataΔ\Delta (percentage points)
Simulated, four measurement functions of the paper−0.39 to 0.17
Simulated, non-Gaussian or heteroscedastic observation noise0.01 to 0.03
Simulated, sin⁡(2x)\sin(2x)1.58 to 2.00
Simulated, sin⁡(4x)\sin(4x)3.46 to 6.97
FRED-MD [6], exchange-rate group0.31 ± 0.17

Δ\Delta averaged over the indicator triples within each cluster, with its standard error for FRED-MD; simulated data of a comparable size.

In light of your suggestions, we have provided the four real-data experiments (Section 6.2.1, with the datasets and their reference causal graphs in Appendix G.2) and the applicability test (Appendix E.3) in the updated version.

References

[1] Byrne. Structural Equation Modeling with AMOS. Routledge, 2016.

[2] Zeng et al. Causal discovery with multi-domain LiNGAM for latent factors. IJCAI, 2021.

[3] Jöreskog and Sörbom. LISREL (example data distributed with the software). Scientific Software International.

[4] Silva. Measuring latent causal structure. arXiv, 2010.

[5] Himi et al. Multitasking behavior and its related constructs: Executive functions, working memory capacity, relational integration, and divided attention. Cognition, 2019.

[6] McCracken and Ng. FRED-MD: A monthly database for macroeconomic research. Journal of Business & Economic Statistics, 2016.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.