- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Hessian Rank Constraint (HRC) for nonlinear latent causal discovery. The authors provide theoretical results showing how the rank of the cross-Hessian of the observed log-density can identify the dimension of latent sets separating disjoint groups of observed variables, under specific assumptions. They then integrate HRC into FOFC for latent variable identification under a pure measurement model and into a PC-based algorithm to recover causal relations among latent variables. The method is evaluated on synthetic datasets with different nonlinear functions and latent dimensions. An additional experiment on a real-world macroeconomic dataset provides an illustration of the approach.
Critical strengths
I see two main strengths in this paper that support my positive assessment of the proposed approach:
- I find this work interesting, well written, and clearly presented. The paper addresses a timely and important research topic, as nonlinear causal discovery in the presence of latent variables remains a challenging and active research area. I particularly appreciate the theoretical contribution, which extends rank-based constraints to nonlinear settings, as well as the pedagogical presentation of the assumptions and theoretical results. I consider this a valuable contribution to the field.
- I appreciate the extensive theoretical developments. The experiments on synthetic data cover different nonlinear functions, latent dimensions, and sample sizes, with promising results compared to existing methods. This provides encouraging evidence for the proposed approach within the considered setting.
Critical weaknesses
-
The scope (of the nonlinear extension & of the algorithmic and experimental validation) : Although the proposed HRC provides an interesting theoretical contribution, I have some concerns about the generality of the nonlinear setting considered. In particular, the rank characterization relies on an approximation whose accuracy depends on the conditional-score residuals being sufficiently small relative to the spectral gap. The nonlinear constructions provided by the authors are interesting, but they mainly illustrate particular mechanisms for which the conditional scores are exactly affine, or settings where the approximation error can be controlled under additional conditions. I would have appreciated a clearer characterization of the classes of nonlinear models covered by these assumptions, as well as their limitations. In particular, experiments involving stronger nonlinearities, such as polynomial functions, highly oscillatory functions, or interactions between several latent factors, would help better understand the boundaries of the proposed approach. Although HRC is introduced as a more general theoretical criterion, its algorithmic application relies on the pure one-factor measurement model, where each observed variable has at most one latent parent. I acknowledge that the authors explicitly present FOFC+PC as an application of HRC rather than a restriction of the theoretical contribution. Nevertheless, the experimental evaluation is conducted within this specific measurement setting. I would have appreciated a discussion of how HRC could be extended to more general measurement structures, particularly when observed variables depend on multiple latent factors. This is especially relevant given the recent developments in causal representation learning involving more complex latent measurement structures.
-
Experiments and analysis: I have several concerns regarding the experimental evaluation and the interpretation of the results.
- Dong et al. (2024) is discussed alongside Silva et al. (2006) and Kummerfeld and Ramsey (2016), but is not included in the experimental comparisons. Given its relevance to latent variable identification and causal structure discovery, I would have appreciated either its inclusion or an explanation of its exclusion.
- The performance gap between HRC and Prashant et al. (2025) is substantial, but its origin is not sufficiently discussed. It remains unclear to what extent the improvement comes from the proposed Hessian rank criterion rather than differences in the overall causal discovery procedures.
- I would also appreciate clarification of the experimental protocol used for Kummerfeld and Ramsey (2016), since FOFC primarily addresses latent variable clustering rather than the complete recovery of a latent causal graph. How were the graphs evaluated in Tables 1 and 2 obtained for this baseline? PC-algorithm?
- Figure 4 provides an interesting comparison between Hessian-based and covariance-based constraints within FOFC. However, it does not establish an advantage over other nonlinear rank-based criteria, such as the Jacobian-based approach of Prashant et al. (2025). A comparison on the latent-variable identification task, if technically feasible, would be informative. I also noticed some non-monotonic variations in clustering performance with sample size, which deserve further discussion.
- More generally, I find that the analysis of the experimental results could be more developed, particularly to explain the differences observed between the baselines and the behavior of the proposed method across the considered settings.
- I am not fully convinced by the contribution of Section 5.2 to the experimental validation. The groups of observed variables are predefined using economic knowledge, rather than identified by the proposed clustering procedure. Consequently, this experiment mainly provides an illustration of the recovered causal structure, without evaluating the complete framework or independently validating the inferred causal relations. I would have preferred a more informative experimental analysis, particularly given the space devoted to this section. Or, I would have preferred the detailed related-work discussion to be included in the main paper rather than in the appendix, as positioning the contribution with respect to existing approaches is particularly important for this work. I also believe that Montagna et al. (2025), “Score Matching Through the Roof: Linear, Nonlinear, and Latent Variables Causal Discovery” (CLeaR 2025), is a relevant missing reference. Although the objectives and theoretical guarantees differ, this work also exploits derivatives of the observed log-density for causal discovery in the presence of latent variables. Discussing it would help clarify the novelty and positioning of HRC. Finally, the discussion of Kong et al. (2023) could be more nuanced. The authors suggest that assuming latent variables are deterministic functions of observed variables is often violated when each observed variable has its own noise term. However, the presence of observation noise alone does not necessarily imply that latent variables cannot be recovered deterministically from the observations. This depends on the observation mechanism and the noise structure. I would suggest clarifying this statement.
Questions for authors
- Could the authors clarify the scope of the nonlinear models covered by HRC? In particular, beyond the examples with exactly affine conditional scores, for which classes of nonlinear measurement mechanisms can the approximation error be controlled relative to the required spectral gap? How restrictive are these conditions in practice?
- To what extent could HRC be applied beyond the pure one-factor measurement setting? In particular, could it be used when observed variables have multiple latent parents? I understand that FOFC+PC is presented as an illustrative application, but I would appreciate clarification of which restrictions come from HRC itself and which are specific to the proposed algorithmic implementation.
- Could the authors clarify why Dong et al. (2024) was not included among the experimental baselines, despite being discussed alongside Silva et al. (2006) and Kummerfeld and Ramsey (2016)? Is there a methodological reason that would make this comparison inappropriate?
- Could the authors provide more details on the experimental protocol used for Kummerfeld and Ramsey (2016)? Since FOFC primarily addresses latent variable clustering, how were the latent causal graphs evaluated in Tables 1 and 2 obtained for this baseline? Was an additional causal discovery procedure applied?
- How do the authors explain the substantial performance gap between HRC and Prashant et al. (2025)? To what extent can this improvement be attributed to the Hessian rank criterion itself, rather than differences in the downstream algorithms or the experimental setting?
- Would it be possible to evaluate HRC under stronger nonlinearities, such as polynomial or highly oscillatory measurement functions, where the conditional scores may deviate more substantially from affine behavior? Such experiments would help assess the practical scope of the theoretical assumptions and could materially strengthen the empirical conclusions.
20 comments
Response to Reviewer y6t8 (2/17): W1.2
W1.2 The nonlinear constructions provided by the authors are interesting, but they mainly illustrate particular mechanisms for which the conditional scores are exactly affine, or settings where the approximation error can be controlled under additional conditions. I would have appreciated a clearer characterization of the classes of nonlinear models covered by these assumptions, as well as their limitations.
Thank you for this constructive suggestion. We would like to kindly note that the paper delineates the covered models at two levels and states their limitations: Assumption 2 and its discussion (Section 4.1) place the condition on the conditional scores rather than on the measurement functions, with exactly affine nonlinear measurements in Figure 2 and Appendix E.2; Theorem 3 (Section 4.2) gives sufficient conditions under which approximately affine scores keep the error small; and Appendix I states that these conditions are not claimed for arbitrary nonlinear models. We hope this addresses your concern. Prompted by your comment, we have further characterized the two classes and their limitations explicitly.
Measurements with exactly affine conditional scores. The constructions of the paper are instances of a complete description: the conditional score of a group of observed variables is exactly affine in the latent vector if and only if, given the latent vector, the group follows an exponential family whose natural parameter is the latent vector (Proposition 2, Appendix E.3). This restricts only the measurement: the latent causal model may be arbitrarily nonlinear, subject to the regularity of Assumption 1 and the non-degeneracy of Assumption 3, and the separator may have any dimension. It covers every post-nonlinear measurement with a smooth invertible and Gaussian noise , as well as the constructions that stack several such layers.
Measurements with approximately affine conditional scores. Beyond these models, Theorem 3 only asks the conditional scores to be close to affine over the latent values that the posterior covers, which depends on how nonlinear and how noisy the measurement is over that neighbourhood. This covers every smooth measurement whose curvature stays small relative to its slope at the given noise level, for example tanh, softplus and sinh over moderate input ranges, polynomials such as , whose curvature relative to the slope decreases as the input grows, and periodic functions such as sin while the data stay within a range where the function is invertible. Without noise and with an invertible measurement, the scores are affine over the posterior however strong the nonlinearity.
Limitations. The assumptions can fail when the measurement is nearly flat, non-invertible or not smooth over the range of the latent, as for a saturating, periodic or clipped sensor, or when the observation noise is large relative to the loading, since the conditional scores are then far from affine over the posterior.
The experiments illustrate this characterization. We scale the input of each measurement function by a factor ( is the reference scale), which moves the data along from the covered class towards these limitations.
| Measurement function | Skeleton F1 of HRC |
|---|---|
| , | 0.98 |
| , | 0.93 |
| softplus, | 0.98 |
| softplus, | 0.62 |
| sinh, | 1.00 |
| , | 0.88 |
| , | 0.12 |
HRC stays accurate while the measurement remains steep relative to its curvature over the data, even for the strongly curved at , degrades as a nearly flat region enters the data (softplus at ), and fails, as the characterization predicts, when different latent values give the same measurement ( at ).
In light of your suggestions, we have provided this characterization of the covered models and their limitations (Appendix E.2) and the corresponding experiments (Appendix G.4, Table A8) in the updated version.
Response to Reviewer y6t8 (3/17): W2
W2. In particular, experiments involving stronger nonlinearities, such as polynomial functions, highly oscillatory functions, or interactions between several latent factors, would help better understand the boundaries of the proposed approach.
Thank you for this valuable suggestion. Following it, we have added experiments on the three kinds of stronger nonlinearity you mention: polynomial measurement functions, highly oscillatory measurement functions, and interactions between latent factors, both inside a measurement function and in the latent causal mechanisms.
Polynomial measurements. We add terms with unit coefficients up to degree five.
| Measurement function | Skeleton F1 of HRC |
|---|---|
| 0.98 | |
| 0.96 | |
| 0.82 |
HRC remains accurate as the polynomial grows to degree five.
Oscillating measurements. We increase the frequency of and add a fast oscillation to a linear trend.
| Measurement function | Skeleton F1 of HRC |
|---|---|
| 0.97 | |
| 0.88 | |
| 0.98 | |
| 0.12 |
HRC remains accurate under slow oscillations and under fast oscillations around a trend, and degrades only when the measurement becomes non-invertible within the range of the latent, as for .
Interactions inside a measurement function. A second latent enters the nonlinearity of one indicator with weight .
| Interaction | Skeleton F1 of HRC |
|---|---|
| None | 0.91 |
| 0.76 | |
| 0.76 | |
| 0.62 |
HRC remains accurate for a weak interaction and degrades gradually as the second latent gains weight, since the indicator then has two latent parents, which lies outside the pure measurement model of the FOFC + PC application rather than outside HRC itself.
Interactions in the latent causal mechanisms. In latent graphs where some latent has two or more parents, we add pairwise products of the parents with strength to its mechanism.
| Latent mechanism | Skeleton F1 of HRC |
|---|---|
| Linear | 0.67 |
| Pairwise products, | 0.62 |
| Pairwise products, | 0.31 |
HRC handles moderate interactions among the latent causes and degrades as they become strong.
In light of your suggestions, we have provided these experiments (Appendices G.4 and G.5, Tables A9, A10, A12 and A13) in the updated version.
Response to Reviewer y6t8 (1/17): W1.1
Response to Reviewer y6t8 (1/18): W1.1
We sincerely thank you for your detailed and thoughtful review, and for finding our work interesting, well written and a valuable contribution that extends rank-based constraints to nonlinear settings. Your comments on the scope of the nonlinear extension, the experimental analysis and the positioning of HRC have helped us substantially improve the paper. We address your comments one by one below.
Thank you for this insightful comment. We would like to kindly note that the paper characterizes when this residual is small: Theorem 3 (Section 4.2) gives explicit sufficient conditions through posterior concentration and the curvature of the conditional scores, and the sensitivity study in Appendix G.3 (Tables A6 and A7) examines them by varying the noise level and the latent range. We hope this addresses your concern. Prompted by your comment, we have further detailed what Theorem 3 implies for the residual relative to the spectral gap and measured this residual in practice.
If the posterior of the separating latents lies in a ball of radius δ, Theorem 3 bounds the residual by terms of order δ3 (Equation 10), while the spectral gap σd∗(HA,Baff) grows with the score slopes times the posterior variance, of order δ2. The residual relative to the gap is therefore of order δ times the ratio of the curvature to the slope of the conditional scores, so it is small when the posterior is narrow, that is, when the observation noise is small and the measurement is steep. The requirement is thus local rather than global: it depends on how noisy and how nonlinear the measurement is over the posterior neighbourhood, not on how nonlinear it is over the whole range of the latent. Without noise and with an invertible measurement, the residual vanishes however strong the nonlinearity; it becomes large only where the slope of the measurement approaches zero within the range of the latent.
To see how small this residual is in practice, we measured it directly on the conditional score in the generator of the paper. Around each latent value, the valid neighbourhood is the largest interval over which the score is affine in the latent up to 1% of its variation; the needed range is the range covered by the posterior (±2 posterior standard deviations), over which Theorem 3 requires the score to be close to affine (medians over latent values and evaluation points):
In the settings of the paper, the needed range lies inside the valid neighbourhood at almost every point, so the residual is small relative to the affine signal that carries the rank. A stronger but steep nonlinearity, such as x+x3 with its input scaled by 4, keeps it inside and HRC accurate, whereas a measurement whose slope approaches zero, such as the saturating tanh with its input scaled by 4, widens the needed range beyond the valid neighbourhood and HRC degrades.
In light of your suggestions, we have provided this analysis of the residual relative to the spectral gap and its measurement (Appendix G.6, Table A14) in the updated version.