- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose fDAG-ACDC, a methodology focusing on estimating the number of latent components in a causal representation learning model. The proposed methodology doesn’t learn the causal DAG itself, but rather employs any compatible fDAG learning method (in particular ABCDEFG) for a given number of factors m and then uses the modified criterion ACDC to assess the quality of the estimation. The method is evaluated in two synthetic experimental settings with a fixed fDAG learning method and comparing 4 distinct criteria for assesing the choice of m. Finally it is evaluated on a real world dataset where it shows improvement over Validation NLL.
Critical strengths
The authors study a narrow but important and motivated problem in the CRL literature. Their estimation of the number of the latent factors was previously “ignored” in the literature.
The method shows improvement in two experimental settings and in two real datasets. The improvement is in the choice of factors m comparing to other criteria that can be used to choose m.
Critical weaknesses
In theory, I am afraid that when the number m of factors is large, then the complexity of searching it in a brute force way rises significantly. The theory provides a guarantee for identifying the minimum m0, and it is mentioned that under an identifiability guarantee the unique generating factor can be identified (lines 110-112). The latter is never analyzed.
The experimental analysis is quite narrow. Only two experimental setups are considered with a small number of factors m. The baselines chosen as criteria are only three and other criteria are excluded with no supportive justification. In particular AIC/BIC were avoided as they are not well defined for neural models (as claimed in the text), but the experiments use a linear SEM (Eq. 34). Moreover, such important experimental details regarding the data generation (Eq. 34), should be in the main text rather in the appendix.
Questions for authors
What is the complexity of looking for all m’s? Wouldn’t that be a bottleneck in problems with significantly large m?
Can you please explain the identifiability guarantee for estimating the unique generating factor number? Please see weknesses above.
It is a bit confusing to me why F1 and SHD are called oracles whereas they don’t achieve optimal performance.
Can you please explain why you didn’t use AIC/BIC on a linear SEM setup? Also, can you please confirm that the data you generate come from a linear SEM?
4 comments
Response to Reviewer qTZ1 (Q3)
Q3: It is a bit confusing to me why F1 and SHD are called oracles whereas they don’t achieve optimal performance.
F1 and SHD require access to the true graph. Both metrics compare an estimated graph with the true graph, so they cannot be computed in real-world applications where the true graph is unknown. The term “oracle” refers to this access to ground truth. It does not guarantee optimal factor-count selection: maximizing F1 or minimizing SHD targets graph recovery, which need not yield the generating factor count.
Checkpoint selection aims to improve candidate-model quality. In the original simulation experiments, we retained the checkpoint with the highest graph-recovery F1 for each candidate factor count and fitting replicate. This was intended to obtain candidate models with good graph-recovery quality and reduce the influence of imperfect fitting when assessing factor-count selection using ACDC. This use of the true graph occurs during checkpoint selection; the ACDC score and factor-count selection rule do not directly use the true graph.
We have clarified checkpoint selection in Appendix B.2 and made the F1/SHD factor-count selection rules explicit in Appendix B.4 (Equation (41)).
Response to Reviewer qTZ1 (Q4)
Q4: Can you please explain why you didn’t use AIC/BIC on a linear SEM setup? Also, can you please confirm that the data you generate come from a linear SEM?
The simulation data come from a linear SEM. Section 4.1 (p. 8, Equation (23)) now states the linear SEM and additive noise. Table 5 (p. 17) details the interventions; Table 6 (p. 18) specifies the linear generative mechanism used for fitting.
AIC/BIC are not plug-in criteria for the fitted pipeline. Linearity of the simulator does not by itself make textbook AIC/BIC applicable to the fitted latent-variable pipeline. Section 1 (p. 2) already discusses the effective-parameter-count requirement and its interpretation in overparameterized CRL models. The classical formulas are
where is the observed-data training log-likelihood, is its maximum-likelihood estimate, is the training sample size, and is the dimension of the statistical model under the relevant regularity assumptions—not simply the latent count or the number of entries in the training code (Akaike, 1974; Schwarz, 1978).
There are two concrete obstacles to applying these formulas directly to our retained fits.
The reported NLL is not the required maximized training likelihood. First, ABCDEFG is fitted using a masked variational autoencoder with learned graph masks, variational KL terms, and sparsity penalties (Appendix B.2, Figure 4 and Table 6, p. 18), rather than by maximizing an unpenalized observed-data likelihood for each fixed classical SEM. The validation NLL reported in Appendix B.4, Equations (37)–(38) (p. 21), is explicitly the held-out decoder reconstruction term. It is not the maximized training log-likelihood obtained by integrating out the latent variables. Adding or to that reconstruction score would therefore not produce classical AIC or BIC.
A suitable statistical model dimension is needed. Second, a linear input–output mechanism does not make an overparameterized latent-factor representation a regular, uniquely parameterized statistical model. Counting all trainable weights would include variational inference parameters and redundant parameterizations, whereas counting only selected graph edges would omit other distributional parameters. Redundant factors, learned masks, and factorized linear maps can also create nonidentifiability or singularities. The distinction between regular and singular models—and the resulting limitations of ordinary parameter-count corrections—is established in singular learning theory (Watanabe, 2010, 2013). Thus, the concern is not that software cannot count parameters, but that the resulting count and likelihood must have the statistical meaning required by the criterion.
The selection targets differ. There is also a difference in the selection target. AIC estimates global predictive performance, while classical BIC approximates Bayesian model evidence under its regular-model assumptions. Our target is instead the smallest count attaining optimal local structural adequacy under the chosen criterion (Section 3.3, Equation (15)), including settings where added latent structure may absorb misspecification. Neither classical construction automatically guarantees recovery of this target. For these reasons, AIC/BIC are not directly applicable as plug-in baselines for the present fitting-and-scoring protocol.
References
Akaike, H. (1974). A New Look at the Statistical Model Identification. IEEE Transactions on Automatic Control, 19(6), 716–723.
Schwarz, G. (1978). Estimating the Dimension of a Model. The Annals of Statistics, 6(2), 461–464.
Watanabe, S. (2010). Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory. Journal of Machine Learning Research, 11, 3571–3594.
Watanabe, S. (2013). A Widely Applicable Bayesian Information Criterion. Journal of Machine Learning Research, 14, 867–897.
We have moved the linear-SEM data-generation details to Section 4.1 (Equation (23)) and made the validation-NLL definition and selector explicit in Appendix B.4 (Equations (37)–(38)).
Thank you again for your thoughtful questions and constructive suggestions. We hope that our responses have addressed your concerns and clarified the points you raised.
Reply to rebuttal
I thank the authors for their time and effort in carefully addressing my questions and concerns. In particular, I appreciate the clarification of the identifiability guarantees, the computational complexity of the proposed approach, and the rationale for not directly applying AIC/BIC to the considered fitting pipeline. I am also glad to see that these clarifications and the additional theoretical conditions have been incorporated into the revised manuscript. Overall, the response has addressed my main concerns, and I therefore increase my score to 3 (Weak Accept).
Response to Reviewer qTZ1 (Q1–Q2)
We sincerely thank you for your time and effort in reviewing our paper.
Q1: What is the complexity of looking for all m’s? Wouldn’t that be a bottleneck in problems with significantly large m?
Candidate enumeration is linear in grid size at fixed per-candidate cost. With K=∣M∣ candidates and S paired fitting replicates, this requires SK candidate fits. We evaluate a finite grid of candidate counts, fitting each candidate with ABCDEFG. This is a sweep over factor counts rather than exhaustive enumeration of graph structures. A validation-NLL comparison using the same candidate grid and replicates shares the underlying fitting cost.
Post-fitting complexity. Candidate enumeration scales linearly with K at fixed per-candidate costs; dependence on m enters through model size and local neighborhoods. For the additional post-fitting computation, let d be the number of observed nodes and B the number of bootstrap resamples. With at most R scorable environments per local unit, n observations per environment, and k variables per neighborhood, dense covariance estimation and Gaussian KL evaluation cost O(nk2+k3) per local environment. Writing Tgraph and Tpp for per-fit bounds on graph/neighborhood construction and predictive generation, the post-fitting cost is bounded by
Tpost=O(SK[Tgraph+Tpp+dR(nk2+k3)]+BKSlogS).
The bootstrap term comes from sorting S values per candidate and resampling cached criterion curves.
Large models or wide grids can become computational bottlenecks. Yes, substantially larger candidate models or a wider candidate grid can make candidate-model fitting and evaluation a computational bottleneck. A wider grid increases the number of fits, while larger m can increase fitting, predictive-generation, and local-discrepancy costs. Thus, the dependence on K is linear at fixed per-candidate costs, while overall scalability also depends on the size of each candidate model. Candidate fits can run independently in parallel.
We have added a scalability summary to Section 4.1 and training-time measurements and complexity analysis to Appendix B.2 (Tables 8–9, Equation (36)).
Q2: Can you please explain the identifiability guarantee for estimating the unique generating factor number? Please see weaknesses above.
Sufficient conditions: local adequacy and separation from smaller candidates. Let m⋆∈Msel be the minimal, nonredundant generating count. First, the population fitted model at m⋆ must reproduce every scored local distribution, giving C(m⋆)=0. Second, each smaller candidate must mismatch at least one scored local distribution with positive environment weight. For the Gaussian implementation, a difference in a local mean or covariance is sufficient, assuming positive-definite covariances. For such a local unit j and environment r, nonnegativity of KL gives
C(m)≥dkj,mwjmrKL[Pjmr∥Qjmr]>0=C(m⋆),m<m⋆.
Thus m0=m⋆, with a positive underfitting gap on the finite candidate grid.
Linear–Gaussian illustration. Consider candidates with a shared linear decoder X=LmZ+bm+ϵ, where Lm∈Rd×m, bm and the independent, mean-zero Gaussian observation noise are environment-invariant, and interventions shift Gaussian factor means. At the true count, suppose L⋆ has full column rank and the factor-mean contrasts span Rm⋆. Write μr=E[X∣r] and νr=E[Z∣r]. Across environments r=0,…,R, the observed mean-contrast matrix satisfies
= L_\star[\nu_1 - \nu_0, \ldots, \nu_R - \nu_0], \qquad \operatorname{rank}(D) = m^\star.$$ Every $m$-factor candidate in this shared-decoder family has a predictive mean-contrast matrix of rank at most $m$. Hence a candidate with $m < m^\star$ cannot reproduce all environment-specific observed means. If all these environments are scored with positive weight, some observed node has a mean mismatch in its own local unit, ensuring positive local Gaussian KL. Together with an accurate population predictive fit at $m^\star$, this verifies the two conditions above. The rank requirement expresses an intuitive intervention condition: the shifts must reveal every factor direction through the observations. **Recovery follows under the consistency assumptions.** Under these sufficient conditions and the respective assumptions of Theorems 1 and 2, the pointwise and contiguous selectors recover $m^\star$ with probability tending to one as $S$ increases. We have added the sufficient conditions linking the generating count to the consistency target in Appendix D.9 (Equation (100)).