acceptodds
Under review as a conference paper at ICLR 2027

Sparse Neural SIV: Learning Nonlinear Causal Relationships with Synthetic Instruments

Abstract

We consider learning nonlinear causal relationships between multiple treatments and an outcome under unobserved confounding. We leverage information in the joint treatment distribution to learn these relationships without external instruments, assuming that only a sparse, unknown subset of treatments has a causal effect on the outcome. We develop a latent-factor framework under which valid instrumental variables can be derived from the treatments themselves. These synthetic instruments (SIVs) enable identification of nonlinear causal relationships through functional sparsity and a plurality rule. For estimation, we propose Sparse Neural SIV, which incorporates treatment-level sparsity into neural instrumental variable regression to jointly select active treatments and estimate the causal relationship. We fit the network by matching outcomes to predictions averaged over treatments sampled conditional on the synthetic instruments. Under suitable regularity conditions, we establish an estimation-error bound for a global minimizer of the regularized fitting criterion and give conditions for recovering the active treatments. Simulations show lower estimation error than the competing methods in the interaction settings studied under Gaussian and mixture treatment distributions. A mouse-genomics application illustrates nonlinear fitted relationships between gene expression and body weight.

Primary area:
causal reasoning
Keywords:
causal inference, nonparametric instrumental variables, latent confounding, sparse neural networks, variable selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UYJCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The article extends the synthetic instruments approach of Tang et al. (2026) to a nonparametric treatment model. Identification is established under some assumptions. An estimator based on risk minimization over sieves of sparse neural networks is proposed, and some risk bounds are proven under additional assumptions. The method is shown to work well in correctly specified settings.

Critical strengths

Timeliness

The paper picks up an idea of a recent high-profile publication in statistics and extends it to the nonparametric setting. This is timely and interesting.

Good method design

The general design of the estimation method using sparse neural networks and minimizing appropriate risk surrogates is nice.

Critical weaknesses

Heavy assumptions, hardly discussed

The paper makes a long list of fairly strong assumptions that do all the heavy lifting, but the main paper hardly discusses any of them.

  1. The causal model in Section 2.1 itself is barely motivated, and I don't really see why something like this should hold in applications. This deserves substantial discussion.
  2. The sparsity and plurality assumptions are very restrictive, but hidden in the appendix and barely discussed. What comes in Appendix A should be much more central to the paper in my opinion.
  3. The assumptions made for Theorem 2 are almost as long as the proofs (ergo hidden in the appendix), and several of them are questionable; see my questions below.

Extremely favorable experimental settings

The experimental design is favorable in many ways:

  1. Larger variances are assigned to active treatments.
  2. The models are largely specified correctly; no attention is paid to violations of the causal assumptions.
  3. The sampler underlying Sparse Neural SIV actually matches the true model, while leaving competitors with a misspecified choice. A general-purpose method should not rely on knowledge of the data-generating process to such a degree.
Questions for authors
  1. What is the exact relation between your work and Tang et al. (2026)? They seem to have come up with the basic model, ideas, plurality arguments, sparsity assumptions, ... From what I can see, you're putting the "nonparametric" part on top, which causes a few extra difficulties. The paper should make clear which parts you took from previous work and which are the genuinely new contributions.
  2. a. The stability in Assumption 5, eq (11), already fails in simple linear Gaussian models for networks approximating the sign function. What changes if we weaken this by allowing, e.g., other exponents than 1 on the right hand side?
    b. Assumption 7 requires one network to simultaneously have small error for h0h_0 and small regularized loss compared to h0,αh_{0, \alpha}. Why should we expect such a network to exist?
    c. Assumption 8 is something I have never seen, and it is not verified for a concrete model. I see that it is convenient in the proofs and extends ideas from linear models. But why should we expect this to hold in the neural network world?
4 comments
AuthorsOpenReview ↗

Response to Reviewer UYJC (Part 1/2)

Response to Reviewer UYJC

Thank you for identifying gaps in the estimation assumptions and experimental comparisons. The revised proof removes the uniform ReLU rotation bound and assumed cross-support curvature, and derives objective approximation from approximation of h0h_0. We address these changes and the empirical concerns separately. References below are to the revised manuscript.

Q1. Contribution beyond Tang et al.

In response to your question, Section 1 now separates the inherited framework from our contributions. We retain Tang et al.’s (2026) treatment factor model and synthetic-instrument construction, but make the strategy applicable when nonlinear causal relationships and interactions cannot be specified in advance, while jointly selecting active treatments.

As also recognized by Reviewer QyuP, this is non-trivial: it requires functional identification with unknown support and solving an ill-posed inverse problem with estimated instruments. Sections 3–4 develop sparse functional identification—including a result without plurality under stronger sparsity—and a sparse neural estimator using conditional treatment sampling.

The estimation and support-recovery theory accounts for generated instruments, sampling error, neural approximation and regularization, narrowing the gap between the synthetic-instrument strategy and practical use.

Q2. Revisions to the estimation theory

In response to your three concerns, we revised the proof to remove the uniform rotation bound and the cross-support curvature assumption, and to derive objective approximation from approximation of h0h_0 (Section 4.4).

(a) Instrument stability.

We no longer require Equation (11). We replace it with approximate moment validity under estimated instruments and stability of a source representer g0g_0 satisfying h0=T0∗g0h_0=T_0^*g_0 (Appendix B.2, Assumptions 4 and 6). The proof excludes incorrect supports, then controls estimation on the true support through Tikhonov regularization.

Under the revised assumptions, including βiv>1\beta_{\mathrm{iv}} > 1, the loading-estimation term changes from 1/(nNαn2)1/(n_N\alpha_n^2) to 1/(nNαn)1/(n_N\alpha_n), and the bias term becomes αn\alpha_n. Here αn\alpha_n is the regularization parameter and nNn_N the nuisance-sample size; Theorem 2 gives the full rate.

(b) Neural approximation.

We no longer require a separate bound on excess regularized loss relative to h0,αh_{0,\alpha}. We assume a true-support network approximates h0h_0 in L2L_2; the Tikhonov normal equation and perturbed bias bound then yield objective approximation (Appendix B.2, Assumption 6; Appendix C.2, Lemmas 5–6). Corollary 2 in Appendix D.5 gives the resulting neural rate.

(c) Curvature across supports.

We no longer assume it. For a fixed population, fixed dimensions and a fixed output bound, identification and weak compactness yield a positive gap across the finitely many incorrect supports. On the true support, the quadratic Tikhonov identity gives curvature directly in function space, accommodating nonconvex neural parameterization.

The theorem establishes exact support recovery under its assumptions, including k=sk=s, unknown support, independent nuisance fitting and a global minimizer (Appendix C.2, Lemmas 4–6).

Q3. Causal motivation and identification assumptions

In response to your request, Sections 2–3 now state the causal and identification assumptions explicitly. A motivating model is X=ΛU+ϵX=\Lambda U+\epsilon and Y=h0(X)+g(U)+ηY=h_0(X)+g(U)+\eta: shared latent states influence gene expression and weight, while h0h_0 allows nonlinear gene effects and interactions. Independence of UU and ϵ\epsilon, centered g(U), and E(η∣U,ϵ)=0E(\eta\mid U,\epsilon)=0 imply the maintained disturbance restriction.

Our response to Q5 below examines sensitivity to treatment–confounder interactions beyond this additive model (Appendix G.12, Table 12).

Different genes participate in different biological processes, motivating a sparse causal model in which only a subset directly affects weight. With p=17p=17 and q=2q=2, s<p−qs < p-q requires at least three nulls. We also prove identification within Hk\mathcal H_k without additional plurality whenever 2k≤p−q2k\le p-q, under moment validity and candidate completeness.

Under these assumptions and the corresponding bound on true sparsity, this result applies to all budgets k=1,…,6k=1,\ldots,6 considered in mouse CV. The main figure uses CV-selected k=1k=1; the fixed-k=2k=2 two-gene fit is in Appendix G.7.

Sections 2–3 now state the factor-space, completeness, sparsity and plurality conditions explicitly. Appendix A.3, Proposition A.2 gives the support-union proof. Appendix G.11 reports conservative factor selection using the maximum of two selectors and sensitivity across q=1,2,3q=1,2,3, including instability across training sets.

AuthorsOpenReview ↗

Response to Reviewer UYJC (Part 2/2)

Q4. Balancing active and inactive treatment variances

Motivated by your concern, we added 50 paired replications at n=3,000n=3{,}000, equalizing active and inactive groups’ average marginal variance while keeping loadings and total treatment variance unchanged. Sparse Neural SIV retains an advantage: RelL2 is 0.888 versus 1.026 for Linear SIV and 1.813 for Dense Neural SIV. Paired differences are −0.138 [95% CI: −0.211, −0.067] and −0.925 [−0.997, −0.856].

Absolute error increases, but the comparisons support benefits from nonlinear modeling and treatment-level sparsity beyond the original variance allocation (Appendix G.13, Table 14).

Q5. Sensitivity to causal-model misspecification

Motivated by your question, we added a moment-violating experiment:

Y=h0(X)+δsin⁡(0.75X1)U1/c+0.5ηY=h_0(X)+\delta\sin(0.75X_1)U_1/c+0.5\eta, where cc standardizes the interaction. This introduces latent treatment-effect modification while preserving the marginal intervention mean h0h_0. The baseline δ=0\delta=0 has no latent-factor confounding.

Across 50 paired replications at n=5,000n=5{,}000 with k=2k=2, mean structural RMSE is 0.437 at baseline and 0.443 at δ=0.1\delta=0.1, a paired increase of 0.007 [95% CI: 0.003, 0.012]. Exact input-mask recovery remains 96%, compared with 98% at baseline. Increasing the departure fivefold to δ=0.5\delta=0.5 yields RMSE 0.525, a paired increase of 0.088 [0.073, 0.104], with support recovery of 48%.

These results highlight robustness at the modest departure and greater sensitivity of treatment selection under the stronger perturbation. Results at n=3,000n=3{,}000 show the same direction. Appendix G.12 describes the design, and Table 12 reports the results for both sample sizes.

Q6. Comparing methods with shared conditional samplers

Motivated by your concern about comparability, we added experiments in which all three SIV methods use the same fitted conditional sampler, with 50 paired replications at n=3,000n=3{,}000 under Gaussian treatment noise and a three-component scale mixture. Within each design, all three SIV methods share the same fitted first stage under either a single-Gaussian or two-component Gaussian-mixture sampler.

Both sampler families are misspecified for the three-component design.

Under mixture noise, Sparse Neural SIV retains its advantage: RelL2 is 0.278/0.276 across samplers, versus 1.431/1.352 for Dense Neural SIV and 1.029/1.049 for Linear SIV. Both paired Sparse-minus-Dense confidence intervals exclude zero. Changing the sampler gives no clear improvement for Sparse Neural SIV (difference −0.002 [95% CI: −0.014, 0.011]).

The advantage therefore persists with shared samplers, including these misspecified cases (Appendix G.8, Table 7).

I thank the authors for revising the paper and generating a response. I see that half of the assumptions and several pages of proofs for the main estimation theorem have substantially changed. The initial paper clearly wasn't ready, and expecting the reviewers to check all this during the rebuttal period is unreasonable, so I will keep my score.

Thank you for the follow-up. We would like to clarify the scope of this revision. The identification conditions (Assumptions 1–3), the identification result (Theorem 1), and the Sparse Neural SIV estimator are the same as in the submission. You suggested giving the identification conditions a more central place in the paper, so we moved their statements from Appendix A into the main text. This accounts for part of the change visible on the page.

The three steps you asked about in Q2 are needed for the estimation guarantee in Theorem 2. The identification argument in Theorem 1 and the construction of the Sparse Neural SIV estimator do not rely on them. Furthermore, the proof follows the same overall strategy. It uniformly controls the difference between the empirical and population criteria, uses curvature to control estimation error, and bounds the regularization bias. In response to Q2, we replaced the uniform stability assumption in Eq. (11) with conditions on the functions used in the proof. In Appendix C.2, Lemma 4 establishes a uniform moment gap on incorrect supports, Lemma 5 gives the exact curvature on the true support, and Lemma 6 bounds the Tikhonov regularization bias under instrument perturbation. The empirical-process bounds in Appendix D.1–D.4 and the Gaussian sampler verification in Appendix E use the same arguments as before.

We also added experiments addressing your three empirical questions about equalized treatment variances, departures from the causal model, and a common conditional sampler across SIV methods.

If any particular step remains unclear, we’d be happy to point to the relevant lines in the revised proof and explain the argument.

Reviewer_QyuPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends the identification arguments in Tang et. al. (2026) to nonparametric models. Motivated by the identification results, it provides nonparametric estimator based in part on Li et. al. (2024). Convergence rates are provided under standard conditions, along with simulation evidence of its accuracy. I elaborate below.

Identification:

The paper considers settings in which p>1p>1 treatments are available. Confounding between treatments and outcomes is explained by q<pq<p latent factors. The paper suggests that the low-dimensional confounding be partialled out, which can be achieved using say, PCA. The problem then, is that after partialling out this confounding, one is left with only p−qp-q dimensional residual variation, i.e., a length p−qp-q vector of `synthetic instruments'. These p−qp-q instruments are valid (under the assumptions of the paper) but even in a linear model, this would be insufficient to identify the effects of the pp treatments. However, the paper assumes that \textbf{strictly} fewer than p−qp-q of the treatments actually have a causal effect on the outcome. If only \textbf{weakly} fewer than p−qp-q treatments impact the outcomes, and this subset of treatments were known, then causal effects could be identified: both in linear models, and indeed in additive nonparametric models under a statistical completeness condition (a la Newey and Powell 2003). It is the extension to nonparametric models that differentiates these identification results from Tang et. al. In reality, the identifies of these treatments are unknown, however, one can apply a plurality vote, in a similar spirit to Guo et. al. (2018). Consider all subsets SS of the treatments of size p−qp-q, for each of these there is a corrersponding population IV estimate, and if the subset contains the true <p−q<p-q treatments, that population IV estimate is the true causal effect. Because there are at least q+1q+1 such subsets, then at least q+1q+1 of these corresponds to the same estimate. If q>p/2−1q>p/2-1 then a majority must provide the same true causal effect, but otherwise a plurality vote will provide the correct answer so long as there is not by coincidence a larger subset that provides wrong estimates that all happen to agree.

Estimation:

The partialling out of the shared confounders occurs in a first stage. Note that the factor model is linear (in contrast to the structural function h0h_0 which is treated as nonparametric). This step yields the synthetic instruments. These are then used in a manner similar to the external instruments in penalized DeepIV. However, a key distinction is that only a subset of size at most p−q−1p-q-1 treatments may enter the structural function. This is achieved by imposing sparsity in the first layer of the relevant neural network in a data-driven way. The approach is analogous to Tang et. al.'s methods for linear models.

Efficacy:

The asymptotic results are standard. Consistency of the matrix that orthogonalizes treatments with respect to the shared confounders is simply assumed as a high-level condition.

The simulation results suggest the method performs better than alternatives in large samples when the true model is non-linear. This is not surprising, all the other methods should be inconsistent in such settings whereas the data-generating process satisfy the assumptions underpinning the method presented in this work.

Critical strengths

The paper is a non-trivial extension of Tang et. al. (2026). Whereas the latter assumes a fully linear model, the present work allows for a non-linear (indeed, non-parametric) structural relationship between treatments and outcomes. The underlying logic of the identification strategy remains essentially the same. The paper uses this identification approach to motivate a nonparametric estimation method similar to penalized neural instrument variables (IV) regression, but with some crucial alterations (as I describe in the summary above). The authors provide some asymptotic analysis and Monte Carlo experiments. I found it quite well-written and easy to follow but then I am very familiar with this area and so it is hard for me to judge how clear it would be to someone who is newer to this literature.

Critical weaknesses

I don't think there is anything ground-breaking here (certainly there is less conceptual innovation than in Tang et. al. (2026)). That said, the extension to npnparametric models is non-trivial.

Personally, I am a somewhat skeptical of plurality vote (and indeed majority vote) approaches to IV estimation in general. In practise, both parametric and non-parametric IV models are always misspecified (for one thing, why would we ever assume the causal model is additively separable?) and if they are misspecified, there will generally not be something that resembles a clear voting winner. Given that the prposed method uses sparsity rather than an explicit plurality vote, I wondered how the present method may perform under some small degree of misspecification.

Questions for authors

It was unclear to me how the proposed estimator actually implements something like a plurality vote. It seems that (given a fixed number of treatments that may impact the outcome) one simply fits a sparse neural network rather than perform any actual plurity vote. It is not really clear how these are connected. The same can be said for Tang et. al. These approaches seem to me to resemble more the Lasso approach to IV selection (as in e.g., https://arxiv.org/abs/2208.05278), and this requires some more elaboration I think. I would be curious to hear more about the relationship between the plurality-vote motivation and the actual estimation method, which imposes sparsity.

I wondered whether it might be possible to simply control for the extracted UU (of course, only identified up to rotations) in a sparse nonparametric regression? In linear IV, two-stage least-squares is equivalent to partialling out the IV from the treatments and then adding this estimated residual as an additional control in estimation. With majority/plurality vote in linear models, this approach would be close to https://arxiv.org/abs/2208.05278. In non-parmaetric models, this could perhaps avoid some of the difficulties associated with nonparametric IV (e.g., ill-posedness). That said, there could be some big reason this would not work, but I am curious.

2 comments
AuthorsOpenReview ↗

Response to Reviewer QyuP (Part 1/2)

Response to Reviewer QyuP

Thank you for the questions about plurality, factor adjustment and misspecification. We agree that the central identification strategy builds on Tang et al. (2026). Our main contribution is to substantially narrow the gap between that strategy and its practical application, by making it applicable to a broader range of settings in which nonlinear effects and interactions cannot be specified in advance.

We retain the treatment factor model while allowing an unknown nonlinear causal response, including interactions, and jointly selecting the active treatments. As the reviewer recognizes, this extension is non-trivial: it requires moving from finite-dimensional identification to an ill-posed functional inverse problem, and developing an estimation procedure and theory that accommodate both nonlinear responses and unknown treatment support.

References below are to the revised manuscript.

Q1. Why sparse fitting implements plurality

Your question prompted us to clarify how sparse fitting implements the voting argument. Sparse fitting implements the population plurality rule through a single constrained moment criterion, so an explicit vote-counting step is unnecessary. Under the identification conditions, candidate voting and the sparsity-constrained criterion identify the same causal relationship (Section 3.2, Theorem 1; Appendix A.1).

This replaces fitting and comparing a model for every voter with one constrained optimization problem. Two observations explain why.

First, consider three treatments, one factor and one active treatment. Each candidate imposes a different null-effect hypothesis, βj=0\beta_j=0. Although their synthetic-instrument coordinates differ, they are invertible transformations of the same Z=B⊥⊤XZ=B_\perp^\top X and therefore generate the same conditional-expectation operator.

In the linear case, their first-stage treatment projections coincide; denote this common projection by X~\widetilde X.

Second, in the linear example, if treatment 1 is active, the candidates imposing β2=0\beta_2=0 and β3=0\beta_3=0 both return the true vector β0\beta_0, while the candidate imposing β1=0\beta_1=0 returns an incorrect vector. Under candidate uniqueness and plurality, the same population winner is obtained by minimizing E(Y−X~⊤β)2E(Y-\widetilde X^\top\beta)^2 subject to ∣β∣0≤1\\|\beta\\|_0\le1.

Any other one-sparse minimizer would solve two candidate regressions and therefore receive two votes, contradicting plurality. This is the common-projection and sparse-fitting insight of Tang et al. (2026, Sections 3.3–3.4).

Our nonparametric formulation uses the same logic with the operator Th=Eh(X)∣ZTh=E\\{h(X)\mid Z\\}. For s≤k<p−qs\le k < p-q, any zero-loss function supported on at most kk treatments receives at least (p−∣S(h)∣q)≥q+1\binom{p-|S(h)|}{q}\ge q+1 votes. Plurality makes h0h_0 the unique minimizer of ∣Th−r∣22\\|Th-r\\|_2^2 over this sparse class (Section 3.2, Theorem 1; Appendix A.1).

Thus a single constrained optimization implements the population identification rule. For nonlinear hh, the relevant prediction is Eh(X)∣ZE\\{h(X)\mid Z\\}, generally different from hE(X∣Z)h\\{E(X\mid Z)\\}; this is why our estimator averages network outputs over conditionally sampled treatments.

This population equivalence concerns the hard-constrained moment criterion; the regularized empirical estimator is analyzed separately in Theorem 2.

Multiply robust estimation rather than select-then-estimate.

Under the stated identification conditions, the same criterion works across admissible sparse configurations, regardless of which treatments have zero effects. Unlike selecting valid instruments and then constructing a post-selection IV estimator, as in https://arxiv.org/abs/2208.05278, we optimize a common criterion without first deciding which candidate hypotheses are correct.

This avoids fitting a separate ill-posed nonparametric IV model for every candidate and thresholding agreement between fitted functions. (Appendix A.1).

AuthorsOpenReview ↗

Response to Reviewer QyuP (Part 2/2)

Q2. Why not adjust directly for estimated factors?

Tang et al. (2026, Section 5.2 and Supplement S.8) show why direct factor adjustment is not generally a substitute for SIV in nonlinear outcome models. Their factor adjustment methods, including the linear-score and prespecified cubic adjustment methods, show persistent error over the sample sizes examined, whereas SIV error decreases.

Motivated by your question, we expanded the comparison with factor and residual adjustment. Our Sparse Neural U^\widehat U comparator fits hθ(X)+aϕ(U^)h_\theta(X)+a_\phi(\widehat U) with neural networks in both branches. In the main Gaussian Interaction design at n=10,000n=10{,}000, RelL2 is 0.292 for Sparse Neural SIV versus 1.126 for this comparator (Section 5.1; Appendix G.4, Table 2).

A paired 2×2 experiment crossing Gaussian/mixture noise with additive/interacting confounding shows where the advantage holds (32 replications, n=3,000n=3{,}000; Appendix G.4, Table 4). Both adjustment methods outperform SIV under additive confounding.

Under Gaussian interacting confounding, RelL2 is 0.372 for SIV versus 0.544 for either adjustment; under mixture interacting confounding, it is 0.346 versus 0.449 for factor adjustment and 0.376 for residual control, with both mixture comparisons inconclusive. All four designs preserve the IV moment.

The key distinction is that estimated factors are proxies, not observed confounders. In the model Y=h0(X)+g(U)+ηY=h_0(X)+g(U)+\eta, adjustment must account for Eg(U)∣XE\\{g(U)\mid X\\}; replacing UU by its estimated score does not generally reproduce this quantity. For example, E(U13∣X)≠E(U1∣X)3E(U_1^3\mid X)\ne\\{E(U_1\mid X)\\}^3.

Thus even knowing gg does not justify applying it to estimated scores; Tang et al. (2026, Supplement S.8) explain why the linear adjustment–SIV equivalence need not extend to nonlinear relationships.

This highlights an important advantage of SIV: under the maintained identification conditions, we need not specify or estimate how UU affects YY. Under the sufficient model above, ZZ is independent of UU, so the conditional moment removes any centered, integrable g(U)g(U) (Section 2.2; Tang et al., 2026, Section 4.3).

This matters because UU is unobserved: unlike adjustment for observed covariates, a linear effect of UU cannot be assessed directly, and good outcome-model fit using estimated scores does not validate that latent-effect assumption.

Q3. Model scope and mild misspecification

In response to your concern about plurality, we have added an identification result that removes this requirement under stronger sparsity: under moment validity and candidate completeness, the causal relationship is uniquely identified within the class of functions depending on at most kk treatments whenever 2k≤p−q2k\le p-q (Section 3.2; Appendix A.3, Proposition A.2).

As explained in our response to Reviewer 3zd3, Q1, this permits budgets up to k=7k=7 in the mouse application with p=17p=17 and q=2q=2. Cross-validation selected k=1k=1, well below this threshold (Section 5.2; Appendix G.7, Table 5).

We agree that valid IV analysis requires a valid causal model. Compared with the additive linear model of https://academic.oup.com/jrsssb/article/80/4/793/7048331, we allow unknown nonlinear treatment effects and interactions. The maintained outcome restriction is E(u0∣ϵ)=0E(u_0\mid\epsilon)=0. A sufficient model is Y=h0(X)+g(U)+ηY=h_0(X)+g(U)+\eta, with U⊥ϵU\perp\epsilon, centered g(U)g(U) and E(η∣U,ϵ)=0E(\eta\mid U,\epsilon)=0 (Section 2.2).

Motivated by your question about model misspecification, we added a latent treatment-effect-modification experiment. The results provide encouraging evidence of robustness to a modest departure from additive separation: estimation accuracy remains close to baseline and exact support recovery remains high. Specifically, we generate Y=h0(X)+δsin⁡(0.75X1)U1/c+0.5ηY=h_0(X)+\delta\sin(0.75X_1)U_1/c+0.5\eta, where cc standardizes the interaction term.

This introduces latent treatment-effect modification while preserving the marginal intervention mean h0h_0. Across 50 paired replications at n=5,000n=5{,}000 with k=2k=2, setting δ=0.1\delta=0.1 changes mean structural RMSE from 0.437 to 0.443 while exact support recovery remains 96%, compared with 98% at baseline.

With the fivefold larger departure δ=0.5\delta=0.5, RMSE is 0.525, a paired increase of 0.088 [0.073, 0.104], and support recovery is 48%. Thus the modest departure has little practical effect here, while the stronger departure materially affects performance (Appendix G.12, Table 12).

We also added near-tie and approximate-sparsity checks (Appendix G.12, Table 13). As (X1+aX2)2(X_1+aX_2)^2 approaches an exact tie at a=1a=1, Neural RelL2 rises from 0.676 to 0.805 to 0.899; at the tie, the causal relationship is not uniquely identified. Adding a weak third effect while fitting two treatments changes RelL2 from 0.676 to 0.674 and 0.724, with both paired difference intervals including zero.

The quadratic benchmark is more sensitive (0.039 to 0.724). Both checks retain the IV moment.

Reviewer_3zd3OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to learn nonlinear causal relationships from multiple treatments when confounders are unobserved and no external instruments are available. It constructs synthetic instruments by removing shared latent-factor variation from the treatments, then uses sparsity and a plurality condition to identify the response and which treatments it depends on. The proposed neural estimator uses conditional treatment samples and two independent Monte Carlo batches. The paper gives estimation and support-recovery guarantees under additional assumptions, evaluates the method on artificial factor-model data, and illustrates fitted relationships between gene expression and mouse body weight.

Critical strengths
  1. The writing is relatively clear overall. In particular, the explanation of factor projection and the distinction between linear and nonparametric IV regression help readers follow the construction.

  2. The identification argument and estimation analysis appear mathematically sound under the stated assumptions. The worked Gaussian example and primitive sufficient conditions result provide useful support beyond the high-level plurality condition.

  3. The estimator accounts for the lower-dimensional conditional treatment distribution. Using independent Monte Carlo batches to avoid the variance bias of a single-batch squared loss is also a well-motivated construction.

Critical weaknesses
  1. Assumptions 1–3 are formally stated mainly in the appendices, with insufficient discussion of their causal implications or applicability to the mouse data. Gaussian examples and genericity establish mathematical coexistence, rather than scientific plausibility. In the four-treatment Gaussian example, changing the response to

    h0(x)=(x1+x2)2h_0(x)=(x_1+x_2)^2

    makes (x3+x4)2\left(x_3+x_4\right)^2 an equally sparse moment solution despite candidate completeness, producing a 2:2 voting tie. This violates Assumption 3 rather than the theorem, but shows that benign causal structures can be excluded.

  2. The dimension (p≥2q+1,s<p−qp\ge2q+1, s<p-q) and row-deletion requirements need a causal interpretation.

    Deleting any loading row must leave two disjoint full-rank q×qq\times q blocks. This requires redundant representation of latent directions, while the sparsity bound requires at least q+1q + 1 structural null treatments. For one factor, row deletion requires at least three nonzero loadings, excluding confounding localized to only one or two treatments. These restrictions can be plausible, but their applicability to the 17 preselected genes remains insufficiently motivated.

  3. Assumption 5’s uniform relative operator-Lipschitz bound, Equation (11), can fail for elementary bounded ReLU networks. In the same Gaussian geometry, let d=(X1−X2)/2d=(X_1-X_2)/\sqrt2 and construct

    at(d)=min⁡1/2,max⁡(0,d−t),gt=a0−at.a_t(d)=\min\\{1/2,\max(0,d-t)\\},\qquad g_t=a_0-a_t.

    The function norm is of order t, while its conditional expectation’s change per unit small instrument rotation is of order t\sqrt t, forcing the relative constant to diverge. Appendix E verifies sampler accuracy, not this condition; the concern is the theorem’s applicability.

  4. Reusing fixed Monte Carlo draws (L1993) creates an indefinite empirical objective: with zero outcome and observed-function value, batch averages t and −t give loss −t² and zero function penalty. Clipping bounds this incentive but does not remove it. The implementation also uses CV-selected sparsity, shared nuisance/structural samples, and fixed penalties, unlike the theorem’s global minimizer with known sparsity, independent nuisance fitting, and vanishing regularization. The practical effects remain unquantified.

Questions for authors

Addressing one point on causal implications (1 or 2) and one technical issues (3 or 4) can make an acceptance.

  1. Present the essential assumptions and explain “active treatments” and conditional sampling earlier. Give a scientifically motivated causal example, and explain what supports the disturbance restriction, completeness, and exclusion of competing sparse explanations in the mouse application.

  2. Explain which causal structures provide the required loading redundancy and structural null treatments. For the mouse data, what justifies one latent factor, and how sensitive are the conclusions to factor dimension, weak loadings, and residual treatment dependence?

  3. Please give a concrete causal model and a sequence of sparse ReLU network classes whose approximation error tends to zero, and verify that the estimation assumptions, including Equation (11), hold simultaneously for that model and sequence. Or, explain what restrictions or weaker bound replace Equation (11) and how the rate changes.

  4. Assess sensitivity to draw count, independent draw realizations, resampling, regularization, and optimization. Compare the implemented procedure with the theorem’s split-sample, known-sparsity setup to isolate these effects.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3zd3 (Part 1/2)

Response to Reviewer 3zd3

Thank you for your insightful comments. We have added identification without plurality under stronger sparsity, replaced the uniform rotation bound, and expanded the factor-specification and implementation experiments. References below are to the revised manuscript.

Q1. Identification assumptions and the four-treatment tie

Motivated by your four-treatment example, we have added an identification result that does not require plurality under stronger sparsity. Under moment validity and candidate completeness, the causal relationship is uniquely identified among functions depending on at most kk treatments whenever 2k≤p−q2k\le p-q (Section 3.2; Appendix A.3, Proposition A.2).

Importantly, this stronger sparsity requirement can still be mild in applications with many treatments. Our mouse analysis has p=17p=17 and q=2q=2, so the condition allows any k≤7k\le7. The main analysis selects k=1k=1 by cross-validation, while the appendix also reports a fixed-k=2k=2 analysis (Section 5.2; Appendix G.7, Table 5 and Figure 5); both are well below this threshold.

As another example, Tang et al.’s separate mouse analysis used p=2,819p=2{,}819, estimated q=3q=3 and s=5s=5, also well below the corresponding half-sparsity threshold of 1,408.

Disturbance restriction.

We assume that shared latent factors account for the confounding between gene expression and body weight. For example, an unobserved physiological state may affect both multiple genes and weight. After accounting for these shared factors, the outcome disturbance is assumed to be mean-independent of the remaining gene-specific expression variation: E(u0∣ϵ)=0E(u_0\mid\epsilon)=0 (Section 2.2; Appendix G.11).

Completeness.

A sufficient condition is that XS∣ZX_S\mid Z belongs to a conditional exponential family with a one-to-one sufficient statistic and natural-parameter support containing an open set (https://doi.org/10.1111/1468-0262.00459, Theorem 2.2). For an rr-dimensional natural parameter, a full-row-rank derivative in the p−qp-q instrument coordinates requires r≤p−qr\le p-q.

Each mouse candidate has r=p−q=15r=p-q=15 treatments, meeting the dimension requirement. The additional full-rank condition ensures completeness in a Gaussian factor model (Appendix A.4).

For the broader class with support size below p−qp-q, identification uses plurality. In the Gaussian subclass, Corollary 1 shows that, for a fixed structural function under its stated conditions, exceptional loading matrices form a Lebesgue-null set (Section 3.2; Appendix A.4). Your symmetric example belongs to this set.

Near violations still matter: competing sparse functions can produce conditional moments too similar to distinguish reliably. This parallels faithfulness versus strong faithfulness (https://doi.org/10.1214/12-AOS1080). Our near-tie experiments show increasing estimation error as competing explanations approach the same moments (Appendix G.12, Table 13).

Q2. Factor specification and the mouse analysis

We have replaced the fixed factor dimension with data-driven selection and added sensitivity analyses across q=1,2,3q=1,2,3. The revised mouse analysis selects q=2q=2 from the expression data and k=1k=1 by cross-validation. Dscam’s full-sample contrast remains positive across the additional q=1,2,3q=1,2,3 sensitivity fits. We also make the biological and factor-model assumptions explicit (Section 5.2; Appendices G.7 and G.11).

A sufficient causal model is X=ΛU+ϵX=\Lambda U+\epsilon and Y=h0(X)+g(U)+ηY=h_0(X)+g(U)+\eta, with U independent of ε, centered g(U), and E(η∣U,ϵ)=0E(\eta\mid U,\epsilon)=0. It implies E(u0∣ϵ)=0E(u_0\mid\epsilon)=0. Shared physiological states can influence multiple genes and body weight, motivating recovery of shared variation from the treatments.

Different genes also participate in different biological processes, motivating structural nulls whose identities need not be known. A null gene can remain associated with weight through shared causes or another included gene. Here s<p−qs < p-q requires at least three such nulls.

Loading redundancy is essential: after deleting a treatment, two disjoint full-rank loading blocks must remain under the classical diagonal-noise factor model. For q=1q=1 this requires at least three nonzero loadings. A hidden cause affecting only one gene and the outcome cannot generally be recovered from other genes.

Our orthonormal projection avoids dividing by individual weak loadings, but weak factor directions can still destabilize loading-space estimation (Section 2.2, Assumption 1; Appendix G.11).

We choose qq as the maximum of Onatski’s estimate and permutation parallel analysis to reduce the risk of leaving shared confounding in the instruments. Both selected q=2q=2 in the full sample; their maximum was also two in all 50 training sets from ten repeated five-fold splits (Appendix G.11).

AuthorsOpenReview ↗

Response to Reviewer 3zd3 (Part 2/2)

Q2. Factor specification and the mouse analysis (continued)

At q=2q=2, both five-fold CV partitions selected k=1k=1, and the common full-sample refit retained Dscam with a +2.60 g contrast (Section 5.2, Figure 4; Appendix G.7, Table 5). In fixed-k=2k=2 sensitivity fits, Dscam’s full-sample contrast stayed positive across q=1,2,3q=1,2,3, although selected genes varied across training splits (Appendix G.11, Table 11 and Figure 7). Appendix G.7, Figure 5 provides the fixed-k=2k=2 two-gene curves.

Genes may remain correlated after the shared factors are removed, and the reason for this correlation matters. If it comes from shared laboratory noise unrelated to body weight, the IV moment may still hold, but factor estimation needs to allow correlated gene-specific noise. If it comes from an additional biological factor that affects both expression and weight, that factor remains a source of confounding.

A third possibility is that one gene regulates another. With gene regulation along X1→X2→YX_1\to X_2\to Y, our curve for X1X_1 holds X2X_2 fixed and excludes the mediated effect through X2X_2. Appendix G.11 discusses these mechanisms and factor sensitivity.

Q3. Replacing the uniform instrument-rotation bound

Your ReLU counterexample prompted a revised proof that no longer requires the earlier Equation (11). We replace it with approximate moment validity under estimated instruments and stability of a source representer g0g_0 satisfying h0=T0∗g0h_0=T_0^*g_0 (Appendix B.2, Assumptions 4 and 6). These conditions concern the disturbance and true function, rather than the entire growing neural class.

The proof first rules out incorrect supports, then controls estimation on the true support through Tikhonov regularization.

Under the revised assumptions, including βiv>1\beta_{\mathrm{iv}} > 1, the loading-estimation term changes from 1/(nNαn2)1/(n_N\alpha_n^2) to 1/(nNαn)1/(n_N\alpha_n), and the regularization-bias term from αnmin⁡βiv,2\alpha_n^{\min\\{\beta_{\mathrm{iv}},2\\}} to αn\alpha_n. Here αn\alpha_n is the regularization parameter and nNn_N the nuisance-sample size.

Section 4.4, Theorem 2 and Appendices B.2/C.2 give the full rate, assumptions and proof.

Q4. Monte Carlo, optimization and implementation choices

We added sensitivity experiments on Monte Carlo sampling, optimization, regularization, sample splitting and sparsity selection. Too few draws worsen estimation; regularization and sparsity choices matter; independent nuisance fitting has an accuracy cost at a fixed structural sample size.

Increasing the draw budget beyond our 48-draw baseline or changing how draws are refreshed produced no clear improvement in these experiments.

Monte Carlo sensitivity is concentrated at the small draw budget.

In 40 paired Gaussian Interaction replications at n=3,000n=3{,}000, increasing draws per batch from 12 to 48 reduced mean structural RelL2 from 0.680 to 0.556, a paired improvement of 0.123 [95% CI: 0.030, 0.220]. Comparisons with 192 draws and alternative draw seeds did not detect clear differences from the 48-draw baseline.

Per-epoch resampling, which mitigates the risk of overfitting to fixed draws you describe, also showed no clear difference in RelL2 (+0.036 [95% CI: −0.022, 0.110]), and the mean squared gap between the two batch averages, the negative discrepancy term underlying the indefinite objective, was similar to that under fixed draws (0.0058 vs. 0.0055) (Appendix G.9, Table 8).

Regularization and sparsity selection matter more than simply training longer.

Continuing optimization lowered training loss but increased structural RelL2 by 0.032; removing Tikhonov regularization increased it by 0.023 (Appendix G.9, Table 9). In the separate n=10,000n=10{,}000 scan, doubling the group penalty increased RelL2 from 0.283 to 0.342 and reduced exact input-mask recovery from 80% to 2% (Appendix G.10, Table 10).

Relative to the theorem’s known-sparsity setup, CV-selected sparsity slightly improved structural error, from 0.556 to 0.541, while reducing exact support recovery from 15% to 5%, often selecting k=1k=1. Thus accurate estimation of the causal relationship and recovery of every active treatment respond differently to tuning.

Sample splitting involves an accuracy tradeoff.

With the structural sample held at 1,500 observations and k=2k=2, fitting the nuisance components on an independent sample increased RelL2 from 0.544 to 0.605. The independent arm uses an additional 1,500 nuisance observations, whereas the shared arm reuses the structural observations. This comparison isolates independent nuisance fitting at the same structural sample size.

Together, these experiments support using an adequate Monte Carlo budget and tuning regularization and sparsity carefully; extra optimization alone does not ensure better causal estimation. Appendix G.9, Table 9 reports the sample-splitting and sparsity-selection results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.