acceptodds
Under review as a conference paper at ICLR 2027

SC-PFN: A Causal Foundation Model with Amortized Statistical Correction

Abstract

Prior-data fitted networks (PFNs) amortize causal effect estimation, but prediction loss averaged across tasks does not ensure unbiased estimation within each task. Neyman orthogonality motivates residual corrections that cancel first-order nuisance sensitivity in population estimating moments. We introduce SC-PFN (Statistical Correction PFN), a causal foundation model that jointly learns effect prediction and context-conditioned residual weighting. The model predicts observed variables to form residuals and assigns weights to context rows. Weighted residual averages give regularized coefficients, which query weights combine into an update of the base effect estimate. A single model is pretrained across back-door, front-door, and instrumental-variable settings under stated identifying assumptions and applied without dataset-specific parameter updates. We characterize the coefficients as constrained ridge solutions and bound the correction's response to residual perturbations with row and query weights held fixed. Experiments on synthetic and semi-synthetic causal benchmarks show lower mean effect-estimation error than evaluated baselines. Component comparisons and score-shuffling controls support the contribution of the correction and its observation-specific weights.

Primary area:
causal reasoning
Keywords:
Causal Inference, Treatment Effect Estimation, Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AuEBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a residual correction to causal pfn foundation model, inspired by orthogonal (debiased) estimation. Theoretical guarantees are not established. Proposed model is implemented and empirical comparison to existing pfn and traditional methods is done.

Critical strengths

The idea of debiasing applied to causal pfns is very timely and worthy of investigating.

Critical weaknesses

There are several major issues I see with this paper:

  1. Theoretical motivation: I do think the idea of debiasing causal pfns is very timely and interesting. However, traditionally, orthogonalization is against sensitivity to (first order) nuisance estimation error. With pfns being a full end-to-end one-step estimation procedure, no nuisances are explicily estimated and the theoretical motivation needs to be justified much more carefully. This is not recognized in the paper at all
  2. Theoretical guarantees: In fact, the proposed residual corrected pfn is not supported with any theoretical guarantees analogous to those used to motivate this work. We are left purely to empirical results to assess the proposal (see below for points on results).
  3. Design soundness: i) The two head approach for base + correction does not guarantee the individual learned heads correspond to this decomposition. The loss only ever trains their sum, placing no constraint that ensures the decomposition is true. This is never empirically tested either. ii) regularization terms are not theoretically grounded. iii) More generally, the learned residuals, row weights, and query weights are not constrained to correspond to nuisance functions, Riesz representers, influence functions, or any other statistical objects invoked in the motivation.
  4. Experiment design: Beyond raw predictive performance, the experiments do not validate any aspect of the correction interpretation. Within-task and repeated-context bias is mentioned but never evaluated. Bias variance decomposition is not evaluated. There is no direct test that the learned weights approximate the statistical objects motivating the method, no evaluation of the fully trained model with the correction switched off at inference, and no generic parameter-matched additive-correction baseline.
  5. Experiment rigor: For results to be meaningfully interpretable, the comparison needs to be fair (unconfounded): the comparison to CausalFM is not matched in causal-label exposure or parameter count, while training-data and compute budgets cannot be verified as matched (same should be at least attempted for CausalPFN and Do-PFN too). Especially the amount of training data is critical for fair comparison here.
  6. Writing quality: The exposition is poor. Related work is very sparse and missing bib keys (L71 three times). Text is difficult to follow. Notation is not introduced consistently spread out through the entirety of text and could certainly be simplified. Carried thread of though throughout paragraph between sentences is almost non-existent.

In sum, I find substantial issues with this submission in almost every aspect. The most valuable part is the idea itself, which in my eyes has not been done justice with the exploration of it that is submitted here. I would encourage authors to rewrite the entirety of the submission, run new and improved experiments, and actually develop theory exploring this idea and supporting their empirical results.

A basic llm inspection of the code accompanying the submission found the following concerns:

  • Query-treatment leakage: Corrected predictions use unmasked query treatments.
  • Residual target leakage: Residual heads see the values they are asked to predict.
  • Unauditable prior: Training data and task generation are missing, so prior size and mixture cannot be verified.
Questions for authors

Please see above weakness. How does the data and compute budget of SC-PFN training compare to that of CausalFM?

Reviewer_gMttOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SC-PFN, a single pretrained model for amortized causal effect estimation across BD, FD, and IV settings. The model corrects a base effect prediction by combining predicted row and query weights with observed residuals, motivated by AIPW estimation. Experiments on five benchmarks demonstrate improved PEHE over the evaluated baselines.

Critical strengths

Amortized statistical correction. The main novelty of the paper is that the row-level correction weights are not derived from fitted nuisance functions (e.g., inverse propensity weights), but instead are predicted by the network and learned end to end. This avoids deriving setting-specific weights and allows a single model to apply corrections across three different settings. However, this also means the weights may not guarantee a statistically valid correction (see Weaknesses).

Consistent empirical gains. SC-PFN attains the lowest mean PEHE across all five benchmarks. However, three of the margins are modest (Jobs, IV binary, and FD). The shuffled scores row in Table 3 demonstrates that the model relies on the learned row weights, while the ablations support the proposed modeling choices. Additional experiments under nonlocal-context contamination and spurious association demonstrate improvements over CausalFM.

Critical weaknesses

Unclear statistical basis for the correction. Much of the motivation rests on the similarity between the weight-residual product in AIPW estimation and the model estimate. However, AIPW uses Riesz-based weights so that the orthogonality condition in Equation 4 holds. In the proposed model, the weights H^θ,r\hat{H}_{\theta, r} are network outputs, and there is no guarantee that they approximate the Riesz representative. It is also unclear why the residuals from B^\hat{B} would even be informative of the effect head error, since it is not a plug-in built from B^\hat{B} as in standard AIPW. Could the authors further motivate the choice of the correction structure?

To me, it seems that the empirical gains from the correction are mainly due to added inductive bias that helps the model learn, rather than to a statistical correction in the double machine learning sense. This calls into question the framing of the work.

Limited novelty. The authors note that the setting GG is not explicitly supplied as input to the model. However, each setting has its own masking pattern, so the setting is effectively given by which variables are present. Training a single PFN across causal structures has been explored (Do-PFN), and the priors, benchmark, backbone architecture, and effect head follow from CausalFM. The main new component is therefore the correction module, whose statistical motivation is unclear.

Untested Bayesian framing. The model predicts a full density over the effect (Equation 7). However, every evaluation only uses its mean as a point estimate. It is unclear why the model is trained to predict a density at all, rather than the mean directly. If it is for uncertainty quantification, it should be tested with calibration or coverage.

Polish. I found the paper to generally lack polish in the presentation of ideas, particularly in Section 5. The components of the correction are defined mathematically, but there is little discussion of what they are meant to represent or why the correction is designed this way. The joint training objectives are not clearly motivated, and the model architecture is not fully described. The loss weights λRS\lambda_{RS} and λmom\lambda_{mom} are not specified, nor are key training details. Line 71 contains several missing citations, and Figure 1 appears to contain many artifacts of generative AI.

Questions for authors
  1. Do the learned weights H^\hat{H} resemble the weights used in standard AIPW? The only diagnostic is in-sample and BD-only.
  2. Could the authors compare SC-PFN against applying the same correction with AIPW weights?
  3. SC-PFN and the controls are each trained once. Could the authors report results over multiple training seeds to assess training sensitivity?
  4. Apart from Jobs, all evaluation datasets are generated from the same prior as pretraining. How does SC-PFN perform on additional differently generated data (e.g., ACIC 2016, as reported by CausalFM), or under prior misspecification?
Reviewer_v3hbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SC-PFN adds learned residual correction to a causal prior-data fitted network. A shared transformer predicts an effect distribution, observed-variable predictions, residual weights, and query weights. Clipped, regularized coefficients aggregate residuals and shift the effect distribution. One model covers back-door, front-door, and instrumental-variable tasks without deployment-time parameter updates. The paper characterizes the coefficients as constrained ridge solutions and bounds residual sensitivity at fixed weights. It reports the lowest mean PEHE across five benchmark settings, supported by matched ablations, score shuffling, and context stress tests. Implementation discrepancies, however, complicate interpretation.

Critical strengths
  1. Explicit residual aggregation provides an inexpensive, closed-form correction while retaining fixed pretrained parameters across identifying regimes. The distinctive contribution over CausalFM is the jointly learned correction, potentially improving estimation without separate dataset-specific fitting.

  2. Table 3 matches the backbone, prior, data, optimization, and instantiated parameter budget. Relative to predictive-only training, reported PEHE improves from 0.528 to 0.440 on BD-Synthetic and from 0.592 to 0.456 on IV-Continuous. Auxiliary-only, no-update, and no-moment variants help isolate contributions; inference-time shuffling additionally tests row-score correspondence. These are useful controls, conditional on resolving the implementation concerns.

Critical weaknesses
  1. The released implementation conflicts with the stated interface and objective.

The manuscript specifies covariates-only queries, but model/sc_pfn.py reconstructs treatment from unmasked inputs (515–516), enabling treatment-dependent corrections (418–424, 466). test.py supplies full treatments, and the loader leaves query treatments unmasked. Predictions can therefore use excluded information.

This code-path finding does not establish an accuracy effect. The experimental implementation and impact on reported PEHE remain unknown; authors should identify the version used.

Equations (17)–(18) average squared task moments; utils/losses.py instead squares batch-averaged moments (27–36, 161–185), allowing cancellation. The denominator floor on sum(H^2) (461) also differs from Equation (9)’s additive ridge, potentially with small numerical effects.

  1. Evidence for statistical correction and reliable gains is limited.

Proposition 1 fixes constructed weights and establishes neither full-estimator robustness, Neyman orthogonality, nor reduced repeated-context bias. Appendix B.4 acknowledges this, yet Figure 1 calls the estimate debiased. Equation (6)’s bias is not directly measured; lower PEHE could reflect variance reduction.

Appendix D.7 calibrates and evaluates on identical perturbations using raw scores rather than the final normalized, query-weighted correction, leaving its statistical validity unestablished.

Evaluation uses ten synthetic datasets per setting, one training seed, and permutations of one Jobs dataset. Gains over CausalFM are small: FD 0.847→0.832; Jobs 0.478→0.461.

Questions for authors
  1. Which code version and checkpoint produced Tables 2-3? Can predictions be shown invariant to withheld query treatments? If the released path was used, please provide corrected evaluations.

  2. Was the moment penalty computed per task or after batch averaging? Was the denominator an additive ridge or a floor? Please reconcile the equations and experimental implementation.

  3. Do corrected gains persist across training seeds and paired datasets? Can fixed-SCM, repeated-context experiments separate bias from variance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.