acceptodds
Under review as a conference paper at ICLR 2027

ProximalFM: Amortized Proximal Causal Inference under Hidden Confounding

34 views

Abstract

Standard causal identification methods often assume no unmeasured confounding and can fail when relevant confounders are unobserved. Proximal causal inference instead uses proxy variables to identify effects under hidden confounding and additional assumptions. However, nonparametric proximal estimation can be challenging in practice: recovering causal estimands such as the conditional average treatment effect (CATE) requires solving an ill-posed integral equation that is data-hungry, hyperparameter-sensitive, and optimization-unstable. Bayesian inference for such models provides a desirable alternative, mitigating these difficulties by regularizing through the prior. However, computing a posterior is itself challenging, as a typical likelihood function will include latent variables. Following the recent success of tabular foundation models in backdoor, instrumental variable, and frontdoor settings, we propose that prior-data fitted networks (PFNs) are uniquely suited to resolve this bottleneck. Indeed, by training on synthetic data sampled from compliant structural causal models with access to oracle counterfactuals, we simplify the task substantially, amortizing the implied Bayesian operator inversion into a single transformer forward pass. Compared to prior literature that focuses primarily on point estimation, our model, ProximalFM, explicitly targets the Bayesian posterior distribution of the CATE. One unique aspect of this problem is that we need to provide Monte Carlo estimates of the oracle CATEs, leading to a novel variation of PFNs that accounts for the added stochastic error. ProximalFM is competitive with existing estimators across a diverse set of proximal regimes, with its clearest advantage under substantial latent confounding and informative proxies. It attains this performance without dataset-specific tuning, in a single amortized forward pass.

Primary area:
foundation or frontier models, including LLMs
Keywords:
causal inference, tabular foundation models, proxy learning, amortized inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_MRXSOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces PROXIMALFM, a prior-data fitted network for proximal causal inference under hidden confounding, trained on synthetic structural causal models with proxy variables to directly predict a posterior over the CATE in a single forward pass. The key idea is to adapt the PFN prior to the proxy setting-specific SCM. Empirically, PROXIMALFM is competitive with existing proximal estimators while avoiding dataset-specific retraining and hyperparameter tuning.

Critical strengths

Relevant and timely problem. The paper addresses causal effect estimation under hidden confounding, a practically important setting where standard unconfoundedness-based methods can fail, and connects this with the rapidly growing literature on causal foundation models.

Practical motivation. Proximal methods can be computationally demanding and sensitive to tuning, whereas PROXIMALFM provides amortized inference in a single forward pass without dataset-specific optimization.

Experimental evaluation. The paper compares against several proximal and non-proximal baselines across diverse semi-synthetic settings and additionally considers posterior calibration and a physical-system benchmark.

Critical weaknesses
  • Limited novelty. The main idea is essentially to apply existing ideas combining causal inference and PFNs (e.g., CausalPFN, do-PFN, CausalFM) to a different identification setting. While this is in principle interesting, the authors seem to omit key challenges in the proxy setting (see below) and the contribution seems to be a rather straightforward adaption of a different causal graph. Previous work such as CausalFM explicitly mentions that their approach could be straightforwardly adapted to different causal graphs/ identification settings. The contribution here therefore seems to be more in implementing the actual model for the proxy setting than in introducing a new conceptual idea.

  • Identification issues. While PROXIMALFM does encode the proximal graph, exclusion restrictions, and positivity, it does not guarantee the key proxy-specific completeness condition, relying instead on heuristics. Bridge-function existence is also assumed in the theory but not clearly guaranteed by the prior. This creates a gap between the theoretical identification result and the actual training distribution. Typically this either yields large posterior uncertainty for a flexible prior, or informative intervals come at the cost of a misspecified prior. Neither is quantified as far as I can tell.

  • On a positive note, these two points may have a common remedy. If the authors could construct a reliable prior that is guaranteed to satisfy the proxy-spcific identifiability conditions, this would both address the identification concern and strengthen the novelty. Alternatively, partial identification approaches could potentially also be explored as in e.g., https://arxiv.org/abs/2605.12924. This could lead to contributions genuinely specific to the proximal setting, rather than simply encoding a different causal graph within the existing frameworks. I encourage the authors to explore this direction.

Questions for authors
  • What is different compared to just adapting the existing causal PFNs to the proxy graph? (which could also be considered potential baselines for the experiments)
  • What are we loosing by ignoring part of the identifiability assumptions in the prior construction? Are there partial identification results?
3 comments

We thank the reviewer MRXS for their thoughtful comments on the novelty of ProximalFM and for raising the questions of completeness, bridge-function existence, and partial identification.

Novelty and Relevance.

We agree that existing work, such as CausalPFN or CausalFM, has already applied PFNs to causal effect estimation under other causal assumptions. Naturally, ProximalFM builds on existing architecture (TabICLv2) by adjusting its inputs and prediction head for the proximal causal inference problem. Our main contributions lie in proposing and studying solutions to the challenges that specifically relate to the proximal causal inference setting.

For example, we introduce a simple training strategy to estimate a posterior over a conditional average rather than a distribution of individual-level outcomes. We use it to target the CATE rather than individual treatment effects (ITEs), but the strategy is broadly applicable to other estimands, such as the conditional mean E[Y∣X=x]\mathbb{E}[Y\mid X=x].

Furthermore, PFN priors typically need to generate noisy individual data, as they need to imitate observational context on which the PFN will later be applied to. In causal inference, this is individual potential outcomes. However, if one trains the PFN directly on these values (e.g. on ITE = difference of potential outcomes), the PFN will learn the posterior distribution of the individual effect. This individual effect has the right mean, but its uncertainty is wider than the one of the quantity of interest (ITE = CATE + noise). That is why we introduce a novel CATE approximation using a Monte Carlo average of ITEs. This approximation is then used as target in a newly introduced noise-aware loss, with which we account for Monte Carlo approximation noise.

The new ablation in Appendix F.2 shows that the noise-aware loss behaves as expected. When the Monte Carlo count is high, the need to account for noise is negligible. However, for small Monte Carlo repetitions, one needs to account for the approximation noise otherwise the learned posterior distribution is too wide.

Finally, we would like to emphasize the relevance of the problem: Nonparametric proximal methods typically require solving an ill-posed bridge estimation problem, which may require an appropriate regularization against weak proxy informativeness. This problem is notably difficult, especially for small sample sizes. Our PFN approach does not require any dataset-specific training or tuning. And the pretraining of ProximalFM is also simplified as it leverages the oracle quantities provided by the simulator, therefore not requiring to explicitly solve for the bridge function at any time. Finally, the SCM prior also provides an implicit regularization, avoiding the need to select a regularization parameter for each dataset.

Completeness and Partial Identification.

We agree that potential misspecifications in the prior may lead to loss of performance (larger posterior uncertainty and informative, but misspecified intervals). We choose a prior specification that trades off feasibility, compute and performance. ProximalFM is very competitive across different settings, in particular for the relevant case of high confounding and low proxy informativeness (Figure 4).

More importantly, we are excited to see reviewer MRXS recognise the importance of partially identifying the CATE in the proximal learning setting, e.g. when the full identification criteria are not guaranteed to be fulfilled. Naturally, we have started parallel work that addresses this exact question of partial identification, which will form part of a separate publication.

Indeed, it will also take inspiration from the work done in Balazadeh et al. (2026), and combine this with initial closed-form solutions into bounding of proximal causal effects by Ghassami et al. (2023).

Crucially, we found that identifying the challenges of misspecification that appear during the development of ProximalFM to be indispensable first step towards developing bounding solutions of those causal effects. We are looking forward to presenting and discussing partial identification results in our upcoming submission, for which ProximalFM serves as a foundation.


Balazadeh, V., Kamkari, H., Barath, M., Silva, R., & Krishnan, R. G. (2026). IV-ICL: Bounding Causal Effects with Instrumental Variables via In-Context Learning. arXiv preprint arXiv:2605.12924.

Ghassami, A., Zhang, Y., Shpitser, I., & Tchetgen Tchetgen, E. (2023). Partial identification of causal effects using proxy variables. arXiv preprint arXiv:2304.04374.

Thank you for your response. I am still trying to understand the methodological novelty of ProximalFM and would appreciate further clarification on which challenges specific to proximal causal inference the paper addresses beyond what existing causal PFNs already propose. Candidates:

  • Different causal graph: Adapting the SCM prior to the proximal graph seems to be a relatively straightforward extension of CausalFM, which explicitly proposes this approach for different identification settings.
  • Targeting causal estimands: This also seems to offer limited novelty, as CausalPFN already learns posterior distributions over conditional expected potential outcomes.
  • Partial identification: This would be a genuinely interesting proximal-specific contribution. However, the review concerns the current submission rather than planned future work.
  • Monte Carlo noise correction: This seems to leave the noise-aware loss as the main methodological contribution. Could the authors elaborate on its novelty beyond standard measurement-error correction?

Overall, could the authors clarify which methodological challenges specific to the proximal setting they solve that go beyond straightforward adaptations of existing causal-PFN approaches?

We thank Reviewer MRXS for their prompt follow-up.

We agree that earlier causal PFNs already provide a framework for adapting priors to new identification assumptions. In proximal inference, specifying the DAG still leaves open how to construct a prior that spans a range of proxy informativeness and confounding strengths, and how to leverage the proxy variables in the observational context while targeting a CATE conditional on X alone.

Differences with CausalFM and CausalPFN.

CausalFM provides a framework to build PFN under different causal identification settings. However, its posterior is over individual treatment effect, whose uncertainty is generally not identifiable, leading to deceptive uncertainty interpretability. CausalPFN proposes a PFN for backdoor inference, targeting the conditional expected potential outcomes (CEPOs). We are interested in the CATE whose uncertainty cannot be retrieved from the CEPOs posterior alone. One could train directly on their difference when both CEPOs are available for each simulated SCM (as they are under CausalPFN’s backdoor prior). In our proximal simulator, the same latent confounder generates the proxies, treatment assignment, and outcomes. We obtain CATE labels by averaging paired counterfactual differences at fixed X=xX=x. This gives us an approximate CATE target with an estimated Monte Carlo error, which we account for in our noise-aware loss.

Synthetic prior challenges.

We agree that it is straightforward to adapt an existing prior to be compliant with the proximal DAG. But the proximal DAG only specifies which dependencies are allowed, while the prior determines how much those dependencies vary across simulated datasets. As widely acknowledged in the literature, the prior must provide diverse SCMs. For example, providing every features W into the outcome mechanism can give wider W blocks more influence on Y because they occupy more of the input. This would undersample compliant SCMs with wide W but having little influence on Y. We subsample features along (W to Y) and (Z to A) edges to correct this. We also vary the share of U in the inputs to the treatment and outcome mechanisms, allowing a low-dimensional U to induce substantial confounding (Appendix B.2.3). These choices allow proxy influence and confounding strength to vary more independently of block width, improving prior diversity. These are essential and novel prior adjustments we developed throughout this paper and find worthwhile sharing in the literature.

Architectural challenges.

One difference with current causal PFNs is that proximal inference has more distinct variable roles. The context and the query do not consist of the same variables: we observe (X,W,Z) but want to predict a CATE quantity depending on (X) alone. This requires specific role embedding for X,W,Z, as well as an appropriate masking at inference, a crucial extension for the proximal setting.

Learning conditional expectations from noisy Monte Carlo targets.

We provide a simple framework: generate replicates of individual draws, estimate the conditional mean by Monte Carlo average, estimate the approximation noise and account for it in the loss. That simple principle allows to train a PFN on conditional expectations when its simulator generates individual draws. For example, a tabular prior that can generate repeated outcomes at fixed X=x (e.g. slightly adjusted TabICL or TabPFN priors) can supervise a posterior over E[Y | X=x]. For ProximalFM, the procedure is applied to the CATE by averaging paired counterfactual differences. In Appendix F.2, we observe that this correction reduces the overcoverage caused by training on a small number of draws, an extension that we find necessary to be discussed in the causal PFN literature.

Limits of proxy adjustment with pretrained causal PFNs.

It is a natural starting point to apply an existing pretrained causal PFN to proximal inference. We evaluated CausalPFN by supplying (X,W,Z) in the observational context, and then averaging its predicted contrasts over nearest neighbours of pairs (W,Z) from the point X=x. This yields a CATE estimate depending on x alone. CausalPFN with proxy marginalization is competitive in some settings. However, in the nonlinear setting with strong latent confounding and low proxy reliability, increasing the context size improves its estimates of effect heterogeneity while leaving substantial error in the average effect (Figure D.2). ProximalFM has lower normalized squared ATE error, and additional training on longer contexts reduces this error further on these large context sizes. We adapted earlier causal PFN frameworks to train a model on a proximal prior. ProximalFM as far as we know is the first model that gives practitioners a pretrained model for this setting, so they can estimate a CATE posterior from data with treatment and outcome proxies without designing their own simulator, pretraining a model, or tuning it for each dataset.

Reviewer_r3moOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ProximalFM, a prior-data fitted network that amortizes CATE estimation under the proximal causal framework. The model is trained on synthetic proximal SCMs using Monte Carlo CATE targets and a noise-aware distributional objective, enabling it to produce a posterior over CATE in a single forward pass without solving a separate bridge-function inverse problem for each dataset. The method is evaluated on semi-synthetic benchmarks, a controlled proxy-informativeness study, and a physical causal system.

Critical strengths

The paper addresses an important and technically challenging problem by combining proximal causal identification with amortized in-context inference. Its methodology is carefully developed, including causal-role-aware encoding, direct CATE posterior prediction, and explicit treatment of Monte Carlo target noise. It also provides comprehensive evaluations, including experiments that compare against a broad range of proximal and causal-learning baselines.

Critical weaknesses
  1. For proximal identification conditions, enforcing the completeness condition is notably difficult. The dimensional requirements and proxy-relevance diagnostics used in the prior are informative heuristics, but they are not sufficient in general and are not shown to be equivalent to completeness under the proposed generator. It would be more transparent to add controlled sensitivity check to examine the impact of the weakening or the failure of completeness condition.

  2. Let Q^θ,N(x)\widehat Q_{\theta,N}(x) be the trained PFN estimate, QN(x)Q_N(x) the oracle posterior estimand under the synthetic prior, and Q⋆(x)Q^\star(x) the true CATE. Then the error can be decomposed as

    \begin{aligned} \left|\widehat Q_{\theta,N}(x)-Q^\star(x)\right| \le \left|\widehat Q_{\theta,N}(x)-Q_N(x)\right| + \left|Q_N(x)-Q^\star(x)\right|. \end{aligned}

    the second term is justified in Appendix A. It seems that the control of the first term is not established in the paper, and I believe that this term may not necessarily vanish as N→∞N \to \infty. The algorithm seems to have a partial control on this term by introducing the mean square term in the loss, but some error analysis would add more transparency and facilitate better understanding of what drives the estimation error.

  3. Most episodes follow the same block-level proximal template. Although a complex SCM may sometimes be summarized by the same block structure (X,U,Z,W,A,Y)(X,U,Z,W,A,Y), this does not imply that the learned PFN is invariant to the underlying coordinate-level graph. The prior could be enriched with constrained micro-DAGs that preserve the proximal restrictions while varying (i) between-block structure, such as distinct latent variables serving as shared confounders, treatment-specific causes, outcome-specific causes, or effect modifiers, and (ii) within-block structure, such as sparse causal graphs among baseline covariates or latent variables. Evaluation on held-out micro-DAG families would provide stronger evidence of structural generalization.

Questions for authors
  1. Pretraining uses (dX,dW,dZ)≤10(d_X,d_W,d_Z) \leq10, and the main benchmark uses only five columns per block, even when source datasets contain up to 124 features. How does accuracy and runtime behave for substantially larger dX,dW,dZd_X, d_W, d_Z, and does the four-CLS-token compression become a bottleneck?
  2. Since credible intervals become increasingly overconfident on semi-synthetic data as context size grows, what interpretation should practitioners attach to the reported posterior? How can this overconfidence be detected or mitigated?
  3. Since nPEHE is the primary benchmark metric but conflates treatment-effect level error and heterogeneity error, could you report shape R2R^2 alongside nPEHE in all main benchmark tables? This would clarify whether performance differences reflect improved average-effect estimation, improved recovery of CATE heterogeneity, or both.
2 comments

We thank the reviewer for their careful reading and thoughtful suggestions. We have updated the manuscript with the following results, figures, tables, and theoretical analysis in Appendix F. We will include some references to these in the main body of the text.

Increasing benchmark dimension.

Appendix F.3 provides new results in higher dimensions. First, we expand the same benchmark as Figure 4, for high confounding and low proxy informativeness, on a sweep of dimensions. We observe that ProximalFM's performance is almost dimension-agnostic, unlike, e.g. P-learner and KPV, which perform best for low dimension settings.

Then, we select a subset of high-dimensional datasets and test ProximalFM on them. While we observe that performance degrades with higher dimensions, we do not observe a sudden fall for dimensions d>10d>10, which were never observed in the pretraining.

We don't expect the 4 CLS-token to be a bottleneck, as this is the same configuration as the TabICLv2 architecture, which has proven to perform well in general tasks (with more dimensions that considered here).

Posterior calibration.

We agree that the undercoverage of the credible intervals is a practical concern. Appendix F.4 proposes a possible explanation for the cause of this undercoverage. As the context grows, the posterior uncertainty may shrink faster than the bias in the posterior mean. In that case, the coverage can fall even as point estimates improve.

Recalibration is a difficult task as we never observe the outcomes of interest, i.e. the treatment effect, but only the potential outcomes. Because of this, typical methods of recalibration or conformal prediction typically fails. For instance, CausalPFN introduces a recalibration based on observed outcomes which applies temperature scaling to each treatment arm predictor separately. ProximalFM predicts the CATE directly, not arm-specific quantities, and thus cannot use a similar approach. Direct coverage assessment is possible on synthetic or semi-synthetic tasks with known CATEs. An external randomized study can also provide a reference for prespecified subgroup average effects, subject to sampling uncertainty and comparability of the populations.

Treatment-effect heterogeneity.

Appendix F.5 reports paired nPEHE and shape R2R^2. For query errors ei=Q^i−Qi⋆e_i=\widehat Q_i-Q_i^\star, we use

=1-\frac{\operatorname{Var}_i(e_i)} {\operatorname{Var}_i(Q_i^\star)}.$$ A score of $1$ means perfect recovery up to a constant offset while a constant predictor scores $0$. Shape $R^2$ improves at every context step in all eight settings. From $N=128$ to $2048$, its overall mean rises from $0.339$ to $0.775$, while nPEHE falls from 0.902 to 0.545. Improved heterogeneity recovery accounts for 84.1% of the reduction in mean squared nPEHE, with effect-level accuracy accounting for 15.9%. We also added a new visualisation of the error decomposition (ATE + R2-shape), and we thank the reviewer for this suggestion, as we find Figure F.6 informative. For instance, it clearly displays that ridge and TabICL P-learners estimate the average treatment effect about equally right, but TabICL recovers more of the heterogeneity. **Completeness.** We agree that the current heuristics do not fully guarantee completeness, and that this is a difficult task to enforce it in practice. The reviewer can refer to our responses to reviewers PRdM and MRXS for a deeper discussion on this topic. **Learned--oracle error.** Under the synthetic prior, the oracle posterior mean minimises the population squared-error penalty (Appendix F.6). The difference between the trained PFN's population penalty and the oracle's equals their mean squared distance in standardised outcome units. We agree that this gap may persist as the context size $N$ grows. Its size depends on the PFN's capacity and how well it learns from the synthetic training tasks. A systematic study of how the gap varies with $N$ is beyond the scope of this rebuttal. **Micro-DAG suggestion.** Thank you for suggesting constrained micro-DAGs. This is a promising extension of the prior’s structural generalization, but currently remains out of scope of our current work due to the time and compute needed for retraining. We will mention this as a possible direction for future work in the manuscript.

Thank you for your thorough response. Your clarifications and the revisions to the manuscript, particularly the additional experiments and theoretical results, have sufficiently addressed my previous concerns. I appreciate the considerable effort you have devoted to addressing these issues. Therefore, I will maintain my score.

Reviewer_PRdMOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ProximalFM, a tabular foundation model for causal effect estimation under hidden confounding. The method uses proxy variables to identify conditional average treatment effects (CATE). It is trained on synthetic data generated from structural causal models with known counterfactual outcomes. A noise-aware training objective is introduced to account for Monte Carlo estimation errors. ProximalFM predicts the posterior distribution of CATE in a single forward pass without task-specific tuning. Experiments on synthetic, semi-synthetic, and physical datasets demonstrate competitive performance against existing causal inference methods, particularly under strong hidden confounding.

Critical strengths
  1. The paper extends prior-data fitted networks (PFNs) to proximal causal inference with hidden confounding. Unlike existing causal foundation models that mainly rely on backdoor, instrumental variable, or frontdoor identification, ProximalFM explicitly incorporates treatment-inducing and outcome-inducing proxies. This allows the model to estimate CATE without assuming that all confounders are observed.
  2. The authors construct a synthetic prior over structural causal models (SCMs) tailored to proximal identification. The data generation process enforces latent exchangeability, proxy exclusion restrictions, and positivity through structural constraints. It also introduces dimensionality and categorical heuristics to encourage completeness. This provides a practical way to generate diverse training tasks under proximal causal assumptions.
  3. The paper introduces Monte Carlo CATE targets and a noise-aware training objective. Instead of using individual treatment effects as supervision, the authors estimate CATE by averaging multiple counterfactual realizations for each query. The proposed objective explicitly accounts for the sampling variance of these estimates through Gaussian convolution. This helps prevent Monte Carlo noise from inflating the predicted posterior uncertainty.
  4. The experimental evaluation covers different confounding strengths, proxy informativeness levels, and linear and nonlinear causal mechanisms. ProximalFM achieves competitive performance across these settings, with particularly strong results under substantial hidden confounding. The paper also reports inference efficiency, posterior calibration, and performance on a physical causal system. These experiments provide a detailed assessment of both the advantages and limitations of the proposed method.
Critical weaknesses
  1. The synthetic prior relies on dimensionality and categorical heuristics to approximate the completeness assumption, but these heuristics do not guarantee identification under general nonlinear mechanisms. More importantly, the empirical analysis in Appendix B.3.4 shows that the weakest latent confounder dimension has a median incremental R2R^2 of only approximately 0.05, with nearly 60% of datasets falling below 0.1. Although weak proxy recoverability does not necessarily imply a violation of completeness, these results raise concerns about the effectiveness of the proposed heuristics. The paper lacks experiments that systematically examine how completeness violations or near-violations affect the learned estimator.
  2. The proposed noise-aware objective is not sufficiently validated through controlled experiments. Section 3.2 introduces Gaussian convolution to account for Monte Carlo noise in CATE supervision, but the experiments do not directly compare this objective with a standard noise-unaware distributional loss. The paper also does not investigate how the number of Monte Carlo replicates KK affects estimation accuracy and posterior calibration. Since the noise-aware objective is a central methodological contribution, these comparisons are necessary to establish its practical benefits beyond the theoretical motivation.
  3. The paper introduces several modifications to the TabICLv2 architecture, including structural role embeddings, a treatment-aware target encoder, and a direct CATE posterior prediction head. However, no ablation experiments isolate the contribution of these components. For example, it remains unclear whether explicitly encoding the roles of XX, ZZ, and WW provides substantial improvements over simpler feature-group representations. Similarly, the benefit of treatment-aware outcome encoding is not evaluated against conventional treatment and outcome embeddings. Without these comparisons, it is difficult to determine which architectural choices are essential to the reported performance.
  4. The experimental results show that ProximalFM does not consistently outperform existing methods, especially as the context size increases. In Table D.11, under nonlinear mechanisms with strong confounding and weak proxies, ProximalFM achieves an nPEHE of 0.964 at a context size of 8,192, while P-learner (TabICL) achieves 0.660. Extending ProximalFM's training context range to 5,000 observations only improves its nPEHE to 0.930. These results suggest that the performance gap cannot be fully explained by the limited context sizes used during pretraining. Further analysis is needed to understand whether the model struggles to capture treatment-effect heterogeneity or to exploit additional observations.
Questions for authors

See weakness

2 comments

We thank the reviewer for their thorough review and positive feedback on our work. Additional experiments were added in Appendix F to complement the discussion.

Completeness.

We agree that completeness remains unverified under our prior. Our heuristics give the proxies the capacity to carry information about U, but how much information they actually carry depends on the SCM sampled. Figure D.4 varies the joint population R2R^2 for predicting U from (W,Z) from 0.01 to 0.99, holding treatment and outcome mechanisms fixed. For low proxy strengths, the performance of ProximalFM is close to that of regression on X alone. Its error generally falls as proxy information increases. It outperforms proxy-adjusted OLS at most levels. OLS overtakes it when the proxies nearly reveal U and proximal inference is not needed any longer.

In the prior audit (Figure B.17), the least recoverable coordinate of U has a median incremental R2R^2 of about 0.05. The same audit shows higher average proxy relevance at larger dimension slack dP−dUd_P-d_U for P∈{W,Z}P\in\{W,Z\} (panel b), supporting our dimensional heuristic. Requiring at least dU+kd_U+k dimensions (k>0) per proxy block would strengthen the heuristic at the cost of excluding smaller blocks.

In an additional experiment (Appendix F.1), we discretize U, W, and Z and compare the smallest singular values of P(U=u∣Z=z,A=a)P(U=u\mid Z=z,A=a) and P(W=w∣U=u,A=a)P(W=w\mid U=u,A=a) against a pure-noise control (Figure F.1). The comparison provides evidence of proxy signal for partitions with up to k=16k=16 joint bins. At k=32k=32, some one-dimensional SCMs still show larger values than those typically seen under the control, while the curves for d=3d=3 and d=5d=5 largely overlap it. Finer partitions leave fewer observations per cell, so the results leave open whether the weakest population directions are weakly represented or absent.

Noise-aware loss.

We added an ablation using a simpler proximal SCM prior. We compare K=250 and K=16 draws per training target with and without variance correction, plus an uncorrected K=1 variant. The loss-ablation figure in Appendix F.2 shows the calibration curves and the mean width of the CATE predictive distributions of each variant, when evaluated on datasets provided from the same prior as the training.

We observe the expected outcomes. With K=250, the coverage curves are similar with and without correction. With K=16, correction narrows intervals at all three context sizes. At N=256, it reduces the mean 90% interval and corrects its coverage from 96.2% to 90.7%. At the same context size, the uncorrected K=1 intervals cover 99.9% of reference CATEs and are about three times as wide as the default’s. These results show how a small Monte Carlo budget can inflate CATE intervals when training ignores target noise.

Architectural ablations.

Appendix F.2 compares five variants trained for 15000 updates on a simplified proximal prior. The default tags the X, W, and Z blocks and uses a treatment-specific outcome projection at both the column and in-context learning (ICL) stages. We remove the role tags, replace the treatment-specific projections with the sum of a shared outcome projection and a treatment embedding, or supply the treatment and outcome at only one of the two stages.

Each ablation either shows a negative or a negligible impact on performance (Table F.1). We only considered using a single CATE output head. We considered using two heads that predict separate marginal posteriors for the conditional means of the potential outcomes but one would also need to model their posterior dependence to recover the CATE posterior. Convolving the marginals to obtain the distribution of the difference would assume posterior independence, a strong assumption because both means depend on the same unknown SCM. The direct head learns the posterior of the difference itself instead (Appendix C.1).

Large context sizes.

We agree that the gap to the P-learner (TabICL) is substantial at N=8192 in Figure D.2. The original ProximalFM checkpoint was trained on contexts of at most 2048 observations. Therefore, we expected its performance to stall beyond that range.

The post-training experiment was a compute-limited pilot to test whether adapting to longer contexts could improve performance (Appendix C.3.1). At N=8192, mean nPEHE improves from 0.964 to 0.930, compared with 0.660 for the P-learner (Table D.11). The gain suggests some room for improvement with longer-context training, while the extent to which it can close the gap remains unknown. Training the full model on contexts up to 81928192 exceeds the time and compute available for this rebuttal.

The error decomposition in Figure D.2, panel (b), shows that ProximalFM continues to recover treatment-effect heterogeneity as context grows beyond 2048, while its error in the average treatment error changes only little. The small gain from post-training comes from a better estimate of the average effect.

My concerns have been well solved. I will raise my score to 4. Good luck!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.