- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes fluctuation-supervised pretraining (FSP): instead of labeling synthetic tables with the true effect θ(P), it uses the oracle influence-function target θ(P) + P_n ψ_P. The goal is to teach a frozen model efficient repeated-sampling behavior. Theory characterizes label learnability, connects pretraining error to inferential guarantees, and establishes lower bounds. Experiments include a four-stratum simulator, low-dimensional continuous covariates, semisynthesis, and two coarsened randomized studies.
Critical strengths
- The matched-backbone ablation clearly illustrates how supervision changes the learned estimator’s behavior. Under effect shift, Raw FSP improves RMSE from .1364 to .0624 relative to latent-effect supervision.
- The theory is careful within its stated setting and distinguishes label-prediction risk from causal-effect estimation risk.
- The reported deployment speedups provide a concrete computational motivation for amortizing estimation.
Critical weaknesses
-
The motivation for the theoretical objective is insufficiently established.
The tension between prior-averaged prediction accuracy and fixed-population frequentist behavior is well known. The contribution therefore needs to go beyond identifying shrinkage and showing how to reproduce an efficient estimator’s sampling behavior. The paper does not sufficiently explain why this is the most useful objective for causal pretraining, or which unmet statistical need it addresses relative to existing estimation and correction procedures. Shrinkage can be appropriate under a well-specified prior and a prediction-risk objective; its presence alone does not establish a deficiency that requires removing it. My concern is primarily the significance of the chosen theoretical problem, rather than the correctness of the analysis. -
The theory does not sufficiently explain the statistical benefit of pretraining.
A central motivation for causal pretraining is that structure learned across tasks can improve estimation with scarce deployment data. FSP’s theory characterizes how well a pretrained model can approximate an oracle influence-function target, but does not identify when this yields a finite-sample advantage over per-table AIPW/DML or what transferable structure produces that advantage. In particular, the label-prediction result concerns the constructed training target, while deployment ATE risk remains of order . This distinction is acknowledged, but leaves the practical significance of the faster label-learning rate unclear. Computational amortization is valuable; it should be distinguished more clearly from a statistical benefit due to transfer. -
The empirical evidence is insufficient to resolve these concerns.
The paper includes useful effect-shift, overlap, and smoothness experiments, but evaluation remains limited to low-dimensional settings (d ≤ 4) and coarsened real data. This provides limited evidence of transfer across substantially different confounding structures or realistic high-dimensional and mixed-covariate tables. The results are also mixed: S-learner has lower RMSE than both FSP variants in all three stratified mechanisms in Table 3, although FSP improves on the reported AIPW/DML baseline. The continuous experiments show gains at trained lengths, but S-learner overtakes FSP in aggregate at n = 1024. Raw FSP’s native coverage of .706 in the weak-overlap diagnostic further shows that learning the mean response does not ensure reliable uncertainty estimates. These findings support a narrower contribution than a broadly established advance in transferable causal estimation.
Questions for authors
- What specific unmet need motivates learning the efficient sampling response through pretraining, rather than using per-table AIPW/DML or correcting a pretrained estimator at deployment? Is the intended primary contribution computational amortization, improved finite-sample estimation, or inferential calibration?
- What structure is transferred under FSP, and can you identify a regime where that transfer yields a statistical advantage over per-table estimation? A quantitative comparison involving pretraining size, deployment sample size, complexity, and mismatch between training and deployment would help establish the theoretical contribution’s significance.
- How robust are the benefits under substantially different confounding mechanisms, higher-dimensional or mixed covariates, and broader benchmarks such as IHDP or ACIC?
- Beyond the existing null and coverage diagnostics, how do the methods compare on small-sample null rejection rates, interval width, and power using their native uncertainty estimates? This would clarify the inferential tradeoff rather than treating reduced attenuation as sufficient evidence of improvement.
Answers to Questions 1–3 could change my assessment.
11 comments
Thanks for the review and here is the kind response(cont'd)
Do these distinctions change observable performance? The matched Raw experiment holds architecture, simulator stream, optimizer, and training budget fixed, changing only the label. Near the prior center, latent-effect supervision has lower point RMSE (0.0176 versus 0.0621), exactly where shrinkage can be beneficial. When the effect shifts, the ordering reverses: FSP lowers RMSE from 0.1364 to 0.0624 (54.2%), reduces sampling defect by 99.0%, and moves the sampling-response slope from −0.058 to 0.972 (Figure 3, Table 3). Thus the experiment does not merely show that shrinkage exists; it isolates a change in which statistical procedure the same network has learned. The seven-level fluctuation ablation tests the predicted supervision mechanism, and the larger weak-overlap experiments identify settings in which the learned rule retains point-accuracy gains. At deployment, the computational distinction is also measurable: on the tested Criteo-visits workload, design FSP takes 5.57–5.61 ms per table on one CPU thread, versus 8.66 ms for optimized AIPW, without per-table nuisance refitting (Appendix P.3).
The significance of the problem is instead the missing bridge between amortized causal prediction and fixed-population efficient inference. FSP supplies a task-aligned supervision target, a counterintuitive label-learnability result, a finite-pretraining-to-inference analysis, and controlled evidence that the learned sampling response—not simply the amount of shrinkage—determines when a frozen model can deliver accurate, reusable causal estimation.
W2. When does EIF-supervised pretraining yield a statistical transfer benefit?
“The theory does not sufficiently explain the statistical benefit of pretraining.”
Response to W2. Pretraining can reduce the cost of learning an adjustment from a small deployment table when tasks share prognostic structure. Our shared-prognostic model gives an explicit threshold: the frozen estimator has lower risk than deployment-only least squares when its offline adjustment error is smaller than the nuisance-learning penalty paid on that table. Appendix Q.5 (pp. 61–62) gives the complete derivation.
What transfers, and when does it help? Consider independent randomized tasks with , , , and mutually independent, with known :
\qquad \varepsilon\sim N(0,\sigma^2).$$ The prognostic coefficient $\beta$ is shared, while the ATE $\theta$ may change arbitrarily between tasks. Pretraining yields $\widehat\beta_M$, independent of the deployment table. Within the frozen residual-score family $f_b(D_n)=2n^{-1}\sum_i Z_i(Y_i-X_i^\top b)$, let $\mathcal E_M=\mathbb E_{\rm pre}\|\widehat\beta_M-\beta\|^2$. Direct calculation gives $$\begin{aligned} R_{\rm frozen}(M,n) &=\frac{4\sigma^2}{n}+\frac{4\mathcal E_M}{n},\\ R_{\rm table\text{-}LS}(n) &=\frac{4\sigma^2}{n} \left(1+\frac{d}{n-d-2}\right), \qquad n>d+2. \end{aligned}$$ Here table-only least squares jointly estimates $\theta$ and the $d$ prognostic coefficients; in this Gaussian model it attains the minimax risk over unrestricted $(\theta,\beta)$ for estimators using only the deployment table. Consequently, $$\boxed{ R_{\rm frozen}(M,n)<R_{\rm table\text{-}LS}(n) \quad\Longleftrightarrow\quad \mathcal E_M<\frac{\sigma^2d}{n-d-2}. }$$ The same sufficient condition beats the explicitly specified two-fold, known-propensity AIPW comparator with foldwise pooled linear nuisance fits; its risk is at least $\frac{4\sigma^2}{n}\bigl(1+\frac{d}{n/2-d-1}\bigr)$ for even $n$ and $n/2>d+1$. Appendix Q.5 specifies the foldwise least-squares nuisance fits used in this comparison. **Why can EIF labels recover the shared adjustment?** For a synthetic training table of size $m$, write $u_j=2m^{-1}\sum_i Z_{ji}Y_{ji}$ and $v_j=2m^{-1}\sum_i Z_{ji}X_{ji}$. Its exact oracle EIF teacher is $T_j=\theta_j+2m^{-1}\sum_i Z_{ji}\varepsilon_{ji}$, so $$u_j-T_j=v_j^\top\beta.$$Thanks for the review and here is the kind response(cont'd)
The task-specific effect and outcome noise cancel in the supervision equation. This identifies how EIF labels recover the shared adjustment within the stated score family. Within this specified score family, empirical minimization of exact EIF-label loss therefore recovers almost surely with independent Gaussian-design training tables. With independent mean-zero label noise of variance , it instead yields for . Thus, when and is sufficiently larger than , pretraining replaces the table-only adjustment cost of order by an offline-learning cost of order . The gain can be even more pronounced when : with negligible offline error, . The oracle variance remains ; the gain comes from reducing the additional nuisance-learning cost. Because the risk is independent of , the gain persists under arbitrary treatment-effect shifts. Under prognostic shift, the same threshold applies with measured against the deployment coefficient.
How does this relate to teacher learnability? The two results answer complementary questions. Theorem 4.2 shows, in a separate finite-stratum model, that a full-EIF label can be predicted from the table with ; the observable stratum estimator already witnesses this rate without pretraining. Theorem 4.3 explains why the rate matters for a frozen model: in the exact decomposition
its label-learning contribution to the scaled remainder is , while approximation, optimization, finite-episode cost (proportional to ), and mechanism transfer must also be controlled. The shared-prognostic specialization goes beyond this recovery guarantee: it identifies a concrete transferable structure and a threshold for strict finite- improvement over fitting it from scratch.
Does the effect appear beyond the specialization? In Figure 19, holding the shifted deployment mechanism and fixed while raising from 2,048 to 32,768 reduces sampling defect from 2.163 to 0.050, moves from 1.616 to 0.998, and improves native nominal-95% coverage from 91.1% to 94.5%. In the separate matched transfer experiment, FSP has lower point RMSE in all 12 tested cells against each of the logistic and saturated AIPW specifications, with favorable pointwise paired intervals. Appendix Q.1 reports the full grid and the separate sampling-response diagnostics. The statistical benefit of pretraining is therefore precise: shared information learned offline can remove a quantifiable nuisance-learning penalty at deployment, while EIF supervision provides a way to learn and reuse the corresponding first-order adjustment. Computational amortization is an additional, separate benefit.
W3. Transfer beyond simple simulations, strong baselines, and calibrated inference
“The empirical evidence is insufficient to resolve these concerns.”
Response to W3. The prespecified trained-length continuous grid gives a direct majority comparison: FSP wins 16 of 24 cells against the strongest tested classical method, including all 12 weak-overlap cells. Against cross-fitted AIPW/DML it wins 24/24 (Appendix N.10). We also test mixed real covariates, external outcome benchmarks, larger tables, and all 82 encoded ACIC inputs.
1. Transfer beyond four-stratum inputs. On Adult/Bank tables with 12 continuous and categorical features, nonlinear assignment, interactions, and heterogeneous effects, the matched 1.30M-parameter FSP achieves 0.0449 RMSE under weak overlap and shifts of . Matched latent supervision gives 0.0534 and polynomial T-learning 0.0521: reductions of 15.8% and 13.7%, respectively. The paired RMSE-difference interval against T-learning is [−0.0116, −0.0024], conditional on the checkpoints. FSP's mean absolute shift-tracking error is 0.0048 versus 0.0271 for latent supervision (Appendix P.1).
On larger randomized tables, design-aware FSP lowers Criteo visit RMSE by 9.2%–24.6% relative to the tested Hájek AIPW over 16,384–262,144 bootstrap draws, scored against the released-data empirical contrast (Appendix P.3).
Thanks for the review and here is the kind response
W1. Why is the efficient sampling response a meaningful objective for causal pretraining?
Response to W1. Shrinkage can improve prior-averaged prediction under a well-specified prior. Our objective is reliable repeated-sample estimation in a fixed deployment population. The unmet need arises when a causal foundation model is pretrained across synthetic populations but deployed to analyze repeated samples from one fixed, unknown population. For that population, the questions are whether estimates have the right repeated-sampling center, whether their fluctuations reflect the available information, and whether reported uncertainty has its intended coverage. A hospital analyzing successive treatment cohorts, for example, needs its estimates and intervals to behave correctly for its own patient population, not merely to score well across simulated hospitals. Low risk averaged over training mechanisms does not, by itself, answer any of these fixed-population questions—particularly when the deployment effect differs from the training-prior center. Our simpler scientific question is: Can we train a causal foundation model once to act like an efficient statistical estimator on new tables, rather than only to predict their latent effects?
Why the EIF, rather than an arbitrary anti-shrinkage penalty? For a regular, asymptotically efficient ATE estimator under the specified causal model, the canonical first-order expansion at a fixed mechanism P is
θ(Dn)=θ(P)+PnψP+oP(n−1/2),
where ψP is the efficient influence function. This identifies the sampling response we want a frozen model to reproduce; the matched-label comparisons evaluate this response separately from point RMSE. In particular, a model can have low point error by attenuating sample-to-sample movement, yet fail to reproduce the first-order behavior required for efficient frequentist inference. The objective is therefore tied to a specific statistical use case, not chosen merely because the EIF is familiar.
Why does this remain an open problem when classical correction already exists? AIPW, cross-fitted DML, and related one-step or targeted corrections can construct efficient estimators by estimating nuisances and correcting the estimate on each new table. More directly related, the PFN one-step posterior correction of Melnychuk et al. (2026) recovers nuisance posteriors and corrects the PFN-based procedure at deployment. These are important, often stronger alternatives, not deficiencies we claim to have removed. They address a different computational arrangement: construct or correct the estimator at test time. We ask whether synthetic pretraining can amortize that first-order correction into the native frozen map, so that new tables require neither nuisance refitting nor an additional EIF correction. Simulator-known mechanisms are used to construct supervision only during pretraining; they are not available to, or assumed known by, the deployed model. This distinction matters when many cohorts or experimental tables must be processed repeatedly, while also creating a new statistical question: when does the learned correction survive finite pretraining and a change of population?
What is mathematically new and difficult? We supervise each synthetic table with TP(Dn)=θ(P)+PnψP, rather than the mechanism-level constant θ(P). Surprisingly, the fluctuating target can be easier to learn from the table: Theorem 4.2 proves sharp n−2 minimax label-prediction risk for full fluctuation in our finite-stratum class, while every fixed partial-fluctuation level below one retains an n−1 term. The gain comes from cancellation of first-order label ambiguity, not from assuming the unknown effect becomes easier to estimate. The next obstacle is to pass from an oracle synthetic label to an actual trained network. Theorem 4.3 bounds the sampling defect of the uncorrected frozen output using label learnability, network approximation, finite training episodes, optimization, and explicit mechanism-transfer conditions; it then relates that defect to fixed-population bias, variance, Gaussian approximation, and coverage conditional on variance-head accuracy. Complementary lower bounds (Theorems 4.4–4.6) distinguish the information supplied by pretraining episodes from the irreducible n−1 risk of estimating the deployment effect. This is not a standard one-step analysis with consistent test-time nuisance fits: the object whose sampling law must be controlled is a pretrained, frozen table-to-estimate rule.