- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a residual correction to causal pfn foundation model, inspired by orthogonal (debiased) estimation. Theoretical guarantees are not established. Proposed model is implemented and empirical comparison to existing pfn and traditional methods is done.
Critical strengths
The idea of debiasing applied to causal pfns is very timely and worthy of investigating.
Critical weaknesses
There are several major issues I see with this paper:
- Theoretical motivation: I do think the idea of debiasing causal pfns is very timely and interesting. However, traditionally, orthogonalization is against sensitivity to (first order) nuisance estimation error. With pfns being a full end-to-end one-step estimation procedure, no nuisances are explicily estimated and the theoretical motivation needs to be justified much more carefully. This is not recognized in the paper at all
- Theoretical guarantees: In fact, the proposed residual corrected pfn is not supported with any theoretical guarantees analogous to those used to motivate this work. We are left purely to empirical results to assess the proposal (see below for points on results).
- Design soundness: i) The two head approach for base + correction does not guarantee the individual learned heads correspond to this decomposition. The loss only ever trains their sum, placing no constraint that ensures the decomposition is true. This is never empirically tested either. ii) regularization terms are not theoretically grounded. iii) More generally, the learned residuals, row weights, and query weights are not constrained to correspond to nuisance functions, Riesz representers, influence functions, or any other statistical objects invoked in the motivation.
- Experiment design: Beyond raw predictive performance, the experiments do not validate any aspect of the correction interpretation. Within-task and repeated-context bias is mentioned but never evaluated. Bias variance decomposition is not evaluated. There is no direct test that the learned weights approximate the statistical objects motivating the method, no evaluation of the fully trained model with the correction switched off at inference, and no generic parameter-matched additive-correction baseline.
- Experiment rigor: For results to be meaningfully interpretable, the comparison needs to be fair (unconfounded): the comparison to CausalFM is not matched in causal-label exposure or parameter count, while training-data and compute budgets cannot be verified as matched (same should be at least attempted for CausalPFN and Do-PFN too). Especially the amount of training data is critical for fair comparison here.
- Writing quality: The exposition is poor. Related work is very sparse and missing bib keys (L71 three times). Text is difficult to follow. Notation is not introduced consistently spread out through the entirety of text and could certainly be simplified. Carried thread of though throughout paragraph between sentences is almost non-existent.
In sum, I find substantial issues with this submission in almost every aspect. The most valuable part is the idea itself, which in my eyes has not been done justice with the exploration of it that is submitted here. I would encourage authors to rewrite the entirety of the submission, run new and improved experiments, and actually develop theory exploring this idea and supporting their empirical results.
A basic llm inspection of the code accompanying the submission found the following concerns:
- Query-treatment leakage: Corrected predictions use unmasked query treatments.
- Residual target leakage: Residual heads see the values they are asked to predict.
- Unauditable prior: Training data and task generation are missing, so prior size and mixture cannot be verified.
Questions for authors
Please see above weakness. How does the data and compute budget of SC-PFN training compare to that of CausalFM?