acceptodds
Under review as a conference paper at ICLR 2027

Beyond Remasking: Probabilistic Tabular Prediction under Recording Shift

Abstract

A tabular model can reconstruct hidden entries accurately yet predict the wrong distribution for genuinely missing values. We address recording shift, where selective data collection makes observed outcomes unrepresentative of those requiring prediction. Our approach uses additional information retained in labelled source records and an unlabelled deployment sample, even when that information is unavailable for individual predictions. Building on existing identification results, we develop two probabilistic adaptation routes under conditional stability and source support. Direct reweights learning or labelled contexts. Supported fine-law adaptation (SFLA) combines state-conditional distributions using deployment probabilities. We implement these routes through deployment-targeted flow matching (DT-FM) for joint continuous prediction and through adaptation of frozen tabular foundation models. Our analysis separates correctable population mismatch from information lost at inference and explains how estimation errors affect the benefit of correction. Controlled experiments recover conditional changes invisible to marginal correction and associations lost by point prediction. Experiments on real datasets under imposed recording shifts demonstrate gains over native predictors, with controls identifying the contribution of recording correction.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Tabular foundation models, flow matching, missing data, distribution shift, probabilistic prediction, selection bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z2a4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to predict missing tabular values when the recorded cases are not representative of the cases that need predictions, which the authors call "recording shift." It assumes that labelled source records and an unlabelled deployment sample both contain extra information C (for example, why a test is missing) that is not available at prediction time. It also assumes the outcome behaves the same in both groups once the inputs and C are known. Building on the identification result of Boeken et al. (2023), it proposes two corrections. Direct reweights the training loss, or the in-context examples of a frozen tabular foundation model, using an estimated source-to-deployment ratio. SFLA fits one predictive distribution per state of C and mixes them using deployment state probabilities estimated from the unlabelled sample. Both are implemented in a weighted flow-matching model (DT-FM) and in frozen TabPFN, TabICLv2 and TabDPT. The analysis splits prediction error into information lost by omitting C and a population mismatch that can be corrected, and gives a formula for when re-mixing fitted components helps or hurts. In a synthetic task where native models cannot tell two deployments apart, SFLA reaches the best attainable Brier risk (0.42 vs. 0.50). On five real datasets with imposed recording, updating the mixture clearly helps on Household and Metro. However, native TabPFN stays better on Beijing and Appliances, and an independent Electricity test shows no clear gain.

Critical strengths

Strength: The paper identifies a real failure of native missing-value handling and shows it with a clean construction.

  1. It explains clearly why training on artificially hidden values can give the wrong distribution for genuinely missing ones when recording is selective (Section 1; Appendix A.2).
  2. The unchanged-marginals example (Table 2) gives two deployments with the same contexts, inputs and missingness patterns but opposite optimal predictions. All three native TFMs stay at the 0.50 bound, while SFLA reaches the attainable risk without changing any pretrained weights.
  3. The correction works through interfaces that frozen foundation models already have (context sampling and per-state queries; Table C.1), so it can be applied without retraining.
Critical weaknesses

Weakness 1: The real-data evidence is narrow and does not show the method is competitive.

  1. The authors impose the recording themselves, so the bridge assumption holds by construction (Sec. 5.1). On Gas and Appliances, the state is even computed from the responses (Sec. 5.1). No experiment uses natural missingness, so the paper tests estimation, not whether the method works in a real deployment.
  2. The strongest gains from updating the mixture appear on two of five datasets, Household and Metro (Table 3). Each comes from a single fit and was examined during development (Table C.2; App. F.2). The prespecified primary Household test failed, while the reported SFLA result was a secondary comparison using a different grouping protocol (App. F.1.2). The only untouched dataset, Electricity, shows no clear gain (Sec. 5.2).
  3. The correction often loses to not correcting. Native TabPFN beats SFLA on Beijing and Appliances (Table F.2). Uniform contexts, which apply no correction, give the best scores on Gas and Appliances (Table 3). On Adult and the clinical data, the gains are mixed or uncertain.
  4. Selection is inconsistent across applications. In the incomplete-source flow study, the source-based rule retains 14 originals and two source blends across 16 cases (Table F.23), even though some alternative candidates have lower evaluation risk. For frozen TFMs, selection removes the mixture-update benefit on Beijing and Household but preserves substantial gains on Metro (Table F.14a).
  5. DT-FM, a named contribution, is weak. It never gives the best score on any dataset in Table 3, and its Direct and SFLA versions are often worse than plain DT-FM. In the continuous synthetic experiment (Table D.4), a simple Gaussian-component model matches or slightly outperforms DT-FM under SFLA, raising questions about the added benefit of the flow model in this controlled setting.
  6. Key ablations are missing. Every experiment uses a binary state C, although the method is claimed for continuous and mixed states (Sec. 2; App. A.5). The weight estimator is one fixed setup. Auxiliary sample size, label budget and context size are never varied on real data.
  7. The reproducibility statement claims that fitting and prediction costs are reported in Appendix C.8, but the appendix provides configurations rather than measured runtimes or memory use. The computational overhead of state-conditional queries, mixture construction and joint sampling is therefore unclear, particularly as the number of states increases.
  8. Overall, the evidence supports one narrow claim: re-mixing helps when a binary recording shift is built into a dataset where the state predicts the outcome. It does not show that the method beats native foundation models or holds on independent data.

Weakness 2: The novelty is thin, and the most relevant competitors are missing.

  1. The method combines known parts. Identification comes from Boeken et al. (2023), Direct is importance weighting (Shimodaira, 2000), and SFLA is the law of total probability with new mixing weights, as in pattern-mixture models (Little, 1993). Context Direct has no guarantee; the paper admits it "need not implement weighted risk minimization" (App. G.2).
  2. The theory is standard algebra. Propositions 1 and 2 expand squared distances. Proposition B.1 says that a model given identical inputs gives identical outputs. None of the results says when SFLA should beat an existing method.
  3. The title promises to go "beyond remasking", but the main real-data table includes no remasking method (Table 3). ReMasker appears only as a point predictor in side experiments, where it is scored on spread and correlation it cannot produce (Fig. 3). CSDI is never run.
  4. The strongest competitors are not run. Boeken's doubly robust estimator is cited but not evaluated. The joint learner that the paper itself names as the right comparison is also not evaluated (App. C.4). Where simple baselines are run, they keep up. Repeated regression ties SFLA on the binary task and has the best RMSE on two of five datasets (Table 2; Table F.4). Logistic regression with the same correction matches the foundation models (Table D.3).
  5. Overall, without these baselines it is unclear what is new beyond moving a known correction to distributions and foundation models, or whether SFLA is the best way to use the auxiliary sample.
Questions for authors

Major questions:

  1. Is SFLA the best way to use the auxiliary sample? Two natural competitors are missing.
    • The joint learner, a TFM given the auxiliary deployment (H, C) rows in its context, is named in App. C.4 but not run.
    • Boeken et al.'s doubly robust estimator is cited but not evaluated (Sec. 1; App. G.1).
    • Can the authors compare both with SFLA on the controlled task (Table 2), Household and Metro?
  2. Does remasking actually fail under recording shift on real data? The title targets remasking, yet no remasking method appears in Table 3. ReMasker is scored only as a point predictor (Fig. 3; App. F.4–F.5), and CSDI is never run. Can the authors add a distributional remasking-style imputer to the main comparison, so the gap the title promises can be measured?
  3. Does the method work under natural missingness? Every real-data shift, including the clinical study, is imposed, so the bridge holds by design (Sec. 5.1; App. F.7). Can the authors evaluate one real setting with a representative auxiliary sample and a credible bridge? If no such dataset exists, how common is this setting in practice?
  4. Are the Household and Metro gains robust?
    • These are the only datasets with clear gains (Table 3). Each is one fit on data examined during development (Table C.2; App. F.2).
    • How many designs were tried before these five were fixed?
    • Do the gains replicate over new allocations and refits? The independent Electricity test shows no clear gain (Sec. 5.2).
  5. When should a practitioner apply the correction? Native TabPFN beats SFLA on Beijing and Appliances (Table F.2), and the paper's own validation mostly returns the uncorrected predictor (Table F.23; Table F.14a). Prop. 2 depends on the unknown component error. Does the source-label gain prediction in App. E.2 work on the real datasets, so that a user without deployment labels can decide?
  6. What is new beyond Boeken et al.? Identification, weighting and repeated regression come from Boeken et al. (2023), and repeated regression ties SFLA on the binary task (Table 2). Under what conditions, stated formally, should SFLA beat repeated regression or Direct for continuous or joint outcomes? Is there a result beyond the squared-norm expansions in Props. 1 and 2?
  7. Why is DT-FM a separate contribution? DT-FM never gives the best score in Table 3, a Gaussian model matches it (Table D.4), and App. A.2 says CFMI can implement either route. How does DT-FM differ from CFMI trained with the same weights?

Minor questions:

  1. Have the authors tried non-binary C? All experiments use two states (Sec. 2). Results with |C| ≥ 4 or a continuous C (App. C.1) would test the general claim.
  2. How sensitive are the results to the weight estimator and the auxiliary sample size on real data? Both are fixed or varied only on synthetic data (App. C.1; App. E.3).
  3. Do the gains hold with larger contexts? The real-data runs use 2,048 examples per context (Table C.3). Do the gains persist at 8k~16k labels for TabPFN 3.5 and TabICLv2?
  4. Where are the costs? App. C.8 is cited for fitting and prediction costs but lists only settings. Can the authors report time per 1,000 queries for SFLA vs. Native, and how it grows with |C|?
  5. Is the corrected prediction calibrated? Coverage stays below nominal, e.g. for TabPFN SFLA on Metro and Appliances (App. F.1.3) and for the flows on Appliances (Table F.17). Does the correction change calibration, and could a conformal step fix it?
  6. Please clarify notation and reporting. Several symbols have more than one meaning (Sec. 4; App. B). Table 3 mixes 128-draw and exact CRPS, and Table F.7(c) omits Metro. Matched-budget results would help for Adult (Table F.24) and Table 2.
5 comments
AuthorsOpenReview ↗

Weakness 1, evidence and competitiveness

Thank you for the very detailed review. It set the agenda for this revision, and we try to answer every point. Your overall concern was that the evidence "does not show that the method beats native foundation models or holds on independent data". We believe that it now does: on 29 datasets outside the submission, with ten random splits each, the final prediction beats Native TabPFN trained on all source labels by 0.0045 CRPS [0.0038, 0.0051], and SFLA beats Native on identical labels by 0.0098 [0.0093, 0.0104] (common response, 2/2).

Imposed recording; no natural missingness. We added NHANES 2015–16, where clinical practice decided who was tested and the survey measured everyone. Gain over Native are 0.0256 [0.0216, 0.0295], with the protocol fixed before the cycle was opened. Gas and Appliances are now labelled as constructed designs with response-derived states.

Household and Metro are single development fits; Electricity is null. The 580-run study replaces single fits with replication on independent datasets, and Household, Metro and Beijing are now refitted ten times. Against all-label Native, the final prediction gains 0.0075 [0.0056, 0.0094] on Metro (significant) and 0.0035 on Household (not significant after Holm); on Beijing it does not improve (−0.0029 [−0.0144, 0.0087]), which we have now included in the revised appendix.

The correction often loses to not correcting. The final prediction keeps Native and uniform contexts as candidates and chooses without deployment outcomes. On TabPFN, it is never worse than its Native candidate (15 datasets better, 14 tied, 0 worse) and beats uniform contexts in 26 of 29 (0 worse). Against all-label Native, the per-dataset record is 8 better, 12 tied, 9 worse. Here, the gain is concentrated where the state is informative (it leads in five of the six datasets with an update gain above 0.01).

Selection is inconsistent. Selection is now part of the method and is evaluated as such. The final TabPFN prediction beats an equally sized stack that is validated the same way, but corrects with source frequencies by 0.0061 [0.0056, 0.0066], with no dataset worse.

DT-FM is weak. We refit DT-FM and CFMI to assess sensitivity to retraining and test whether adaptation benefits both architectures. This gives 60 fits across five applications × two flow families × three routes × two refits. The adapted mixture improves joint energy score over the unadapted flow in 9 of 10 application–family comparisons, and 6 remain significant after Holm correction, three per family. Without recording shift, selected DT-FM and CFMI achieve the lowest mean energy score in 7 of 8 completion queries across four datasets. These results support DT-FM as an effective joint-prediction implementation of the framework, whilst we do not claim architectural superiority.

Missing ablations. We now added:

  • four-state and continuous-state designs;
  • one-, three- and five-dimensional states on nine real datasets;
  • three state estimators with fixed TFM components;
  • auxiliary samples from 128 to 4,096 records;
  • source-label budgets from 1,000 to 6,000;
  • contexts of 2,048 and 8,000 examples with TabICLv2.

No measured costs. TabPFN 3.5 on GPU with a 2,048-example context: 9.9–10.6 s per 1,000 queries for Native and 19.4–20.8 s for two-state SFLA, with 4.1–6.7 GiB peak memory. On CPUs, the cost grows more slowly than the number of states, because the context is encoded once: 1.9× Native at four states and 3.8× at sixteen. TabICLv2 and TabDPT are timed as well.

AuthorsOpenReview ↗

Weakness 2, novelty and competitors

"The method combines known parts." We agree on the parts and now credit them explicitly. Identification and mean estimators to Boeken et al., weighting to Shimodaira, mixtures to pattern-mixture models. Our contribution is the predictive method built from them:

  • full distributions through frozen-TFM interfaces;
  • a theory of the fitted errors;
  • a validated final prediction with a risk bound.

For Context Direct, we claim what it does, which is to build contexts that resemble a deployment sample. We do not claim that a pretrained network minimises a weighted risk.

"The theory is standard algebra; none of the results says when SFLA should beat an existing method." The revision adds results that do, and tests them.

  • Against mean corrections. If a mean correction with a fixed distributional form has unavoidable misfit Δ, and SFLA has fitting error e, then SFLA's risk is lower by at least Δ − e. In a prespecified simulation, the bound held in all 1,680 replications, and SFLA won in 10 of 10 cells where e < Δ was predicted.
  • Against importance weighting. SFLA's variance is lower by a post-stratification term. The realised gap matched the predicted one within 1% at large samples.
  • For the final prediction. An oracle inequality bounds the stack's risk against the best mixture of its candidates, and also covers a choice among fitting configurations made on the same validation records.
  • For the practitioner. The outcome-free gain estimate predicted TabPFN's realised update gain with rank correlation 0.95.

The paper also reports where predictions fell short: the predicted switch-over point is conservative, the variance result is first order and reverses in small samples with a rare state, and bands built from fitted components undercover.

"No remasking method in the main table." We now compare against distributional remasking methods. Across ten refits on Household, Metro and Beijing, our final predictor reduces CRPS relative to TabCSDI by 9.6–21.9% under recording shift. Improvements are significant in all six comparisons—with and without shift—after Holm correction. We also evaluate native DiffPuter on incomplete-source California. Its mixture correction gives no clear improvement under informative recording and worsens performance in the control. We have removed distributional rankings of ReMasker, which produces point predictions.

"The strongest competitors are not run." They are now.

  • Boeken's published doubly robust correction. Implemented from the published formula and checked against the authors' code. On the controlled task, it ties, as you anticipated. Brier risk 0.4236 for SFLA, 0.4236 for repeated regression and 0.4237 for the published correction (attainable risk is 0.4200).
  • Across 145 OpenML dataset–conditions (statistical learners; better / tied / worse for the final prediction):
CompetitorBetterTiedWorse
Boeken repeated regression †14500
Augmented doubly robust mean correction †14230
Distributional repeated regression130150
Joint learner (MICE)13870
Joint learner (FCS)131140
Best prior method, chosen with deployment outcomes103420
Stack of prior methods90550

† Mean estimators, scored as distributions by shifting the location of the source distribution.

  • On actual TabPFN, distributional repeated regression with a forest outer learner is far behind the final prediction: CRPS is 0.332 against 0.217.

The simple baselines do keep up on the binary task, and we say so. The advantage appears for full distributions, where a mean correction cannot follow a change of shape.

AuthorsOpenReview ↗

Major questions

Q1. Is SFLA the best way to use the auxiliary sample? No single route is best everywhere, which is why the final prediction combines them. That combination is never worse than the best prior method chosen with deployment outcomes in any of 145 OpenML dataset–conditions (see table above).

  • Boeken's doubly robust estimator: evaluated faithfully; cf. above.
  • The joint learner: MICE and FCS joint learners that receive the auxiliary (H, C) rows are also in the table above. A comparison with a pretrained joint learner, TabImpute, is still running; we will add it to the revised paper once it finishes.

Q2. Does remasking fail on real data? The gap is now measured against native missing-value handling on a real TFM: SFLA beats Native TabPFN with identical labels by 0.0098 across 29 datasets. Against a dedicated remasking imputer, the final prediction lowers TabCSDI's CRPS by 9.6–21.9% on Household, Metro and Beijing. The formal point stands independently: a model trained only on source records cannot tell apart two deployments that give it identical inputs, while auxiliary deployment records can.

Q3. Natural missingness, and how common is the setting? NHANES provides evidence under naturally occurring screening histories: the survey supplies auxiliary covariates and independently measured outcomes for evaluation. Our final predictor improves on Native by 0.0256 CRPS, and the diagnostic for the mean implication of stability is not rejected. Trial transport gives a concrete motivation for the information split. When planning for a new recruitment population, detailed baseline assessments can be collected from a sample before follow-up outcomes become available, while forecasts use routinely available patient characteristics. This makes the auxiliary sample useful without requiring detailed assessments for every prediction. We evaluate this arrangement in the International Stroke Trial, using separate patients’ recorded outcomes to assess transfer to three held-out countries. Our final predictor reduces joint Brier risk by 11.7–15.3% against Native.

Q4. Are the Household and Metro gains robust?

  • Designs tried: the records identify 17 other imposed-recording designs; five have no recorded reason for exclusion. The paper states this history and does not call the five an untouched benchmark.
  • Replication: in ten refits, the final prediction improves significantly on all-label Native on Metro; the smaller Household gain is not significant after correction. The 580-run study replicates the effect on 29 independent datasets.

Q5. When should a practitioner apply the correction? Let validation determine whether and how much to correct. Our final predictor evaluates and combines candidates, including Native, using source labels and unlabelled deployment records. The gain estimates are informative on real data:

  • TabPFN, 29 datasets (new Figure 2): predicted update gain +0.0148, realised +0.0144, rank correlation 0.95; the sign is right in every dataset where the effect exceeds 0.005. The same holds for the choice between SFLA and Native (rank correlation 0.96).
  • NHANES HbA1c: predicted −0.0019, realised −0.0020 [−0.0032, −0.0007]. The estimate flagged the one harmful update in advance.

These estimates rely on stability. When that assumption is doubtful, sensitivity analysis assesses how fragile the recommendation is, and a representative outcome audit provides evidence for checking and revising the prediction.

Q6. What is new beyond Boeken et al.? Three formal results, each with its limit.

  • SFLA against mean corrections: SFLA wins when its fitting error is below the mean correction's unavoidable misfit (W2 above). This does not apply against unrestricted distributional regression.
  • SFLA against importance weighting: lower variance by a post-stratification term, with known state probabilities.
  • Coincidence and choice: with cellwise estimators the three routes coincide. With fitted learners they differ, and the oracle inequality justifies choosing among them by validation.

Q7. Why is DT-FM a separate contribution? DT-FM makes the framework operational for joint continuous prediction through deployment-targeted flow training and joint sampling. We present this as an implementation contribution, without claiming architectural superiority over CFMI. Both flow families now receive the same adaptation, yielding Holm-significant improvements on three of five applications for each family. These results demonstrate that the adaptation benefits extend across flow backbones.

Reviewer_ZRGPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies probabilistic tabular prediction under recording shift and proposes two strategies: importance-weighted learning and conditional distribution assembly. The paper corrects the mismatch with theoretical analysis and empirical evaluation.

Critical strengths

The paper addresses an important problem in tabular prediction under recording shift. The theoretical analysis provides useful insights into the limitations of existing methods and the conditions under which adaptation can help.

Critical weaknesses

The experiments mainly focused on synthetic recording shifts, where the conditional stability assumption holds by construction. It is not very clear how the proposed methods could be generalize to naturally missing.

Questions for authors

See weakness

2 comments

Thank you for the positive assessment, and for stressing the scope of our initial experimental evaluations. The experiments imposed the recording, so stability held by construction. The revision now tests the method where recording arises naturally, tests the assumption itself, and shows what to do when it fails.

A natural-recording study. In NHANES, clinical practice decided who had a cholesterol test in the previous year (source) and who did not (deployment). The survey measured everyone's cholesterol, so predictions for the untested can be scored. C is a prior diagnosis of high cholesterol, known in the records and withheld from each prediction. We developed on 2011–2014 and fixed the protocol before opening 2015–2016.

NHANES 2015–2016CRPS gain [95% interval]
Final prediction versus Native0.0256 [0.0216, 0.0295]
Final prediction versus a stack of prior methods0.0014 [0.0004, 0.0025]
Deployment mixture versus source mixture, same components0.0004 [−0.0014, 0.0021]

The first two pass the prespecified Holm test. Intervals use the survey design, over twenty random splits with every model refitted. Here, most of the gain comes from the complete predictor, and the mixture update alone is small.

The assumption is tested, and not assumed. The stability test is not rejected (−1.3 mg/dL [−4.6, 2.1]).

The theory flagged, in advance, the one correction that would hurt. For HbA1c, the source-based estimate predicted a loss of 0.0019 from the update; the realised loss was 0.0020 [0.0007, 0.0032].

When stability fails, the paper now says what happens and what to do.

  • Detection. The same test applies wherever outcomes for a small audit sample of deployment records are available.
  • Bounds. We give tight bounds on the deployment distribution for a stated degree of violation, and report the degree at which each conclusion would change.
  • Repair. Under outcome-dependent recording, observing outcomes for 10% of the deployment sample cuts CRPS from 0.4591 to 0.4016, a 12.5% reduction over 30 random splits. Where stability holds, the same check costs at most 0.0012.

I thank the authors for their reply, and they have addressed my concern. I will maintain my score. I am not an expert in this field, so my assessment of the novelty and coverage of related work may be limited.

Reviewer_tJu4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies recording shift in tabular data. The authors point out that artificially masked values and truly missing values may have different distributions. To address this problem, they propose two methods: Direct Adaptation and Supported Fine-Law Adaptation (SFLA). The methods are applied to flow matching and several tabular foundation models. The paper also provides theoretical analysis and experiments to evaluate these methods.

Critical strengths

The paper studies an important problem in tabular prediction. The motivation is clear, and the problem is relevant to real-world applications.

The proposed framework can be used with different models. It also considers both single-variable and joint probabilistic prediction.

The experiments include several baselines and ablation studies. These comparisons help explain how different parts of the methods affect the results.

Critical weaknesses
  • Limited novelty. The main methods are closely related to existing techniques. Direct Adaptation uses importance weighting, while SFLA uses weighted mixtures of conditional distributions. The theoretical results also seem to follow from existing statistical properties. The authors should explain more clearly what is new compared with previous work.

  • Inconsistent improvements. The proposed methods do not always perform better than the original models. For example, in Table 3, SFLA improves TabPFN 3.5 on only two out of five datasets. This raises concerns about how well the methods work across different tasks.

  • Limited real-world validation. Most experiments use recording shifts created by the authors. It is unclear whether the methods would work as well on naturally occurring recording shifts. The methods also require strong assumptions and additional deployment data. These conditions may be difficult to satisfy in practice.

Questions for authors

Please see weakness

2 comments

Thank you for the encouraging assessment of the problem and the framework. Your three concerns led directly to the main new results.

1. "Inconsistent improvements: SFLA improves TabPFN 3.5 on only two out of five datasets."

We replaced the five single fits with a 580-run replication on 29 datasets in the main paper (see also our common response).

  • Gains without additional labels. Using identical labelled training data, SFLA reduces CRPS relative to Native TabPFN by 0.0098 [0.0093, 0.0104], with 19 wins, 9 ties and 1 loss across 29 datasets.
  • The final prediction beats Native trained on all source labels by 0.0045 [0.0038, 0.0051], p < 0.001.
  • Addressing inconsistent gains. The two-of-five result highlights why a fixed correction is not enough. Its benefit depends on the shift, the predictive value of C, and estimation error. The revised procedure uses source validation to combine predictors, including Native, without accessing deployment outcomes. With statistical learners, it achieves 118 wins against Native across 145 dataset–conditions, with no losses under the reported comparison criterion.

2. "Limited real-world validation; strong assumptions and additional deployment data."

  • Natural recording. In NHANES 2015–16, clinical practice decided who had a cholesterol test in the previous year; the survey then measured everyone. The final prediction improves on Native by 0.0256 CRPS [0.0216, 0.0295], and on a stack of prior methods by 0.0014 [0.0004, 0.0025]. The protocol was fixed before this cycle was opened. In the International Stroke Trial, it lowers joint Brier risk by 11.7–15.3% against Native in three held-out countries.
  • The assumption can be checked. The test of stability is not rejected in NHANES (−1.3 mg/dL [−4.6, 2.1]). The revision also bounds the deployment distribution under a stated degree of violation. In a controlled study where stability fails, observing outcomes for 10% of the deployment sample cuts CRPS by 12.5%.
  • Small unlabelled samples can support useful adaptation. The method uses a representative deployment sample containing both routine inputs H and detailed information C, without requiring outcome labels. Each new prediction needs only H. On Household and Metro, gains persist with just 128 auxiliary records, down from 4,096. Depending on the application, C could come from existing triage notes, referral records or screening histories.

3. "Limited novelty; the theory seems to follow from existing statistical properties."

We now state plainly what is prior work and what is ours.

Prior workThis paper
Boeken et al.: identification; mean estimatorsFull predictive distributions, including joint ones (DT-FM)
Importance weighting; mixture identitiesThe same corrections through frozen-TFM interfaces, with no weight updates
StackingA doubly robust stack for the deployment population, with an oracle inequality that also covers a choice among fitting configurations on the same validation records
Fitted-error comparisons that say when each correction wins, and an outcome-free prediction of the update's gain

The theory makes quantitative predictions, and a prespecified simulation confirms them:

  • the predicted variance advantage of SFLA matches the realised one within 1% at large samples;
  • SFLA beat the mean corrections in 10 of 10 cells where our condition predicted it.

On actual TabPFN, the outcome-free estimate predicted the update's gain as +0.0148 against +0.0144 realised, with rank correlation 0.95 across the 29 datasets.

Thank you for the detailed response.

Overall, I think the authors have addressed my main concerns, and the revised paper is much stronger. I am willing to increase my rating from 2 to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers on what the revision adds

We thank all three reviewers. The reviewers asked whether the correction works on a real foundation model across more datasets, whether it works when we do not impose the recording ourselves, what is new relative to Boeken et al., and whether the paper can be made easier to read. We have run the experiments you asked for. The answers are positive, which, we believe, makes the paper stronger now.

Your concerns and updated results

ConcernNew evidenceResult
Gains on only 2 of 5 datasets, from single fits (R1, R3)Actual TabPFN 3.5 on 29 datasets not in the initial submission, 10 fresh random splits each, every model refitted: 580 runsThe final prediction beats Native trained on all source labels by 0.0045 CRPS [0.0038, 0.0051], p < 0.001
Is it the correction, or just ensembling? (R3)Same study: the same fitted TabPFN components, mixed with source versus deployment frequenciesThe mixture update alone gains 0.0144 [0.0136, 0.0152] under shift and 0.000 at equal recording rates, i.e. it switches off when there is nothing to correct
The recording is imposed by the authors (R1, R2, R3)NHANES 2015–16: clinical practice decided who had a cholesterol test, and the survey measured everyone0.0256 [0.0216, 0.0295] better than Native; the test of the stability assumption is not rejected
Missing competitors (R3)Boeken's published corrections, distributional repeated regression, MICE/FCS joint learners, and stacks of prior methodsOn 145 OpenML dataset–conditions (statistical learners): 103 better, 42 tied, 0 worse than the best prior method chosen with deployment outcomes; ours with TabPFN has 9.6–21.9% lower CRPS than the remasking imputer TabCSDI
When should a practitioner correct? (R3)The outcome-free risk estimate that builds the final prediction (new Figure 2)On TabPFN, it predicts the update's realised gain: +0.0148 predicted, +0.0144 realised, rank correlation 0.95 across datasets
The theory is standard algebra (R1, R3)A prespecified simulation testing each claim against the quantity it predictsPredicted variance gap matched within 1%; SFLA won in 10 of 10 cells where the theory said it would
DT-FM is weak (R3)60 refits of DT-FM and CFMI on five applicationsThe adapted mixture lowers joint energy in 9 of 10 comparisons, 6 significant after Holm
Costs, ablations, calibration (R3)Runtimes for three TFMs up to 16 states; state, weight-estimator and budget ablations; coverageFor TabPFN, four-state SFLA costs 1.9×
Hard to read (all)Main text rewritten; clearer algorithm presentation; consolidated appendix

What is new in this paper

Boeken et al. (2023) identify the deployment distribution and estimate its mean. We build the predictive method on that foundation.

  1. Full predictive distributions from frozen foundation models. Direct and SFLA use only interfaces that TFMs already have, i.e. context sampling and per-state queries, with no weight updates. DT-FM extends them to joint continuous prediction.
  2. A theory of when each correction helps. We give each route's fitted error, the condition under which SFLA beats mean corrections, and an outcome-free prediction of the update's gain, now confirmed on TabPFN.
  3. A final prediction with a risk guarantee. A doubly robust stack mixes Native, prior methods and our corrections without deployment outcomes. Under the stability assumption, its risk is within a sampling term and a product of estimation errors of the best mixture of its candidates. Native stays among the candidates, so the method can decline to correct.
AuthorsOpenReview ↗

TabPFN replication experiment and scope

Design. TabPFN 3.5 on 29 OpenML-CTR23 regression datasets. The recording state is chosen without outcomes. Ten random splits, under shift and under equal recording rates.

CRPS gain of the second over the firstUnder shiftDatasets better / tied / worseEqual recording rates
Source mixture → deployment mixture (SFLA), same fitted components+0.0144 [0.0136, 0.0152]6 / 22 / 1−0.0004 [−0.0010, +0.0001]
Native → SFLA, identical labels+0.0098 [0.0093, 0.0104]19 / 9 / 1+0.0070 [0.0064, 0.0076]
Equally sized stack that corrects with source frequencies → final prediction+0.0061 [0.0056, 0.0066]6 / 23 / 0+0.0000 [−0.0002, +0.0003]
Native with all source labels → final prediction+0.0045 [0.0038, 0.0051]8 / 12 / 9+0.0027 [0.0018, 0.0036]

Interpretation.

  • The mechanism works on a real TFM. Changing only the mixing weights, with every fitted component fixed, gains 0.0144 under shift and nothing when recording rates are equal.
  • The gain is adaptation, not ensembling. A stack of the same size that corrects with source frequencies is 0.0061 worse under shift, and identical at equal rates.
  • The final prediction is a safe default. Against its own Native candidate it is better in 15 datasets, tied in 14 and worse in none.
  • It is beneficial where the detailed information matters. Native gains 0.0061 from the labels our procedure holds out for validation. The final prediction still leads it in five of the six datasets where the update gain exceeds 0.01, and in five of the other 23: hence 8 better, 12 tied, 9 worse after Holm adjustment. With statistical learners, cross-fitting removes this cost: the final prediction is worse than all-label Native in 2% of runs, down from 30%.

Scope.

  • Natural recording and transfer. NHANES measures every participant's outcome, which is what lets us score predictions for people whose test was never ordered. In the International Stroke Trial (IST) data, the final prediction lowers joint Brier risk by 11.7–15.3% against Native in three held-out countries; repeated regression and a stack of prior methods do as well there.
  • Comparison with remasking imputer. Ten refits on Household, Metro and Beijing datasets from the initial submission: the final prediction lowers the CRPS of TabCSDI, a diffusion imputer trained by remasking, by 9.6–21.9% under shift, and all six comparisons are significant after Holm correction. Native TabPFN also beats TabCSDI; against it the final prediction improves on Metro (0.0075 [0.0056, 0.0094]) and ties on Household and Beijing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.