- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how to predict missing tabular values when the recorded cases are not representative of the cases that need predictions, which the authors call "recording shift." It assumes that labelled source records and an unlabelled deployment sample both contain extra information C (for example, why a test is missing) that is not available at prediction time. It also assumes the outcome behaves the same in both groups once the inputs and C are known. Building on the identification result of Boeken et al. (2023), it proposes two corrections. Direct reweights the training loss, or the in-context examples of a frozen tabular foundation model, using an estimated source-to-deployment ratio. SFLA fits one predictive distribution per state of C and mixes them using deployment state probabilities estimated from the unlabelled sample. Both are implemented in a weighted flow-matching model (DT-FM) and in frozen TabPFN, TabICLv2 and TabDPT. The analysis splits prediction error into information lost by omitting C and a population mismatch that can be corrected, and gives a formula for when re-mixing fitted components helps or hurts. In a synthetic task where native models cannot tell two deployments apart, SFLA reaches the best attainable Brier risk (0.42 vs. 0.50). On five real datasets with imposed recording, updating the mixture clearly helps on Household and Metro. However, native TabPFN stays better on Beijing and Appliances, and an independent Electricity test shows no clear gain.
Critical strengths
Strength: The paper identifies a real failure of native missing-value handling and shows it with a clean construction.
- It explains clearly why training on artificially hidden values can give the wrong distribution for genuinely missing ones when recording is selective (Section 1; Appendix A.2).
- The unchanged-marginals example (Table 2) gives two deployments with the same contexts, inputs and missingness patterns but opposite optimal predictions. All three native TFMs stay at the 0.50 bound, while SFLA reaches the attainable risk without changing any pretrained weights.
- The correction works through interfaces that frozen foundation models already have (context sampling and per-state queries; Table C.1), so it can be applied without retraining.
Critical weaknesses
Weakness 1: The real-data evidence is narrow and does not show the method is competitive.
- The authors impose the recording themselves, so the bridge assumption holds by construction (Sec. 5.1). On Gas and Appliances, the state is even computed from the responses (Sec. 5.1). No experiment uses natural missingness, so the paper tests estimation, not whether the method works in a real deployment.
- The strongest gains from updating the mixture appear on two of five datasets, Household and Metro (Table 3). Each comes from a single fit and was examined during development (Table C.2; App. F.2). The prespecified primary Household test failed, while the reported SFLA result was a secondary comparison using a different grouping protocol (App. F.1.2). The only untouched dataset, Electricity, shows no clear gain (Sec. 5.2).
- The correction often loses to not correcting. Native TabPFN beats SFLA on Beijing and Appliances (Table F.2). Uniform contexts, which apply no correction, give the best scores on Gas and Appliances (Table 3). On Adult and the clinical data, the gains are mixed or uncertain.
- Selection is inconsistent across applications. In the incomplete-source flow study, the source-based rule retains 14 originals and two source blends across 16 cases (Table F.23), even though some alternative candidates have lower evaluation risk. For frozen TFMs, selection removes the mixture-update benefit on Beijing and Household but preserves substantial gains on Metro (Table F.14a).
- DT-FM, a named contribution, is weak. It never gives the best score on any dataset in Table 3, and its Direct and SFLA versions are often worse than plain DT-FM. In the continuous synthetic experiment (Table D.4), a simple Gaussian-component model matches or slightly outperforms DT-FM under SFLA, raising questions about the added benefit of the flow model in this controlled setting.
- Key ablations are missing. Every experiment uses a binary state C, although the method is claimed for continuous and mixed states (Sec. 2; App. A.5). The weight estimator is one fixed setup. Auxiliary sample size, label budget and context size are never varied on real data.
- The reproducibility statement claims that fitting and prediction costs are reported in Appendix C.8, but the appendix provides configurations rather than measured runtimes or memory use. The computational overhead of state-conditional queries, mixture construction and joint sampling is therefore unclear, particularly as the number of states increases.
- Overall, the evidence supports one narrow claim: re-mixing helps when a binary recording shift is built into a dataset where the state predicts the outcome. It does not show that the method beats native foundation models or holds on independent data.
Weakness 2: The novelty is thin, and the most relevant competitors are missing.
- The method combines known parts. Identification comes from Boeken et al. (2023), Direct is importance weighting (Shimodaira, 2000), and SFLA is the law of total probability with new mixing weights, as in pattern-mixture models (Little, 1993). Context Direct has no guarantee; the paper admits it "need not implement weighted risk minimization" (App. G.2).
- The theory is standard algebra. Propositions 1 and 2 expand squared distances. Proposition B.1 says that a model given identical inputs gives identical outputs. None of the results says when SFLA should beat an existing method.
- The title promises to go "beyond remasking", but the main real-data table includes no remasking method (Table 3). ReMasker appears only as a point predictor in side experiments, where it is scored on spread and correlation it cannot produce (Fig. 3). CSDI is never run.
- The strongest competitors are not run. Boeken's doubly robust estimator is cited but not evaluated. The joint learner that the paper itself names as the right comparison is also not evaluated (App. C.4). Where simple baselines are run, they keep up. Repeated regression ties SFLA on the binary task and has the best RMSE on two of five datasets (Table 2; Table F.4). Logistic regression with the same correction matches the foundation models (Table D.3).
- Overall, without these baselines it is unclear what is new beyond moving a known correction to distributions and foundation models, or whether SFLA is the best way to use the auxiliary sample.
Questions for authors
Major questions:
- Is SFLA the best way to use the auxiliary sample? Two natural competitors are missing.
- The joint learner, a TFM given the auxiliary deployment (H, C) rows in its context, is named in App. C.4 but not run.
- Boeken et al.'s doubly robust estimator is cited but not evaluated (Sec. 1; App. G.1).
- Can the authors compare both with SFLA on the controlled task (Table 2), Household and Metro?
- Does remasking actually fail under recording shift on real data? The title targets remasking, yet no remasking method appears in Table 3. ReMasker is scored only as a point predictor (Fig. 3; App. F.4–F.5), and CSDI is never run. Can the authors add a distributional remasking-style imputer to the main comparison, so the gap the title promises can be measured?
- Does the method work under natural missingness? Every real-data shift, including the clinical study, is imposed, so the bridge holds by design (Sec. 5.1; App. F.7). Can the authors evaluate one real setting with a representative auxiliary sample and a credible bridge? If no such dataset exists, how common is this setting in practice?
- Are the Household and Metro gains robust?
- These are the only datasets with clear gains (Table 3). Each is one fit on data examined during development (Table C.2; App. F.2).
- How many designs were tried before these five were fixed?
- Do the gains replicate over new allocations and refits? The independent Electricity test shows no clear gain (Sec. 5.2).
- When should a practitioner apply the correction? Native TabPFN beats SFLA on Beijing and Appliances (Table F.2), and the paper's own validation mostly returns the uncorrected predictor (Table F.23; Table F.14a). Prop. 2 depends on the unknown component error. Does the source-label gain prediction in App. E.2 work on the real datasets, so that a user without deployment labels can decide?
- What is new beyond Boeken et al.? Identification, weighting and repeated regression come from Boeken et al. (2023), and repeated regression ties SFLA on the binary task (Table 2). Under what conditions, stated formally, should SFLA beat repeated regression or Direct for continuous or joint outcomes? Is there a result beyond the squared-norm expansions in Props. 1 and 2?
- Why is DT-FM a separate contribution? DT-FM never gives the best score in Table 3, a Gaussian model matches it (Table D.4), and App. A.2 says CFMI can implement either route. How does DT-FM differ from CFMI trained with the same weights?
Minor questions:
- Have the authors tried non-binary C? All experiments use two states (Sec. 2). Results with |C| ≥ 4 or a continuous C (App. C.1) would test the general claim.
- How sensitive are the results to the weight estimator and the auxiliary sample size on real data? Both are fixed or varied only on synthetic data (App. C.1; App. E.3).
- Do the gains hold with larger contexts? The real-data runs use 2,048 examples per context (Table C.3). Do the gains persist at 8k~16k labels for TabPFN 3.5 and TabICLv2?
- Where are the costs? App. C.8 is cited for fitting and prediction costs but lists only settings. Can the authors report time per 1,000 queries for SFLA vs. Native, and how it grows with |C|?
- Is the corrected prediction calibrated? Coverage stays below nominal, e.g. for TabPFN SFLA on Metro and Appliances (App. F.1.3) and for the flows on Appliances (Table F.17). Does the correction change calibration, and could a conformal step fix it?
- Please clarify notation and reporting. Several symbols have more than one meaning (Sec. 4; App. B). Table 3 mixes 128-draw and exact CRPS, and Table F.7(c) omits Metro. Matched-budget results would help for Adult (Table F.24) and Table 2.
5 comments
Weakness 2, novelty and competitors
"The method combines known parts." We agree on the parts and now credit them explicitly. Identification and mean estimators to Boeken et al., weighting to Shimodaira, mixtures to pattern-mixture models. Our contribution is the predictive method built from them:
- full distributions through frozen-TFM interfaces;
- a theory of the fitted errors;
- a validated final prediction with a risk bound.
For Context Direct, we claim what it does, which is to build contexts that resemble a deployment sample. We do not claim that a pretrained network minimises a weighted risk.
"The theory is standard algebra; none of the results says when SFLA should beat an existing method." The revision adds results that do, and tests them.
- Against mean corrections. If a mean correction with a fixed distributional form has unavoidable misfit Δ, and SFLA has fitting error e, then SFLA's risk is lower by at least Δ − e. In a prespecified simulation, the bound held in all 1,680 replications, and SFLA won in 10 of 10 cells where e < Δ was predicted.
- Against importance weighting. SFLA's variance is lower by a post-stratification term. The realised gap matched the predicted one within 1% at large samples.
- For the final prediction. An oracle inequality bounds the stack's risk against the best mixture of its candidates, and also covers a choice among fitting configurations made on the same validation records.
- For the practitioner. The outcome-free gain estimate predicted TabPFN's realised update gain with rank correlation 0.95.
The paper also reports where predictions fell short: the predicted switch-over point is conservative, the variance result is first order and reverses in small samples with a rare state, and bands built from fitted components undercover.
"No remasking method in the main table." We now compare against distributional remasking methods. Across ten refits on Household, Metro and Beijing, our final predictor reduces CRPS relative to TabCSDI by 9.6–21.9% under recording shift. Improvements are significant in all six comparisons—with and without shift—after Holm correction. We also evaluate native DiffPuter on incomplete-source California. Its mixture correction gives no clear improvement under informative recording and worsens performance in the control. We have removed distributional rankings of ReMasker, which produces point predictions.
"The strongest competitors are not run." They are now.
- Boeken's published doubly robust correction. Implemented from the published formula and checked against the authors' code. On the controlled task, it ties, as you anticipated. Brier risk 0.4236 for SFLA, 0.4236 for repeated regression and 0.4237 for the published correction (attainable risk is 0.4200).
- Across 145 OpenML dataset–conditions (statistical learners; better / tied / worse for the final prediction):
| Competitor | Better | Tied | Worse |
|---|---|---|---|
| Boeken repeated regression † | 145 | 0 | 0 |
| Augmented doubly robust mean correction † | 142 | 3 | 0 |
| Distributional repeated regression | 130 | 15 | 0 |
| Joint learner (MICE) | 138 | 7 | 0 |
| Joint learner (FCS) | 131 | 14 | 0 |
| Best prior method, chosen with deployment outcomes | 103 | 42 | 0 |
| Stack of prior methods | 90 | 55 | 0 |
† Mean estimators, scored as distributions by shifting the location of the source distribution.
- On actual TabPFN, distributional repeated regression with a forest outer learner is far behind the final prediction: CRPS is 0.332 against 0.217.
The simple baselines do keep up on the binary task, and we say so. The advantage appears for full distributions, where a mean correction cannot follow a change of shape.
Major questions
Q1. Is SFLA the best way to use the auxiliary sample? No single route is best everywhere, which is why the final prediction combines them. That combination is never worse than the best prior method chosen with deployment outcomes in any of 145 OpenML dataset–conditions (see table above).
- Boeken's doubly robust estimator: evaluated faithfully; cf. above.
- The joint learner: MICE and FCS joint learners that receive the auxiliary (H, C) rows are also in the table above. A comparison with a pretrained joint learner, TabImpute, is still running; we will add it to the revised paper once it finishes.
Q2. Does remasking fail on real data? The gap is now measured against native missing-value handling on a real TFM: SFLA beats Native TabPFN with identical labels by 0.0098 across 29 datasets. Against a dedicated remasking imputer, the final prediction lowers TabCSDI's CRPS by 9.6–21.9% on Household, Metro and Beijing. The formal point stands independently: a model trained only on source records cannot tell apart two deployments that give it identical inputs, while auxiliary deployment records can.
Q3. Natural missingness, and how common is the setting? NHANES provides evidence under naturally occurring screening histories: the survey supplies auxiliary covariates and independently measured outcomes for evaluation. Our final predictor improves on Native by 0.0256 CRPS, and the diagnostic for the mean implication of stability is not rejected. Trial transport gives a concrete motivation for the information split. When planning for a new recruitment population, detailed baseline assessments can be collected from a sample before follow-up outcomes become available, while forecasts use routinely available patient characteristics. This makes the auxiliary sample useful without requiring detailed assessments for every prediction. We evaluate this arrangement in the International Stroke Trial, using separate patients’ recorded outcomes to assess transfer to three held-out countries. Our final predictor reduces joint Brier risk by 11.7–15.3% against Native.
Q4. Are the Household and Metro gains robust?
- Designs tried: the records identify 17 other imposed-recording designs; five have no recorded reason for exclusion. The paper states this history and does not call the five an untouched benchmark.
- Replication: in ten refits, the final prediction improves significantly on all-label Native on Metro; the smaller Household gain is not significant after correction. The 580-run study replicates the effect on 29 independent datasets.
Q5. When should a practitioner apply the correction? Let validation determine whether and how much to correct. Our final predictor evaluates and combines candidates, including Native, using source labels and unlabelled deployment records. The gain estimates are informative on real data:
- TabPFN, 29 datasets (new Figure 2): predicted update gain +0.0148, realised +0.0144, rank correlation 0.95; the sign is right in every dataset where the effect exceeds 0.005. The same holds for the choice between SFLA and Native (rank correlation 0.96).
- NHANES HbA1c: predicted −0.0019, realised −0.0020 [−0.0032, −0.0007]. The estimate flagged the one harmful update in advance.
These estimates rely on stability. When that assumption is doubtful, sensitivity analysis assesses how fragile the recommendation is, and a representative outcome audit provides evidence for checking and revising the prediction.
Q6. What is new beyond Boeken et al.? Three formal results, each with its limit.
- SFLA against mean corrections: SFLA wins when its fitting error is below the mean correction's unavoidable misfit (W2 above). This does not apply against unrestricted distributional regression.
- SFLA against importance weighting: lower variance by a post-stratification term, with known state probabilities.
- Coincidence and choice: with cellwise estimators the three routes coincide. With fitted learners they differ, and the oracle inequality justifies choosing among them by validation.
Q7. Why is DT-FM a separate contribution? DT-FM makes the framework operational for joint continuous prediction through deployment-targeted flow training and joint sampling. We present this as an implementation contribution, without claiming architectural superiority over CFMI. Both flow families now receive the same adaptation, yielding Holm-significant improvements on three of five applications for each family. These results demonstrate that the adaptation benefits extend across flow backbones.
Weakness 1, evidence and competitiveness
Thank you for the very detailed review. It set the agenda for this revision, and we try to answer every point. Your overall concern was that the evidence "does not show that the method beats native foundation models or holds on independent data". We believe that it now does: on 29 datasets outside the submission, with ten random splits each, the final prediction beats Native TabPFN trained on all source labels by 0.0045 CRPS [0.0038, 0.0051], and SFLA beats Native on identical labels by 0.0098 [0.0093, 0.0104] (common response, 2/2).
Imposed recording; no natural missingness. We added NHANES 2015–16, where clinical practice decided who was tested and the survey measured everyone. Gain over Native are 0.0256 [0.0216, 0.0295], with the protocol fixed before the cycle was opened. Gas and Appliances are now labelled as constructed designs with response-derived states.
Household and Metro are single development fits; Electricity is null. The 580-run study replaces single fits with replication on independent datasets, and Household, Metro and Beijing are now refitted ten times. Against all-label Native, the final prediction gains 0.0075 [0.0056, 0.0094] on Metro (significant) and 0.0035 on Household (not significant after Holm); on Beijing it does not improve (−0.0029 [−0.0144, 0.0087]), which we have now included in the revised appendix.
The correction often loses to not correcting. The final prediction keeps Native and uniform contexts as candidates and chooses without deployment outcomes. On TabPFN, it is never worse than its Native candidate (15 datasets better, 14 tied, 0 worse) and beats uniform contexts in 26 of 29 (0 worse). Against all-label Native, the per-dataset record is 8 better, 12 tied, 9 worse. Here, the gain is concentrated where the state is informative (it leads in five of the six datasets with an update gain above 0.01).
Selection is inconsistent. Selection is now part of the method and is evaluated as such. The final TabPFN prediction beats an equally sized stack that is validated the same way, but corrects with source frequencies by 0.0061 [0.0056, 0.0066], with no dataset worse.
DT-FM is weak. We refit DT-FM and CFMI to assess sensitivity to retraining and test whether adaptation benefits both architectures. This gives 60 fits across five applications × two flow families × three routes × two refits. The adapted mixture improves joint energy score over the unadapted flow in 9 of 10 application–family comparisons, and 6 remain significant after Holm correction, three per family. Without recording shift, selected DT-FM and CFMI achieve the lowest mean energy score in 7 of 8 completion queries across four datasets. These results support DT-FM as an effective joint-prediction implementation of the framework, whilst we do not claim architectural superiority.
Missing ablations. We now added:
No measured costs. TabPFN 3.5 on GPU with a 2,048-example context: 9.9–10.6 s per 1,000 queries for Native and 19.4–20.8 s for two-state SFLA, with 4.1–6.7 GiB peak memory. On CPUs, the cost grows more slowly than the number of states, because the context is encoded once: 1.9× Native at four states and 3.8× at sixteen. TabICLv2 and TabDPT are timed as well.