acceptodds
Under review as a conference paper at ICLR 2027

TabACE: Robust Inference in Tabular Foundation Models via Adaptive Context Ensembling

Abstract

Tabular Foundation Models (TFMs) have demonstrated remarkable performance across prediction tasks through their in-context learning capabilities. However, perturbations or shifts in the context can easily cause predictions for a query to fluctuate drastically. Leveraging the configurability of TFM contexts, we propose TabACE (Adaptive Context Ensembling), a model-agnostic and training-free inference strategy to enhance prediction robustness. TabACE aggregates diverse alternative contexts and modifies the default prediction (Base) only when their agreement and a task-specific gate support the update. We provide a theoretical risk bound guaranteeing that this aggregation and integration effectively limits harmful updates. Across OpenML benchmarks spanning five context perturbations and seven distribution shifts, fully synthetic settings, and natural shifts, TabACE consistently mitigates performance degradation across foundation and conventional predictors while preserving clean performance. Additionally, caching reusable context computations provides a nearly sixfold speedup for repeated inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Foundation Models, In-Context Learning, Ensemble Learning, Robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bPdSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TabACE, a training-free wrapper for frozen tabular foundation models. For each query, it runs the backbone on the standard ("Base") context and on R alternative contexts. These alternatives are built by row filtering or reweighting, feature dropping or clipping, kNN label repair, and source-to-target transport. TabACE then aggregates the alternative predictions using weights based on confidence, agreement, and a learned per-constructor prior, and blends the aggregate with Base through a gate α = α_max·U·E. Fusion parameters are calibrated out-of-fold on other datasets.

Critical strengths

Backbone-agnostic design. TabACE wraps frozen predictors without retraining, demonstrating gains across four TFMs (LimiX-2M, TabICL-v2, TabPFN-v3, TabDPT v1.2) and conventional models (kNN, Logistic/Ridge, CatBoost, MLP), making it easy to adopt.

Broad evaluation. The study covers classification and regression, 12 change families at five strengths, approximately 100 OpenML datasets, synthetic experiments with bootstrap confidence intervals, and natural shifts on Folktables.

Informative ablations. Context-budget, aggregation rule, and component ablations isolate key design choices, with Holm-corrected tests for the latter. The gate analysis further clarifies when predictions change. The author also reports failure cases, including a 43.8% worsening rate in one regression setting.

Critical weaknesses

Unjustified context size. The experiments use only 256 context rows without justification, despite several CC18 datasets being much larger (e.g., CIFAR-10 has 60K rows and Devnagari-Script about 92K). This is particularly concerning given that TabICLv2 and TabPFNv3 support much larger contexts (48K rows or more). Such aggressive subsampling may degrade model performance and increase prediction instability, potentially favoring ensembling. The authors should justify this choice and evaluate larger context sizes.

Inadequate ensemble baselines. TabACE is an ensemble method over TFMs, yet the paper does not compare it against established or even naive ensemble methods. A single forward pass with a randomly sampled context is a potentially unstable and weak baseline, especially with only 256 context rows. Thus, the reported gains may reflect generic ensemble benefits rather than TabACE's specific contributions.

Limited generalizability due to perturbation and constructor matching. The perturbations appear closely matched to their corresponding constructors: Shortcut corrupts the top feature, while its constructor drops the top-correlated feature; Regression Concept introduces smooth drift, while its constructor removes the strongest smooth component. Gains diminish when these matched constructors are absent and are most pronounced when shifts appear in the Shared bank. Without such a match (i.e. when the perturbation/shift is not included in the Shared Constructor bank), performance is nearly identical to Base. The gain is also minimal in Folktables settings when the shift is not by design. These results raise concerns about generalizability beyond the specifically matched perturbations and constructors.

Questions for authors
  • Figure 5 reports latency for random contexts but not their accuracy. Could the authors report accuracy to enable a fair comparison?

  • Why use 256 labeled rows for classification but up to 2,048 for regression? Do the reported gains persist with larger context sizes?

  • Folktables evaluation. Which family was specified for the Specific setting? Could the authors report variance across random seeds and compare against DistPFN and Drift-Resilient TabPFN?

  • Regression gate robustness. Since Ureg increases with Base–aggregate disagreement, updates are largest when the alternatives disagree most with Base. How does the method behave when these alternatives are correlated and jointly incorrect?

  • Failure under confident errors. In classification, a low-entropy but incorrect Base prediction closes the gate, potentially preventing useful corrections. How frequently does this occur, and how does it affect overall performance?

8 comments
AuthorsOpenReview ↗

Part 1: Context size and larger-context evaluation

Weakness 1 — justification for the context budget

The experiments use only 256 context rows without justification, despite several CC18 datasets being much larger (e.g., CIFAR-10 has 60K rows and Devanagari-Script about 92K). This is particularly concerning given that TabICLv2 and TabPFNv3 support much larger contexts (48K rows or more). Such aggressive subsampling may degrade model performance and increase prediction instability, potentially favoring ensembling. The authors should justify this choice and evaluate larger context sizes.

Question — classification/regression context budgets and persistence of gains

Why use 256 labeled rows for classification but up to 2,048 for regression? Do the reported gains persist with larger context sizes?

Thank you for highlighting the context-budget choice. We used a compact 256-row setting to keep the broad robustness evaluation computationally manageable, informed by the small-context setting examined in TabDPT [1] (its inference ablation explicitly includes 256 rows). Classification and regression used separate sampling protocols, with up to 2,048 source-context rows for regression. We have now extended the evaluation to common context caps of 1,024, 4,096, and 8,192 rows for both tasks. The following TabICL-v2 results cover 72 classification and 35 regression datasets for clean/corrupted data, and 35 classification and 33 regression datasets for shifts.

Classification (Acc / AUC; Appendix E.2, Tables 20–21, p. 38).

Context capMethodClean (72 datasets)Five corruptions (72)Seven shifts (35)
1024Base0.8254 / 0.92180.5332 / 0.70150.7635 / 0.8474
1024Shared0.8268 / 0.92230.6586 / 0.80440.7762 / 0.8661
1024Specific0.8289 / 0.92290.6917 / 0.83740.8047 / 0.8767
4096Base0.8381 / 0.93040.5446 / 0.71050.7607 / 0.8554
4096Shared0.8402 / 0.93060.6790 / 0.81160.7738 / 0.8724
4096Specific0.8436 / 0.93080.7071 / 0.84330.7982 / 0.8806
8192Base0.8416 / 0.93140.5469 / 0.71070.7625 / 0.8576
8192Shared0.8433 / 0.93150.6814 / 0.81240.7746 / 0.8734
8192Specific0.8456 / 0.93160.7115 / 0.84400.8042 / 0.8824

Regression (R2R^2 / nRMSE).

Context capMethodClean (35 datasets)Five corruptions (35)Seven shifts (33)
1024Base0.7557 / 0.4680−1.0547 / 1.25980.2891 / 0.6659
1024Shared0.7570 / 0.46620.4779 / 0.68030.5816 / 0.5533
1024Specific0.7573 / 0.46330.5968 / 0.60510.7322 / 0.4569
4096Base0.7590 / 0.4647−1.0380 / 1.25600.3265 / 0.6379
4096Shared0.7599 / 0.46270.4800 / 0.67930.5980 / 0.5325
4096Specific0.7604 / 0.46220.5985 / 0.60400.7512 / 0.4306
8192Base0.7596 / 0.4642−1.0317 / 1.25450.3287 / 0.6310
8192Shared0.7603 / 0.46230.4809 / 0.67900.5864 / 0.5328
8192Specific0.7609 / 0.46210.5989 / 0.60380.7649 / 0.4109

Shared and Specific continue to outperform Base under both corruptions and shifts at every evaluated context cap, while maintaining close clean performance. The larger contexts also strengthen Base on clean data, yet the robustness benefits remain. These results directly address the concern that TabACE’s gains arise only from a small, unstable 256-row context.

AuthorsOpenReview ↗

Part 2: Ensemble baselines and random-context accuracy

Weakness 2 — inadequate ensemble baselines

TabACE is an ensemble method over TFMs, yet the paper does not compare it against established or even naive ensemble methods. A single forward pass with a randomly sampled context is a potentially unstable and weak baseline, especially with only 256 context rows. Thus, the reported gains may reflect generic ensemble benefits rather than TabACE’s specific contributions.

Question — accuracy of random-context ensembles

Figure 5 reports latency for random contexts but not their accuracy. Could the authors report accuracy to enable a fair comparison?

Thank you for suggesting these comparisons. We now report the predictive performance of random-context ensembles alongside the native ensemble and learned static weighting. Native-6 enables TabICL-v2’s six-estimator native ensemble; Random-6 averages predictions from six independently sampled contexts; Convex Stacking-6 learns fixed nonnegative weights summing to one on calibration data. Shared uses one Base and five alternative-context predictions. All four ensemble methods use the same backbone and six predictions per query. The table reports classification Acc/AUC and regression R2R^2/nRMSE under clean data, the five corruptions, and the seven shifts.

TaskMethodCleanFive corruptions (avg.)Seven shifts (avg.)
ClassificationBase0.804 / 0.9010.494 / 0.6760.763 / 0.826
ClassificationNative-60.803 / 0.9010.507 / 0.6830.763 / 0.825
ClassificationRandom-60.800 / 0.8970.523 / 0.6960.764 / 0.828
ClassificationConvex Stacking-60.800 / 0.8980.559 / 0.7040.766 / 0.827
ClassificationShared0.804 / 0.9010.575 / 0.7200.772 / 0.831
RegressionBase0.739 / 0.476−1.077 / 1.2700.273 / 0.730
RegressionNative-60.740 / 0.487−0.597 / 1.1450.287 / 0.721
RegressionRandom-60.742 / 0.505−0.397 / 1.0890.351 / 0.702
RegressionConvex Stacking-60.733 / 0.499−0.446 / 1.1040.401 / 0.682
RegressionShared0.738 / 0.4780.363 / 0.7830.467 / 0.659

Shared outperforms native ensembling, random-context averaging, and static learned weighting on both classification metrics and both regression metrics under corruptions and shifts. Random-context ensembling improves over Base in several settings, but does not reproduce Shared’s robustness. This directly addresses the concern that the improvements merely result from repeated backbone predictions with a small context.

We additionally evaluated DENSE [2], GES, robust aggregation, TabPFN PHE [3], RF-PFN, and OligoICP [4]. The complete baseline tables and budget conventions are provided in Appendix E.1, Tables 16–19, pp. 36–37, covering both TabICL and TabPFN classification and regression. The Random-6 row above supplies the predictive-performance counterpart to the random-context latency comparison in Figure 5.

AuthorsOpenReview ↗

Part 3: Generalization beyond perturbation–constructor matching

Weakness 3 — matched mechanisms and limited generalizability

The perturbations appear closely matched to their corresponding constructors: Shortcut corrupts the top feature, while its constructor drops the top-correlated feature; Regression Concept introduces smooth drift while its constructor removes the strongest smooth component. Gains diminish when these matched constructors are absent and are most pronounced when shifts appear in the Shared bank.

Without such a match (i.e., when the perturbation/shift is not included in the Shared Constructor bank), performance is nearly identical to Base. The gain is also minimal in Folktables settings when the shift is not by design. These results raise concerns about generalizability beyond the specifically matched perturbations and constructors.

Thank you for raising this concern. We directly tested whether TabACE remains effective without a matching constructor or calibration exposure to the evaluated mechanism.

Separating calibration coverage from constructor matching (Appendix E.3, Table 22, p. 39). We conduct a 2×22\times2 Shortcut ablation, independently varying these two factors, with the same available supervision across configurations. C/D use nonmatching constructions; B/D exclude Shortcut from policy calibration.

ConfigurationShortcut in calibrationMatched constructorAccAUC
Base——0.517190.65690
AYesYes0.743940.87909
BNoYes0.707770.85370
CYesNo0.711870.85757
DNoNo0.693330.84662

C/D outperform Base without the matched feature-dropping constructor; D retains this advantage even when Shortcut is also absent from calibration. These results show that useful corrections can arise from the remaining contexts rather than requiring a directly matched repair branch.

Problem-aware construction is also a useful option when problem knowledge is available: SPN [5] constructs strategic in-context examples to adapt frozen TFMs to post-manipulation inputs. Specific uses available family knowledge to construct additional candidates, while Shared requires no mechanism identifier and remains effective without dedicated matching constructors, as tested below.

Mechanisms outside the Shared bank (Appendix E.4, Tables 23–24, p. 39). We evaluate three new corruptions and three new shifts, all excluded from policy calibration and without adding dedicated matching constructors. Results average strengths {0.2,0.5,0.8,1.0}\{0.2,0.5,0.8,1.0\} and the listed datasets, with the same available supervision and no additional target labels. The corruptions modify source contexts; rotation and nonlinear sensor transformations modify classification queries or regression source-context features; interaction-dependent sampling reweights complete source rows with their labels.

Unseen mechanismDatasets (Cls./Reg.)Classification Acc: Base / SharedRegression R2R^2: Base / Shared
Heavy-tailed noise64 / 350.7875 / 0.81760.6380 / 0.6757
Column misalignment72 / 350.7824 / 0.81860.6179 / 0.6751
Feature quantization64 / 350.7070 / 0.75760.3743 / 0.5023
Feature rotation31 / 310.7609 / 0.78610.3375 / 0.4663
Nonlinear sensor transformations32 / 330.8002 / 0.82080.1009 / 0.4413
Interaction-dependent sampling35 / 330.8214 / 0.82880.6487 / 0.6860

Shared improves both classification Acc and regression R2R^2 in all six settings. This directly addresses the statement that performance is nearly identical to Base when the evaluated mechanism is absent from the constructor bank: the additional evaluations demonstrate gains under precisely that condition.

Natural shifts. We agree that some Folktables gains are modest and depend on the task and migration setting. We therefore broadened the evaluation to bidirectional hospital-system transfer on PhysioNet Sepsis 2019 and admission-source transfer on UCI Diabetes 130-US Hospitals (non-emergency-room → emergency-room). Shared and Specific improve OOD Acc in all three transfer settings; for example, readmission Acc is 57.27% / 58.97% / 59.24% for Base / Shared / Specific. Together with the unseen-mechanism tests, these clinical results provide further evidence of generalization beyond the specifically matched synthetic conditions. Complete clinical results for the 1:1 case–control evaluation are provided in Appendix E.5.4, Table 27, p. 42.

Reviewer_BV6fOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TabACE is an inference framework for improving the robustness of frozen tabular foundation models. It constructs heterogeneous alternative contexts and combines their predictions using confidence, agreement, and calibrated reliability weights. A query-dependent gate controls the departure from the Base prediction. The paper provides a conditional risk bound and evaluates classification and regression across four foundation models. Validation includes controlled perturbations, distribution shifts, synthetic data, natural shifts, and component ablations. The reported results show reduced degradation with largely preserved clean performance.

Critical strengths
  1. The method addresses an important practical problem. Context sensitivity limits the reliability of tabular foundation models. TabACE separates the selection of useful alternative predictions from the decision to modify Base. This is a reasonable design for balancing correction and unnecessary intervention.

  2. The controlled evaluation provides meaningful evidence of effectiveness. Improvements appear across four backbones. Dataset-level out-of-fold calibration supports transfer to unseen datasets. Fixed-bank ablations provide additional evidence for the components. These results make the method worth considering beyond a single-model demonstration.

Critical weaknesses
  1. The independent contribution of the fusion mechanism remains unclear. DENSE, TabPFN PHE, and OligoICP provide relevant ensemble strategies. The paper needs comparisons that match context membership, calibration information, and inference budget. The mean and median baselines also remove the Base gate, so their gaps do not isolate the weighting rule. The component ablations help, but some signals remain active elsewhere after their weighting coefficients are removed. Specific’s deployed branch counts and selection procedure are also missing. These gaps make it difficult to separate gains from better contexts, learned reliability, and the proposed query-dependent fusion.

  2. The robustness evidence is narrower than the overall results suggest. Calibration covers the same registered mechanisms used in evaluation. The natural-shift study contains only two migration settings and three tasks. Gains also vary substantially across families. Based on Table 15, Domain and Temporal account for roughly 86% of Shared’s total regression improvement across the seven families. The mechanism has relevant limitations: the entropy gate suppresses corrections to confident Base errors, and median-based support scoring does not prevent an extreme regression branch from pulling the weighted mean. These cases matter for the paper’s robustness motivation.

Questions for authors

Q1:Please report the deployed context lists and branch counts for Specific. How are these lists selected? Which optimization settings are used to fit the fusion policy?

Q2: With the same context bank, calibration data, Base gate, and inference budget, how does the full method compare with static learned weights and uncertainty-only weighting? This would clarify the value of query-dependent agreement.

Q3: How often can TabACE correct high-confidence Base errors? How sensitive is the regression output to one extreme alternative prediction? Are branch predictions bounded or clipped?

Q4: Which datasets and conditions are used to calibrate these policies? Please provide the evaluation sample sizes and uncertainty estimates for the reported gains.

8 comments
AuthorsOpenReview ↗

Part 1: Independent contribution of query-dependent fusion

Weakness 1 — matched ensemble comparisons and component isolation

The independent contribution of the fusion mechanism remains unclear. DENSE, TabPFN PHE, and OligoICP provide relevant ensemble strategies. The paper needs comparisons that match context membership, calibration information, and inference budget. The mean and median baselines also remove the Base gate, so their gaps do not isolate the weighting rule. The component ablations help, but some signals remain active elsewhere after their weighting coefficients are removed.

Question 2

With the same context bank, calibration data, Base gate, and inference budget, how does the full method compare with static learned weights and uncertainty-only weighting? This would clarify the value of query-dependent agreement.

Thank you for highlighting the need to distinguish ensemble competitiveness from the contribution of the fusion rule. We address these through comparisons with established ensembles and fixed-bank component ablations.

Comparison with ensemble methods. We added DENSE [1], TabPFN PHE [2], OligoICP [3], and learned static stacking, alongside the native and random-context ensembles. The representative results below use TabICL-v2 or TabPFN-v3, evaluated on classification (Acc/AUC) and regression (R2R^2/nRMSE). The listed ensemble configurations deploy six predictions per query; OligoICP 60→6 selects six members from 60 candidates. Complete results, adaptation markers, and candidate/deployment budget conventions are provided in Appendix E.1, Tables 16–19, pp. 36–37.

BackboneMethodFive corruptions: classificationSeven shifts: classificationFive corruptions: regressionSeven shifts: regression
TabICL-v2DENSE-60.531 / 0.6950.764 / 0.826——
TabICL-v2Convex Stacking-60.559 / 0.7040.766 / 0.827−0.446 / 1.1040.401 / 0.682
TabICL-v2Shared0.575 / 0.7200.772 / 0.8310.363 / 0.7830.467 / 0.659
TabPFN-v3PHE-60.464 / 0.6260.763 / 0.816−0.387 / 1.1110.424 / 0.659
TabPFN-v3OligoICP 60→6——−0.370 / 1.1200.333 / 0.703
TabPFN-v3Shared0.553 / 0.6970.770 / 0.8170.237 / 0.8340.460 / 0.656

We find that Shared outperforms the highlighted established ensembles and static stacking under both corruptions and shifts. These comparisons address the missing-baseline concern; to examine fusion independently of context construction, we turn to the fixed-bank experiment below.

Contribution of query-dependent weighting. Appendix D.3 fixes each fold's five-context bank and uses the corresponding outer-training folds for policy fitting, retaining the remaining aggregation and gate mechanisms when removing one component. The results equally weight the five-corruption and seven-shift tracks after averaging over nonzero strengths.

Fixed-bank configurationClassification AccRegression nRMSE
Uniform prior (πr=1/R\pi_r=1/R)0.6360.835
Without agreement weighting (ηS=0\eta_S=0)0.6390.805
Full fusion0.6720.739

In classification, setting ηS=0\eta_S=0 leaves confidence-based branch weighting and the gate in place. Adding agreement to branch weighting improves performance even when the context bank and the remaining gate mechanisms are retained; all displayed differences from full fusion are significant under the appendix's Holm-corrected paired tests. The uniform-prior comparison further shows that learned context reliability and query-dependent agreement contribute complementary information.

We also appreciate your observation that some signals remain active in other paths. These are weighting-term ablations: setting ηS=0\eta_S=0 removes agreement from branch weighting while retaining it in context evidence, and setting ηQ=0\eta_Q=0 similarly retains confidence in the gate. We have made these definitions explicit in Appendix D.3. This interpretation directly isolates the incremental value of the weighting terms within the deployed architecture; the mean/median comparisons in Appendix D.2 evaluate alternative aggregation pipelines, including their different gating choices.

AuthorsOpenReview ↗

Part 2: Specific context selection and policy optimization

Weakness 1 — deployed branch counts and selection procedure

Specific’s deployed branch counts and selection procedure are also missing. These gaps make it difficult to separate gains from better contexts, learned reliability, and the proposed query-dependent fusion.

Question 1

Please report the deployed context lists and branch counts for Specific. How are these lists selected? Which optimization settings are used to fit the fusion policy?

Thank you for asking us to clarify the deployed configuration. Appendix B.4 specifies the Shared registry (Table 8), family-directed Specific constructors (Tables 9–10), and candidate selection and aggregation (B.4.6); Appendix B.5, Table 11, summarizes implementation settings. We summarize the TabACE settings below.

Candidate construction and selection. For a declared family zz, Specific retains the Shared registry and adds the registered family-directed candidates:

\qquad R_z=|\Phi_z|, \qquad N_{\mathrm{passes}}=1+R_z.$$ Here $\Phi_z$ is the member list selected on the outer-training datasets and frozen before evaluating the held-out dataset. The candidate pool and deployed list are distinct: registered coverage, repair, feature-subspace, and transport levels define separate candidates, while $R_z$ counts only deployed alternatives. - **Information supplied**: Declared family $z$, labeled source context, and available unlabeled target features; no target labels or corruption-strength input - **Representative family-directed candidates**: Shortcut: predictive-feature removal, weak-feature subspaces, clipping, and stable-row filtering; Boundary: neighborhood label repair and stable subsets; Domain/Covariate: target matching and location-scale or quantile transport - **Selection**: Five-fold dataset-level OOF; select the member list and fit the family-specific policy using the other four folds, then freeze both for the held-out dataset - **Policy objective**: Classification cross-entropy; regression squared error, over calibration episodes with strengths $\{0,0.2,0.5,0.8,1\}$ - **Parameters fitted**: Context priors $\pi_r$, evidence coefficients $\eta_{Q,r},\eta_{S,r}$, and integration cap $\alpha_{\max}$; regression additionally calibrates $\tau_y$ and uses agreement coefficients **Fusion-policy fitting.** Equation 46 defines calibration of the final prediction loss with the backbone frozen. The parameterization uses softmax for normalized context priors, softplus for nonnegative evidence coefficients and positive regression scale, and sigmoid for $\alpha_{\max}\in[0,1]$. **The family identifier determines the admissible constructor registry; the selected member list and fusion policy are fixed before held-out inference.** The Shortcut constructor-matching ablation and the uniform-prior and agreement-weighting ablations in Appendix D.3 show how **context construction supplies effective alternative predictions, learned reliability priors reduce the influence of unreliable branches, and agreement weighting favors predictions supported by other branches for the current query**. We now explicitly report the context categories and branch-count convention in Appendix B.4.5: Shared uses five alternatives plus Base for both classification and regression; Specific uses the frozen selected list $\Phi_z$, with $R_z=|\Phi_z|$ alternatives and $1+R_z$ predictions including Base. Appendix E.5.1, p. 40 enumerates the deployed Folktables list: five Shared alternatives and eight Specific alternatives, giving six and nine total predictions, respectively. Candidate selection for the main benchmark is described in Appendix B.4.6.
AuthorsOpenReview ↗

Part 3: Generalization coverage and distribution of gains

Weakness 2 — calibration coverage, natural shifts, and concentrated regression gains

The robustness evidence is narrower than the overall results suggest. Calibration covers the same registered mechanisms used in evaluation. The natural-shift study contains only two migration settings and three tasks. Gains also vary substantially across families. Based on Table 15, Domain and Temporal account for roughly 86% of Shared’s total regression improvement across the seven families.

Thank you for pointing out both the calibration coverage and the uneven distribution of gains. We have added mechanism-held-out ablations, six unseen mechanisms, and additional natural-shift evaluations, and examine the regression results by family.

Transfer beyond calibration and constructor matching (Appendix E.3, Table 22, p. 39). We conduct a 2×22\times2 Shortcut experiment, independently varying whether calibration includes Shortcut and whether a matched constructor is available, with the same available supervision across configurations.

ConfigurationShortcut in calibrationMatched constructorAccAUC
Base——0.517190.65690
AYesYes0.743940.87909
BNoYes0.707770.85370
CYesNo0.711870.85757
DNoNo0.693330.84662

B/D outperform Base when Shortcut is absent from calibration, and D also excludes the matched constructor, showing that the benefit extends beyond direct calibration–constructor matching.

We further evaluate Shared on three unseen corruptions and three unseen shifts, all excluded from policy calibration and supplied with no dedicated matching constructors or mechanism identifiers, at strengths {0.2,0.5,0.8,1.0}\{0.2,0.5,0.8,1.0\}. Shared improves classification Acc and regression R2R^2 in all six settings, extending the evidence beyond the registered calibration mechanisms. The complete results are provided in Appendix E.4, Tables 23–24, p. 39.

Additional natural shifts. Beyond the Folktables geographic and temporal evaluations, we test bidirectional hospital-system transfer on PhysioNet Sepsis 2019 and non-emergency-room to emergency-room admission-source transfer on UCI Diabetes 130-US Hospitals. The table reports OOD Acc for the 1:1 case–control evaluation (Appendix E.5.4, Table 27, p. 42).

DatasetTransfer taskBaseSharedSpecific
PhysioNet Sepsis 2019Sepsis A→B61.59%62.14%63.04%
PhysioNet Sepsis 2019Sepsis B→A63.79%64.60%64.91%
UCI Diabetes 130-US HospitalsReadmission57.27%58.97%59.24%

Shared and Specific improve Base in all three medical OOD settings, extending the natural-shift evidence beyond Folktables.

Family-specific gains. We agree that the magnitude of improvement varies substantially across families, with Domain and Temporal contributing the largest share of the aggregate regression gain. Appendix Table 15 already reports each family separately: averaged over the four nonzero strengths, Shared improves R2R^2 over Base in all seven families. Together with the mechanism-held-out and clinical results above, this shows that the benefit extends beyond the two families contributing most to the aggregate improvement.

Reviewer_SbVzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TabACE, an ensemble method where the same tabular foundation model (TFM) is run on few alternative contexts obtained with some augmentation. Examples of augmentations are resampling, filtering or transforming the labeled rows. The "alternative" predictions are merged together with the original prediction with a weighted average, where the weights are determined a learned per-context prior, confidence and agreement. The evaluation includes four TFMs on OpenML benchmarks with synthetic corruptions and shifts, on fully synthetic data, and on Folktables natural shifts. Importantly, it also improves non-neural methods such as CatBoost.

Critical strengths
  1. The method is simple, training-free and model-agnostic. It also improves conventional predictors (e.g., kNN, CatBoost) (Tab 3).
  2. Performance with a uncorrupted context is maintained, while the improvements under the considered corruptions and shifts are large on average.
  3. The evaluation is broad, and on OpenML the weighting policy is always calibrated on the datasets not being evaluated.
Critical weaknesses
  1. The evaluation seems circular: the corruptions are introduced by the authors, and the proposed alternative contexts look like direct countermeasures. From my understanding, the experiments show that a policy fitted to these corruptions can partly undo them, however, it remains unclear how well it generalizes to other unseen corruption. For example:
  • The Shortcut corruption corrupts the most predictive feature, and one of the five Shared classification contexts simply drops that feature.
  • The weighting policy is calibrated on the same corruption types and strengths
  • the Specific variant is even told which one is present.
  • The only natural-shift benchmark is Folktables, where the improvements are mostly small and half of the times similar to the improvements observed on the unshifted test data, suggesting a generic ensembling effect.
  1. I am also not sure why these context operations/augmentation should be hardcoded at inference time instead of learned. I think a baseline is a TFM fine-tuned on contexts with the same corruptions, which should learn to ignore those and possibly to generalize other similar corruptions.
  2. The comparisons should report the performance when ensembling also the baselines, i.e., the main backbone (TabICL-v2) is used a single estimator, without its own ensembling.

I am willing to raise my score if: (a) TabACE can generalize across shifts unseen during calibration, and (b) a comparison with a TFM fine-tuned with similar data-augmentations is added.

Questions for authors
  1. How does TabACE perform on corruptions unseen during calibration? And does it work across other real-world shifts, such as in the medical domain between eICU and MIMIC?
  2. Would a TFM fine-tuned on contexts with the same corruptions be as robust, and does TabACE still improve its performance?
  3. How does TabACE compare with the base model using its own ensembling at the same number of forward passes?
7 comments
AuthorsOpenReview ↗

Part 1: Generalization beyond calibration and constructor matching

Weakness 1 — corruption–constructor matching and calibration coverage

The Shortcut corruption corrupts the most predictive feature, and one of the five Shared classification contexts simply drops that feature.

The weighting policy is calibrated on the same corruption types and strengths

Thank you for raising this important question about whether the gains depend on matching the evaluation conditions. We have examined this directly through the following additional experiments.

Calibration and constructor matching. To examine these two factors directly, we conducted a 2×22\times2 Shortcut ablation, complementing our weighting-component ablations. We exclude Shortcut from policy calibration in B/D and use nonmatching context constructions in C/D, with the same available supervision across conditions. We find that C achieves 0.71187/0.85757 Acc/AUC; even with both calibration coverage and the matched constructor absent, D achieves 0.69333/0.84662, compared with Base's 0.51719/0.65690.

ConfigurationShortcut in calibrationMatched constructorAcc ↑\uparrowAUC ↑\uparrow
Base——0.517190.65690
AYesYes0.743940.87909
BNoYes0.707770.85370
CYesNo0.711870.85757
DNoNo0.693330.84662

We believe these results directly address both concerns: constructor matching and calibration on the evaluated corruption types. C/D outperform Base without the matched feature-dropping constructor, showing that the benefit extends beyond direct corruption–constructor pairing. B/D also outperform Base when Shortcut is excluded from calibration; together with the six unseen-mechanism results below, this demonstrates that the learned weighting policy transfers to mechanisms absent from calibration.

Weakness 1 — mechanism information supplied to Specific

the Specific variant is even told which one is present.

Specific and Shared. We would like to clarify the different deployment settings of the two variants. We designed Specific to use mechanism knowledge when it is available, allowing additional contexts to reflect the problem structure. Problem-aware context construction also appears in SPN (Lv et al., ICML 2026) [1], which constructs examples for strategically modified inputs. We also evaluate Shared, which receives no mechanism identifier and uses fixed constructor rules. Thus, Specific provides an option for exploiting available problem knowledge, while Shared establishes performance when that knowledge is unavailable. In the experiments below, we further show that Shared consistently outperforms Base under three unseen corruptions and three unseen shifts.

Question 1 — generalization to corruptions unseen during calibration

How does TabACE perform on corruptions unseen during calibration?

Unseen mechanisms. We further tested Shared on three additional corruptions and three distribution shifts, excluding all six mechanisms from policy calibration and providing no dedicated matching constructors. We evaluated strengths {0.2,0.5,0.8,1.0}\{0.2,0.5,0.8,1.0\} and report averages over these strengths and the datasets listed below. All comparisons use the same available supervision, with no additional target labels or mechanism identifiers.

We apply the three corruptions to source contexts. For feature rotation and nonlinear sensor transformations, we modify query features in classification and source-context features in regression. Interaction-dependent sampling reweights complete source rows while retaining their associated labels.

Unseen mechanismDatasets (Cls./Reg.)Base: Acc / AUCShared: Acc / AUCBase: R2R^2 / nRMSEShared: R2R^2 / nRMSE
Heavy-tailed noise64 / 350.7875 / 0.89320.8176 / 0.90680.6380 / 0.54170.6757 / 0.5161
Column misalignment72 / 350.7824 / 0.89200.8186 / 0.91000.6179 / 0.55910.6751 / 0.5170
Feature quantization64 / 350.7070 / 0.87820.7576 / 0.89020.3743 / 0.70700.5023 / 0.6452
Feature rotation31 / 310.7609 / 0.82170.7861 / 0.84200.3375 / 0.75270.4663 / 0.6810
Nonlinear sensor transformations32 / 330.8002 / 0.85240.8208 / 0.86000.1009 / 0.82350.4413 / 0.6665
Interaction-dependent sampling35 / 330.8214 / 0.86760.8288 / 0.87790.6487 / 0.53500.6860 / 0.5042

We observe that Shared outperforms Base on all four metrics in every setting, despite receiving no indication of which mechanism is present. These results directly answer your question about unseen corruptions: Shared retains its benefits beyond the mechanisms represented in calibration and beyond those addressed by dedicated constructors.

AuthorsOpenReview ↗

Part 2: Robustness under real-world distribution shifts

Weakness 1 — natural shifts and generic ensembling effects

The only natural-shift benchmark is Folktables, where the improvements are mostly small and half of the times similar to the improvements observed on the unshifted test data, suggesting a generic ensembling effect.

Thank you for asking us to examine whether the natural-shift gains can be explained by ordinary ensembling. We repeated the Folktables experiments and observed consistent gains across geographic shifts (California →\rightarrow Michigan, 2018) and temporal shifts (California, 2014 →\rightarrow 2018), on Employment, Income, and TravelTime. We also compared against DistPFN–TabICL v2 and Drift-Resilient, using the same geographic and temporal task definitions; DistPFN uses TabICL v2 as its backbone (Appendix E.5.2, Table 25, p. 41).

ShiftTaskDomainBaseDistPFN–TabICL v2Drift-ResilientSharedSpecific
GeographicEmploymentID0.79220.78920.78850.79520.8089
GeographicEmploymentOOD0.80430.80340.80220.81060.8245
GeographicIncomeID0.78880.79300.78560.81180.8201
GeographicIncomeOOD0.76550.77680.77330.78230.7914
GeographicTravelTimeID0.59120.57310.57230.62250.6569
GeographicTravelTimeOOD0.54720.58020.56670.58180.6012
TemporalEmploymentID0.77040.77700.76990.78610.7994
TemporalEmploymentOOD0.78470.78040.77420.79220.8030
TemporalIncomeID0.79260.80090.80100.81260.8240
TemporalIncomeOOD0.78440.78940.78670.80000.8106
TemporalTravelTimeID0.59510.59370.59560.62370.6526
TemporalTravelTimeOOD0.59080.57310.56920.60450.6350

We find that both Shared and Specific outperform Base, DistPFN–TabICL v2, and Drift-Resilient in all 12 ID/OOD comparisons, with Specific giving the best accuracy throughout. These repeated results show that the gains persist across tasks and both types of natural shift, rather than arising from an isolated favorable setting.

We address the generic-ensembling explanation directly in Part 4, using native ensembling, random-context ensembling, and learned static weighting under the same prediction budget.

Question 1 — other real-world shifts

And does it work across other real-world shifts, such as in the medical domain between eICU and MIMIC?

Thank you for suggesting medical shifts. We do not yet have access authorization for eICU and MIMIC, so we evaluated two publicly available clinical datasets: PhysioNet Sepsis 2019 and UCI Diabetes 130-US Hospitals. For Sepsis, we test hospital-system transfer in both directions, A →\rightarrow B and B →\rightarrow A; ID patients come from the source system and OOD patients from the target system. For readmission, we follow the TableShift admission-source domain definition: non-emergency-room admission sources form the source/ID domain, while emergency-room admissions form the OOD domain. We report Accuracy on 1:1 case–control evaluations (Appendix E.5.4, Table 27, p. 42).

DatasetTaskDomainBaseSharedSpecific
PhysioNet Sepsis 2019Sepsis A →\rightarrow BID70.20%70.71%71.72%
PhysioNet Sepsis 2019Sepsis A →\rightarrow BOOD61.59%62.14%63.04%
PhysioNet Sepsis 2019Sepsis B →\rightarrow AID76.19%77.38%78.57%
PhysioNet Sepsis 2019Sepsis B →\rightarrow AOOD63.79%64.60%64.91%
UCI Diabetes 130-US HospitalsReadmissionID54.82%55.26%56.87%
UCI Diabetes 130-US HospitalsReadmissionOOD57.27%58.97%59.24%

We observe Specific > Shared > Base in all six medical ID/OOD settings. These results directly address your request for evidence beyond Folktables: we obtain improvements under naturally occurring hospital-system and admission-source shifts, without constructing synthetic corruptions for these evaluations.

AuthorsOpenReview ↗

Part 3: Comparison and complementarity with robust fine-tuning

Weakness 2 — inference-time context construction versus learning

I am also not sure why these context operations/augmentations should be hardcoded at inference time instead of learned. I think a baseline is a TFM fine-tuned on contexts with the same corruptions, which should learn to ignore those and possibly to generalize other similar corruptions.

Question 2 — robustness after fine-tuning and additional TabACE gains

Would a TFM fine-tuned on contexts with the same corruptions be as robust, and does TabACE still improve its performance?

Thank you for suggesting this comparison. FT (fine-tuning) denotes updating the pretrained backbone on corruption-augmented training episodes. We fine-tuned TabICL-v2 using 20% clean and 80% matched-corruption training episodes, with strengths {0.2,0.5,0.8,1.0}\{0.2,0.5,0.8,1.0\}. We updated the last four ICL blocks and prediction components using AdamW, selected checkpoints on held-out training data, and excluded the final evaluation queries from training and selection. We then froze the FT checkpoints and applied Shared or Specific at inference time, yielding FT+Shared and FT+Specific (Appendix E.6, Table 28, p. 42).

Within this comparison, we evaluate all methods on the same dataset splits and query sets, under identical perturbation/shift conditions and strength grids, using the same evaluation metrics. The detailed supplementary protocol is given in Appendix E.6, Table 28, p. 42.

TaskMethodCleanFive corruptionsSeven shifts
ClassificationBase0.8033 / 0.90070.4973 / 0.67290.7634 / 0.8246
ClassificationShared0.8048 / 0.90300.5818 / 0.72530.7747 / 0.8392
ClassificationSpecific0.8041 / 0.90130.6132 / 0.76610.8197 / 0.8576
ClassificationFT0.7970 / 0.89980.5399 / 0.69780.7687 / 0.8301
ClassificationFT+Specific0.8018 / 0.90200.6135 / 0.77060.8220 / 0.8682
RegressionBase0.7425 / 0.4803−1.0763 / 1.26850.2746 / 0.7288
RegressionShared0.7400 / 0.48250.3637 / 0.78450.4652 / 0.6564
RegressionSpecific0.7519 / 0.47100.4855 / 0.70690.6531 / 0.5365
RegressionFT0.6942 / 0.52750.3102 / 0.84940.3897 / 0.6898
RegressionFT+Specific0.7253 / 0.49940.4401 / 0.71990.6101 / 0.5693

We agree that FT is effective against the tested corruptions. We also find that Shared already achieves comparable or better performance without backbone fine-tuning. Adding Shared or Specific further improves the FT model, while the combined variants reach broadly similar performance levels to their direct inference-time counterparts. We therefore believe these results address your question: inference-time context construction remains valuable when corruption-based fine-tuning is available, and the two approaches can also be combined.

We additionally measured the training overhead. We ran 175 dataset–track fine-tuning jobs on eight NVIDIA GeForce RTX 5090 GPUs (32 GB each), with one GPU per job. Training runs for the selected models averaged 41.4 seconds per model; the recorded candidate training runs required 147.8 GPU-minutes in total, summed across GPUs rather than measured as wall-clock time. Applying TabACE directly to the pretrained backbone avoids this additional backbone-optimization stage. Its computational advantage here is therefore lower upfront adaptation cost while achieving broadly comparable predictive performance.

For comparison, the paper reports Shared median inference latencies of 872 ms for classification and 141 ms for regression, using one Base and five alternative-context passes. Direct TabACE requires no backbone fine-tuning, avoiding the training cost above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Discussion Summary

We thank the reviewers for carefully reading our paper and offering thoughtful suggestions. We appreciate your recognition of TabACE's simple, backbone-agnostic design (SbVz and bPdS), the practical relevance of context sensitivity (BV6f), and the breadth of the evaluation and value of the ablations (BV6f and bPdS). SbVz also noted that the method maintains clean performance while improving robustness under corruptions and shifts.

Your feedback guided additional comparisons, broader generalization tests, and a closer examination of the method's behavior. We summarize these updates below; new appendix material is highlighted in blue.

Comparisons and context budgets

  • Ensemble baselines (SbVz, BV6f, bPdS). We now compare native and random-context ensembling, static learned weights, DENSE, PHE, RF-PFN, and OligoICP across TabICL/TabPFN classification and regression, specifying deployment budgets and adaptation settings (Appendix E.1, Tables 16–19).

  • Larger contexts (bPdS). We evaluate Base, Shared, and Specific with 1,024, 4,096, and 8,192 context rows under clean data, corruptions, and shifts for both tasks (Appendix E.2, Tables 20–21).

  • Fine-tuning (SbVz). We test corruption-based fine-tuning alone and with Shared/Specific, reporting predictive performance and training overhead to assess their complementarity (Appendix E.6, Table 28).

Generalization across mechanisms and domains

  • Unseen mechanisms (SbVz, BV6f, bPdS). A 2×2 Shortcut ablation separates calibration coverage from constructor matching. We also evaluate three unseen corruptions and three unseen shifts without dedicated matching constructors (Appendices E.3–E.4, Tables 22–24).

  • Natural and clinical shifts (SbVz, BV6f, bPdS). We extend Folktables comparisons to DistPFN–TabICL-v2 and Drift-Resilient TabPFN, report repeated-run variability, and evaluate clinical transfer on PhysioNet Sepsis 2019 and UCI Diabetes 130-US Hospitals (Appendix E.5, Tables 25–27).

Method analysis and implementation details

  • Failure modes and refinements (BV6f, bPdS). We measure the frequency and impact of high-confidence Base errors, extreme regression predictions, and correlated branch errors, and evaluate improvements with frozen backbones (Appendix E.7, Tables 29–31).

  • Deployment and ablations (BV6f, bPdS). We detail context categories, branch counts, candidate selection, and policy fitting, and clarify which weighting terms each ablation removes (Appendices B.4–B.5 and D.3).

We provide point-by-point responses to each review below. We believe these revisions address most of the concerns raised, and we thank the reviewers for their constructive suggestions and welcome further discussion. We would also appreciate the Area Chair’s assessment of how the additional results and clarifications address the main concerns raised in the reviews.

Note for the Area Chair

For ease of assessment, we briefly summarize the current discussion status.

Reviewer SbVz explicitly stated that they would be willing to raise their score if (i) TabACE could generalize across shifts unseen during calibration and (ii) a comparison with a TFM fine-tuned using similar data augmentations was added. We directly addressed both conditions through the unseen-mechanism, unseen-shift, and clinical-transfer evaluations (Appendices E.3–E.5, Tables 22–27), as well as the fine-tuning comparison (Appendix E.6, Table 28). We also added compute-matched ensemble baselines (Appendix E.1, Tables 16–19) to address the related comparison concern. Although we have not yet received a follow-up from Reviewer SbVz, we believe that these additional results substantially address the key concerns and the specific conditions identified in their review.

(Reviewer SbVz subsequently provided follow-up comments before the deadline, and we responded with further clarifications and additional experiments. Owing to the limited remaining discussion time, we could not complete every newly requested comparison experiment. If the paper is accepted, we will complete these comparisons and incorporate the results and corresponding analysis into the camera-ready version.)

Reviewer BV6f stated that most of their concerns had been addressed, while Reviewer bPdS stated that the larger-context evaluations, budget-matched ensemble comparisons, and unseen-mechanism tests addressed their principal concerns.

We would be grateful for the Area Chair’s assessment of the complete evidence and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.