acceptodds
Under review as a conference paper at ICLR 2027

Beyond Oracle Headroom: Evaluating Context Selection in Tabular Foundation Models

Abstract

Better contexts in hindsight need not yield better decisions at deployment. We study this gap in tabular foundation models using TabPFN and TabICL. Development evaluations show mean Raw Oracle Headroom of 0.318 in negative log-likelihood (NLL), yet expanding a nested candidate bank increases headroom without corresponding gains for the tested selectors. An exact decomposition separates fixed candidate-rule advantage from observed per-query switching opportunity and shows that similar headroom can arise from different candidate-loss structures across datasets. On ten prospectively selected holdout datasets, simple geometric pre-outcome policies beat uniform random choice by 5.69–8.89 percentage points among bank-positive queries. However, post-hoc frequency matching and a direct P2 fixed-rule comparison leave added rule-identity assignment value unestablished. P2 also has exact score ties on 99.80% of queries: tied contexts often share score-determining bottlenecks yet yield different predictions, and its lexicographic tie-break outperforms uniform choice among minima on both models. Once bank predictions are available, predictive centrality (PC) improves over CRUMB in the original formal test, while an advantage over same-bank averaging is not established. On the fresh holdout, gains over averaging are supported only for TabICL, and neither backbone establishes an advantage over CRUMB. Mean gains can also coexist with losses on many individual queries. Together, these results separate hindsight opportunity, policy-level success, and deployment value, showing that context-selection claims depend critically on the comparator and decision stage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation models, in-context learning, context selection, distribution shift, oracle evaluation, reliability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YMMCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds having a context that performs well does not mean we can identify or use it in practice. Over fixed bank of 16 context authors report development headroom, headroom splits as H = G + S, pre-inference policies gains mostly come from choosing the same rule again and again and post-inference methods shows consistent gains.

Critical strengths
  1. The paper uses strong controls and separates different evidences, revealing that P2's gains is because of choosing a fixed rule.
  2. The paper compares complex selection method against probability averaging, showing extra computation can bring limited benefit and harm performance on many queries.
Critical weaknesses
  1. Novelty is limited, paper mainly reports negative results, without proposing new method or how theory can be turned into practical gains.
  2. Experiments use small support pools and hand-build candidate banks and does not reflect read-world use of TabPFn and TabICL on larger datasets. It's unclear whether findings generalize to real settings.
Questions for authors
  1. On fresh holdout, how does always using CLASS_BALANCED_KNN compare with the development selected fixed reference (STANDARDIZED_KNN/RANDOM_SLOT_6) in NLL?
  2. Does the conclusion, little added value from per-query selection beyond a good fixed rule or averaging, hold with larger support pools say 1000-3000 rows, and larger K?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YMMC

We thank the reviewer for the careful reading and the two concrete questions. Section and appendix numbers refer to the revised PDF. All numbers are NLL differences with 95% dataset-bootstrap intervals unless stated; positive favours the first-named method.

W1: "mainly negative results, without a new method or how theory turns into practical gains."

The paper's contribution is the evaluation logic: a selection strategy must beat the strongest alternative that uses the same information, and hindsight headroom does not tell whether it will. The revision adds a case where this logic leads to a gain. Row-level signals failed to identify the right context; the query's observed-feature set does identify what its context should contain. Constructing the context from it (Sec. 7) improves on full support by +0.051 [+0.018, +0.096] on four backbones and beats a mismatched construction of the same form by +0.063 [+0.024, +0.117]. We state its limits plainly: the underlying principle dates to 2007, the cost is 16 to 55 times that of full support, and on TabPFN-3.5 a cost-matched NaN-masking ensemble comes close. ICRM constructs contexts along the feature dimension; it does not rescue row selection, whose negative findings stand.

W2: small support pools and hand-built banks.

We added pools of 1,024 to 3,072 rows with K = 256/512 on thirteen datasets and four backbones (Sec. 6.2, App. K), and compared against full support, which uses no bank at all. App. G.2 varies how the bank is composed (resampled candidate identities; mostly random candidates): the changes are largely unresolved, so we do not claim invariance to bank construction. ICRM needs no candidate bank.

Q1: always using CLASS_BALANCED_KNN versus the development-selected reference on the fresh holdout.

CBKNN is better on all four backbones: TabPFN v2 +0.043 [+0.001, +0.100], TabPFN-3.5 +0.046 [+0.001, +0.106], TabICL v1 +0.068 [+0.027, +0.116], TabICLv2 +0.036 [+0.010, +0.064] (App. K.2). P2 selects that rule on 99.22% of queries, and no additional gain of P2 over fixed CBKNN was established. One caveat: on the ten additional CC18 datasets (App. K.2) the CBKNN advantage over the development reference is close to zero for the two TabPFN backbones (about -0.002 and -0.000) and positive for the two TabICL backbones (+0.035 and +0.050, only the last interval excluding zero), so we do not claim a general ranking of fixed rules.

Q2: does "little added value beyond a good fixed rule or averaging" hold with 1,000 to 3,000 rows and larger K?

Yes, at pool 3,072 and K = 512 (thirteen datasets, first seed).

(1) P2 adds no established gain over fixed CBKNN on any backbone (e.g. TabPFN v2 +0.0005 [-0.0001, +0.0015]).

(2) Predictive-centrality selection and averaging are equivalent within the prespecified margin of 0.01 NLL on all four backbones (differences 0.0014 to 0.0030; all 90% intervals inside the margin).

(3) Averaging beats a strong 64-estimator class-balanced context only on TabICLv2 (+0.029 [+0.005, +0.056]).

(4) Full support (all 3,072 rows) improves on averaging on three backbones (+0.014 to +0.018) but not on TabICLv2.

The limitation is that this primary comparison uses one seed.

AuthorsOpenReview ↗

Follow-up on the revised experiments and remaining concerns

Thank you again for the detailed review. We wanted to briefly follow up because the revision now directly addresses the two experimental questions you raised: the fresh-holdout fixed-CBKNN comparison and the 1,000–3,000-row / larger-K setting, in addition to the new feature-dimension construction in Sec. 7.

If you have had a chance to look at the revision, we would greatly appreciate knowing whether these results resolve your main concerns, or whether there is a remaining issue that you think is still central to your assessment.

Thank you for the revision. The additional experiments addresses my scalability concern and shows that row-level selection offers limited benefit over simpler methods. The fixed-CBKNN results suggest P2/P3 gains mainly comes from rule quality, which should be clarified in the abstract.

My novelty concern still remains, and I suggest narrowing the claims in Section 7 regarding natural missingness. Given the careful experiments, I have raised my presentation and overall scores to 3.

Reviewer_DxgjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether oracle headroom in context selection can translate into practical improvements for tabular foundation models. Using TabPFN and TabICL, the authors analyze different sources of oracle gains and evaluate both pre-inference selection policies and post-bank predictive centrality (PC). The results show that large oracle gains do not necessarily lead to better deployable selection strategies.

Critical strengths

1.The paper addresses an important but often overlooked question: whether oracle improvements are actually achievable at inference time.

2.The experimental analysis is fairly thorough. I particularly appreciate the fresh holdout evaluation, comparisons with probability averaging, and discussion of negative results.

Critical weaknesses

1.The novelty is somewhat limited. The decomposition of oracle headroom is straightforward, and the main conclusion is closely related to known observations in algorithm selection. The paper could better clarify what general insights go beyond the specific context banks studied.

2.The practical value of the proposed selection strategies remains unclear. P2 selects the same candidate rule on 99.22% of queries and shows no established improvement over that fixed rule. PC also fails to consistently outperform probability averaging, despite the additional inference cost.

3.The main conclusions are still tied to a limited set of backbones and experimental settings. Several supplementary analyses are post-hoc, making the generalizability of the findings harder to assess.

Questions for authors

Please see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer Dxgj

We thank the reviewer for the constructive assessment, and for noting the fresh-holdout evaluation and the comparisons with probability averaging. Section and appendix numbers refer to the revised PDF. All numbers are NLL differences with 95% dataset-bootstrap intervals unless stated; positive favours the first-named method.

W1: limited novelty; what general insight goes beyond the specific banks?

We agree that the headroom decomposition is an accounting identity and that the oracle/achievable distinction is known from algorithm selection; the revision says so explicitly. What the paper adds are controls specific to tabular foundation models, and what they show:

(1) A fixed rule already builds a different context per query, so beating random choice does not show useful query-to-rule assignment. P2 beats random but chooses one rule on 99.22% of queries and adds +0.0001 [-0.0005, +0.0010] over fixing it.

(2) A learned router's query-specific signal did not transfer. The positive TabICL v1 interaction (+0.011) became -0.004 [-0.016, +0.008] on twelve datasets held out from fitting, although the router changed its choice on 85% of queries.

(3) Selection versus aggregation of identical predictions. Selection is equivalent to averaging within a prespecified margin at the largest budget, and averaging itself does not consistently beat a strong single-context ensemble.

(4) The same evaluation principle can return a positive answer. For a different pre-inference signal, the query's observed features, the matched construction beats full support (+0.051) while a mismatched construction of the same form does not (-0.014). The interventions are not identical to the row-rule assignment tests, but the principle is the same: compare with the strongest alternative that uses the same information, and break the query correspondence as a control.

The general insight is that deployable value depends on whether information available when choosing determines what the context should contain, not on the size of the hindsight gap.

W2: practical value of P2 and PC is unclear.

We agree, and the revision no longer presents them as recommended methods. P2's apparent gain over random is largely a fixed-rule preference; PC is equivalent to averaging at the largest budget and worse than it on 29% of queries in the original test. Their role is as test cases. The practical findings are:

(1) A strong fixed rule or averaging captures what the tested selectors capture.

(2) Full support is a strong alternative whenever it fits, in the settings studied.

(3) Under query-side missingness, matching the context to the query's observed features gives a large gain at a large inference cost. Using the model's native missing-value indicator with NaNs in the context gives a smaller gain (+0.031 [+0.004, +0.070] over full support) at no extra cost on TabPFN backbones.

W3: few backbones and settings; several analyses are post hoc.

Revision experiments extend the comparisons to four backbones, supports of up to 3,072 rows, and additional datasets. We separate prespecified from post-hoc evidence throughout. The formal test, the fresh holdout's conversion comparison, and the targeted router repetition had rules fixed before their outcomes were evaluated. The ICRM validation and its later controls had rules fixed before their own outcomes were evaluated; the controls reuse the validation datasets and are not independent holdouts. Analyses added afterwards are labelled as such, and unmet criteria are reported (App. K).

Thank you for the detailed response and the additional experiments.

I appreciate the authors' efforts to address my concerns.

The new experiments with four backbones and larger context sizes improve the experimental evaluation. The new ICRM results are also interesting and provide useful evidence for the value of query-specific information.

Overall, I think the revised paper is stronger, and the authors have addressed several of my concerns. Therefore, I am willing to increase my rating from 2 to 3, although some limitations remain.

Reviewer_8okWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper proposes an empirical analysis of context selection on predictive performance in tabular foundation models. The authors consider several context selection strategies, including random sampling, knn and CRUMB. The evaluations, which focus on TabPFN v2 and TabICL v1, shows the effectiveness depend on decision stage.

Critical strengths

none

Critical weaknesses
  • The paper is really hard to read in its current form
  • The experiments lack support of recent tabular foundation models with larger context size
  • The paper discussion seems to focus exclusively on performance gain while context selection strategies have also been proposed to address other issues
Questions for authors

cf. critical weaknesses

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8okW

We thank the reviewer for the direct feedback. We have revised the paper substantially in response to each of the three weaknesses. Section and appendix numbers refer to the revised PDF. Numbers are NLL differences; positive favours the first-named method.

W1: the paper is hard to read.

We rewrote the abstract, introduction, framework, results, and conclusion. Concretely:

(1) The introduction now starts from the observed phenomenon (a 0.32-nat hindsight gap) and the reason it is ambiguous, rather than from a list of findings.

(2) Sections 4 to 6 are each titled by the question they answer and end by stating what remains open.

(3) The two-stage logic (before versus after candidate inference) is introduced once in Sec. 2 and then followed in order.

(4) Numbers are reported at the precision the argument needs, and methods are introduced by what they do before their acronyms.

(5) Appendix material that no longer supports any conclusion (early null-model diagnostics and a stylized analysis) was removed, with a short note in App. C explaining why those diagnostics cannot identify structure.

(6) Statistical statements distinguish "not established" from "equivalent within a prespecified margin".

The main text remains nine pages.

W2: recent tabular foundation models with larger context size.

We added TabPFN-3.5 and TabICLv2. The original formal tests remain on the two original backbones. The revision comparisons (fixed-rule control, selection versus averaging, larger budgets, learned routing, and the main comparison of Sec. 7) are reported on all four backbones; the mismatched-mask and NaN-masking controls of Sec. 7 cover two backbones each.

Supports were extended from 128 rows to 1,024 to 3,072 rows, with K up to 512 (Sec. 6.2). At that scale the conclusions hold: there is no established gain of query-to-rule assignment over a fixed rule, and selection is equivalent to averaging within a margin of 0.01 NLL.

In addition, a descriptive check with 10,240-row supports on the two newer backbones (six datasets, 64 queries each for ICRM; App. L) found a larger loss from query-side missingness than at 1,024 rows (about 0.09 to 0.10 versus 0.06), and the feature-matched construction still recovered most of it. With six datasets this is an observation, not a test of the effect of context length.

We do not cover supports of tens of thousands of rows or backbones designed for them, and we now say so in the limitations.

W3: the discussion focuses exclusively on performance gain.

Beside mean NLL, the revision reports the following.

(1) Query-level harm. A method that improves the mean can be worse on 29% to 59% of queries (Sec. 6.3, App. I). We also test whether a conservative router can identify risky queries: it reduces exposure but passes the rate-matched random-fallback criterion on only one of four backbones.

(2) Measured compute for the main comparisons. For example, the original 16-candidate bank used about 100 times the model calls of a one-context method, and ICRM uses 16 to 55 times the GPU time of full support.

(3) A setting in which context construction addresses a deployment problem other than raw accuracy on clean inputs, namely a mismatch between complete support rows and incomplete queries (Sec. 7).

We agree that context selection also serves memory and scalability goals. Related Work now notes that predictive gain is not the only purpose of context selection and that our comparisons test predictive gains against stated alternatives and resources.

AuthorsOpenReview ↗

Follow-up to Reviewer 8okW

We would like to make it as easy as possible to check whether the revision addresses your concerns, and to ask one question.

A short reading path. If time is limited, the following four places show most of what changed:

(1) The abstract and the last two paragraphs of the introduction (page 1 to 2) state the question, the answer, and the contribution.

(2) Figure 1 (page 2) gives the logic of the whole paper in four lines: what a selector is compared with before inference and after inference.

(3) Table 2 (page 6) covers your second point: four backbones including TabPFN-3.5 and TabICLv2, with supports of 3,072 rows.

(4) Section 7 and Table 3 (pages 7 to 8) are new: a context constructed from each query's observed features improves on full-support inference by +0.051 NLL [+0.018, +0.096] across four backbones.

One question. Your review rates soundness as a significant concern, but the listed weaknesses concern readability, model coverage, and scope. If there are specific soundness issues, for example in the statistical analysis, the evaluation protocol, or the separation of prespecified and post-hoc evidence, we would be grateful to know which, so that we can address them within the discussion period.

Thank you for your time.

Thank you for the review. We substantially revised the paper in response to your concerns about presentation and scope: the main argument was rewritten around the scientific questions, we added TabPFN-3.5 and TabICLv2 together with larger-support experiments, and we now report query-level harm and measured computational cost in addition to mean predictive gain.

If you have an opportunity to revisit the revision, we would appreciate knowing whether these changes address your main concerns, or which concern remains decisive for your recommendation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their time and feedback. Three points recur in the reviews: the paper is hard to read, the evidence covers few backbones and small supports, and a diagnostic study leaves open how its insight turns into practical gain. The revised PDF addresses each. Section and appendix numbers below refer to the revised PDF. All numbers are NLL differences with 95% dataset-bootstrap intervals unless stated; positive favours the first-named method.

  1. The paper has been rewritten around its question.

The main text now follows one line of reasoning: how large the hindsight opportunity is and what it consists of (Sec. 4); whether information available before inference can claim the query-specific part (Sec. 5); whether selecting is better than averaging once predictions exist (Sec. 6); and whether any decision-time signal does yield deployable value (Sec. 7). Sections 4 to 6 are titled by the questions they answer, and each ends by stating what remains open. Implementation detail is in the appendix, which was shortened (the PDF went from 51 to 44 pages) by removing material that no conclusion depends on. App. C states why the early null-model diagnostics are no longer used.

  1. Broader evidence.

The original formal tests use TabPFN v2 and TabICL v1. Revision experiments repeat the main comparisons on two further backbones (TabPFN-3.5 and TabICLv2), on additional datasets, and with supports of up to 3,072 rows and K up to 512 (Sec. 6.2, App. K).

  1. From diagnosis to a deployable gain.

The diagnosis says that the tested row-level signals do not identify which context a query needs. We therefore asked whether any signal available before inference does. In our setting queries have masked features and the support is complete, yet every policy treats imputed values as observed. Building each query's context from the features it actually observes (ICRM, Sec. 7) improves on full-support inference by +0.051 [+0.018, +0.096] across four backbones on eleven validation datasets (three of them had appeared in an earlier descriptive analysis of MCAR damage, as disclosed in App. L).

(a) The same construction with another query's mask is worse than full support (-0.014 [-0.027, -0.007]); ICRM beats it by +0.063 [+0.024, +0.117] (TabPFN v2 and TabICL v1). This supports the interpretation that matching the context to the query, rather than column deletion as such, produces the gain; the control does not decompose the gain into separate causes.

(b) ICRM beats the backbones' native missing-value handling when the context also contains NaNs (+0.021 [+0.010, +0.034], two TabPFN backbones).

(c) Limits, stated in the paper. The principle is the reduced-feature model of Saar-Tsechansky and Provost (2007), which ICRM carries out in context without training. It costs 16 to 55 times the inference time of full support. At matched cost it still wins on TabPFN v2 (+0.020 [+0.008, +0.036]) but is undecided on TabPFN-3.5 (+0.010 [-0.0002, +0.021]). Evidence is limited to synthetic MCAR missingness, and about 23% of queries get worse by more than 0.01 NLL. This study was run after the main analyses and is reported as such.

Individual responses to each reviewer follow below their reviews.

AuthorsOpenReview ↗

Update to all reviewers: three results obtained after the revision

We report three results obtained after the revised PDF was uploaded. Details are in our replies to Reviewer 8okW; we summarize them here because the first one bears on our earlier responses to all reviewers. The PDF is unchanged.

  1. Section 7 does not carry over to naturally occurring missing values.

Our earlier responses presented the feature-matched construction of Section 7 (ICRM) as a case in which a decision-time signal yields deployable value. We then ran a follow-up whose design, datasets, and decision rule were fixed in advance, on 13 new datasets with naturally occurring missing values in both support and query rows, with TabPFN-3.5 and TabICLv2.

In that setting ICRM did not improve on inexpensive alternatives at matched cost (median imputation, imputation with missing-value indicators, iterative imputation, and the models' native handling of missing values). Its mean difference was negative against every one of them on both backbones, between -0.005 and -0.019 NLL, and by our pre-specified rule the inexpensive alternatives are sufficient. A second analysis with complete support rows and real query missingness, on the eight eligible datasets, was inconclusive on both backbones.

The gain reported in Section 7 should therefore be read as specific to complete support rows with synthetically masked queries, which is the limitation stated in the paper. Please weigh our earlier statements about Section 7 accordingly. Sections 4 to 6 are not affected.

  1. Larger pools (relevant to Reviewer YMMC's W2 and Q2, and to Reviewer 8okW).

In an exploratory check on five tables with more than 100,000 rows, including Folktables ACSIncome, we compared full-support inference with a retrieved context of the 1,024 nearest rows per query as the pool grew from 1,024 to 65,536 rows (one seed, 200 queries per table, no intervals). Mean NLL of full support fell from 0.471 to 0.425 on TabPFN-3.5 and from 0.473 to 0.436 on TabICLv2. Retrieval was level with full support up to 16,384 rows and behind it at 65,536 rows, by 0.011 and 0.007 NLL. Up to the largest scale we could run, selecting a context did not improve prediction over using the whole pool; at that scale its case is a smaller context at a small loss in accuracy.

  1. Conventional baselines (requested by Reviewer 8okW).

On the 13 datasets of item 1, with the same 2,048 support rows and 256 queries, LightGBM and XGBoost with default hyperparameters reached mean NLL 0.607 and 0.689, against 0.443 for TabPFN-3.5 and 0.457 for TabICLv2 (accuracy 79.3% and 78.7% against 81.6% and 81.2%). The boosted trees were not tuned, so part of the NLL gap reflects calibration.

Items 2 and 3 were not pre-specified and are reported as post-hoc analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.