acceptodds
Under review as a conference paper at ICLR 2027

Understanding Context Quality and Improving Batch Retrieval for Tabular Foundation Models

Abstract

Tabular foundation models predict new samples from labeled reference examples without task-specific model training. Using a separate context for each query or query cluster incurs substantial computational cost, motivating the selection of an effective shared context for an entire query batch under a limited budget. Existing retrieval methods typically use fixed selection rules whose performance varies substantially across tasks. Through exhaustive synthetic experiments and controlled analyses on real datasets, we find that a sample's predictive contribution is closely related to local class support, support accumulation, allocation across queries, and class composition. Based on these observations, we propose Preference-Aware Integrated Retrieval (PAIR), an adjustable retrieval framework that combines support accumulation, coverage allocation across queries, and class reservation, with configurations selected through validation. Its coverage objective evaluates candidates according to existing coverage and exhibits diminishing returns, enabling efficient greedy optimization with an approximation guarantee. Across 51 classification tasks, PAIR achieves the best mean rank of 1.91 among seven methods, with its untuned default ranking second, and the gains persist across four frozen models, multiple context budgets, and regression tasks. Its mean inference time is approximately one twenty-fifth that of per-query retrieval, with corresponding differences in predictive quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation model, context quality

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_iZauOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

TFMs are an important discovery for tabular tasks. They have to rely on in-context examples for task awareness/definition, and batching is required for deployment efficiency. Under these constraints, the RQ is how to choose the right context for best performance for any query batch. Overall, this poses an interesting problem of how to fill in the context with the best examples to maximize the task performance.

The authors essentially create a value function for contexts given a query batch, f(c, Q_batch), and use a retrieval formulation + submodular function for the final objective leading to an efficient context construction mechanism for every Q_batch.

Proposed method: "we propose Preference-Aware Integrated Retrieval (PAIR), which values a candidate's contribution according to the coverage each query already receives. Its three adjustable choices, namely how coverage accumulates over multiple neighbors, how additional coverage is valued across queries, and how much of the budget is reserved across classes, define interpretable retrieval preferences selected through validation."

Core factors that govern context quality, as per the authors: "(i) local class support: examples from the query's class tend to provide greater predictive benefit when they are closer to the query. (ii) support accumulation: marginal gains diminish with accumulated support, so that the same candidate yields different gains in different contexts. (iii) allocation across queries: under a shared budget, strengthening one query's support can come at the cost of another's. (iv) class composition: the context's class composition shifts predictions even with local support held constant."

Critical strengths
  • They contribute an integrated formulation for scoring S across all four proposed factors.
  • Simple intuition based and it works. It is clear why this approach works well, and it is interpretable in terms of what's helping the prediction. AUC-ROC gains are strong compared to other baselines.
Critical weaknesses

Critical Weaknesses

  1. Section 3.2 / 3.3 lacks clarity on the problem setup. Numbers, problem setup and analysis definitions are lacking and formulations not shown properly. E.g. how Fig 2 metrics are computed, or Table 1: what are these metrics?
  2. Final conclusions from Section 3.2 and 3.3 are trivial and intuitive, unclear what we gained from there.
  3. "Section 3 identifies four factors that govern context quality, with response magnitudes that vary across tasks." Claim not well supported. Cyclical self-justification. These four factors were the intuitive hypothesis over what context properties might affect the predictor's performance, not an exhaustive search process to "identify" factors that govern quality.
  4. This is simply in-context learning; it's obvious that finding examples similar to the query and putting a diverse set in the context would help, so solution novelty is unclear. Also, the problem itself is not a totally unsolved problem; a lot of work has been done on exemplar selection in in-context learning. Would like to see those methodologies properly brought into the picture for comparison.
  5. AUC-ROC missing for untuned PAIR in Figure 3. That is critical to show.
  6. The same conundrum of whether this is a worthwhile problem in the long run. Same issue as LLMs: prompt tuning or in-context learning that was a big hype a year ago are not useful problems to solve anymore. Show gains with other newer / larger TFMs (newer version of TabPFN, Kumo, LimiX etc.): whether the problem itself exists there, and then whether complex solutions are required or a simple policy of nearest samples for given queries, constrained by class balance and sample diversity in the shared context, suffices.
  7. Dependent on an external representation method for the similarity function between queries and candidate examples. Details for this are not provided. How is this trained in data-scarce regimes like tabular prediction tasks?
Questions for authors

Questions For Authors

  1. How fast is the approach and how easy is the setup? Can c be reformulated for any Q_b, any incoming task?
  2. Please share details of how to setup the similarity function for each task.
3 comments

We appreciate the reviewer's time and the direct feedback on both the presentation and the motivation of the paper. Our answers follow point by point.

W1. "Section 3.2 / 3.3 lacks clarity on the problem setup. Numbers, problem setup and analysis definitions are lacking and formulations not shown properly. E.g. how Fig 2 metrics are computed, or Table 1: what are these metrics?"

Thank you for pointing this out. We have rewritten Section 3. It now opens with the problem setup: the labeled pool, the query batch, the budget, and what a selector chooses. Section 3.2 states what each entry of Table 1 is, namely the difference between two contexts that differ only in the named property, measured as the log-loss difference of the focal query for the first four columns and as the change in true-class probability for class composition. The captions of Figure 2 and Table 1 define every panel and column, and Appendix D describes how the contexts are constructed.

W2. "Final conclusions from Section 3.2 and 3.3 are trivial and intuitive, unclear what we gained from there."

What we gain from Sections 3.2 and 3.3 is how the properties differ across tasks. We agree that each property is intuitive when considered in isolation, but this is not where our contribution lies. The properties compete for the same context budget, and tasks differ in how strongly they respond to each of them, and even in the direction of the response (Table 1). These differences are large enough to change which retriever performs well, which is why the performance of existing selectors depends strongly on the dataset: Facility Location and Pooled kNN each win on a substantial share of our 51 tasks. The four properties therefore characterize a task in terms that a retriever can act on.

PAIR is built on this: it makes the properties adjustable for each task, and it ranks first in the main comparison and under all seven TFMs that we evaluate. The rewritten Section 3 presents the argument in this order to make it clearer.

W3. "'Section 3 identifies four factors that govern context quality …' Claim not well supported. Cyclical self-justification. These four factors were the intuitive hypothesis over what context properties might affect the predictor's performance, not an exhaustive search process to 'identify' factors that govern quality."

We agree that the four properties were chosen and were not found by a search. In the revised paper they are the terms in which we describe how tasks differ in what a shared context needs. To make this clear, we have rewritten Section 3 and the corresponding sentences in the abstract, the introduction, and the conclusion.

We have also added a data-driven check on 46 tasks outside the development set (new Appendix D.5). We computed 39 candidate properties of contexts that were not produced by PAIR, including diversity, density, and distribution-matching properties, and fitted models that explain the batch loss on held-out tasks. The four properties alone reach a held-out R² of 0.64, against 0.70 for a flexible model over all 39 properties, and removing any of the other groups of properties changes the fit only marginally.

W4. "This is simply in-context learning; it's obvious that finding examples similar to the query and putting a diverse set in the context would help, so solution novelty is unclear. Also, the problem itself is not a totally unsolved problem; a lot of work has been done on exemplar selection in in-context learning. Would like to see those methodologies properly brought into the picture for comparison."

Solution novelty. The novelty of PAIR is that it adapts the balance between similarity and diversity to each task. Finding examples similar to the query and putting a diverse set in the context are each a fixed policy in our comparison, namely Pooled kNN and Facility Location. Neither of them is consistently better than the other across tasks, and PAIR improves on both.

We have also added a comparison in which the parameters of these two baselines are tuned on validation data, namely the kernel scale of Facility Location and the neighborhood depth of Pooled kNN (new Appendix K). PAIR remains first in this comparison.

The problem. The problem that we study is different from exemplar selection in in-context learning. Exemplar selection typically chooses a few demonstrations for a single query, whereas here B rows are chosen out of a large labeled pool once for a whole query batch. One context therefore serves many queries at once, and the classes that it contains determine which classes receive probability.

Comparison. The methodologies for this problem are in our comparison: Pooled kNN, Facility Location, CRUMB, and VIP-COp. To our knowledge, these are the available methods for constructing a shared context for a TFM.

W5. "AUC-ROC missing for untuned PAIR in Figure 3. That is critical to show."

Thank you for this suggestion. In Figure 3, the PAIR curve at B≥500 is already the untuned default, which the caption now states, so the two curves differ only at B=200. The AUC values are given below; the untuned default is above both baselines at every budget.

Mean AUC, 17 binary datasets, four predictorsB=2005001,0002,000
PAIR (validation-selected at B=200)0.7880.8080.8210.830
PAIR, untuned default0.7830.8080.8210.830
Pooled kNN0.7610.7990.8150.827
Facility Location0.7660.7980.8140.824

W6. "The same conundrum of whether this is a worthwhile problem in the long run. … Show gains with other newer / larger TFMs (newer version of TabPFN, Kumo, LimiX etc.): whether the problem itself exists there, and then whether complex solutions are required or a simple policy of nearest samples for given queries, constrained by class balance and sample diversity in the shared context, suffices."

Whether the problem persists. A TFM predicts only through its context. For an LLM, in-context examples are one of several ways to specify a task, whereas a TFM has no other access to the training data. The size of the context therefore determines the inference time and memory of any such model, and selecting what enters a bounded context remains necessary as the models become larger. The closer analogy in LLMs is the handling of long contexts, which has remained an active problem. We have rewritten the opening of the introduction to make this point.

Newer TFMs. As you suggested, we evaluated three more recent TFMs: TabPFN-3.5, LimiX-2, and Kumo Tabular (new Appendix L). The problem is present on all three, in the sense that the selectors still differ clearly, and PAIR ranks first on each of them, both with the configurations selected on TabPFN-v3 and with configurations reselected on the new model.

Mean rank of PAIR among seven methods, B=100Configurations from TabPFN-v3Configurations reselected
TabPFN-3.52.161.91
LimiX-22.162.42
Kumo Tabular2.062.00

Simple policies. The nearest-sample policy and the diversity policy are Pooled kNN and Facility Location in this comparison, and the advantage of PAIR over both is statistically significant on each of the three models. A fixed configuration of PAIR without any search, the untuned default, ranks second on every model, ahead of all the other methods. A simple policy therefore captures part of the gain, and selecting the configuration on validation data adds to it.

W7. "Dependent on an external representation method for the similarity function between queries and candidate examples. Details for this are not provided. How is this trained in data-scarce regimes like tabular prediction tasks?"

PAIR does not use an external or learned representation, and nothing is trained. The similarity is computed directly from the standardized Euclidean distance between the features of a query and of a candidate, or from the rank of this distance. As this was stated only in the appendix, Section 4.1 now says it explicitly. The method therefore applies unchanged to small datasets, including the 15 small tasks in our evaluation. The details are given under Q2.

Q1. "How fast is the approach and how easy is the setup? Can c be reformulated for any Q_b, any incoming task?"

Speed. Selecting the contexts for a complete test set takes about two seconds, out of about 49 seconds in total (Appendix H, Table 26). In a controlled benchmark, selection stays within a few seconds for training pools of 500,000 rows.

Setup. Nothing is trained. PAIR can be used directly with its untuned default, which requires no search and ranks second in Table 2, ahead of all the other methods. Alternatively, a configuration is selected once per task by a validation search of several minutes per split, and is then reused for all later query batches.

New batches and tasks. The coverage c is computed from the features of the incoming query batch and of the pool, so it is defined for any batch of any task without modification.

Q2. "Please share details of how to setup the similarity function for each task."

No task-specific setup is required. The features are standardized with training-set statistics, and the similarity is a decreasing function either of the Euclidean distance or of its rank among the neighbours of the query. Which of the two is used is selected on validation data together with the other parameters (Appendix C). On the 46 tasks outside the development set, the rank-based similarity is selected on 27 tasks and the distance-based similarity on 19.

We thank the reviewer again and hope that these clarifications and the new results address the concerns.

Reviewer_9LQ9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on context selection for tabular foundation models. As tabular foundation models often operate on batches of test queries, that share the same "retrieval" context, the paper identifies a problem of context construction for the prediction batches.

The paper identifies multiple context propeties like local class support, support accumulation, allocation across queries, and class composition that change predictions, and then use proxies for those properies in the propose context construction procedure (PAIR)

PAIR is then tested on subsets of tabarena and openml benchmarks comparing to context selection methods from prior work and a simple pooled knn baseline.

Critical strengths

The paper identifies a timely problem -- I believe context selection is an important direction to explore in foundation models and the shared batch inference is an interesting aspect that makes context selection procedure potentially non trivial. I also like that paper aproaches the problem through analysis first.

Critical weaknesses

The core weakness is the actual magnitude of the improvement and the setting in which the method is evaluated. The budget B at which the proposed method improves over the much simpler Pooled-kNN baseline is 500 for classification, and the difference diminishes with increased context (and stays very small from the beggining on regression problems).

The analysis alone I think does not provide enough novelty and understanding to justify for lacking empirical improvements in my view.

I see potential for changing my opinion if the method is demonstated useful on larger B (e.g. on tabarena v0.1 aka BeyondArena medium-to-large scale subsets)

The writing at in sections 3-5 is not sufficiently clear and at times is overloaded with dense descriptions of experimental results. The section 3 introducing the context properties and the proxies could also be significantly simplified in my view, this would greatly benifit the paper at this stage.

Questions for authors

How does the method work on larger datasets, where context selection is more important for efficiency?

2 comments

We thank the reviewer for the careful and constructive review. Our responses to the individual points are below.

W1. "The budget B at which the proposed method improves over the much simpler Pooled-kNN baseline is 500 for classification, and the difference diminishes with increased context (and stays very small from the beginning on regression problems)."

With validation selection, PAIR improves over Pooled kNN at every budget up to B=4,000, the largest budget we evaluated (see the table below). In the submission, PAIR at B≥500 was run with its untuned default configuration, without the validation selection used for the main table, so Figure 3 showed the default and not the full method at these budgets; Section 5.4 now states this.

The narrowing of the difference with increased context is expected, and it does not affect our contribution, because the setting of the paper is inference under limited resources, that is, small B, which is where the advantage of PAIR is largest. The narrowing is common to all selectors: a larger context contains more of the training pool regardless of how it is chosen, so every selector improves and the differences among them shrink. Over the same range, the spread among the four baselines falls from 0.029 to 0.008 AUC, while the margin of PAIR over Pooled kNN goes from +0.041 to +0.014 and remains positive at every budget.

Mean AUCB=5001,0002,0002,5003,0003,5004,000
PAIR0.7670.7810.7940.7900.7960.7970.798
Pooled kNN0.7260.7480.7680.7720.7780.7820.784
Facility Location0.7200.7500.7690.7750.7790.7800.785
Uniform Random0.7120.7420.7620.7660.7710.7780.780
CRUMB (K=1)0.6970.7310.7550.7630.7680.7740.777
PAIR − Pooled kNN+0.041+0.033+0.026+0.018+0.018+0.015+0.014
Spread among the four baselines0.0290.0190.0140.0120.0110.0080.008

The 13 binary tasks among the 15 larger tasks of W3.

For regression, we agree that the margins are smaller. Only the coverage mechanisms of PAIR are active in regression, since class reservation and the class-frequency correction do not apply. PAIR nevertheless ranks first at every budget and under every predictor (Table 15).

W2. "The analysis alone I think does not provide enough novelty and understanding to justify for lacking empirical improvements."

The understanding that the analysis provides is that the four properties, each of which looks simple when taken alone, differ in strength from one dataset to another, and that these differences are large enough to change which retriever performs well. This is what makes the properties consequential for retrieval, and it is the reason why the performance of existing selectors depends strongly on the dataset:

  1. How much each of the four properties matters, and even its direction, changes from task to task (Table 1).
  2. Consistently with this, no existing selector is best across tasks. Facility Location and Pooled kNN each win on a substantial share of the 51 tasks, under all four predictors (table below).
  3. On 46 further tasks, the four properties explain most of the explainable variation in loss (held-out R² 0.64, against 0.70 for a flexible model over 39 properties; Appendix D.5).
PredictorTasksFacility Location betterPooled kNN better
TabPFN-v3513318
TabICL-v2513516
TabFM482919
LimiX-16M483117

This is what motivates the design of PAIR: Pooled kNN and Facility Location keep these properties fixed, whereas PAIR selects them for each task. To make this argument clearer, we have rewritten Section 3 around it.

On the empirical side, the new results under W3 show that the improvement holds at larger budgets and on larger tasks.

W3. "I see potential for changing my opinion if the method is demonstrated useful on larger B (e.g. on tabarena v0.1 aka BeyondArena medium-to-large scale subsets)."

Following your suggestion, we evaluated PAIR on 15 medium-to-large classification tasks from BeyondArena (21k–396k training rows) at budgets from 500 to 4,000. PAIR ranks first among the five selectors at every budget, and its rank advantage over both Pooled kNN and Facility Location is statistically significant at every budget. The results are summarized below and reported with confidence intervals in the new Appendix G.8 (Table 21).

Mean rankB=5001,0002,0002,5003,0003,5004,000
PAIR1.201.271.331.601.531.601.47
Facility Location3.073.003.133.133.203.333.07
Pooled kNN3.133.403.003.332.873.133.27
Uniform Random3.333.333.403.273.533.003.20
CRUMB (K=1)4.274.004.133.673.873.934.00

15 tasks, official TabArena metric, TabPFN-v3, three splits.

We would also like to note that the original 51 tasks already include the 23 classification tasks of TabArena v0.1 that fit our protocol; this is now stated in Section 5.1.

W4. "The writing in sections 3–5 is not sufficiently clear … Section 3 … could also be significantly simplified."

Thank you for this suggestion. We have rewritten Section 3 so that it follows a single line of argument (the three steps listed under W2). It is now about one third shorter, and the detailed numbers have been moved to Appendix D. In Section 4, we added a short explanation of the decision that each mechanism controls, and in Section 5 we replaced enumerations of ranks with descriptions of the overall pattern.

Q1. "How does the method work on larger datasets, where context selection is more important for efficiency?"

PAIR retains its advantage on larger datasets. On the 15 medium-to-large BeyondArena tasks under W3, whose training pools contain 21k–396k rows, it ranks first among the five selectors at every budget from 500 to 4,000. The advantage also holds on the largest of them: on the four tasks with more than 100k training rows (132k–396k), PAIR has the highest mean AUC at every budget.

Mean AUCB=5001,0002,0002,5003,0003,5004,000
PAIR0.6960.7160.7350.7310.7380.7400.742
Pooled kNN0.6580.6750.6980.7040.7110.7140.719
Facility Location0.6550.6790.6970.7080.7140.7170.723
Uniform Random0.6600.6840.7010.7070.7140.7200.724
CRUMB (K=1)0.6460.6780.6950.7050.7060.7110.717

The same holds on the seven binary tasks with more than 50k training rows.

Regarding efficiency, larger datasets are also where a selected context becomes necessary: on resource-limited devices, full-context inference exceeds a 60-second per-batch budget from 10k–200k training rows (new Appendix H.4).

Thank you again for your time and for the constructive feedback.

Reviewer_F7fzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fixed-batch context retrieval for frozen tabular foundation models under a limited context budget. Given a labeled pool and a fixed query batch, the goal is to select one labeled context shared by the complete batch in a single prediction call. Through synthetic experiments and controlled analyses on real datasets, the authors identify four factors associated with context quality: local class support, support accumulation, allocation across queries, and class composition. Based on these observations, the paper proposes Preference-Aware Integrated Retrieval (PAIR), which combines query coverage, allocation across queries, and class reservation, with its configuration selected using validation data. PAIR uses greedy context construction with a conditional approximation guarantee. Across 51 classification tasks, several frozen predictors, context budgets, and additional regression tasks, PAIR achieves competitive predictive performance. The paper also reports substantially lower end-to-end inference time than per-query kNN on five datasets.

Critical strengths
  1. The paper addresses a relevant practical problem for tabular foundation models. Constructing a separate context for each query can require repeatedly processing different labeled contexts. Selecting a compact context shared within a fixed query batch is therefore meaningful when context capacity and inference cost are constrained.

  2. The paper provides an interpretable analysis of context quality. The decomposition into local class support, support accumulation, allocation across queries, and class composition offers a useful perspective for understanding why fixed retrieval heuristics behave differently across tasks.

  3. PAIR provides a coherent framework for combining several retrieval considerations. The method integrates accumulated support, coverage allocation across queries, and class reservation in a unified objective. The conditional submodularity result also provides a theoretical justification for greedy construction once the similarities and reserved set are fixed.

  4. The experimental evaluation is systematic and shows competitive results within the studied setting. The paper evaluates multiple baselines, context budgets, and frozen TFMs, and includes an untuned configuration, component ablations, and supplementary regression experiments. Within this scope, PAIR achieves consistently strong performance, including the best average rank at the primary budget.

Critical weaknesses
  1. The efficiency evidence is limited and does not isolate PAIR’s contribution. The 25.1×25.1\times speedup is reported on only five datasets against per-query kNN. The gain may largely result from sharing one context across a query batch, whereas per-query kNN constructs a different context for each query and cannot evaluate the entire batch with one shared-context prediction call. The paper does not separate similarity computation, greedy construction, and inference costs, or establish whether the result generalizes across scales and implementations.

  2. The distinction between fixed query batches and query-cluster retrieval is unclear. PAIR divides the test set into multiple fixed batches and uses one context and predictor call for each batch. This appears operationally similar to query-cluster retrieval, which also groups queries and assigns a shared context to each group. The paper does not clearly explain how the two settings differ beyond the grouping and context-selection strategies, or compare them under matched numbers and sizes of query groups.

  3. The evaluation remains limited relative to the breadth of the claimed setting. While the study includes 51 classification tasks, this is a relatively narrow evaluation for a general retrieval framework intended to operate across tabular foundation models. The capped validation, test, and query-batch sizes further limit evidence on large and structurally diverse datasets.

  4. The comparison with full-context inference is incomplete. Full Context is mainly used as an accuracy reference on a feasible subset. A more complete comparison of predictive performance, runtime, memory, and context-construction overhead is needed to establish when PAIR is preferable to direct full-context or larger-context inference.

  5. Model dependence and evaluation stability are insufficiently characterized. PAIR selects its configuration using TFM validation performance, while test-time context construction relies on a geometric proxy. The paper does not establish whether this proxy reflects the context preferences of different TFMs or clarify whether configurations are reselected for each predictor. It also does not report the stability of selected configurations and final results across data splits and repeated runs.

  6. The reliance on standardized Euclidean distance limits generality. This similarity may not reflect context utility for mixed-type, sparse, interaction-heavy, or distribution-shifted data. The paper does not establish whether its findings transfer to learned or model-derived similarities.

Questions for authors

1.The reported 25.1x speedup is based on only five datasets. Could the authors evaluate efficiency across a broader range of tasks and query scales, and separately report context-selection and TFM-inference costs? This would help determine whether the speedup generalizes and how much is attributable to PAIR rather than to shared-context inference itself.

2.How does fixed-batch retrieval differ operationally from query-cluster retrieval beyond the grouping strategy? Could the authors compare them under matched numbers and sizes of query groups and predictor evaluations?

3.How do predictive quality and runtime change as query-batch size increases while the test set and context budget remain fixed? How sensitive are the results to batch composition, such as random versus similarity-based grouping?

4.On datasets where Full Context is feasible, could the authors compare PAIR and Full Context in terms of predictive performance, runtime, memory use, and context-construction overhead?

5.In the cross-predictor experiments, are PAIR’s configurations selected independently for each dataset, split, budget, and TFM, or transferred across predictors? If selection is repeated, what is its computational cost and how stable are the selected configurations across validation splits? More generally, what evidence shows that the geometric test-time objective reflects the context preferences of different TFMs—for example, do its proxy marginal gains correlate with model-specific loss reductions?

6.Could the authors evaluate PAIR on a larger and more structurally diverse collection of classification tasks? In particular, how robust are the conclusions on large, high-dimensional, sparse, or mixed-type datasets, and when standardized Euclidean distance is replaced by learned or model-derived similarities?

5 comments

We thank the reviewer for the detailed review and for the specific comparisons that were requested, which have made the evaluation more complete. Where a weakness and a question concern the same issue, we address the weakness briefly and provide the detailed response under the corresponding question to avoid repetition.

W1. "The efficiency evidence is limited and does not isolate PAIR's contribution. … The gain may largely result from sharing one context across a query batch … The paper does not separate similarity computation, greedy construction, and inference costs, or establish whether the result generalizes across scales and implementations."

We agree that most of the speedup comes from sharing one context across a query batch, and we have revised the abstract and Section 5.3 to say so. The setting largely determines the cost, and the selector determines the predictive quality that is obtained at that cost. This is why we compare selectors within the shared-context setting, where they have similar running times and PAIR attains the lowest log loss. The stage-level breakdown and the results on more datasets and scales are given under Q1.

W2. "The distinction between fixed query batches and query-cluster retrieval is unclear. … The paper does not clearly explain how the two settings differ beyond the grouping and context-selection strategies, or compare them under matched numbers and sizes of query groups."

The two settings differ in what they assume about the queries, and this changes what the selector has to do. Query-cluster retrieval presupposes a large set of queries that can be grouped into compact clusters; each cluster then receives a context that is close to a per-query context, and the saving comes from the number of clusters being small. We study the case in which the batch is arbitrary: in our experiments the queries are assigned to batches at random, so the queries of a batch are spread over the feature space. A single context then has to balance closeness to individual queries against coverage of the whole batch and of the classes, and this balance is what the mechanisms of PAIR control. We have rewritten the corresponding paragraph of Section 2 to state this difference. The matched comparison is given under Q2.

W3. "The evaluation remains limited relative to the breadth of the claimed setting. … 51 classification tasks … is a relatively narrow evaluation for a general retrieval framework intended to operate across tabular foundation models. The capped validation, test, and query-batch sizes further limit evidence on large and structurally diverse datasets."

The breadth of our evaluation is in line with that of the closest related work, and it uses the same benchmark. Our 51 classification tasks include the 23 classification tasks of TabArena v0.1 that fit our protocol, and TabArena is the benchmark on which CRUMB is evaluated. For comparison, CRUMB uses the 51 classification and regression datasets of TabArena, and VIP-COp evaluates classification on 38 datasets.

To broaden the evaluation further, we have added two sets of experiments. First, 15 larger classification tasks with 21k–396k training rows at budgets up to 4,000, on which PAIR ranks first at every budget (results under Q6). Second, three more recent TFMs in addition to the four of the submission, namely TabPFN-3.5, LimiX-2, and Kumo Tabular; PAIR ranks first under each of them (results under Q5).

Regarding the capped validation, test, and query-batch sizes: the caps bound the number of scored validation and test queries, while the training pool from which the context is selected is used in full and contains up to 396k rows in the new tasks. The effect of the query-batch size is examined under Q3.

W4. "The comparison with full-context inference is incomplete. … A more complete comparison of predictive performance, runtime, memory, and context-construction overhead is needed to establish when PAIR is preferable to direct full-context or larger-context inference."

Full-context inference is limited by the resources of the device, and the setting of our paper is inference under limited resources. PAIR is preferable whenever the training pool exceeds the memory or latency budget of the device, because full-context inference is then infeasible and a selected context is the only option; on CPU-only and small-GPU devices this happens from 10k–200k training rows (Q4). In this setting a bounded context is selected for every query batch, and PAIR addresses how to select it.

Full-context inference serves in the paper as a reference for analysis, and the revised Section 5.4 states this role explicitly. The comparison of predictive performance, runtime, memory, and context-construction overhead under matched conditions is given under Q4.

W5. "Model dependence and evaluation stability are insufficiently characterized. … The paper does not establish whether this proxy reflects the context preferences of different TFMs or clarify whether configurations are reselected for each predictor. It also does not report the stability of selected configurations and final results across data splits and repeated runs."

The geometric proxy reflects the context preferences of each TFM, because its parameters are selected by the validation performance of that TFM. Configurations are reselected for each predictor, so the context that is constructed at test time is the one that this predictor preferred on validation data, and PAIR ranks first under each of the seven TFMs that we evaluate.

The final results are also stable across data splits: when each of the three outer splits is ranked separately, PAIR ranks first on every one of them (new Table 34 in Appendix K).

Mean rank among seven methods, 51 tasksSplit 0Split 1Split 2
PAIR2.172.112.28
PAIR, untuned default2.922.873.01
Facility Location3.533.873.57
Pooled kNN4.303.934.55

The remaining details are given under Q5.

W6. "The reliance on standardized Euclidean distance limits generality. … The paper does not establish whether its findings transfer to learned or model-derived similarities."

Standardized Euclidean distance in the original feature space is the more general choice, and it is the setting of most retrieval methods for TFMs. A similarity computed from the features needs only the data, so it applies to any TFM, requires no training, and is available at any dataset size, whereas a learned or model-derived similarity is tied to the model that produces it. LoCalPFN retrieves neighbours in raw standardized features and reports that this is very competitive with encoder-based alternatives, CRUMB clusters queries by Euclidean distance on standardized features, and Pooled kNN and Facility Location operate in the same space.

Our new experiments on three more recent TFMs support this: with the configurations selected on TabPFN-v3, PAIR ranks first on each of them, so contexts selected in this space carry over across models (results under Q5).

Our study therefore fixes this retrieval scenario and analyses it. The revised Section 3 describes, along four properties, how the contexts built by different retrieval methods differ and how tasks differ in what they need; these properties are the axes of this description, and we have rewritten the section to make this clear. Retrieval under learned or model-derived similarities is a separate object of study, which the conclusion names as future work.

Regarding mixed-type and other structurally diverse data, the 15 larger tasks include high-dimensional, missing-value, and mixed-type data, and PAIR ranks first on them with standardized Euclidean distance (Q6).

Q1. "Could the authors evaluate efficiency across a broader range of tasks and query scales, and separately report context-selection and TFM-inference costs?"

The table below separates the stages on the five timed datasets (new Table 26 in Appendix H). Per-query kNN needs 1,775 predictor calls per test set, against 8 for every shared-context method, and this difference accounts for the speedup. Selection takes 2.3 of the 48.7 seconds of PAIR.

MethodPredictor callsNeighbor cacheSelectionPredictor startupInferenceTotal (s)
PAIR81.42.326.918.148.7
Pooled kNN81.50.127.417.846.9
Facility Location81.46.627.516.852.4
Per-query kNN1,7751.50.127.71,194.51,223.9

The same measurement on a set of 19 datasets gives the same picture: the shared-context methods take 40–46 seconds per test set and differ from one another by a few seconds.

For larger scales, we measured selection in a controlled benchmark with training pools of up to 500,000 rows and batches of up to 1,024 queries, where selecting one context with PAIR takes at most about 7 seconds. Running time is determined by the sizes involved (rows, features, budget, and batch size) and not by the content of the data, so these measurements carry over to other datasets of the same size.

Q2. "How does fixed-batch retrieval differ operationally from query-cluster retrieval beyond the grouping strategy? Could the authors compare them under matched numbers and sizes of query groups and predictor evaluations?"

Operationally, the main difference lies in how the groups are formed. Query-cluster retrieval, as in CRUMB, forms the groups itself, by clustering the queries so that those within a group are similar. In our setting the batches are random: the queries are assigned to batches at random, without regard to their features, and the method only chooses the context (see W2).

For the matched comparison, each 256-query batch is partitioned by the same KMeans clustering into 1, 4, or 16 groups, every selector chooses 100 rows per group, and the predictor is called once per group (new Table 27 in Appendix H).

Log loss1 group per batch4 groups16 groups
PAIR1.7461.4971.385
Pooled kNN3.2132.2331.830
Facility Location2.7482.2181.933
CRUMB6.8755.9804.276

Log loss is the official TabArena metric for these multiclass tasks, and PAIR has the lowest log loss at every number of groups. Table 27 additionally reports accuracy and running time.

Reviewer_gjiUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PAIR, a preference-aware context selection framework for tabular foundation models. The central motivation is that, when multiple query examples are processed together, the choice of in-context examples can affect the predictions produced by the model. Rather than selecting a single context using a fixed heuristic, PAIR combines multiple context-selection preferences, including class reservation and geometric or neighborhood-based criteria, to construct contexts intended to improve predictive performance.

The paper evaluates PAIR across a broad collection of tabular classification tasks, primarily using TabPFN-v3, and compares it against several context-selection baselines and variants. It also studies the influence of class composition, context size, batch regrouping, and the computational cost of context selection. The theoretical component establishes a monotonicity and submodularity result for the proposed coverage objective, yielding a conditional greedy approximation guarantee for that surrogate objective.

The empirical results suggest that PAIR can improve performance over simpler context-selection strategies, with the reported gains varying across binary and multiclass tasks. The paper also investigates when class composition and context geometry influence the selected contexts and resulting predictions.

Critical strengths
  1. Careful, transparent empirical design: Exhaustive enumeration isolates addition, exchange, and interaction effects; real-data treatments were frozen in advance; and evaluations span 51 tasks, four frozen TFMs, multiple budgets, and regression panels, accompanied by extensive robustness checks and per-task tables (Table 18).
  2. A useful unifying view: Expressing CB-Random, Pooled kNN, and Facility Location as corner settings of a single parameterized, interpretable family successfully explains why different baselines win or lose across tasks.
Critical weaknesses

Class-composition effects are not fully isolated. The experiments intended to investigate the effect of class composition also modify other properties of the selected context. In particular, the real-data intervention exchanges examples while changing their feature composition, and the paper explicitly acknowledges that other background features also change. Therefore, the observed differences cannot be attributed exclusively to class composition. A more controlled intervention would keep the feature vectors fixed and change only the labels or class assignments, where methodologically appropriate.

The interpretation of augmented explanatory power requires further validation. The large increase in augmented R2R^2 reported in Table 10 is potentially interesting, but its interpretation is complicated by the extremely small probabilities assigned to absent classes and the resulting clipping behavior. It is not sufficiently clear how much of the explanatory power comes from geometric support information versus the distinction between class-present and class-absent observations.

The computational comparison does not fully characterize the cost of obtaining a reusable PAIR configuration. The reported runtime comparison is useful, but it excludes the earlier validation search. Although the paper correctly distinguishes the number of scored prediction rules from the number of distinct contexts and TFM calls, the total cost of configuration selection and the amortized cost across subsequent inference workloads remain unclear.

The theoretical guarantee is disconnected from the predictive objective. The theorem establishes a greedy approximation guarantee for the coverage surrogate, not for the predictive loss of the tabular foundation model. The paper acknowledges this distinction, but the theoretical result therefore provides limited direct assurance about the predictive performance of PAIR.

The generality of the diagnostic findings remains uncertain. Some diagnostic analyses are conducted on a small set of development tasks. It is unclear whether the identified conditions reliably predict which configurations will work best on previously unseen tasks.

Questions for authors
  1. The primary evaluation uses ROC AUC for binary tasks and log loss for multiclass tasks. Could the authors provide complementary metrics such as accuracy, balanced accuracy, macro-F1, and ECE for the main comparisons in Tables 2 and 16, with the decision threshold and calibration protocol specified? More importantly, could the authors assess the sensitivity of the multiclass results to probability clipping (e.g., 10−410^{-4}) and report log loss separately for queries whose true class is present versus absent from the selected context? This would clarify whether the reported gains reflect improvements beyond the handling of absent classes.

  2. Across the 46 tasks not used for diagnostic development, how frequently does validation select an interior configuration of (f,p,α)(f,p,\alpha), rather than a boundary or corner configuration? More importantly, can the factor responses measured independently on each task predict its validation-selected configuration or its relative performance? Such an analysis would help establish whether the proposed diagnostics generalize beyond the five development tasks, rather than merely explaining the configurations selected on those tasks.

  3. The class-composition experiments are informative, but the real-data intervention also changes other background features. Could the authors provide a more controlled intervention that isolates class composition while preserving the feature distribution and query-support geometry as much as possible? A synthetic intervention with fixed feature vectors and controlled label assignments may be useful for this purpose. Additionally, could the authors explain the negative composition effect observed on Connect-4? In particular, what property of this task causes class-balanced allocation to move the context composition away from the query-batch composition, and why does this appear to be detrimental?

  4. The paper motivates its class-frequency correction by citing prior work on prior-probability adjustment, including Saerens et al. (2002) and Lee (2026), but does not directly compare the proposed correction against these approaches. Could the authors benchmark the validation-selected power correction against EM-based prior re-estimation and the relevant DistPFN correction? More generally, why is the restricted family β∈{0,0.5,1}\beta\in\{0,0.5,1\} sufficient, given that sensitivity to context class frequencies may vary across predictors, context sizes, and tasks? A comparison across different model and context-size settings would help establish the robustness of this design choice.

  5. The runtime comparison in Section 5.3 reports end-to-end inference costs, but appears to exclude the preceding validation search used to select PAIR's configuration. Could the authors report the total configuration-selection cost, including inner validation, and provide an amortized comparison for different numbers of subsequent query batches? Additionally, although CRUMB with K=4K=4 and K=16K=16 is already evaluated in Figure 7, could the authors provide the corresponding numerical predictive results and runtime comparisons in a directly comparable table, ideally alongside the main benchmark results?

  6. The primary classification experiments use frozen one-estimator TabPFN-v3. Do the main conclusions hold when using the default multi-estimator TabPFN-v3 configuration? Additionally, could the authors provide a sensitivity analysis with independently validation-tuned baselines, particularly Facility Location's kernel scale ss and Pooled kNN's neighborhood depth? Although the paper includes matched-tuning controls, it is unclear whether the reported advantage is robust to a broader and equally competitive tuning procedure for these baselines.

11 comments

We are grateful to the reviewer for the close reading of the paper and its appendices, and for the precise questions. We answer each point below. Where a weakness and a question concern the same issue, we address the weakness briefly and provide the detailed response under the corresponding question to avoid repetition.

W1. "Class-composition effects are not fully isolated. … the real-data intervention exchanges examples while changing their feature composition … A more controlled intervention would keep the feature vectors fixed and change only the labels or class assignments."

The exchange of rows is the intervention that matches the question of our analysis, which is how the choices of a retriever change the prediction: a retriever acts on a context by exchanging rows, and each row carries its features together with its label. The effect that the exchange measures is therefore the one that a retriever actually produces when it changes the class composition of a context.

Following your suggestion, we have added the label-only intervention as a control in Appendix D. It complements the exchange by showing how much of this effect comes from the labels alone, and the results are given under Q3.

W2. "The large increase in augmented R² reported in Table 10 is potentially interesting, but its interpretation is complicated by the extremely small probabilities assigned to absent classes … It is not sufficiently clear how much of the explanatory power comes from geometric support information versus the distinction between class-present and class-absent observations."

The two are closely related: a class that is absent from the context is the limiting case of local class support, in which the query has no same-class row at all.

To make the decomposition explicit, we have added a second analysis to Table 10. The table now reports the results on all observations and, separately, on the observations in which the true class is present, so that the two contributions can be read off directly. Among the observations in which the true class is present, the label-aware support variables still add explanatory power on every task and in every fold. The sensitivity to probability clipping is addressed under Q1.

W3. "The computational comparison does not fully characterize the cost of obtaining a reusable PAIR configuration. … the total cost of configuration selection and the amortized cost across subsequent inference workloads remain unclear."

The cost of selecting a configuration is paid once and is then shared by every later query batch, and the search can also be skipped altogether. Once a configuration is selected, the cost of each query batch is comparable to that of the other shared-context selectors. Without any search, the untuned default of PAIR already ranks second in Table 2, ahead of all the other methods.

As you requested, we have added the total cost of configuration selection and its amortized cost over subsequent query batches to Appendix H; the numbers are given under Q5.

W4. "The theoretical guarantee is disconnected from the predictive objective. The theorem establishes a greedy approximation guarantee for the coverage surrogate, not for the predictive loss of the tabular foundation model."

The theoretical guarantee is disconnected from the predictive objective because the predictive loss of a TFM is an empirical quantity: it is defined by the forward pass of a trained network and is known only by evaluating it. The direct assurance about the predictive performance of PAIR therefore comes from experiments, and the theorem covers the part that can be analysed, namely the greedy approximation guarantee for the coverage surrogate, which holds under every configuration in the search space.

The experiments connect the coverage surrogate to the predictive loss: the parameters of the surrogate are chosen by the validation loss of the TFM itself, so the predictor is part of the selection loop, and the resulting contexts rank first under all seven TFMs that we evaluate (Section 5.5 and Appendix L).

W5. "The generality of the diagnostic findings remains uncertain. Some diagnostic analyses are conducted on a small set of development tasks. It is unclear whether the identified conditions reliably predict which configurations will work best on previously unseen tasks."

To establish the generality of the diagnostic findings, we have added an analysis on 46 tasks outside the development tasks (new Appendix D.5). On these previously unseen tasks, and with contexts that are not produced by PAIR, the four properties account for most of the explainable variation in batch loss, so the findings extend beyond the five development tasks. The results are given under Q2.

Regarding which configurations will work best on a new task, PAIR determines this by validation on that task, and the four properties define the space in which the validation searches, since each mechanism of PAIR controls one of them. We have rewritten Section 3 to make this role clear.

Q1. "Could the authors provide complementary metrics such as accuracy, balanced accuracy, macro-F1, and ECE for the main comparisons …, with the decision threshold and calibration protocol specified? More importantly, could the authors assess the sensitivity of the multiclass results to probability clipping … and report log loss separately for queries whose true class is present versus absent from the selected context?"

As you requested, we have added accuracy, balanced accuracy, macro-F1, and ECE for the main comparison in the new Appendix I (Table 30), with the protocol specified: predictions use argmax without additional calibration, and ECE uses top-label confidence with 15 equal-width bins. Among these complementary metrics, PAIR has the best balanced accuracy overall, and the best accuracy and macro-F1 on the multiclass tasks. The submission reported ROC AUC and log loss because they are the official metrics of TabArena and therefore the standard for this benchmark; they are threshold-free and score the full predicted distribution, which is exactly what a shared context changes. We keep them as the primary metrics and report the other four as complements.

The reported gains reflect improvements beyond the handling of absent classes. The multiclass results are insensitive to probability clipping: PAIR ranks first at each of the three thresholds that we tested (10⁻¹², 10⁻⁶, and 10⁻⁴). PAIR also ranks first when the comparison is restricted to the queries whose true class is present in the context of every method.

Q2. "Across the 46 tasks not used for diagnostic development, how frequently does validation select an interior configuration of (f, p, α) …? More importantly, can the factor responses measured independently on each task predict its validation-selected configuration or its relative performance?"

Interior configurations. On the 46 tasks, validation selects a configuration with at least one interior parameter on 40 tasks and a baseline corner on 2 (both Facility Location); this is now stated in Appendix C.

Predicting performance. The new Appendix D.5 measures the four properties on contexts drawn by generators that do not use PAIR, on the 46 tasks split into two halves by source dataset, and predicts the batch loss on the held-out half. The four properties alone reach a held-out R² of 0.64, against 0.70 for a flexible model over 39 candidate properties.

Predicting the configuration. The paper does not use the properties to predict the configuration of a task; PAIR obtains it by validation on that task. A fixed configuration also transfers to unseen tasks: a default chosen without the target task is better than Facility Location on 33 of the 51 tasks, and the untuned default of the paper ranks second in Table 2, ahead of all the other methods.

Q3. "Could the authors provide a more controlled intervention that isolates class composition while preserving the feature distribution and query-support geometry …? Additionally, could the authors explain the negative composition effect observed on Connect-4?"

Controlled intervention. We ran this intervention on the synthetic task and on the five real tasks, keeping every feature vector of the context fixed and changing only labels (Appendix D). This separates the contribution of the labels from that of the positions of the exchanged rows, and the results are reported there for each task. The response to an exchange depends on where the added same-class rows lie as well as on their labels, which is the situation that a retriever faces. Section 3 now describes class composition as which classes a context contains and in what proportion.

Connect-4. The label-only change on Connect-4 is small, with an interval that spans zero, so the negative contrast in Table 1 comes from the features of the exchanged rows.

Regarding class-balanced allocation: the three classes of Connect-4 are strongly imbalanced (about 10%, 25%, and 66%), and all three are already present in a context of 100 rows. Balancing the classes therefore adds no class coverage and only moves the class proportions of the context away from those of the query batch, so that the smallest class is predicted far more often than it occurs. On this task, validation selects a configuration of PAIR without class reservation.

Q4. "Could the authors benchmark the validation-selected power correction against EM-based prior re-estimation and the relevant DistPFN correction? More generally, why is the restricted family β ∈ {0, 0.5, 1} sufficient …?"

As you requested, we have benchmarked the validation-selected power correction against EM-based prior re-estimation and the DistPFN corrections in the new Appendix J. Eight corrections are applied to the same predictor outputs on the contexts selected by PAIR, so that the comparison isolates the correction.

Mean rank among eight corrections, 51 tasksAllBinaryMulticlass
β ∈ {0, 0.5, 1}, validation-selected (paper)2.972.653.48
β ∈ {0, 0.25, …, 1.5}, validation-selected3.183.163.20
No correction3.803.893.67
Saerens EM7.427.407.45
EM or none, validation-selected3.744.063.23
DistPFN5.946.035.80
DistPFN-T5.385.005.97
DistPFN, DistPFN-T, or none, validation-selected3.573.813.20

The correction of the paper has the best mean rank overall and on the binary tasks, where it is also first under each of the four predictors.

The restricted family β ∈ {0, 0.5, 1} is sufficient: a denser grid is statistically indistinguishable from it, since the interval of their rank difference spans zero and the two tie on 31 of the 51 tasks. We therefore use the smaller grid, which also keeps the number of scored rules in the validation search small.

Comments on the paper

1 comment
Area_Chair_VXRwOpenReview ↗

Prompting for Discussion

Hi Reviewers,

The authors have replied to your comments. Can you please engage in discussion? Have your views changed based on this reply?

Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.