- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TFMs are an important discovery for tabular tasks. They have to rely on in-context examples for task awareness/definition, and batching is required for deployment efficiency. Under these constraints, the RQ is how to choose the right context for best performance for any query batch. Overall, this poses an interesting problem of how to fill in the context with the best examples to maximize the task performance.
The authors essentially create a value function for contexts given a query batch, f(c, Q_batch), and use a retrieval formulation + submodular function for the final objective leading to an efficient context construction mechanism for every Q_batch.
Proposed method: "we propose Preference-Aware Integrated Retrieval (PAIR), which values a candidate's contribution according to the coverage each query already receives. Its three adjustable choices, namely how coverage accumulates over multiple neighbors, how additional coverage is valued across queries, and how much of the budget is reserved across classes, define interpretable retrieval preferences selected through validation."
Core factors that govern context quality, as per the authors: "(i) local class support: examples from the query's class tend to provide greater predictive benefit when they are closer to the query. (ii) support accumulation: marginal gains diminish with accumulated support, so that the same candidate yields different gains in different contexts. (iii) allocation across queries: under a shared budget, strengthening one query's support can come at the cost of another's. (iv) class composition: the context's class composition shifts predictions even with local support held constant."
Critical strengths
- They contribute an integrated formulation for scoring S across all four proposed factors.
- Simple intuition based and it works. It is clear why this approach works well, and it is interpretable in terms of what's helping the prediction. AUC-ROC gains are strong compared to other baselines.
Critical weaknesses
Critical Weaknesses
- Section 3.2 / 3.3 lacks clarity on the problem setup. Numbers, problem setup and analysis definitions are lacking and formulations not shown properly. E.g. how Fig 2 metrics are computed, or Table 1: what are these metrics?
- Final conclusions from Section 3.2 and 3.3 are trivial and intuitive, unclear what we gained from there.
- "Section 3 identifies four factors that govern context quality, with response magnitudes that vary across tasks." Claim not well supported. Cyclical self-justification. These four factors were the intuitive hypothesis over what context properties might affect the predictor's performance, not an exhaustive search process to "identify" factors that govern quality.
- This is simply in-context learning; it's obvious that finding examples similar to the query and putting a diverse set in the context would help, so solution novelty is unclear. Also, the problem itself is not a totally unsolved problem; a lot of work has been done on exemplar selection in in-context learning. Would like to see those methodologies properly brought into the picture for comparison.
- AUC-ROC missing for untuned PAIR in Figure 3. That is critical to show.
- The same conundrum of whether this is a worthwhile problem in the long run. Same issue as LLMs: prompt tuning or in-context learning that was a big hype a year ago are not useful problems to solve anymore. Show gains with other newer / larger TFMs (newer version of TabPFN, Kumo, LimiX etc.): whether the problem itself exists there, and then whether complex solutions are required or a simple policy of nearest samples for given queries, constrained by class balance and sample diversity in the shared context, suffices.
- Dependent on an external representation method for the similarity function between queries and candidate examples. Details for this are not provided. How is this trained in data-scarce regimes like tabular prediction tasks?
Questions for authors
Questions For Authors
- How fast is the approach and how easy is the setup? Can c be reformulated for any Q_b, any incoming task?
- Please share details of how to setup the similarity function for each task.
3 comments
W3. "'Section 3 identifies four factors that govern context quality …' Claim not well supported. Cyclical self-justification. These four factors were the intuitive hypothesis over what context properties might affect the predictor's performance, not an exhaustive search process to 'identify' factors that govern quality."
We agree that the four properties were chosen and were not found by a search. In the revised paper they are the terms in which we describe how tasks differ in what a shared context needs. To make this clear, we have rewritten Section 3 and the corresponding sentences in the abstract, the introduction, and the conclusion.
We have also added a data-driven check on 46 tasks outside the development set (new Appendix D.5). We computed 39 candidate properties of contexts that were not produced by PAIR, including diversity, density, and distribution-matching properties, and fitted models that explain the batch loss on held-out tasks. The four properties alone reach a held-out R² of 0.64, against 0.70 for a flexible model over all 39 properties, and removing any of the other groups of properties changes the fit only marginally.
W4. "This is simply in-context learning; it's obvious that finding examples similar to the query and putting a diverse set in the context would help, so solution novelty is unclear. Also, the problem itself is not a totally unsolved problem; a lot of work has been done on exemplar selection in in-context learning. Would like to see those methodologies properly brought into the picture for comparison."
Solution novelty. The novelty of PAIR is that it adapts the balance between similarity and diversity to each task. Finding examples similar to the query and putting a diverse set in the context are each a fixed policy in our comparison, namely Pooled kNN and Facility Location. Neither of them is consistently better than the other across tasks, and PAIR improves on both.
We have also added a comparison in which the parameters of these two baselines are tuned on validation data, namely the kernel scale of Facility Location and the neighborhood depth of Pooled kNN (new Appendix K). PAIR remains first in this comparison.
The problem. The problem that we study is different from exemplar selection in in-context learning. Exemplar selection typically chooses a few demonstrations for a single query, whereas here B rows are chosen out of a large labeled pool once for a whole query batch. One context therefore serves many queries at once, and the classes that it contains determine which classes receive probability.
Comparison. The methodologies for this problem are in our comparison: Pooled kNN, Facility Location, CRUMB, and VIP-COp. To our knowledge, these are the available methods for constructing a shared context for a TFM.
W5. "AUC-ROC missing for untuned PAIR in Figure 3. That is critical to show."
Thank you for this suggestion. In Figure 3, the PAIR curve at B≥500 is already the untuned default, which the caption now states, so the two curves differ only at B=200. The AUC values are given below; the untuned default is above both baselines at every budget.
| Mean AUC, 17 binary datasets, four predictors | B=200 | 500 | 1,000 | 2,000 |
|---|---|---|---|---|
| PAIR (validation-selected at B=200) | 0.788 | 0.808 | 0.821 | 0.830 |
| PAIR, untuned default | 0.783 | 0.808 | 0.821 | 0.830 |
| Pooled kNN | 0.761 | 0.799 | 0.815 | 0.827 |
| Facility Location | 0.766 | 0.798 | 0.814 | 0.824 |
W6. "The same conundrum of whether this is a worthwhile problem in the long run. … Show gains with other newer / larger TFMs (newer version of TabPFN, Kumo, LimiX etc.): whether the problem itself exists there, and then whether complex solutions are required or a simple policy of nearest samples for given queries, constrained by class balance and sample diversity in the shared context, suffices."
Whether the problem persists. A TFM predicts only through its context. For an LLM, in-context examples are one of several ways to specify a task, whereas a TFM has no other access to the training data. The size of the context therefore determines the inference time and memory of any such model, and selecting what enters a bounded context remains necessary as the models become larger. The closer analogy in LLMs is the handling of long contexts, which has remained an active problem. We have rewritten the opening of the introduction to make this point.
Newer TFMs. As you suggested, we evaluated three more recent TFMs: TabPFN-3.5, LimiX-2, and Kumo Tabular (new Appendix L). The problem is present on all three, in the sense that the selectors still differ clearly, and PAIR ranks first on each of them, both with the configurations selected on TabPFN-v3 and with configurations reselected on the new model.
| Mean rank of PAIR among seven methods, B=100 | Configurations from TabPFN-v3 | Configurations reselected |
|---|---|---|
| TabPFN-3.5 | 2.16 | 1.91 |
| LimiX-2 | 2.16 | 2.42 |
| Kumo Tabular | 2.06 | 2.00 |
Simple policies. The nearest-sample policy and the diversity policy are Pooled kNN and Facility Location in this comparison, and the advantage of PAIR over both is statistically significant on each of the three models. A fixed configuration of PAIR without any search, the untuned default, ranks second on every model, ahead of all the other methods. A simple policy therefore captures part of the gain, and selecting the configuration on validation data adds to it.
W7. "Dependent on an external representation method for the similarity function between queries and candidate examples. Details for this are not provided. How is this trained in data-scarce regimes like tabular prediction tasks?"
PAIR does not use an external or learned representation, and nothing is trained. The similarity is computed directly from the standardized Euclidean distance between the features of a query and of a candidate, or from the rank of this distance. As this was stated only in the appendix, Section 4.1 now says it explicitly. The method therefore applies unchanged to small datasets, including the 15 small tasks in our evaluation. The details are given under Q2.
Q1. "How fast is the approach and how easy is the setup? Can c be reformulated for any Q_b, any incoming task?"
Speed. Selecting the contexts for a complete test set takes about two seconds, out of about 49 seconds in total (Appendix H, Table 26). In a controlled benchmark, selection stays within a few seconds for training pools of 500,000 rows.
Setup. Nothing is trained. PAIR can be used directly with its untuned default, which requires no search and ranks second in Table 2, ahead of all the other methods. Alternatively, a configuration is selected once per task by a validation search of several minutes per split, and is then reused for all later query batches.
New batches and tasks. The coverage c is computed from the features of the incoming query batch and of the pool, so it is defined for any batch of any task without modification.
Q2. "Please share details of how to setup the similarity function for each task."
No task-specific setup is required. The features are standardized with training-set statistics, and the similarity is a decreasing function either of the Euclidean distance or of its rank among the neighbours of the query. Which of the two is used is selected on validation data together with the other parameters (Appendix C). On the 46 tasks outside the development set, the rank-based similarity is selected on 27 tasks and the distance-based similarity on 19.
We thank the reviewer again and hope that these clarifications and the new results address the concerns.
We appreciate the reviewer's time and the direct feedback on both the presentation and the motivation of the paper. Our answers follow point by point.
W1. "Section 3.2 / 3.3 lacks clarity on the problem setup. Numbers, problem setup and analysis definitions are lacking and formulations not shown properly. E.g. how Fig 2 metrics are computed, or Table 1: what are these metrics?"
Thank you for pointing this out. We have rewritten Section 3. It now opens with the problem setup: the labeled pool, the query batch, the budget, and what a selector chooses. Section 3.2 states what each entry of Table 1 is, namely the difference between two contexts that differ only in the named property, measured as the log-loss difference of the focal query for the first four columns and as the change in true-class probability for class composition. The captions of Figure 2 and Table 1 define every panel and column, and Appendix D describes how the contexts are constructed.
W2. "Final conclusions from Section 3.2 and 3.3 are trivial and intuitive, unclear what we gained from there."
What we gain from Sections 3.2 and 3.3 is how the properties differ across tasks. We agree that each property is intuitive when considered in isolation, but this is not where our contribution lies. The properties compete for the same context budget, and tasks differ in how strongly they respond to each of them, and even in the direction of the response (Table 1). These differences are large enough to change which retriever performs well, which is why the performance of existing selectors depends strongly on the dataset: Facility Location and Pooled kNN each win on a substantial share of our 51 tasks. The four properties therefore characterize a task in terms that a retriever can act on.
PAIR is built on this: it makes the properties adjustable for each task, and it ranks first in the main comparison and under all seven TFMs that we evaluate. The rewritten Section 3 presents the argument in this order to make it clearer.