- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes INCONVENIENT, a benchmark for tabular prediction when the training data come from a biased sample but the test data come from the full population. Starting from 200 public datasets, the authors select training subsets in which the relationship between one feature and the target is made stronger, weaker or reversed, and they provide knobs to control how severe the shift is. Standard tabular models, including gradient boosted trees and TabPFN, lose a large part of their performance under these shifts. A small experiment also suggests that an LLM can partly tell which feature relationships changed the most.
Critical strengths
S1: The setting is worth studying. The deployment population stays fixed and only the training sample is unrepresentative, which differs from the domain split design of WILDS, TableShift and WhyShift. Changing P(Y|X) with a controlled direction and strength is a good idea for a benchmark.
S2: The benchmark is large, with 200 datasets and about 15,500 shifts, and it comes with a command line tool that exposes the severity knobs. The severity analysis is reported honestly, including the finding that the magnitude knob does not reliably control shift difficulty in Section 5.3
Critical weaknesses
W1: The main purpose of the benchmark is not evaluated. The title and abstract present INCONVENIENT as a benchmark for LLM generalization, and Section 5.1 argues that "correction must therefore come from outside the data". However, no LLM based predictor or prior elicitation method is evaluated, although TabLLM, LIFT, AutoElicit, LLM Processes and LoID are all cited in Section 2. The only LLM result is a Recall@5 probe with one model, compared only with random ranking. A simple heuristic that ranks features by the size of their shifted coefficients is needed before the gain can be attributed to domain knowledge. Many of the datasets, such as Titanic, Boston, California Housing, Pima diabetes and COMPAS, may also have been seen during LLM pretraining, which is a contamination problem for exactly the methods the benchmark targets. Without at least one method that closes part of the gap, it is not possible to tell whether the benchmark is solvable or whether it can separate methods.
W2: The construction and the filtering make the main results hard to interpret. (a) The influence score in Eq. (5) is built from each example’s residual, so the selection depends directly on the label, and Appendix A.1 itself calls it residual-conditioned selection. This is closer to targeted data poisoning than to the examples in the introduction, such as one hospital or one platform, where selection mostly depends on covariates or site. (b) can be as small as 0.1, so a convenience set can be a small part of the pool, while the IID model is trained on the whole pool. The reported drop therefore mixes the effect of fewer samples with the effect of a distorted relationship, and a random subset of the same size is needed as a control. (c) Shifts with a performance drop below 0.1 are removed (Section 4.1, step 6), so an average drop of at least 0.1 is guaranteed by design. Table 1 in the appendix also shows different shift counts for different methods under the same configuration. For binary tasks with attenuation, bin 0 and of 0.1, there are 80 shifts for CatBoost, 103 for LightGBM and 56 for logistic regression. This suggests the filter is applied per method, so each method is averaged over a different set of shifts, and statements such as Ridge and MLP "suffer among the largest degradations" are not like for like comparisons.
W3: There are several inconsistencies, and the presentation makes the benchmark hard to assess. (a) Figure 1 shows normalized classification drops of about 0.02 to 0.06. This cannot happen if every retained shift has an absolute drop of at least 0.1, unless Figure 1 uses unfiltered shifts, which is not stated. (b) The conclusion says the baselines "lose between 20% and 45% of their performance", while Section 5.1 reports 20% to 29% for classification and 53% to 82% for regression. (c) In Appendix A.2 the stated counts do not match the lists. Regression from TALENT says 7 but lists 6, binary from OpenML says 23 but lists 25, binary from Kaggle says 11 but lists 9, and multiclass from OpenML says 78 but lists 81. At least 13 datasets appear in both the regression and the multiclass lists, for example Abalone, Airfoil Self Noise, Concrete Compressive Strength, Kin8nm and Wine Quality, and several entries appear to be copies of the same source, such as the Pima diabetes, wine quality and medical insurance files. The real diversity is therefore below 200 datasets. (d) Most results are averaged plots and a configuration table in Appendix A.3 that runs for more than ten pages, with little analysis per dataset or per domain. In Figure 2 most of each panel is empty, and in Figure 1 the methods are separated only by color with the same marker. There is also no figure that shows how a shift is constructed.
Questions for authors
Q1 (see W1): Please evaluate at least two methods that use LLM knowledge, for example TabLLM or LIFT style prediction, or AutoElicit or LoID style priors used to regularize a linear model, together with a strong data only baseline, and report whether any of them reduces the drop. For Section 5.4, please add a baseline that ranks features by coefficient size, and a contamination check, such as asking the LLM to reproduce the reference coefficients of well known datasets from feature names only.
Q2 (see W2): What is the ratio between the convenience set size and the pool size across the retained shifts? Could the authors add a random subset of the same size as a control and report the drop relative to it? Is the 0.1 filter applied per method or per shift? Please report results without the filter and on the set of shifts shared by all methods.
Q3 (see W2): Can the selection procedure be checked against at least one real case where the sampling mechanism is known, for example a site or covariate based selection, or the head and neck cancer example cited in the paper? A covariate based selection mode could also be offered next to the residual based one.
Q4 (see W3 ): Please state which filter is used in Figure 1, reconcile the 20% to 45% in the conclusion with Section 5.1, and explain the dataset counts in Appendix A.2 and how duplicate sources were handled. I also suggest adding a figure that shows how a shift is constructed, moving some analysis per dataset or per domain into the main text, and using distinct markers or line styles with a colorblind friendly palette in Figure 1, so that the methods can be distinguished without relying on color.
5 comments
Thank you for the detailed response. I accept the argument on label-dependent selection, and I have raised my soundness score accordingly
For W1, I would like the authors to clarify one point. In the new AutoElicit experiment, the flat-prior Bayesian baseline seems to perform better on the convenience samples than all baselines in Figure 3. I also suggest adding TabLLM, LoID and AutoElicit to the main results
Some points in my review are not addressed in the response,see w3c. The description of the filter is also inconsistent. The response to Reviewer QPho says the 0.1 filter is computed per shift from the best baseline. However, the number of shifts in Table 1 seems to differ across methods under the same configuration
The PDF on OpenReview has not been updated yet, so I cannot verify the new experiments and revisions in the response. For these reasons, I keep my overall rating.
Thank you for your prompt response!
The updated manuscript is uploaded and we applied all of the requested changes. We would truly appreciate it if you could review it.
-
The experiment in conducted on a randomly selected subset of 1{,}570 datasets. We removed this detail for the sake of brevity, but it is mentioned in the updated version of the manuscript.
-
We thank the reviewer for checking the dataset list carefully. Some datasets with similar names are distinct (for example, they differ in rows, region or time period), but others are copies of the same source obtained from different repositories. We have deduplicated the datasets within each task type by comparing their contents, corrected the counts in Appendix A.2, and rerun all results on the deduplicated set. [The main findings are unchanged: e.g., average degradation changes from X to Y.] Datasets that appear in both the regression and multiclass lists are retained, since each is used with a different target formulation. The benchmark now includes 165 datasets and roughly 13,000 shifts.
-
We exclude baselines that degrade less than 0.1 compared to their own in-distribution performance from the table. Therefore, the shifts that are kept in the dataset have best_iid_performance - best_ood_performance > 0.1. Some methods fail to fit the in-distribution data as well as others. Therefore, their own iid - ood performance falls below 0.1. Table 1 excludes these methods. We acknowledge that this is an important distinction and have mentioned it in the description of Table 1 (Table 3 in the revised manuscript).
Thank you for your insightful review! Please find our responses below:
W1:
We originally excluded LLM-based methods from the benchmark because none of the ones we tested (TabLLM, LoID, AutoElicit) beat the best ML model (on avg 0.018 worse). Each method has a structural handicap. TabLLM sees only a few labeled examples of the train set in its context. AutoElicit and LoID only produce priors for linear models, so comparing them to tree-based models, which can fit more complex functions, is unfair. Their underperformance therefore might not mean that the LLM lacks the relevant knowledge.
Comparing AutoElicit to Bayesian regression with uninformative priors is fair since the methods only differ in the priors used. This isolates the LLM's contribution to downstream performance, and the results confirm it:
Binary tasks: AutoElicit increased AUC 0.682 to 0.811. The full-data fit AUC is 0.821. Regression tasks: AutoElicit raised mean R² from 0.482 to 0.502. The full-data fit R² is 0.526. Coefficients: the elicited priors cut the mean squared error of the coefficients, measured against the full-data coefficients, by 70%. We added the justification for this exclusion and the above analysis to the revised manuscript.
W2-(a):
Selecting on covariates alone can’t create this shift. If selection depends only on X, P(Y|X) doesn’t change, and a well-specified model still learns the right relationship. Label dependence is therefore the mechanism, not an artifact.
Real convenience samples distort relationships through the same mechanism: site effects driven by severity or referral patterns, outcome-dependent recruitment, and collider bias all select on the outcome given the features. The head and neck cancer recurrence cohort is a real example. In this single-institution cohort of 508 patients, a standard model learns that marijuana protects against recurrence, contrary to the clinical literature (Rezaeimanesh et al., 2026). Similarly, early COVID-19 studies of hospitalized patients suggested a spurious protective effect of smoking due to collider bias (Griffith et al., 2020).
INCONVENIENT is a controlled stress test. As in ImageNet-C, synthetic shifts give you ground truth: which relationship changed, in which direction and by how much. Natural-shift benchmarks can’t provide that, and both the severity analysis and the probe depend on it.
The worst-case tilt. We agree that greedy selection leans toward worst-case shifts, but the severity knobs moderate this. Unlike poisoning, no rows are changed or injected. Every example is a real population row, and only the sampling is biased.
W2-(b):
We have added the requested size-matched control to the Appendix. We drew three random subsets of the IID pool, each the same size as the convenience set, and trained all six baselines on them using the same tuning protocol as in the main experiments. The results show that the distortion contributes more to the performance drop:
We also note that small size is itself a defining property of real convenience samples. The total drop therefore reflects the penalty a practitioner would actually face. We now report this in the manuscript.
W2-(c):
>0.1 degradation is a feature of the benchmark rather than an empirical finding: INCONVENIENT contains only shifts with a large OOD–IID gap that standard models cannot close, which leaves room for methods that use information from outside the data. We agree that the wording can be misleading and have revised the introduction accordingly.
Per your suggestion, we also limited our baseline comparison analysis to shifts that are shared by all the baselines. Now we can correctly confirm that ridge and MLP suffer the largest degradations.
W3-(a):
Figure 1 includes all the possible shifts (without the filtering) to show how the knobs effect performance. We added this to the table description.
W3-(b,c):
We apologize for the inconsistencies. Thank you for pointing them out. Section 5.1 reports the correct numbers and the conclusion is now fixed. We also fixed the inconsistencies in the data counts (The dataset count remains 200).
Datasets that appear in more than one category have multiple types of targets. For example, the household income has both a regression type target for income and a classification type target for when whether a household is high income. It is true that this means the 200 datasets are not completely distinct. We will flag this in the manuscript.
W3-(d):
We made all the figures color-blind friendly. We will also add a figure that explains how the shifts are made.