acceptodds
Withdrawn from ICLR 2027

Beyond Fit: Intervention-Aware Hypothesis Search for OOD Tabular Regression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern tabular predictors perform strongly in-distribution but often degrade under out-of-distribution (OOD) shifts, especially when deployment requires extrapolation beyond observed support. A key difficulty is that fitting the source data does not uniquely determine how predictions should continue outside that support: different predictive rules can explain the observed data yet imply very different extrapolations. This suggests that OOD tabular regression requires not only learning or selecting a strong predictor, but also constructing alternative predictive hypotheses and deciding which of them should be used at deployment. We introduce Intervention-Aware Hypothesis Search (IAHS), a framework that separates these two problems by constructing executable hypotheses through representations, predictor families, predictive roles, and compositions, and then using source-derived evidence, permitted unlabeled deployment information, and deployment-specific constraints to guide protocol-conditioned adoption without target labels. On TabularMath, hypothesis construction increases the retrospective mean per-task best OOD from 0.3006 to 0.8177, while Source-score Top-1 selection on the same candidate pool reaches only 0.0625 compared with 0.5829 for IAHS. Across additional regression benchmarks, IAHS improves mean OOD over TabPFN v3 by 0.619 while maintaining comparable non-OOD performance. These results show that reliable extrapolation depends on both which predictive alternatives are made available and how they are selected under the evidence available at deployment.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Tabular data regreession, tabular learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aQs6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Intervention-Aware Hypothesis Search (IAHS), an AutoML-like method for OOD tabular regression. It combines feature generation, model fitting, model selection, and post-processing. On synthetic math-benchmark derived tabular data and other tabular data, it substantially outperforms regular non-OOD-focused regression methods, while being competitive on in-distribution data.

Critical strengths
  • The method performs much better than all compared baselines in the OOD setting, while also being competitive in the IID setting.
Critical weaknesses
  • The core method, IAHS, is only vaguely described in the main paper (Section 3). This may partially be due to its complexity, but I think a better description should be possible when saving space e.g. on vague summaries (section 3, section 3.1). The paper uses non-standard terminology throughout, e.g. source data instead of training data, hypothesis generation instead of model fitting/training, source-derived evidence instead of validation scores/metrics (although this seems to include other criteria), evidence-guided adoption instead of model selection, changing the representation instead of feature engineering, and so on. Equations are generic (e.g. l. 232 and l. 272). Using an algorithm environment could help. While the method to my understanding falls in the field of AutoML, no references to AutoML are made.

  • The benchmarks contain no OOD-specific baselines. I am not an expert in these, but surely something like symbolic regression or methods used in the WildTab paper could be relevant? On the feature engineering part, TabPrep could be relevant, or generally AutoML methods since the present method is, to my understanding, an AutoML method.

  • Runtimes are not reported and limitations are not discussed.

Questions for authors

No questions / see weaknesses.

Reviewer_Hdp2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses extrapolative out-of-distribution (OOD) regression on tabular data. It argues that source fit does not determine how a predictor should continue beyond the observed support, yet a designed adoption policy is needed to extrapolate.

Section 3 : Methhodology

IAHS is a two-stage procedure :

  • Section 3.2 : Hypothesis construction builds complete prediction rules h = (φ, f, r, c) by varying four components: the representation (powers, logarithms, products, safeguarded ratios), the predictor family (linear models, tree ensembles, pretrained tabular models), the predictive role, and prediction-level compositions (bias/affine corrections, source-quantile floors, aggregations).
  • Section 3.3 and Sesction 3.4 : Evidence-guided adoption then selects one rule with a protocol-conditioned policy. The policy combines source-side boundary and pseudo-OOD validation, candidate predictions on the unlabeled deployment batch, and task constraints such as a known shift direction.

Section 4 : Experiments :

On 114 TabularMath tasks, IAHS reaches a target-high OOD mean R² of 0.583 (median 0.998), versus −0.738 for TabPFN v3. IID performance is on par with the strongest baselines. On a second suite of 69 cases (optical dispersion, Tecator, ESOL/FreeSolv, TabArena), IAHS reaches a mean OOD R² of 0.247, versus −0.372 for TabPFN v3. A retrospective oracle analysis reports that construction raises the mean per-task best OOD R² from 0.30 to 0.82. Source-score Top-1 selection on the same pool reaches only 0.06.

Critical strengths

I see two strenghts :

  • A useful decomposition of OOD regression into candidate capacity and deployable selection.
  • Protocol-matched evidence that pretrained tabular models extrapolate poorly and that representation choice matters. Even when TabPFN v3 and TabICL v2 receive the same source-maximum floor, IAHS wins 98/6/10 and 99/5/10 (Table 8). The appendix reports these controls, paired bootstrap intervals and case non-independence transparently. This is the strongest evidence that the results are not purely a protocol artifact, although, as discussed under Weakness 1, a much simpler baseline captures most of the gain.
Critical weaknesses

I have two major concerns for this paper :

  1. IAHS is not specified or designed as a general procedure, and the evidence for generality beyond TabularMath is very weak.
  • The method is a set of protocol-specific policies. It is realized as separate hand-designed policies selected by the protocol label (target-high, IID, reference-based), with unspecified criteria (“sufficient source-side evidence”, “strongly source-supported”). Table 4 lists components (“LLM mode”, “diagnostic-triggered degree-three regression”) that are never described. Without code, the method cannot be reimplemented. The hypothesis already carries a lot of induced knowledge (high target etc), which are then used to better fit OOD data points.
  • Both OOD suites match the method’s inductive bias. TabularMath targets are arithmetic programs largely built from the operations in the feature library. The optical cases (20 of the 28 OOD cases) are one-dimensional dispersion curves in which λ² and 1/λ² are the Sellmeier/Cauchy terms. Yet symbolic regression, automated feature engineering, and polynomial or physics-based fits are not compared.
  1. The headline TabularMath gains rely on label-derived protocol information and on narrowly targeted branches, and they do not survive the paper’s own matched controls.
  • The split leaks label information. The target-high split assigns the top 20% of responses to the test set, so every test response is at least the source maximum by construction. The source-maximum floor (Eqs. 7, 10) and the tail-capability check (Eq. 5) exploit exactly this. Target responses are indeed never used, but the protocol itself is defined by them.
  • This information drives the results. All 114 OOD selections are source-maximum floors (105) or tail-bias rules (9) (Fig. 5). Removing the protocol priority lowers the OOD median from 0.998 to 0.662 (Table 6). Yet Table 1, Fig. 1(d) and Sec. 4.2 compare against baselines that lack this information.
  • Tuning on the evaluation tasks cannot be ruled out. The development protocol is not described, so this hierarchy may have been tuned on the same tasks used for evaluation. This is consequential because the central claim, that construction plus evidence-guided adoption yields reliable extrapolation, rests on these numbers.
Questions for authors
  • How much of the performance remains when the shift direction is withheld or misspecified? And when each protocol-specific policy is applied to the other split type?

Suggestions for a revision or future work

More baselines and benchamarks would be for my concerne mandatory, with a broader range of tests

Reconsider the name “intervention” defined at the end of Sec. 3.4 as changing the deployed prediction rule, which will mislead causal-ML readers. Intervention is a very specific term in causal AI community.

Reviewer_Q8LzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Intervention-Aware Hypothesis Search (IAHS) for out-of-distribution tabular regression. The framework constructs alternative predictive hypotheses through feature transformations, different predictor families, and prediction-level corrections, then selects a hypothesis using source-side evidence and deployment-specific constraints without accessing target labels. Experiments on TabularMath and additional regression benchmarks demonstrate substantial improvements over standard tabular predictors.

Critical strengths

The paper highlights an important limitation of existing tabular predictors: good in-distribution performance does not guarantee reliable extrapolation. Separating hypothesis construction from hypothesis selection provides a useful perspective on OOD tabular regression.

IAHS demonstrates substantial improvements over several strong tabular baselines. The authors also provide useful ablations distinguishing hypothesis construction, adoption policies, and prediction-level corrections.

Critical weaknesses

A substantial portion of the performance gains on TabularMath appears to come from exploiting the known target-high direction through prediction-level corrections. In particular, 105 of 114 OOD cases adopt source-maximum-floor rules. Although IAHS substantially outperforms standard tabular models, a simple degree-two sparse polynomial predictor with the same source-maximum floor already achieves an OOD mean R² of 0.5642, compared with 0.5829 for IAHS. The reported confidence interval for their mean difference includes zero. This raises questions about how much additional benefit is attributable to the proposed hypothesis search and adoption framework.

IAHS relies on different, relatively complex adoption policies for different deployment protocols. The TabularMath evaluation uses target-high-specific rules, while the additional benchmarks adopt a reference-based strategy. Although both share the same conceptual framework, it remains unclear how well a single adoption strategy generalizes to previously unseen shift types. Moreover, the additional OOD evaluation is dominated by optical-dispersion cases, limiting the diversity of independent experimental evidence.

Questions for authors

Given the strong performance of the degree-two polynomial predictor with a source-maximum floor, can the authors further clarify the additional benefit of the hypothesis search and adoption strategy beyond protocol-specific prediction corrections?

How are the adoption policies determined for a previously unseen OOD setting where the shift direction is unknown? Can the authors provide evidence that the framework generalizes without designing protocol-specific selection heuristics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.