- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Intervention-Aware Hypothesis Search (IAHS), an AutoML-like method for OOD tabular regression. It combines feature generation, model fitting, model selection, and post-processing. On synthetic math-benchmark derived tabular data and other tabular data, it substantially outperforms regular non-OOD-focused regression methods, while being competitive on in-distribution data.
Critical strengths
- The method performs much better than all compared baselines in the OOD setting, while also being competitive in the IID setting.
Critical weaknesses
-
The core method, IAHS, is only vaguely described in the main paper (Section 3). This may partially be due to its complexity, but I think a better description should be possible when saving space e.g. on vague summaries (section 3, section 3.1). The paper uses non-standard terminology throughout, e.g. source data instead of training data, hypothesis generation instead of model fitting/training, source-derived evidence instead of validation scores/metrics (although this seems to include other criteria), evidence-guided adoption instead of model selection, changing the representation instead of feature engineering, and so on. Equations are generic (e.g. l. 232 and l. 272). Using an algorithm environment could help. While the method to my understanding falls in the field of AutoML, no references to AutoML are made.
-
The benchmarks contain no OOD-specific baselines. I am not an expert in these, but surely something like symbolic regression or methods used in the WildTab paper could be relevant? On the feature engineering part, TabPrep could be relevant, or generally AutoML methods since the present method is, to my understanding, an AutoML method.
-
Runtimes are not reported and limitations are not discussed.
Questions for authors
No questions / see weaknesses.