- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether LLM agents need to control the entire pipeline for relational forecasting or can instead focus on feature engineering. The authors propose a simple pipeline in which an LLM independently generates SQL feature programs from a static database description, while conventional methods handle model tuning and prediction aggregation. Experiments on 21 RelBench tasks show that this approach outperforms expert-engineered features on average and achieves performance close to RelAgent, while requiring substantially fewer LLM requests and input tokens. The proposed approach is also more robust to replacing a proprietary LLM with a smaller open-source model. Further analyses suggest that interactive database exploration provides limited benefits on most tasks, conventional tuning can improve frozen LLM-generated features, and aggregating independently generated programs can be more effective than selecting a single program based on validation scores. Overall, the findings suggest that LLMs can primarily carry out feature engineering in relational learning, without needing to control the entire modeling pipeline.
Critical strengths
-
Large cost reduction with comparable accuracy and greater robustness to weaker LLMs. The proposed pipeline processes roughly 1,200x fewer input tokens and takes about 100x less LLM time per attempt. Its average AUROC is only 0.86 points below RelAgent, with most of the gap coming from two tasks (driver-dnf and user-ignore). Across the other ten, the mean difference is 0.12 points. When using Qwen3-32B in place of GPT-5.2, the pipeline outperforms RelAgent by 4.8 AUROC points. These results are of practical significance: they suggest that much of the benefit attributed to extended agentic search can be achieved with a substantially cheaper conventional methods.
-
Systematic ablations with a useful negative result about validation. The paper's analysis reveals that validation scores are unreliable for selecting programs, particularly when validation sets are small. Selecting a program based on its validation score performs worse than choosing one at random (74.77 vs. 76.01 AUROC). Even when feedback is available, the deployed program is the best among those saved in only 6 of the 12 classification tasks. These findings raise important questions about how agentic AutoML systems should use validation feedback to guide their search.
Critical weaknesses
-
No variance estimates, and the validation conclusion rests on two small tasks. All results come from a single draw of five programs and one feedback trajectory per task. Several headline effects are small enough to potentially fall within run-to-run noise. The general conclusion that "model selection can be left to conventional methods" rests on two small tasks rather than on evidence across the benchmark.
-
Possible data contamination is not addressed. RelBench and its data-scientist SQL features have been public since 2024, before both LLMs used were released. The models may have seen the schemas, tasks, or even the expert features during pretraining. The paper doesn't discuss this risk nor evaluate on data released after the models' training cutoffs.
Questions for authors
Q. Have the authors evaluated the findings with models that were release prior to the benchmark to isolate potential data contamination?
We thank the reviewer for recognizing the practical value of our approach and for their helpful feedback.
W1. We repeated SQL generation, Optuna tuning, and ensembling 3 times, with 5 programs per task in each repeat. The table below reports the prediction quality of our tuned ensemble across all 21 tasks. For each regression task, we divide our ensemble's test MAE by the test MAE of the single-table LightGBM baseline on the same task, then average these ratios across tasks.
Next, we examined how to use the 5 generated programs. We first compared choosing the program with the best validation score with choosing a program at random. On the 12 classification tasks, validation selection scores 0.09 AUROC points higher on average. However, excluding
rel-f1increases this gain to 0.68 points on 10 tasks. For regression, validation selection also reduced the average error: normalized MAE was 0.842, compared with 0.853 for random selection. We revised the broad claim about validation selection in Section 6.3. However, validation can still select a very poor program on an individual task. For example, in 1 run onrel-f1/driver-dnf, selecting by validation score gave 16.25 points lower AUROC than the expected score of random selection. Combining predictions from all 5 programs avoids relying on this single choice and improves mean AUROC by 1.31 points over validation selection. Most of this gain comes fromrel-f1: excluding it leaves a gain of 0.06 points. We keeprel-f1in the primary analysis because these large losses matter even when the average difference between the alternatives is small.We also repeated the comparison of feedback and independent generation, reported in our response to
iUZT(W1 and Q1). The feedback experiments showed no clear average advantage from feedback, despite 53 higher API charges. Overall, independent SQL generation with conventional tuning and ensembling delivered competitive predictive quality at substantially lower LLM API cost than RelAgent.W2 and Q1. We fully acknowledge that modern LLMs could have seen RelBench schemas, task descriptions, or expert SQL features during training. However, please note that this is not specific only to agentic relational forecasting that we study in our work, but is a general problem for evaluating pretrained models on public benchmarks across machine learning. Checking exactly what a model has seen remains an open problem when its full training data is private. Even existing contamination checks can miss prior exposure [1]. Thus, we argue that the possibility of prior exposure alone is too broad as a reason to dismiss an evaluation. The same concern applies whenever we evaluate an LLM on a public benchmark released before it was trained.
Following your suggestion to evaluate on newer data, we used
march-machine-learning-mania-2026[2] to construct a relational task predicting tournament game winners. The 2026 games took place in March and April, after GPT-5.2's published knowledge cutoff of 31 August 2025 [3]. This provides a test on newer outcomes, although prior exposure to historical data or feature ideas remains possible. We used GPT-5.2 for both methods. Each produced 4 usable programs in 5 attempts, and we selected the best program for each method by AUROC on the 2022–2025 validation seasons. Both predictors were fitted on the same 2010–2021 seasons. We used the official 2026 tournament results as test labels. The table below compares prediction quality and LLM requests for our method and RelAgent on the 134 games played. Brier score is the mean squared error of the predicted probabilities, so lower is better.Our Brier score is 24 lower, with lower error in both the men's and women's tournaments. Our method also used 6.4× fewer input tokens and incurred 28 lower API charges. Appendix K reports this experiment, which extends our evidence beyond RelBench. On this additional task, whose test outcomes occurred after GPT-5.2's published knowledge cutoff, independent SQL generation with conventional training achieved lower prediction error and used fewer LLM resources than RelAgent.
[1] Fu et al., Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions, NAACL 2025
[2] https://www.kaggle.com/competitions/march-machine-learning-mania-2026
[3] https://developers.openai.com/api/docs/models/gpt-5.2