acceptodds
Under review as a conference paper at ICLR 2027

LLMs as Feature Engineers: Simplifying Agentic Relational Learning

Abstract

Relational forecasting tasks, such as predicting customer churn from a database of customers, transactions, and products, can be solved by transforming the database into a single table of entity features and training a standard tabular model on it. The main difficulty of this approach is feature engineering, and recent works delegate it to large language models (LLMs). In particular, RelAgent gives an LLM agent control over the entire pipeline: the agent explores the database, writes SQL feature programs, chooses the downstream model and its hyperparameters, and revises all these decisions using validation feedback over many iterations. In this work, we ask which of these responsibilities actually require an LLM. We study a simple pipeline in which the LLM acts only as a feature engineer. Given a static description of the database, it writes several SQL feature programs independently, using a single request per program and receiving no predictive feedback. Model tuning and aggregation of predictions are handled by standard methods. On classification and regression tasks from RelBench, this pipeline outperforms the features engineered by an expert data scientist on average and stays close to RelAgent ( vs. average AUROC on classification). At the same time, one of our programs requires fewer LLM requests and about fewer processed input tokens than one search rollout of RelAgent. When a top-tier proprietary LLM is replaced with a much smaller open-source one, the average AUROC of our pipeline decreases by points, compared with points for RelAgent. Our analysis shows that a static database description is as informative as interactive exploration on most tasks and that conventional tuning further improves frozen LLM-generated features. It also shows that validation scores are a weak guide for choosing between programs: on classification, averaging independent programs outperforms selecting one of them by validation score. These results suggest that LLMs are most useful in relational learning as feature engineers, while model tuning and selection can be left to conventional methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
relational learning, relational databases, large language models, agents, automated feature engineering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qZjLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether LLM agents need to control the entire pipeline for relational forecasting or can instead focus on feature engineering. The authors propose a simple pipeline in which an LLM independently generates SQL feature programs from a static database description, while conventional methods handle model tuning and prediction aggregation. Experiments on 21 RelBench tasks show that this approach outperforms expert-engineered features on average and achieves performance close to RelAgent, while requiring substantially fewer LLM requests and input tokens. The proposed approach is also more robust to replacing a proprietary LLM with a smaller open-source model. Further analyses suggest that interactive database exploration provides limited benefits on most tasks, conventional tuning can improve frozen LLM-generated features, and aggregating independently generated programs can be more effective than selecting a single program based on validation scores. Overall, the findings suggest that LLMs can primarily carry out feature engineering in relational learning, without needing to control the entire modeling pipeline.

Critical strengths
  1. Large cost reduction with comparable accuracy and greater robustness to weaker LLMs. The proposed pipeline processes roughly 1,200x fewer input tokens and takes about 100x less LLM time per attempt. Its average AUROC is only 0.86 points below RelAgent, with most of the gap coming from two tasks (driver-dnf and user-ignore). Across the other ten, the mean difference is 0.12 points. When using Qwen3-32B in place of GPT-5.2, the pipeline outperforms RelAgent by 4.8 AUROC points. These results are of practical significance: they suggest that much of the benefit attributed to extended agentic search can be achieved with a substantially cheaper conventional methods.

  2. Systematic ablations with a useful negative result about validation. The paper's analysis reveals that validation scores are unreliable for selecting programs, particularly when validation sets are small. Selecting a program based on its validation score performs worse than choosing one at random (74.77 vs. 76.01 AUROC). Even when feedback is available, the deployed program is the best among those saved in only 6 of the 12 classification tasks. These findings raise important questions about how agentic AutoML systems should use validation feedback to guide their search.

Critical weaknesses
  1. No variance estimates, and the validation conclusion rests on two small tasks. All results come from a single draw of five programs and one feedback trajectory per task. Several headline effects are small enough to potentially fall within run-to-run noise. The general conclusion that "model selection can be left to conventional methods" rests on two small tasks rather than on evidence across the benchmark.

  2. Possible data contamination is not addressed. RelBench and its data-scientist SQL features have been public since 2024, before both LLMs used were released. The models may have seen the schemas, tasks, or even the expert features during pretraining. The paper doesn't discuss this risk nor evaluate on data released after the models' training cutoffs.

Questions for authors

Q. Have the authors evaluated the findings with models that were release prior to the benchmark to isolate potential data contamination?

1 comment

We thank the reviewer for recognizing the practical value of our approach and for their helpful feedback.

W1. We repeated SQL generation, Optuna tuning, and ensembling 3 times, with 5 programs per task in each repeat. The table below reports the prediction quality of our tuned ensemble across all 21 tasks. For each regression task, we divide our ensemble's test MAE by the test MAE of the single-table LightGBM baseline on the same task, then average these ratios across tasks.

Task typeTasksMean test score
Classification, AUROC ↑\uparrow121277.35±0.6277.35 \pm 0.62
Regression, normalized MAE ↓\downarrow990.8448±0.00240.8448 \pm 0.0024

Next, we examined how to use the 5 generated programs. We first compared choosing the program with the best validation score with choosing a program at random. On the 12 classification tasks, validation selection scores 0.090.09 AUROC points higher on average. However, excluding rel-f1 increases this gain to 0.680.68 points on 10 tasks. For regression, validation selection also reduced the average error: normalized MAE was 0.8420.842, compared with 0.8530.853 for random selection. We revised the broad claim about validation selection in Section 6.3. However, validation can still select a very poor program on an individual task. For example, in 1 run on rel-f1/driver-dnf, selecting by validation score gave 16.2516.25 points lower AUROC than the expected score of random selection. Combining predictions from all 5 programs avoids relying on this single choice and improves mean AUROC by 1.311.31 points over validation selection. Most of this gain comes from rel-f1: excluding it leaves a gain of 0.060.06 points. We keep rel-f1 in the primary analysis because these large losses matter even when the average difference between the alternatives is small.

We also repeated the comparison of feedback and independent generation, reported in our response to iUZT (W1 and Q1). The feedback experiments showed no clear average advantage from feedback, despite 5353\\% higher API charges. Overall, independent SQL generation with conventional tuning and ensembling delivered competitive predictive quality at substantially lower LLM API cost than RelAgent.

W2 and Q1. We fully acknowledge that modern LLMs could have seen RelBench schemas, task descriptions, or expert SQL features during training. However, please note that this is not specific only to agentic relational forecasting that we study in our work, but is a general problem for evaluating pretrained models on public benchmarks across machine learning. Checking exactly what a model has seen remains an open problem when its full training data is private. Even existing contamination checks can miss prior exposure [1]. Thus, we argue that the possibility of prior exposure alone is too broad as a reason to dismiss an evaluation. The same concern applies whenever we evaluate an LLM on a public benchmark released before it was trained.

Following your suggestion to evaluate on newer data, we used march-machine-learning-mania-2026 [2] to construct a relational task predicting tournament game winners. The 2026 games took place in March and April, after GPT-5.2's published knowledge cutoff of 31 August 2025 [3]. This provides a test on newer outcomes, although prior exposure to historical data or feature ideas remains possible. We used GPT-5.2 for both methods. Each produced 4 usable programs in 5 attempts, and we selected the best program for each method by AUROC on the 2022–2025 validation seasons. Both predictors were fitted on the same 2010–2021 seasons. We used the official 2026 tournament results as test labels. The table below compares prediction quality and LLM requests for our method and RelAgent on the 134 games played. Brier score is the mean squared error of the predicted probabilities, so lower is better.

MethodBrier score ↓\downarrowLLM requests, including failed attempts
Our method0.14870.14871212
RelAgent0.19580.19586262

Our Brier score is 2424\\% lower, with lower error in both the men's and women's tournaments. Our method also used 6.4×6.4\times fewer input tokens and incurred 2828\\% lower API charges. Appendix K reports this experiment, which extends our evidence beyond RelBench. On this additional task, whose test outcomes occurred after GPT-5.2's published knowledge cutoff, independent SQL generation with conventional training achieved lower prediction error and used fewer LLM resources than RelAgent.


[1] Fu et al., Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions, NAACL 2025

[2] https://www.kaggle.com/competitions/march-machine-learning-mania-2026

[3] https://developers.openai.com/api/docs/models/gpt-5.2

Reviewer_kmCYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether relational prediction requires an LLM to control the entire modeling pipeline. Its simpler approach gives the LLM a static description of the database schema, statistics, sample rows, and task, then generates five independent SQL feature programs without predictive feedback, allowing only execution-error repairs. Each program is paired with an Optuna-tuned LightGBM model; predictions are averaged for classification and median-aggregated for regression. On 12 classification and nine regression tasks from RelBench, the authors report better average results than the expert-feature baseline and performance close to RelAgent, with substantially fewer LLM requests and processed input tokens. Further experiments examine exploration, validation feedback, program selection, aggregation, and model tuning, and compare GPT-5.2 with Qwen3-32B. The main contribution is a competitive simple baseline and an empirical analysis of what the additional agent components contribute.

Critical strengths
  1. A simple pipeline achieves competitive results with substantially less LLM use. Across the 21 tasks, its mean classification AUROC is 77.52, close to RelAgent's 78.38, while median requests per attempt are 1 versus 146. This provides a useful baseline with little LLM interaction for relational learning.

  2. The frozen-program comparisons support separating feature generation from modeling. Tuning and aggregation experiments reuse the same SQL programs, reducing variation from feature regeneration. Conventional tuning improves 19/21 tasks, directly supporting the use of standard methods for model adaptation.

Critical weaknesses
  1. The feedback ablation also reflects the candidate-retention policy. Final selection is restricted to programs saved at round ends, and the oracle in Table 11 is sometimes worse than the initial program. The degradation therefore cannot be attributed entirely to validation feedback or unreliable validation scores without a comparison that retains the initial program and all evaluated candidates.

  2. The main performance–cost comparison is not fully matched. RelAgent's performance comes from published results, its costs from a separate reproduction, and Table 4 excludes Optuna tuning. These differences are disclosed, but a direct comparison on the same data version, including shorter agent budgets and full tuning costs, is needed to support stronger cost-effectiveness claims.

Questions for authors
  1. Are the initial program and every evaluated within-round candidate eligible for final selection? Does the feedback result change when selecting the best validation-scoring program from all candidates seen so far? For example, Table 11 gives driver-dnf an initial AUROC of 70.72, above the saved-program oracle of 68.72.

  2. What is the variance in the main differences across independent repetitions of the complete five-program ensemble and feedback trajectories? Could you examine stability by database, particularly the feedback effect and the finding that validation selection is worse than random selection after excluding rel-f1?

  3. Could you report performance and costs for full- and shorter-budget RelAgent runs on the same RelBench version, including Optuna tuning in your pipeline's total runtime? This would distinguish the established savings in LLM use from the less fully tested end-to-end cost-effectiveness advantage.

2 comments

We thank the reviewer for their positive assessment and constructive feedback.

W1 and Q1. The initial program and every later program that completes validation can be selected. At the end of each round, our code saves the program with the highest validation score seen so far. In a new run on 21 tasks, selecting from all evaluated programs gave the same SQL program as selecting from the saved programs on every task.

The initial program's test score in Table 11 of the original submission can exceed the Oracle value because Oracle is computed only over the programs saved at round ends. For example, on rel-f1/driver-dnf, the initial program scored 70.7270.72 AUROC, the selected program scored 66.7366.73, and the best saved program scored 68.7268.72. The initial program can therefore be eligible for selection while being absent from the saved programs used to compute the oracle. The revised paper reports the matched feedback study in Section 6.2.

To measure the effect of feedback separately, we compared generation with and without feedback 3 times per task. In each comparison, both versions started from the same SQL program, generated 4 more programs, and used the same rules for selecting or combining them. We used default LightGBM settings and kept the initial program available for selection. These experiments showed no clear average benefit from feedback, while API charges increased by 5353\\%. When selecting by validation score, the mean AUROC change ranged from −1.18-1.18 points on rel-event to +0.97+0.97 on rel-trial, with the effect still uncertain for every database. These results support independent generation as the default in the tested setting. Please see our response to iUZT (W1 and Q1) for the results and confidence intervals.

W2 and Q3. We ran RelAgent on RelBench 2.1.2, including the corrected rel-event/user-ignore labels, with 5 attempts per task and 10- and 60-step budgets. The table below compares our method with RelAgent on the same benchmark version to measure prediction quality and API cost. For each regression task, we divide each method's test MAE by the test MAE of the paper's single-table LightGBM baseline on the same task.

MethodMean AUROC ↑\uparrowNormalized MAE ↓\downarrowAPI charges, USD
Ours, without Optuna76.6276.620.8540.8549.819.81
Ours, with Optuna77.3577.350.8450.8459.819.81
RelAgent, 10-step budget76.6276.620.8620.862158.29158.29
RelAgent, 60-step budget76.2876.280.8840.884244.47244.47

Without Optuna, our ensemble achieved similar mean AUROC and lower normalized MAE than the shorter RelAgent run, with roughly 16×16\times lower API charges. Optuna improved both metrics without additional LLM requests. Increasing RelAgent's budget raised API charges without improving either mean test metric. These results support competitive predictive quality at substantially lower LLM cost. Next, we compare how long the methods took across all 21 tasks. Even with Optuna, our method took about 52.252.2 job-hours, including 40.440.4 for hyperparameter tuning, compared with 65.465.4 for RelAgent at the 10-step budget and 93.993.9 at the 60-step budget. See our response to iUZT (W3 and Q3) for hardware, memory, and timing details, and our response to iUZT (W2 and Q2) for the SQL feature comparisons under the same training and prediction methods.

Q2. We repeated SQL generation, Optuna tuning, and ensembling three times, with five programs per task in each run. The “Ours, with Optuna” row in W2 and Q3 averages all three complete five-model ensembles per task. Across all 12 classification tasks, the mean AUROC was 77.35±0.6277.35 \pm 0.62. Across all nine regression tasks, the mean normalized MAE was 0.8448±0.00240.8448 \pm 0.0024. We normalize each task’s MAE by the test MAE of the single-table LightGBM baseline before averaging across tasks. Next, we examine whether validation selection and combining predictions improve on choosing a program at random. The table below compares these methods using the same 5 programs in each ensemble. Random selection means choosing each program with equal probability. Its expected score is the average of their individual scores. AUROC differences are in points, and positive values favor the first method in each comparison.

ComparisonAUROC difference, all 12 tasksAUROC difference, 10 tasks excluding rel-f1
Validation selection vs. random selection0.090.090.680.68
Combined predictions vs. validation selection1.311.310.060.06

Selecting by validation score works better outside rel-f1, so we revised the broad claim about validation selection in Section 6.3. Combining predictions improves mean AUROC over validation selection, but this advantage mainly comes from rel-f1. Thus, the repetitions support combining predictions as a practical way to reduce large selection losses on rel-f1.

Thanks for the response. I maintain my positive score.

Reviewer_iUZTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies relational forecasting through SQL-derived entity features and tabular prediction. It asks whether LLMs need to control database exploration, feature construction, model tuning, and validation-driven revision jointly. Its pipeline generates five independent SQL programs from a static database description, permits error-only repair, tunes separate LightGBM models with Optuna, and aggregates their predictions. Experiments cover 12 classification and nine regression tasks on seven RelBench databases, with GPT-5.2 and a classification comparison using Qwen3-32B. The authors report 77.52 average AUROC versus RelAgent’s published 78.38, normalized MAE of 0.844 versus 0.817, and substantially fewer LLM requests and processed input tokens.

Critical strengths
  1. A useful division-of-labor hypothesis: The technical delta is to replace an interactive, validation-guided relational agent with independently generated static-context SQL programs followed by conventional tuning and aggregation (Sec. 3). This is a novel combination/design insight, not a new prediction problem or theoretical result. Relative to RelAgent, it removes joint exploration, model choice, and revision; relative to ReFuGe, it removes iterative feature filtering and feedback; relative to FeatLLM, it generates relational SQL rather than single-table binary rules; relative to Deep Feature Synthesis, it uses task semantics rather than predefined generic primitives (Sec. 2; Appendix A). In my judgment, this could meet ICLR’s novelty threshold if its explanatory evidence is strengthened; new components or state-of-the-art accuracy are not prerequisites.
  2. Potentially valuable reduction in expert and LLM involvement: The targeted workflow includes churn, engagement, and clinical-trial forecasting; the cited manual baseline required 12.3 expert hours per task (Sec. 1). The pipeline exceeds DS+LightGBM on 10/12 classification tasks, while Qwen3-32B retains 74.93 average AUROC versus 77.52 with GPT-5.2 (Sec. 5.1; Table 8). Frozen SQL permits prediction without new LLM calls, but deployment still requires feature computation and five fitted models (Sec. 3). Deployment frequency and realized business savings are not reported in the main text or appendix.
  3. Meaningful frozen-program controls: Selection and aggregation reuse the same five programs, and tuning is compared against defaults on those same features (Appendix F). Aggregation improves classification by 2.75 AUROC points over validation selection, while tuning improves 19/21 tasks (Secs. 6.3–6.4; Table 9). These controls establish component-level effects more convincingly than an undifferentiated full-system comparison.
  4. Substantial reimplementation detail in terms of reproducibility.
Critical weaknesses
  1. The feedback conclusion mixes generation and selection effects: Feedback uses default LightGBM, scalar scores, and validation-based deployment, without the workspace or model adaptation of RelAgent (Sec. 6.2). Only one trajectory per task is run; its “oracle” covers saved round-end programs, not all candidates or the initial program (Appendix H). Both rel-f1 tasks account for more than the entire average classification decrease (Appendix H; Table 11). Consequently, the evidence supports failure of this particular feedback-plus-selector configuration, not a general conclusion that feedback is unnecessary. Compare feedback-guided and independent generation under equal candidate budgets, apply identical aggregation/selection, and repeat trajectories with confidence intervals; repeated-generation variance is not reported in the main text or appendix (Sec. 7).
  2. Headline predictive comparisons are not protocol-matched: Baseline scores are imported, whereas the current experiment uses RelBench 2.1.2; user-ignore was corrected and older scores are explicitly declared incomparable (Sec. 4; Table 7). RelAgent reproduction is used only for cost, with deviations exceeding two points on both rel-f1 tasks (Appendix D). The manual-feature comparison also changes the ensemble and tuning pipeline, so its gain cannot be attributed exclusively to better features (Secs. 3–4). Rerun RelAgent and manual features on identical data, row caps, and downstream budgets, or present corrected-task sensitivity analyses and appropriately narrowed claims.
  3. LLM savings are not complete deployment savings: Table 4 excludes 24 Optuna trials and refitting per program, although these produce the headline predictor; cached input tokens also need not imply proportional monetary savings (Sec. 5.2; Appendix D). Thus, the request/token reduction is well-defined, but total cost-effectiveness remains unresolved. Report full five-program training and prediction latency, hardware, memory, and cached-token charges, plus a quality–cost curve for shortened RelAgent rollouts. These complete costs are not reported in the main text or appendix (Table 4; Appendix D).
  4. Temporal safety remains an unchecked assumption: The validator is explicitly syntactic and does not prove leakage freedom (Appendix B.3), despite point-in-time validity defining the task (Sec. 2). Audit all frozen programs using future-row perturbations and provenance checks; this is a request for verification, not an allegation of observed leakage. Deployment additionally assumes reliable keys/timestamps, sufficiently informative compact schema context, executable economical SQL, permission to disclose sample rows or local hosting, and affordable five-model refitting under the 200,000-row cap (Sec. 3; Appendices B.1–B.6; Ethics Statement). Performance when these assumptions fail is not reported in the main text or appendix.
Questions for authors

Q1. Under equal candidate budgets and identical deployment rules, does repeated feedback-guided generation outperform independent generation?

Q2. Do jointly rerun, corrected-data comparisons retain the reported quality gap?

Q3. What is the complete five-model runtime and monetary cost, including tuning, refitting, inference, and caching, relative to a shortened RelAgent rollout?

Q4. Do all 105 programs pass a test showing that adding or modifying post-prediction records leaves earlier feature vectors unchanged?

2 comments

We thank the reviewer for their careful reading and helpful suggestions for strengthening the experiments.

W1 and Q1. To measure the effect of feedback, we compared generation with and without feedback 3 times on each of the 21 tasks. In each comparison, both versions started from the same SQL program and generated 4 more programs. The version with feedback also received the previous programs and their validation scores. We used the same default LightGBM settings and the same rules for selecting a program or combining predictions. The initial program remained available for both. The table below reports the effect of adding feedback across all 21 tasks. Positive values favor feedback. For regression, we divide the reduction in test MAE from adding feedback by the test MAE of the program both versions started from. Brackets give 9595\\% confidence intervals across tasks.

How the final prediction is madeAUROC change, points (12 tasks)MAE reduction, \\% of initial error (9 tasks)
Select by validation score−0.01;[−1.08,1.06]-0.01\\;[-1.08, 1.06]1.08;[−0.74,2.91]1.08\\;[-0.74, 2.91]
Combine predictions−0.92;[−2.47,0.63]-0.92\\;[-2.47, 0.63]−0.11;[−2.62,2.40]-0.11\\;[-2.62, 2.40]

We found no clear average advantage from feedback in either classification or regression. This holds both when selecting a program by validation score and when combining predictions. All four confidence intervals include zero, and feedback increased API charges by 5353\\% at the same proposal budget. These results support independent generation as the default in the tested setting.

W2 and Q2. We ran RelAgent on RelBench 2.1.2, including the corrected rel-event/user-ignore labels. On rel-event/user-ignore, our method achieved 83.8483.84 AUROC, compared with 81.9181.91 for RelAgent. The table below compares the complete methods and the quality of their SQL features. To compare features, we used the same LightGBM model, 24 Optuna trials, a limit of 200,000 training rows, and the same methods for retraining and combining predictions. For the expert baseline, we compared 1 SQL program generated by the LLM with 1 written by the data scientist for the RelBench User Study. For each regression task, we divide each method's test MAE by the test MAE of the single-table LightGBM baseline on the same task. Positive values favor our method. Brackets give 9595\\% confidence intervals across tasks.

ComparisonAUROC difference, pointsReduction in normalized MAE
Our complete method vs. RelAgent, up to 60 steps1.07;[−0.70,2.83]1.07\\;[-0.70, 2.83] (12 tasks)0.039;[−0.043,0.121]0.039\\;[-0.043, 0.121] (9 tasks)
Our SQL vs. RelAgent SQL, matched training−0.36;[−1.40,0.68]-0.36\\;[-1.40, 0.68] (12 tasks)−0.023;[−0.044,−0.003]-0.023\\;[-0.044, -0.003] (9 tasks)
Our SQL vs. expert SQL, matched training1.73;[0.32,3.15]1.73\\;[0.32, 3.15] (8 tasks)0.008;[−0.009,0.024]0.008\\;[-0.009, 0.024] (6 tasks)

Our complete pipeline delivered competitive quality at substantially lower LLM API cost than RelAgent.

W3 and Q3. We measured API charges using cached-token prices and computation time for feature preparation, model training, and prediction, including Optuna tuning. We also ran RelAgent with 10- and 60-step budgets, using 5 attempts per task. We compared prediction quality and API charges on all 21 RelBench tasks.

MethodMean AUROC ↑\uparrowNormalized MAE ↓\downarrowAPI charges, USD ↓\downarrow
Ours, without Optuna76.6276.620.8540.8549.819.81
Ours, with Optuna77.3577.350.8450.8459.819.81
RelAgent, 10-step budget76.6276.620.8620.862158.29158.29
RelAgent, 60-step budget76.2876.280.8840.884244.47244.47

The 60-step RelAgent run cost 5454\\% more than the 10-step run and scored worse on both test metrics, possibly because the longer search overfit the validation split. Without Optuna, our ensemble achieved similar mean AUROC and lower normalized MAE than the 10-step RelAgent run, with roughly 16×16\times lower API charges. Optuna improved both metrics with additional computation and the same API charges.

Next, we compare runtime across all 21 tasks, from LLM requests through testing.

Time, job-hoursOurs, without OptunaOurs, with OptunaRelAgent, 10-step budgetRelAgent, 60-step budget
Time spent on LLM requests2.32.32.32.322.622.631.931.9
Recorded job-hours11.811.852.252.265.465.493.993.9

We then examine the final training and prediction stages of our tuned method in more detail. The table below shows their median time per task and peak memory per worker. The measurements come from these runtime experiments, which used Intel Xeon Platinum 8568Y+ CPUs.

MeasurementValue
Compute test features4646 seconds
Retrain the 5 models9.29.2 seconds
Generate predictions0.260.26 seconds
Peak memory per worker101101 GiB

W4 and Q4. We thank the reviewer for suggesting this check. It helped us identify issues in how both our method and RelAgent construct training features. These issues, however, do not invalidate the results. All our RelBench experiments use the official training, validation, and test splits. We compute training and validation features using records available up to the validation cutoff. For the final test, we train on the training and validation examples together and compute features using records available up to the test cutoff. On all 21 tasks, we verified that the test snapshot contained no timestamped records dated after the earliest test prediction date. The temporal leakage we found was related only to feature construction for earlier training examples.

Our prompt requires each SQL query to filter records by the example’s prediction date. When this filter is missing, an earlier training example can use training records that became available later within the same database snapshot. For each of the 105 SQL programs in our original experiment, we selected up to 8 training examples from each of 3 dates spanning the training period. We computed their features before and after removing or modifying records dated after each example’s prediction date. After that, we inspected the joins and time filters to explain any changes and found issues in 20 programs. For example, 14 of those 20 joined rel-event user profiles without checking whether the profile existed at the prediction date. We corrected the affected programs and retrained their models with the same hyperparameters. The corrected query paths passed all 60 sampled date-level checks.

In addition, we checked 91 comparable RelAgent programs. Removing later records changed features for earlier training examples in 30 programs, and query inspection confirmed point-in-time issues in 25. However, note that RelAgent used the same RelBench test cutoff, which excluded timestamped records dated after the test prediction date. The table below compares our original and corrected programs against the same 60-step RelAgent results. Positive values favor our method.

Our programsAUROC advantage, points (12 tasks)Mean relative MAE reduction (9 tasks)
Original1.191.194.884.88\\%
Corrected1.021.024.664.66\\%

Correcting the training features changed the aggregate results only slightly. Our method still achieved higher mean AUROC and lower mean MAE than the RelAgent reference. The snapshot checks cover declared timestamps, and the perturbation checks cover the sampled training dates.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated paper

We would like to thank all the reviewers for their valuable feedback. We’ve taken the comments into account and updated the paper accordingly. Namely:

  • We reran RelAgent on the same RelBench 2.1.2 data and expanded our efficiency evaluation to cover API charges, token usage, and worker time. Our pipeline achieved higher mean AUROC and lower normalized MAE than the 60-turn rerun, with roughly 25×25\times lower LLM API charges and 52.252.2 vs. 93.993.9 job-hours.

  • We now report mean performance and variability across three independent runs and include a paired comparison of generation with and without validation feedback.

  • To address concerns about benchmark contamination, we evaluated both methods on 2026 NCAA tournament outcomes that occurred after GPT-5.2’s published knowledge cutoff. Our pipeline achieved a 2424\\% lower Brier score and 2828\\% lower LLM API charges than RelAgent.

  • We also note that our temporal audit found no evidence of test-to-training leakage.

Overall, the new results strengthen the paper and further support our main conclusion that using LLMs in relational learning as feature engineers allows for significantly reduced LLM costs (compared to current more complex agentic pipelines) without sacrificing predictive performance. We believe that our results are important for both the academic researchers (as it can serve as a simple and strong baseline) as well as practitioners (due to a simple procedure and reduced costs).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.