LLMs as Feature Engineers: Simplifying Agentic Relational Learning

Relational forecasting tasks, such as predicting customer churn from a database of customers, transactions, and products, can be solved by transforming the database into a single table of entity features and training a standard tabular model on it. The main difficulty of this approach is feature engineering, and recent works delegate it to large language models (LLMs). In particular, RelAgent gives an LLM agent control over the entire pipeline: the agent explores the database, writes SQL feature programs, chooses the downstream model and its hyperparameters, and revises all these decisions using validation feedback over many iterations. In this work, we ask which of these responsibilities actually require an LLM. We study a simple pipeline in which the LLM acts only as a feature engineer. Given a static description of the database, it writes several SQL feature programs independently, using a single request per program and receiving no predictive feedback. Model tuning and aggregation of predictions are handled by standard methods. On $21$ classification and regression tasks from RelBench, this pipeline outperforms the features engineered by an expert data scientist on average and stays close to RelAgent ($77.52$ vs. $78.38$ average AUROC on classification). At the same time, one of our programs requires $146\times$ fewer LLM requests and about $1,200\times$ fewer processed input tokens than one search rollout of RelAgent. When a top-tier proprietary LLM is replaced with a much smaller open-source one, the average AUROC of our pipeline decreases by $2.59$ points, compared with $8.25$ points for RelAgent. Our analysis shows that a static database description is as informative as interactive exploration on most tasks and that conventional tuning further improves frozen LLM-generated features. It also shows that validation scores are a weak guide for choosing between programs: on classification, averaging independent programs outperforms selecting one of them by validation score. These results suggest that LLMs are most useful in relational learning as feature engineers, while model tuning and selection can be left to conventional methods.