acceptodds
Under review as a conference paper at ICLR 2027

EvoFG: Decoupling Operator Evolution and Feature Binding for Efficient Feature Generation

Abstract

Combining large language models (LLMs) with evaluation feedback has emerged as an effective approach to feature generation for structured data. However, directly generating concrete features typically requires jointly searching over operator structures and input-feature bindings. Consequently, potentially useful structures may receive negative feedback due to mismatched bindings, obscuring the direction of subsequent evolution and limiting LLM utilization efficiency. Some methods also include dataset names and column semantics in their prompts, introducing potential data leakage risks. To address these challenges, we propose EvoFG, a feature generation paradigm that decouples operator structure evolution from feature binding optimization. EvoFG searches over reusable, parameterized operator families, allowing the LLM to generate code without specifying concrete column combinations. Instead, a program enumerates candidate bindings and identifies high-quality feature instances through coarse-to-fine evaluation. The best binding, peak gain, and supporting evidence from other high-performing bindings jointly provide feedback to guide subsequent structural mutations, mitigating the misjudgment of structures caused by binding mismatches and turning each proposal into an exploration of multiple concrete features. This design delegates concrete column selection to programmatic evaluation and does not rely on dataset-specific information, thereby minimizing potential data leakage risks associated with LLM use. The high-quality features generated by EvoFG significantly improve the performance of different downstream models. Evaluations across multiple datasets demonstrate that EvoFG outperforms existing AutoFE methods. With results averaged equally across datasets under the same LLM budget, EvoFG produces over 30 times as many positive-gain candidate features and approximately 2.5 times as many ultimately selected new features as direct LLM-based feature generation, while reducing token consumption per selected feature by approximately 50%. These results demonstrate that organizing search around operator families and optimizing bindings programmatically enables more effective LLM utilization, greater search efficiency, and higher token efficiency, providing an effective search paradigm for feedback-driven evolutionary feature generation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Automated Feature Engineering, Large Language Models, Evolutionary Search, Operator Evolution, Feature Binding Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TeYcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EvoFG proposes LLM-driven feature generation in which the LLM evolves operator families: row-wise programs with unbound input slots. It does not emit concrete features. A local program enumerates bindings (ordered column pairs for binary families) and screens them with a three-subset residual proxy. It then fine-evaluates a shortlist of 64/128 bindings with 3-fold LightGBM CV (Sec. 3.2), and returns a peak gain P and elite support S (Eq. 5) as feedback for structural mutation. The archive is MAP-Elites style, with unary and binary islands (Sec. 3.3). A structure/binding loss decomposition (Eqs. 2–3) motivates the split. Prompts omit dataset names, column semantics and raw samples (Sec. 3.5). The top K = 20 positive-gain features are frozen and evaluated on 27 datasets with LightGBM/XGBoost/MLP (Table 1) and TabPFN-2.5 (Table 2), against AutoFeat, OpenFE, OCTree, CAAFE, FeatLLM, and Direct/Direct-Batch LLM evolution. A separate 15-dataset pilot (Table 3) reports 40.5× more positive candidates, 2.6× more selected features and 48.7% fewer tokens per selected feature than Direct. Component ablations (Table 4), binding diagnostics (Fig. 3) and semantic-context controls (Secs. 4.5–4.6) round out the study.

Critical strengths
  • [Importance of the research problem] The paper identifies two real issues in LLM feature generation: a good structure can be misjudged because of one bad binding, and prompts that include dataset or column semantics risk memorized solutions (Sec. 1).
  • [Novelty of the approach] Evolving operator families with unbound slots, with peak and elite-support feedback (Eq. 5) in a MAP-Elites-style archive, is a clean idea. Eq. 3 states the structure/binding decomposition clearly.
  • [Soundness and feasibility] The pipeline is leakage-aware: search uses outer-training data only, features are frozen before testing, and fine evaluation uses 3-fold LightGBM CV on 64 or 128 bindings (Secs. 3.2 and 4.1).
  • [Soundness and feasibility] The paper is candid about limitations, such as top-K by individual gain ignoring joint utility (Sec. 3.4).
  • [Experiments and practical impact] The evaluation is broad: 27 datasets (18 classification, 9 regression), four downstream learners including TabPFN-2.5, and diagnostics and controls.
  • [Experiments and practical impact] Fine evaluation clearly helps against proxy-only evaluation (+1.22 / +1.72 pp accuracy for LightGBM / XGBoost, Table 4).
  • [Theoretical and application-level implications] The structure/binding decomposition is a useful conceptual lens, and the paper correctly notes that decoupling does not change the optimum.
  • [Theoretical and application-level implications] The semantic-free interface (byte-identical prompts under renaming, Sec. 4.6) suits confidential or regulated tables.
Critical weaknesses
  • [Importance of the research problem] Neither issue is quantified on existing methods before the solution is proposed. The binding-mismatch diagnostic (58 of 80 structures nonpositive under a single binding) comes only later.
  • [Novelty of the approach] As Sec. 2 acknowledges, enumeration, decomposed selection (GRFG), screening (OpenFE, SCOPE-FE), evolutionary LLM feature programs (LLM-FE, ELLM-FT, EFE, TOPOFE) and context-agnostic prompting (OCTree) already exist. The remaining contribution is modest.
  • [Soundness and feasibility] Matched-quota random search rescues the same cases (Sec. 4.5.2), so the benefit appears to come from evaluation coverage, not from the proxy or the feedback.
  • [Soundness and feasibility] The search signal is noisy: label permutation still yields positive candidates, and only 122 of 160 selected features stay positive on recheck (Sec. 4.5.3).
  • [Soundness and feasibility] Programs are limited to unary and binary, and binary enumeration grows as d(d−1).
  • [Experiments and practical impact] In Table 4 (7 tasks; positive values favor full EvoFG), the controls often do better on classification: single binding −1.78 / −1.20 / −1.37, independent proposals −1.23 / −1.36 / −0.92, fixed library −0.84 / +0.23 / −1.87 (LightGBM / XGBoost / MLP).
  • [Experiments and practical impact] The efficiency results come from a separate 15-dataset pilot (25,003 vs. 616 positive candidates, Table 3). Total tokens are higher for EvoFG (3,040,038 vs. 2,572,990), and median search time is 28.32 vs. 1.75 min (Sec. 4.6).
  • [Experiments and practical impact] The main tables have many unavailable comparator entries and no significance tests (Sec. 4.2). Some regression results are erratic (Tesla_Stock: baseline RMSE 31.828, while one method reaches 374.456).
  • [Theoretical and application-level implications] The evidence points to a simpler mechanism, trading LLM calls for local compute, than the feedback-driven narrative suggests.
  • [Theoretical and application-level implications] There are no safeguards or guarantees against selection bias in the noisy search signal. Local compute cost and d(d−1) scaling limit wide tables.
Questions for authors
  1. How do you reconcile Table 4, where single-binding, fixed-library and independent-proposal controls outperform full EvoFG in most classification settings, with the claim that family-level binding search and feedback are what improve features?
  2. Can you compare against Direct with a matched budget of local evaluations, e.g., each Direct proposal randomly re-bound to 64 column tuples and fine-evaluated? Also against a fixed operator library with random binding at matched compute.
  3. Please report total wall-clock and total tokens (not only per selected feature) on the main 27-dataset protocol, and a downstream accuracy-vs-compute comparison with Direct and OpenFE.
  4. Can you add Friedman/Nemenyi or Wilcoxon tests over the datasets where all methods are available, and repeat the feature search with ≥3 seeds?
  5. Why do Tesla_Stock and bike show such large spread across methods? Is there a temporal-split issue?
  6. How does EvoFG compare with LLM-FE under the same LLM (DeepSeek V4.1 Flash) and request budget?
Reviewer_HNHcOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EvoFG, an LLM-based method for automatically generating useful features from tabular data. EvoFG identifies the issue with feature generation where the LLM can suggest transformations, but it may choose poor column combinations. Thus, EvoFG separates feature generation into two stages: one which evolves the operator family and the other to select the right columns. EvoFG is evaluated on 27 tabular datasets, covering classification and regression, and compares it with recent AutoFE methods.

Critical strengths
  • The paper is well-written, well-motivated and easy to follow.

  • The paper evaluates across different tasks, baselines and predictive models.

  • The method is clear, identifies a problem and directly addresses it by separating the column selection from the operator evolution.

Critical weaknesses
  • EvoFG is tested with one LLM backbone, DeepSeek V4.1 Flash. There is no analysis across different LLMs.

  • Methodological novelty seems limited. It does identify a problem, but the solution looks like an engineering solution to solve some bottleneck on top of already established frameworks. This can be okay, but especially given that the field has been well-studied, I think this limits the novelty of the work. My concern is amplified based on the results as well, the results seem to be pretty mixed and does not strongly advocate for the proposed solution.

  • Matched request budgets are used for some comparisons, compute and wall-clock comparisons are mainly limited to EvoFG, Direct, and Direct-Batch. There is no comprehensive accounting of cost and performance for every baseline. This is particularly important as the difference in results in not significant.

  • Results report means, but provide no statistical significance testing. This is a common practice and important as the differences are minimal and could be because of one lucky generation.

  • While obvious dataset recalls are reduced, the authors do not present that EvoFG is free from LLM memorization. A stronger test would use genuinely post-cutoff datasets or synthetic/private datasets.

  • Can the authors evaluate the method on a few more high dimensional datasets?

  • Are there any qualitative examples of how the two stage helps in more interpretable features? Or are the gains primarily associated only with the performance rather than the qualitative interpretability?

Questions for authors

See weaknesses.

Reviewer_R1ZHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes EvoFG, an automated feature generation framework that decouples LLM-based operator evolution from programmatic feature binding optimization. It uses coarse-to-fine evaluation to identify promising bindings and peak–support feedback to guide structural evolution. Experiments on 27 datasets with multiple downstream models demonstrate competitive predictive performance. Additional studies examine feature discovery efficiency, token consumption, and individual component contributions.

Critical strengths
  1. The paper identifies a limitation of direct LLM-based feature generation: a potentially useful operator may be rejected because of an unsuitable feature binding. The controlled binding experiments provide concrete evidence that exploring alternative bindings can recover useful features, including improvements on held-out data.

  2. The paper evaluates EvoFG across 27 datasets and multiple downstream models, including TabPFN. It also includes targeted experiments on binding mismatches, feature discovery, and the reliability of generalization. These evaluations offer useful empirical insights into operator-family-based feature generation and its behavior across different settings.

Critical weaknesses
  1. While the results support the value of evaluating multiple feature bindings, the benefit of the proposed evolutionary feedback mechanism is less clear. In Table 4, the Independent Proposals baseline outperforms full EvoFG across all three classification models and two of the three regression models. The effects of Peak and Elite Support are also mixed, and some ablations modify multiple components simultaneously. Consequently, it remains difficult to determine how much of the observed improvement is due to the proposed structural evolution mechanism rather than broader binding exploration. This distinction is important because feedback-guided operator evolution is a central contribution of the paper.

  2. The reported feature discovery gains are substantial, but the comparison primarily controls the number of LLM requests rather than the total computational effort. EvoFG evaluates considerably more feature bindings per proposal and incurs a higher search cost, with a reported median search time of 28.32 minutes compared with 1.75 minutes for Direct. Moreover, the large increase in positive-gain candidates is measured using training-internal evaluations and does not necessarily translate into corresponding improvements in held-out predictive performance. As efficiency is a main motivation of the work, stronger evidence of the trade-off between computational cost and final predictive quality would be important for assessing its practical contribution.

Questions for authors
  1. Table 4 shows that Independent Proposals performs better than full EvoFG in several settings. Were the initialization, binding evaluation, and final feature selection kept consistent between the two variants? It is currently difficult to tell how much the Peak–Support feedback itself contributes to the final performance.

  2. The efficiency comparison controls the number of LLM requests, but EvoFG requires substantially more local evaluation. How would the methods compare under a similar overall computational budget, for example in terms of runtime or model evaluations? This matters when interpreting the reported efficiency gains, particularly their relationship to final predictive performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.