- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
EvoFG proposes LLM-driven feature generation in which the LLM evolves operator families: row-wise programs with unbound input slots. It does not emit concrete features. A local program enumerates bindings (ordered column pairs for binary families) and screens them with a three-subset residual proxy. It then fine-evaluates a shortlist of 64/128 bindings with 3-fold LightGBM CV (Sec. 3.2), and returns a peak gain P and elite support S (Eq. 5) as feedback for structural mutation. The archive is MAP-Elites style, with unary and binary islands (Sec. 3.3). A structure/binding loss decomposition (Eqs. 2–3) motivates the split. Prompts omit dataset names, column semantics and raw samples (Sec. 3.5). The top K = 20 positive-gain features are frozen and evaluated on 27 datasets with LightGBM/XGBoost/MLP (Table 1) and TabPFN-2.5 (Table 2), against AutoFeat, OpenFE, OCTree, CAAFE, FeatLLM, and Direct/Direct-Batch LLM evolution. A separate 15-dataset pilot (Table 3) reports 40.5× more positive candidates, 2.6× more selected features and 48.7% fewer tokens per selected feature than Direct. Component ablations (Table 4), binding diagnostics (Fig. 3) and semantic-context controls (Secs. 4.5–4.6) round out the study.
Critical strengths
- [Importance of the research problem] The paper identifies two real issues in LLM feature generation: a good structure can be misjudged because of one bad binding, and prompts that include dataset or column semantics risk memorized solutions (Sec. 1).
- [Novelty of the approach] Evolving operator families with unbound slots, with peak and elite-support feedback (Eq. 5) in a MAP-Elites-style archive, is a clean idea. Eq. 3 states the structure/binding decomposition clearly.
- [Soundness and feasibility] The pipeline is leakage-aware: search uses outer-training data only, features are frozen before testing, and fine evaluation uses 3-fold LightGBM CV on 64 or 128 bindings (Secs. 3.2 and 4.1).
- [Soundness and feasibility] The paper is candid about limitations, such as top-K by individual gain ignoring joint utility (Sec. 3.4).
- [Experiments and practical impact] The evaluation is broad: 27 datasets (18 classification, 9 regression), four downstream learners including TabPFN-2.5, and diagnostics and controls.
- [Experiments and practical impact] Fine evaluation clearly helps against proxy-only evaluation (+1.22 / +1.72 pp accuracy for LightGBM / XGBoost, Table 4).
- [Theoretical and application-level implications] The structure/binding decomposition is a useful conceptual lens, and the paper correctly notes that decoupling does not change the optimum.
- [Theoretical and application-level implications] The semantic-free interface (byte-identical prompts under renaming, Sec. 4.6) suits confidential or regulated tables.
Critical weaknesses
- [Importance of the research problem] Neither issue is quantified on existing methods before the solution is proposed. The binding-mismatch diagnostic (58 of 80 structures nonpositive under a single binding) comes only later.
- [Novelty of the approach] As Sec. 2 acknowledges, enumeration, decomposed selection (GRFG), screening (OpenFE, SCOPE-FE), evolutionary LLM feature programs (LLM-FE, ELLM-FT, EFE, TOPOFE) and context-agnostic prompting (OCTree) already exist. The remaining contribution is modest.
- [Soundness and feasibility] Matched-quota random search rescues the same cases (Sec. 4.5.2), so the benefit appears to come from evaluation coverage, not from the proxy or the feedback.
- [Soundness and feasibility] The search signal is noisy: label permutation still yields positive candidates, and only 122 of 160 selected features stay positive on recheck (Sec. 4.5.3).
- [Soundness and feasibility] Programs are limited to unary and binary, and binary enumeration grows as d(d−1).
- [Experiments and practical impact] In Table 4 (7 tasks; positive values favor full EvoFG), the controls often do better on classification: single binding −1.78 / −1.20 / −1.37, independent proposals −1.23 / −1.36 / −0.92, fixed library −0.84 / +0.23 / −1.87 (LightGBM / XGBoost / MLP).
- [Experiments and practical impact] The efficiency results come from a separate 15-dataset pilot (25,003 vs. 616 positive candidates, Table 3). Total tokens are higher for EvoFG (3,040,038 vs. 2,572,990), and median search time is 28.32 vs. 1.75 min (Sec. 4.6).
- [Experiments and practical impact] The main tables have many unavailable comparator entries and no significance tests (Sec. 4.2). Some regression results are erratic (Tesla_Stock: baseline RMSE 31.828, while one method reaches 374.456).
- [Theoretical and application-level implications] The evidence points to a simpler mechanism, trading LLM calls for local compute, than the feedback-driven narrative suggests.
- [Theoretical and application-level implications] There are no safeguards or guarantees against selection bias in the noisy search signal. Local compute cost and d(d−1) scaling limit wide tables.
Questions for authors
- How do you reconcile Table 4, where single-binding, fixed-library and independent-proposal controls outperform full EvoFG in most classification settings, with the claim that family-level binding search and feedback are what improve features?
- Can you compare against Direct with a matched budget of local evaluations, e.g., each Direct proposal randomly re-bound to 64 column tuples and fine-evaluated? Also against a fixed operator library with random binding at matched compute.
- Please report total wall-clock and total tokens (not only per selected feature) on the main 27-dataset protocol, and a downstream accuracy-vs-compute comparison with Direct and OpenFE.
- Can you add Friedman/Nemenyi or Wilcoxon tests over the datasets where all methods are available, and repeat the feature search with ≥3 seeds?
- Why do Tesla_Stock and bike show such large spread across methods? Is there a temporal-split issue?
- How does EvoFG compare with LLM-FE under the same LLM (DeepSeek V4.1 Flash) and request budget?