acceptodds
Under review as a conference paper at ICLR 2027

Co-Evolving LLM Agents and Bayesian Optimizers for Automated Feature Engineering

Abstract

Automated feature engineering (AutoFE) involves searching a combinatorial space of compositional transformations under expensive downstream evaluations, making efficient exploration and effective reuse of search experience essential. We introduce CoEvolve, a collaborative evolution framework that couples an LLM agent for local exploration with a global optimizer for selecting promising feature set anchors. Starting from an anchor, the agent invokes FE-MDP, a surrogate network that predicts one-step performance gains conditioned on the current feature set and candidate action, to guide exploratory beam expansion and collect local search experience. A dual-memory mechanism distills this experience into Conditional Agent Memory (CAM), which captures context-specific evidence about effective and ineffective transformations, and Unconditional Agent Memory (UAM), which accumulates reusable knowledge across feature sets. Together, these memories guide the agent in proposing candidate transformations for downstream evaluation. To coordinate global exploration, we introduce the Deep Set Global Optimizer (DSGO), which combines a permutation-invariant Set Transformer encoder with a Bayesian linear regression posterior to estimate downstream performance and predictive uncertainty. DSGO selects promising anchors through expected improvement, directing local exploration toward promising regions of the search space. As newly evaluated candidates expand the search pool, the surrogate models and agent memories are iteratively updated, allowing global optimization and local exploration to inform one another. Experiments on public tabular benchmarks demonstrate that CoEvolve outperforms existing feature selection and generation methods, achieving average relative performance improvements of 21.68% over the raw-feature baseline and 3.59% over the strongest competing baseline on each dataset.

Primary area:
optimization
Keywords:
automated feature engineering, large language model agents, Bayesian optimization, agent memory, tabular learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jZGHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoEvolve, a framework for automated feature engineering. It combines DSGO for selecting initial feature sets via Bayesian optimization, FE-MDP for estimating transformation gains during local search, and an LLM guided by conditional memory (CAM) and accumulated memory (UAM) to propose further transformations. Evaluated candidates are used to update the models and memory components.

Critical strengths
  1. Conditioning transformations on the current feature set is well motivated. Equation (4) accounts for both generation and selection, and Appendix B demonstrates that the same transformation can benefit one feature set while degrading another. This supports the paper's focus on context-aware feature search.
  2. The experiments cover both classification and regression datasets and compare CoEvolve with a broad set of feature selection and generation methods, providing a reasonably broad empirical basis for evaluating the method.
Critical weaknesses
  1. The abstract describes general challenges in AutoFE but does not identify a specific limitation of existing methods before introducing CoEvolve. The authors should clarify which unresolved problem motivates the proposed method.
  2. Section 2 and Table 1 omit several recent and closely related methods [1-4]. Why are these works neither reviewed nor compared with CoEvolve? Please explain the distinctions from these methods and provide appropriate comparisons under matched evaluation budgets, or justify why such comparisons would be unsuitable. [1] Continuous Optimization for Feature Selection with Permutation-Invariant Embedding and Policy-Guided Search. KDD 2025, 1857-1866. DOI: 10.1145/3711896.3736891. [2] Adaptive Context-Infused Performance Evaluator for Iterative Feature Space Optimization. CIKM 2025, 3448-3457. DOI: 10.1145/3746252.3761123. [3] GPT-FT: An Efficient Automated Feature Transformation Using GPT for Sequence Reconstruction and Performance Enhancement. Web and Big Data (APWeb-WAIM 2025), LNCS 16113, 3-19, 2026. DOI: 10.1007/978-981-95-5640-3_1. [4] Heterogeneous Multi-Agent Reinforcement Learning with Attention for Cooperative and Scalable Feature Transformation. KDD 2026, 2010-2019. DOI: 10.1145/3770854.3780231.
  3. Section 3.2 relies on an LLM for transformations and memory updates, but the manuscript does not identify the model or provide the prompts used. These omissions make it difficult to reproduce the agent's behavior.
  4. Sections 4 and 4.1 omit key experimental details, including the downstream learners, classification metric, train/test split ratio, and baseline tuning procedures. As a result, readers cannot reconstruct the evaluation or assess whether all methods were tested under comparable settings.
  5. Section 4.3 mentions retaining 20 candidates, but this does not fully define the evaluation budget, and neither the total number of evaluations nor the baseline budgets are reported. The reported gains could therefore reflect more evaluations rather than a more effective search strategy.
  6. In Figure 3, the variant without DSGO slightly outperforms CoEvolve on OpenML-586 (0.7282 versus 0.7279), while the full method improves by only 0.0032 and 0.0079 on OpenML-589 and UCI Credit, respectively. Please explain why the simpler variant performs better or similarly on these datasets and whether these small differences persist across independent runs. Without uncertainty estimates or a comparison of computational cost, it remains unclear whether DSGO's additional complexity is justified on these datasets.
Questions for authors
  1. What specific limitation of existing AutoFE methods motivates CoEvolve, and could this gap be stated explicitly in the abstract?
  2. Why were recent related works [1-4] neither reviewed nor compared with CoEvolve? Please clarify how CoEvolve differs from these methods and provide comparisons under matched evaluation budgets, or justify why such comparisons would be unsuitable.
  3. Which LLM, prompts, and generation settings were used to propose feature transformations and update CAM/UAM?
  4. Which downstream learners, metrics, train/test splits, and baseline tuning procedures were used to produce Table 1?
  5. How many cross-validation evaluations and LLM calls did each method use, and does CoEvolve still outperform the baselines under the same evaluation budget?
  6. Why does the variant without DSGO slightly outperform CoEvolve on OpenML-586, while the full method provides only small gains on OpenML-589 and UCI Credit? Do these differences persist across independent runs, and do the gains justify DSGO's additional computational cost?
6 comments
AuthorsOpenReview ↗

Full Ablation Study and Additional Baselines

We appreciate the reviewer’s thoughtful comments. In order to to isolate the contribution of each component in CoEvolve. We have expanded the ablation study to compare CoEvolve with six variants: (1) DSGO + FE-MDP, (2) DSGO + embedding-neighborhood search (ENS), (3) DSGO + LLM + CAM, (4) DSGO + LLM + UAM, and (5) FE-MDP (w/o DSGO) (6) DSGO (w/o FE-MDP). All variants use the same initialization and downstream CV budget. We use the same DSGO configuration wherever DSGO is present. For comparisons that isolate the proposal mechanism, we also hold the anchor-selection procedure, the initial beam search, and the legal action set fixed. By proposal mechanism, we mean how a method chooses which additional feature transformations to evaluate from the selected anchor after the initial beam search. In the DSGO + FE-MDP control, FE-MDP ranks untried legal actions by their predicted one-step gain and selects the highest-ranked actions. In the full CoEvolve model, the LLM selects actions from the same set using CAM, which summarizes which transformations helped or hurt at the current anchor, and UAM, which captures experience across anchors and search rounds. Both methods apply their selected actions to the original anchor and evaluate the resulting feature sets by CV under the same remaining budget.

Table 1\textbf{Table 1}

DatasetCoEvolveDSGO + FE-MDPDSGO + ENSDSGO + LLM + CAMDSGO + LLM + UAMFE-MDP (w/o DSGO)DSGO (w/o LLM, FE-MDP)
OpenML-5860.72790.69880.68910.71480.67050.7282\textbf{0.7282}0.6529
OpenML-5890.7036\textbf{0.7036}0.67340.68480.69780.69570.70040.6215
OpenML-6070.6911\textbf{0.6911}0.66980.68640.67140.63720.65590.6237
OpenML-6200.6857\textbf{0.6857}0.66940.65060.67700.65860.64360.6524
UCI Credit Card0.8061\textbf{0.8061}0.78230.79530.79960.77810.79820.7753
SpamBase0.9520\textbf{0.9520}0.92990.92110.93440.92890.92510.9268
svmguide30.8622\textbf{0.8622}0.82740.79400.83980.81620.79740.8140
German Credit0.7835\textbf{0.7835}0.75650.75410.76720.73860.74000.7494

We also perform additional experiments to compare CoEvolve with more recent AutoFE methods, including CAPS (Liu et al., KDD 2025) and HAFT (Zhe et al., KDD 2026). CAPS uses a permutation-invariant encoder–decoder and a PPO policy to select a subset of existing columns. HAFT (Zhe et al., KDD 2026) uses two feature agents and an operation agent coordinated by a shared critic. Table 2 shows the eperimental results.

Table 2\textbf{Table 2}

DatasetLLM-FECAPSHAFTCoEvolve
OpenML-5860.71340.62150.67580.7279\textbf{0.7279}
OpenML-5890.68250.58150.60360.7036\textbf{0.7036}
OpenML-6070.67100.61470.65480.6911\textbf{0.6911}
OpenML-6200.63590.58740.60450.6857\textbf{0.6857}
UCI Credit0.80680.8076\textbf{0.8076}0.80630.8061
SpamBase0.93170.92820.91920.9520\textbf{0.9520}
svmguide30.82110.83840.84660.8622\textbf{0.8622}
German Credit0.78240.77570.77950.7835\textbf{0.7835}
AuthorsOpenReview ↗

Response to Weakness 1 and Weakness 2

Q1\textbf{Q1}: The abstract describes general challenges in AutoFE but does not identify a specific limitation of existing methods before introducing CoEvolve. The authors should clarify which unresolved problem motivates the proposed method.

A1\textbf{A1}: We agree that the abstract should identify the specific problem behind the large search space and high evaluation cost. Each candidate requires two choices: a feature set to start from and a transformation to apply. Since only a small fraction of these combinations can be evaluated, the challenge is to decide where to spend the next evaluations and what to try there.

CoEvolve addresses this challenge through a divide-and-conquer approach built around two alternating decisions that share evaluation feedback. DSGO first selects a feature set, or anchor, for further exploration. Given that anchor, FE-MDP and the LLM agent choose local transformations to evaluate. When a transformation produces a new feature set, its score informs which set to explore next, while its gain or loss relative to its parent informs which transformations to propose in subsequent rounds. The next global decision can therefore select a set discovered by local search, and the next local decision can draw on evidence gathered at earlier anchors. Our depth-two experiment supports the rationale for choosing the anchor separately. Under the same local evaluation budget, higher-scoring anchors tended to yield higher-scoring successors on all eight datasets. The Spearman correlation between the anchor’s training CV score and its best depth-two successor’s training CV score ranged from 0.3776 to 0.8473, with a mean of 0.5987. Thus, anchor quality provides useful guidance about where to search, even before the local agent determines which transformation to try.

Table 3\textbf{Table 3}

| Dataset | OpenML-586 | OpenML-589| OpenML-607 | OpenML-620 | UCI Credit | SpamBase | svmguide3 |German Credit |

| Spearman | 0.3776 | 0.5172| 0.6724 | 0.5829 | 0.8473| 0.6524 | 0.7128 | 0.4271|

Q2\textbf{Q2}: The “w/o LLM” variant also replaces the local search method with embedding-based neighborhood search. Thus, it does not isolate the benefit of the LLM and its memories. Keeping DSGO and FE-MDP beam search while removing only LLM proposals, under the same evaluation budget, is needed to establish this central contribution.

A2\textbf{A2}: Thank you for identifying these omissions. We perform supplementary experiments to compare CoEvolve with CAPS and HAFT in Table 2.The public implementation of EASE and GPT-FT could not be found, thus they are not included in the supplementary experiment.

In the new comparison, CoEvolve scored higher than both CAPS and HAFT on seven of eight datasets. Across the four regression tasks, the mean gains in �0��0� were 0.1008 over CAPS and 0.0674 over HAFT. On UCI Credit, CoEvolve scored 0.8061, compared with 0.8076 for CAPS and 0.8063 for HAFT. These runs used the same downstream learner, splits, and metrics. The total numbers of downstream search evaluations have not been matched, so the results establish performance under the reported settings. Thus, CoEvolve outperforms CAPS and HAFT.

AuthorsOpenReview ↗

Response to Weakness 3 part 1

Q3\textbf{Q3}: Section 3.2 relies on an LLM for transformations and memory updates, but the manuscript does not identify the model or provide the prompts used. These omissions make it difficult to reproduce the agent's behavior.

A3\textbf{A3}: We agree that Section 3.2 should identify the model and make the prompts easy to locate. Here we provides these details and include them in the appendix of revision. 1.Our benchmark configuration specifies Qwen3-30B-Instruct. 2. System prompts and fixed in-context example We use four system prompts and one fixed in-context example. The example accompanies the Conditional Agent Memory Writer. (1) Conditional Agent Memory Writer — CAM_SYSTEM_PROMPT You are the Conditional Feature Direction Inference Agent. You are given a fixed current feature set S and a collection of observed transitions (S, S_prime, score). Your task is to use these observations to identify promising directions for local search around S. The score of a transition is defined as score = metric(S_prime) - metric(S). A positive score indicates that the change improved performance relative to S. A negative score indicates that the change reduced performance relative to S. Assign each item to one of four priority levels. Priority 1 — Most promising directions to explore A direction appears repeatedly across multiple positive-scoring transitions, or a positive-scoring transition directly supports a single operation. These directions are the most promising candidates for further exploration around S. Priority 2 — Individual contribution unknown; prioritize for ablation A specific feature addition, removal, or replacement is part of the best positive-scoring transition. That transition changes multiple features, so the individual contribution of each component remains unknown. Priority 3 — Lower priority with insufficient evidence of harm The observed gain is small, the evidence is mixed, or the direction has only weak positive support. The direction is a lower priority for the current search, but the evidence gives no clear indication that it is harmful. Priority 4 — Negative directions with low confidence The direction appears only in negative-scoring transitions. The evidence comes from few observations or lacks repeated validation. Lower its search priority. The available evidence does not establish that the direction is harmful across all contexts. Rules 1.Attribute a score to an individual operation only when S and S_prime differ by exactly that operation. 2.If a transition adds or removes multiple features, attribute its observed score only to the combination of changes. Do not assign the score to any individual item. 3.Assign the individual components of the best positive-scoring multi-operation transition to Priority 2. 4.Assign each item to at most one priority level. 5.Return at most K items per priority level. 6.Return fewer than K items when the evidence is insufficient. Do not invent items to fill the list. 7.Limit CAM findings to the current S or similar feature sets. Do not generalize them directly to all feature sets. 8.Avoid claims such as “statistically significant” or “conclusively effective” unless the input includes repeated experiments and variance estimates. 9.Describe each item as a concrete operation or search direction. Examples include adding f1*f2, removing the standalone feature log(f1), or replacing {f1, f2} with {f1/f2}. 10.Return only JSON in the specified format. Output format { "best_anchor": { "transition_id": "", "score": 0.0 }, "analysis": [ { "finding": "", "evidence": [], "confidence": "high|medium|low" } ], "priorities": { "priority1": [ { "rank": 1, "item": "", "reason": "", "next_action": "", "confidence": "high|medium|low" } ], "priority2": [], "priority3": [], "priority4": [] }, "do_not_conclude": [] } (2) Fixed in-context example — CAM_ICL_EXAMPLE K=2 CAM={ "cam_id":"C1", "S":["f1","f2","f3","log(f1)","f1+f2"], "S_prime":[ [["f1","(f1+f2)/log(f1)","f3"],0.20], [["f1","f2","f3","f1+f2"],0.06], [["f1","f2","f3","log(f1)","f1+f2","f2+f3"],0.01], [["f1","f2","f3","log(f1)","f1+f2","sin(f1)"],-0.08] ] }

Reviewer_a8z8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoEvolve for automated feature engineering. It combines Bayesian selection of feature sets, local search guided by a gain predictor, and LLM proposals guided by two types of memory. Evaluated candidates update both the prediction models and the memories. The method achieves the highest reported scores on seven of eight datasets.

Critical strengths
  1. The method connects global selection, local search, and memory updates through measured performance. This provides a clear way to reuse search experience.

  2. The results are promising across both regression and classification tasks. CoEvolve ranks first on seven datasets and improves over the strongest baseline by an average relative gain of 3.59%.

Critical weaknesses
  1. DSGO ranks starting feature sets by expected improvement in their own scores, but the goal is to find useful transformations around them. Many of these scores are already measured. The paper does not clearly explain why this acquisition score reflects the value of further local search.

  2. The “w/o LLM” variant also replaces the local search method with embedding-based neighborhood search. Thus, it does not isolate the benefit of the LLM and its memories. Keeping DSGO and FE-MDP beam search while removing only LLM proposals, under the same evaluation budget, is needed to establish this central contribution.

  3. LLM-FE is a close baseline because it also uses evaluated programs and memory to guide LLM search. It is discussed but not compared in Table 1. A comparison using the same downstream model and evaluation budget would help establish the benefit of the proposed coordination.

Questions for authors

Refer to the weaknesses above.

3 comments
AuthorsOpenReview ↗

Full Ablation Study and Additional Baselines

We appreciate the reviewer’s time and efforts in reviewing our paper. In order to to isolate the contribution of each component in CoEvolve. We have expanded the ablation study to compare CoEvolve with six variants: (1) DSGO + FE-MDP, (2) DSGO + embedding-neighborhood search (ENS), (3) DSGO + LLM + CAM, (4) DSGO + LLM + UAM, and (5) FE-MDP (w/o DSGO) (6) DSGO (w/o FE-MDP). All variants use the same initialization and downstream CV budget. We use the same DSGO configuration wherever DSGO is present. For comparisons that isolate the proposal mechanism, we also hold the anchor-selection procedure, the initial beam search, and the legal action set fixed. By proposal mechanism, we mean how a method chooses which additional feature transformations to evaluate from the selected anchor after the initial beam search. In the DSGO + FE-MDP control, FE-MDP ranks untried legal actions by their predicted one-step gain and selects the highest-ranked actions. In the full CoEvolve model, the LLM selects actions from the same set using CAM, which summarizes which transformations helped or hurt at the current anchor, and UAM, which captures experience across anchors and search rounds. Both methods apply their selected actions to the original anchor and evaluate the resulting feature sets by CV under the same remaining budget.

Table 1\textbf{Table 1}

DatasetCoEvolveDSGO + FE-MDPDSGO + ENSDSGO + LLM + CAMDSGO + LLM + UAMFE-MDP (w/o DSGO)DSGO (w/o LLM, FE-MDP)
OpenML-5860.72790.69880.68910.71480.67050.7282\textbf{0.7282}0.6529
OpenML-5890.7036\textbf{0.7036}0.67340.68480.69780.69570.70040.6215
OpenML-6070.6911\textbf{0.6911}0.66980.68640.67140.63720.65590.6237
OpenML-6200.6857\textbf{0.6857}0.66940.65060.67700.65860.64360.6524
UCI Credit Card0.8061\textbf{0.8061}0.78230.79530.79960.77810.79820.7753
SpamBase0.9520\textbf{0.9520}0.92990.92110.93440.92890.92510.9268
svmguide30.8622\textbf{0.8622}0.82740.79400.83980.81620.79740.8140
German Credit0.7835\textbf{0.7835}0.75650.75410.76720.73860.74000.7494

We also perform additional experiments to compare CoEvolve with more recent AutoFE methods, including CAPS (Liu et al., KDD 2025) and HAFT (Zhe et al., KDD 2026). CAPS uses a permutation-invariant encoder–decoder and a PPO policy to select a subset of existing columns. HAFT (Zhe et al., KDD 2026) uses two feature agents and an operation agent coordinated by a shared critic. Table 2 shows the eperimental results.

Table 2\textbf{Table 2}

DatasetLLM-FECAPSHAFTCoEvolve
OpenML-5860.71340.62150.67580.7279\textbf{0.7279}
OpenML-5890.68250.58150.60360.7036\textbf{0.7036}
OpenML-6070.67100.61470.65480.6911\textbf{0.6911}
OpenML-6200.63590.58740.60450.6857\textbf{0.6857}
UCI Credit0.80680.8076\textbf{0.8076}0.80630.8061
SpamBase0.93170.92820.91920.9520\textbf{0.9520}
svmguide30.82110.83840.84660.8622\textbf{0.8622}
German Credit0.78240.77570.77950.7835\textbf{0.7835}
AuthorsOpenReview ↗

Response to Weakness 1

Q1\textbf{Q1}: The “w/o LLM” variant also replaces the local search method with embedding-based neighborhood search. Thus, it does not isolate the benefit of the LLM and its memories. Keeping DSGO and FE-MDP beam search while removing only LLM proposals, under the same evaluation budget, is needed to establish this central contribution.

A1\textbf{A1}: We agree that an anchor’s own expected improvement (EI) and the value of searching around it are distinct. DSGO computes EI from the predicted score of the anchor itself. When the anchor’s training CV score is already known, we reuse that score and spend the local-search budget on new transformations. The rationale for using anchor quality to allocate this budget is that, given the same amount of local search, a higher-scoring starting set may lead to a higher-scoring successor. We added an experiment to test this premise directly.

For each anchor, we ran the same FE-MDP and LLM-guided local search with a fixed evaluation budget and a search depth of two, keeping the successors within two transformation steps of the anchor. Within each dataset, we computed Spearman’s rank correlation between the anchor’s training CV score and the best successor’s training CV score at depth two. The correlation was positive on all eight datasets, ranging from 0.3776 on OpenML-586 to 0.8473 on UCI Credit, with a mean of 0.5987 (Table 3). Thus, under the same local-search budget, higher-scoring anchors tended to yield higher-scoring successors. This supports using anchor quality as one signal for deciding where to search next.

Table 3\textbf{Table 3}

| Dataset | OpenML-586 | OpenML-589| OpenML-607 | OpenML-620 | UCI Credit | SpamBase | svmguide3 |German Credit |

| Spearman | 0.3776 | 0.5172| 0.6724 | 0.5829 | 0.8473| 0.6524 | 0.7128 | 0.4271|

Table 2 provides complementary end-to-end evidence. Under the same CV budget, the full model scored higher than the variant without DSGO on seven of eight datasets. On OpenML-586, where the anchor–successor correlation was weakest, the two methods were nearly tied (0.7279 with DSGO versus 0.7282 without it). Together, these results support the practical use of DSGO’s EI rule for allocating local evaluations. We will clarify that Table 3 validates the anchor-quality premise: it measures the absolute quality of the successors reached. Whether EI predicts the gain over the anchor, and how much its uncertainty term contributes beyond the observed anchor score, are separate questions.

AuthorsOpenReview ↗

Response to Weakness 2 and Weakness 3

Q2\textbf{Q2}: The “w/o LLM” variant also replaces the local search method with embedding-based neighborhood search. Thus, it does not isolate the benefit of the LLM and its memories. Keeping DSGO and FE-MDP beam search while removing only LLM proposals, under the same evaluation budget, is needed to establish this central contribution.

A2\textbf{A2}: We agree that the original “w/o LLM” ablation changed two components at once: it removed the LLM agent and replaced FE-MDP-guided local search with embedding-neighborhood search (ENS). We therefore relabel this variant DSGO + ENS and add DSGO + FE-MDP as a direct control. The control evaluates untried actions ranked by FE-MDP, while CoEvolve evaluates actions proposed by the LLM using CAM and UAM. Experimental results are presented in Table 1. CoEvolve outperformed DSGO + FE-MDP on all eight datasets. The mean absolute gain was 0.0242 in �0��0� across the four regression tasks and 0.0269 in weighted F1 across the four classification tasks. With the underlying beam search and evaluation budget held fixed, these results support the incremental value of the memory-guided LLM proposal stage.This comparison measures the joint contribution of the LLM agent and its memories.

In addition, In order to complement the ablation study, We also added two dedicated ablations in which the LLM receives only CAM (DSGO + LLM + CAM) or only UAM (DSGO + LLM + UAM) when proposing feature transformations. We compare both variants with the full CoEvolve model under the same DSGO anchor-selection procedure, initial FE-MDP beam search, legal action set, and downstream CV budget. Following experimental observations demonstrate the contribution of the dual-memory mechanism in CoEvolve.

1.CoEvolve v.s. DSGO + LLM + UAM: Adding CAM to the UAM-only control improved scores on all eight datasets, the mean gains are 0.0366 in 1−RAE across the four regression tasks and 0.0355 in weighted F1 across the four classification tasks.

2.CoEvolve v.s. DSGO + LLM + CAM: Adding UAM to the CAM-only control improved scores on six of eight datasets, the mean gains are 0.0118 in 1−RAE and 0.0157 in weighted F1. On OpenML-586 and OpenML-589, CAM alone matched the full model at the reported precision.

These results supports using both memories while showing that the value of adding UAM varies across tasks and that CAM with a larger and more consistent contribution.

Q3\textbf{Q3}: LLM-FE is a close baseline because it also uses evaluated programs and memory to guide LLM search. It is discussed but not compared in Table 1. A comparison using the same downstream model and evaluation budget would help establish the benefit of the proposed coordination.

A3\textbf{A3}: We agree that LLM-FE is a close baseline because both methods use the outcomes of previously evaluated transformations to guide subsequent LLM proposals. We have added a direct comparison on all eight datasets in the Table 2. Both methods use the same LLM backbone, Qwen-30B-Instruct.

CoEvolve achieved a higher score than LLM-FE on seven of the eight datasets. The largest gains were on OpenML-620 (0.6857 vs. 0.6359, +0.0498) and svmguide3 (0.8622 vs. 0.8211, +0.0411). On OpenML-589, OpenML-607, and SpamBase, the gains were approximately 0.02 each. The two methods were close on German Credit, where CoEvolve led by 0.0011, and on UCI Credit, where LLM-FE led by 0.0007.

Reviewer_YEvcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoEvolve, an automated feature engineering framework that integrates LLM agents with Bayesian global optimization and learning-guided local search. The framework consists of three main components: DSGO for global anchor selection, FE-MDP for local exploration, and an LLM agent equipped with contextual reasoning and memory mechanisms to guide feature transformations. The proposed approach achieves competitive performance on several benchmark datasets.

Critical strengths
  1. The paper presents an interesting framework that integrates LLM-based reasoning and experience summarization with Bayesian global optimization and learning-guided local search. This combination provides a structured approach to leveraging historical search experience for automated feature engineering.

  2. The paper provides a well-structured review of relevant literature, covering many recent works.

Critical weaknesses
  1. Figure 3 presents ablation studies for the variants without the LLM agent and without DSGO. However, the "w/o LLM" variant not only removes the LLM agent but also replaces the local search mechanism. Therefore, it is difficult to determine whether the observed performance differences can be attributed specifically to the LLM agent.

  2. The paper introduces CAM and UAM as a dual-memory mechanism, which is presented as one of its key contributions. However, no dedicated ablation studies are provided to demonstrate the effectiveness of these memory components or to justify the necessity of using both CAM and UAM.

  3. The proposed framework combines Bayesian global optimization, learning-guided local search, and LLM-assisted search, each of which builds on established techniques. While their integration is reasonable, the paper does not sufficiently clarify the unique advantages of this combination over existing approaches or demonstrate which critical limitations of prior feature engineering methods it addresses.

  4. Are Figure 1 generated by LLMs/VLMs?

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Full Ablation Study and Additional Baselines

We appreciate the reviewer’s careful review and constructive feedbacks. In order to to isolate the contribution of each component in CoEvolve. We have expanded the ablation study to compare CoEvolve with six variants: (1) DSGO + FE-MDP, (2) DSGO + embedding-neighborhood search (ENS), (3) DSGO + LLM + CAM, (4) DSGO + LLM + UAM, and (5) FE-MDP (w/o DSGO) (6) DSGO (w/o FE-MDP). All variants use the same initialization and downstream CV budget. We use the same DSGO configuration wherever DSGO is present. For comparisons that isolate the proposal mechanism, we also hold the anchor-selection procedure, the initial beam search, and the legal action set fixed. By proposal mechanism, we mean how a method chooses which additional feature transformations to evaluate from the selected anchor after the initial beam search. In the DSGO + FE-MDP control, FE-MDP ranks untried legal actions by their predicted one-step gain and selects the highest-ranked actions. In the full CoEvolve model, the LLM selects actions from the same set using CAM, which summarizes which transformations helped or hurt at the current anchor, and UAM, which captures experience across anchors and search rounds. Both methods apply their selected actions to the original anchor and evaluate the resulting feature sets by CV under the same remaining budget.

Table 1\textbf{Table 1}

DatasetCoEvolveDSGO + FE-MDPDSGO + ENSDSGO + LLM + CAMDSGO + LLM + UAMFE-MDP (w/o DSGO)DSGO (w/o LLM, FE-MDP)
OpenML-5860.72790.69880.68910.71480.67050.7282\textbf{0.7282}0.6529
OpenML-5890.7036\textbf{0.7036}0.67340.68480.69780.69570.70040.6215
OpenML-6070.6911\textbf{0.6911}0.66980.68640.67140.63720.65590.6237
OpenML-6200.6857\textbf{0.6857}0.66940.65060.67700.65860.64360.6524
UCI Credit Card0.8061\textbf{0.8061}0.78230.79530.79960.77810.79820.7753
SpamBase0.9520\textbf{0.9520}0.92990.92110.93440.92890.92510.9268
svmguide30.8622\textbf{0.8622}0.82740.79400.83980.81620.79740.8140
German Credit0.7835\textbf{0.7835}0.75650.75410.76720.73860.74000.7494

We also perform additional experiments to compare CoEvolve with more recent AutoFE methods, including CAPS (Liu et al., KDD 2025) and HAFT (Zhe et al., KDD 2026). CAPS uses a permutation-invariant encoder–decoder and a PPO policy to select a subset of existing columns. HAFT (Zhe et al., KDD 2026) uses two feature agents and an operation agent coordinated by a shared critic. Table 2 shows the eperimental results.

Table 2\textbf{Table 2}

DatasetLLM-FECAPSHAFTCoEvolve
OpenML-5860.71340.62150.67580.7279\textbf{0.7279}
OpenML-5890.68250.58150.60360.7036\textbf{0.7036}
OpenML-6070.67100.61470.65480.6911\textbf{0.6911}
OpenML-6200.63590.58740.60450.6857\textbf{0.6857}
UCI Credit0.80680.8076\textbf{0.8076}0.80630.8061
SpamBase0.93170.92820.91920.9520\textbf{0.9520}
svmguide30.82110.83840.84660.8622\textbf{0.8622}
German Credit0.78240.77570.77950.7835\textbf{0.7835}
AuthorsOpenReview ↗

Response to Weakness 1 and Weakness 2

Q1\textbf{Q1}: Figure 3 presents ablation studies for the variants without the LLM agent and without DSGO. However, the "w/o LLM" variant not only removes the LLM agent but also replaces the local search mechanism. Therefore, it is difficult to determine whether the observed performance differences can be attributed specifically to the LLM agent.

A1\textbf{A1}: We agree that the original “w/o LLM” ablation changed two components at once: it removed the LLM agent and replaced FE-MDP-guided local search with embedding-neighborhood search (ENS). We therefore relabel this variant DSGO + ENS and add DSGO + FE-MDP as a direct control. The control evaluates untried actions ranked by FE-MDP, while CoEvolve evaluates actions proposed by the LLM using CAM and UAM. Experimental results are presented in Table 1. CoEvolve outperformed DSGO + FE-MDP on all eight datasets. The mean absolute gain was 0.0242 in �0��0� across the four regression tasks and 0.0269 in weighted F1 across the four classification tasks. With the underlying beam search and evaluation budget held fixed, these results support the incremental value of the memory-guided LLM proposal stage.This comparison measures the joint contribution of the LLM agent and its memories.

Q2\textbf{Q2}: The paper introduces CAM and UAM as a dual-memory mechanism, which is presented as one of its key contributions. However, no dedicated ablation studies are provided to demonstrate the effectiveness of these memory components or to justify the necessity of using both CAM and UAM.

A2\textbf{A2}: We added two dedicated ablations in which the LLM receives only CAM (DSGO + LLM + CAM) or only UAM (DSGO + LLM + UAM) when proposing feature transformations. We compare both variants with the full CoEvolve model under the same DSGO anchor-selection procedure, initial FE-MDP beam search, legal action set, and downstream CV budget. Following experimental observations demonstrate the contribution of the dual-memory mechanism in CoEvolve. 1.CoEvolve v.s. DSGO + LLM + UAM: Adding CAM to the UAM-only control improved scores on all eight datasets, the mean gains are 0.0366 in 1−RAE across the four regression tasks and 0.0355 in weighted F1 across the four classification tasks. 2.CoEvolve v.s. DSGO + LLM + CAM: Adding UAM to the CAM-only control improved scores on six of eight datasets, the mean gains are 0.0118 in 1−RAE and 0.0157 in weighted F1. On OpenML-586 and OpenML-589, CAM alone matched the full model at the reported precision. These results supports using both memories while showing that the value of adding UAM varies across tasks and that CAM with a larger and more consistent contribution.

AuthorsOpenReview ↗

Response to Weakness 3 and Weakness 4

Q3\textbf{Q3}: The proposed framework combines Bayesian global optimization, learning-guided local search, and LLM-assisted search, each of which builds on established techniques. While their integration is reasonable, the paper does not sufficiently clarify the unique advantages of this combination over existing approaches or demonstrate which critical limitations of prior feature engineering methods it addresses.

A3\textbf{A3}: CoEvolve organizes AutoFE around two recurring decisions: which previously explored feature set should we expand next, and which transformations should we evaluate from that set? A single downstream evaluation informs both decisions in different ways. If action �0��0� transforms �1��1� into �2��2�, the score of$ S′S' $provides a new observation for global anchor selection. The score change from �4��4� to$ S′S'$ provides evidence about action �6��6� at that particular starting point. Appendix B illustrates why this distinction matters. The same action reaches the same final feature set from two different starting sets, yet increases �7��7� by 0.0075 in one case and decreases it by 0.0885 in the other.

CAPS[1] and HAFT[2] each illuminate one part of this design. CAPS searches a permutation-invariant embedding space for a high-performing subset of the supplied columns. CoEvolve also searches over sets, but a selected set serves as a starting point for further construction. A transformation can introduce an expression outside the supplied columns, and the resulting set can become an anchor for another round. HAFT learns which features and operation to select within the current feature space through cooperating agents and a shared critic. CoEvolve makes an additional decision across previously explored sets. DSGO selects an anchor using predicted set scores and uncertainty. At that anchor, FE-MDP guides an initial beam search, after which the LLM uses local outcomes recorded in CAM and experience accumulated in UAM to propose further actions. The search can thus return to an earlier set and try a different transformation in light of evidence acquired elsewhere.

Earlier transformation-graph methods also considered both decisions. Khurana et al[3]., for example, used an RL policy to select a joint action comprising an existing node and a transformation. CoEvolve’s contribution is to model the two decisions separately and connect them through measured outcomes: set-level observations guide where to search, while gains and losses observed at specific starting sets guide what to try there.

Under the settings in Table 2, CoEvolve scored higher than both CAPS and HAFT on seven of eight datasets. On the four regression tasks, the mean gains in �8��8� were 0.1008 over CAPS and 0.0674 over HAFT. On UCI Credit, CoEvolve trailed them by 0.0015 and 0.0002, respectively. The matched-budget ablation provides more direct evidence for the second decision. With the DSGO anchor-selection rule, initial beam, and legal actions held fixed, using the LLM and both memories to allocate the remaining evaluations outperformed continuing down the FE-MDP ranking on all eight datasets. Thus, even with the same rule for where to search, using measured local and cross-anchor experience to decide what to try next improved the reported scores.

[1] Rui Liu, Rui Xie, Zijun Yao, Yanjie Fu, and Dongjie Wang. 2025. “Continuous Optimization for Feature Selection with Permutation-Invariant Embedding and Policy-Guided Search.” In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’25), Vol. 2, pp. 1857–1866.

[2]Tao Zhe, Huazhen Fang, Kunpeng Liu, Qian Lou, Tamzidul Hoque, and Dongjie Wang. 2026. “Heterogeneous Multi-Agent Reinforcement Learning with Attention for Cooperative and Scalable Feature Transformation.” In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’26), Vol. 1, pp. 2010–2019.

[3] Udayan Khurana, Horst Samulowitz, and Deepak Turaga. 2018. “Feature Engineering for Predictive Modeling Using Reinforcement Learning.” In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI ’18), 32(1), pp. 3407–3414.

Q4\textbf{Q4}: Are Figure 1 generated by LLMs/VLMs?

A4\textbf{A4}: Yes, we prepared Figure 1 using GPT 6 Astra Ultra. We have changed it in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.