acceptodds
Under review as a conference paper at ICLR 2027

Automated Feature Selection Algorithm Design as Sequential Decision Making

Abstract

Automated feature selection algorithm design aims to generate algorithms tailored to individual datasets. However, how to construct and iteratively refine executable feature selection algorithms using dataset analysis and evaluation feedback remains underexplored. Fixed algorithm pools restrict the available design decisions, while difficulties in obtaining reliable evaluation feedback hinder iterative refinement. To address this problem, we propose FSDesign, which decouples algorithm design into an environment that supports design and an LLM-based designer agent that makes design decisions. Their interaction constitutes the algorithm design process, which we formulate as sequential decision making. The environment provides an action space in which algorithms are constructed by composing atomic operators, enabling adaptation beyond a predefined pool of complete algorithms. It also evaluates candidates by jointly considering result profiles and downstream predictive metrics, providing comprehensive feedback for iterative refinement. Based on this feedback and the accumulated interaction history, the designer agent repeatedly constructs and refines candidate algorithms. We conduct a series of experiments, demonstrating the effectiveness of FSDesign.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Feature Selection, Automated Algorithm Design, Sequential Decision Making, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yHT8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FSDESIGN, a framework for automated feature selection algorithm design formulated as sequential decision making. It represents candidate feature-ranking algorithms as computational graphs built from atomic operators. An LLM proposes and refines these algorithms based on dataset characteristics and previous interactions. An LLM judge then compares each candidate with the current base algorithm, and the resulting rewards guide a tree-based search process. The framework is evaluated on seven multi-label datasets against automated algorithm design methods and feature selection baselines. The results show the best average rank across four evaluation metrics. The paper also includes ablation studies, case analyses, cost estimates, and statistical tests.

Critical strengths
  1. FSDESIGN does not simply select from a fixed set of existing algorithms. Instead, it combines basic operators to build feature-ranking algorithms for different datasets. This is an interesting approach to automated feature selection.
  2. The experiments cover seven multi-label datasets and four evaluation metrics, with comparisons against several baselines. The authors also include ablation studies, LLM cost estimates, and statistical tests. These experiments provide a useful overall evaluation of the proposed method.
Critical weaknesses
  1. The reliability of the LLM-based reward mechanism is unclear. Reversing the presentation order may reduce positional bias, but the paper does not examine consistency across repeated calls or the correlation between judge rewards and independent test performance. Comparisons with simpler rules, such as Pareto dominance or weighted sums, would help establish whether the LLM judge is necessary. Also, the same negative reward is assigned to different outcomes, including worse candidates and inconsistent judgments, which may weaken the reward signal.
  2. It is unclear how much the improvements come from sequential exploration itself. FSDESIGN combines several components, but the experiments do not fully isolate their contributions. Comparisons with random search and simpler refinement strategies under the same search space and evaluation budget would help clarify the value of tree-based exploration.
  3. The evaluation covers only seven multi-label datasets, so generalization to other settings remains uncertain. Moreover, the omnibus test for Ranking Loss is not significant (p = 0.13761). The best average rank does not establish significant superiority over individual baselines, so the results should be interpreted with caution.
Questions for authors
  1. How consistent are the LLM judge's preferences across repeated calls? Do the assigned rewards correlate with independent outer-test performance? How does the judge compare with simpler multi-objective selection rules?
  2. Can the authors compare random search, linear refinement, tree-based exploration, and the full framework using the same operator space and candidate-evaluation budget?
  3. The final algorithm is selected by maximizing the sum of signed rewards along a path. How sensitive is this selection to path length and search depth? Could deeper paths receive an advantage simply because they accumulate more rewards?
  4. Since the omnibus test for Ranking Loss is not statistically significant, which pairwise differences against the main baselines are statistically supported?
7 comments

R0: Key Revisions to Our Method

We thank the reviewers for their thoughtful questions and constructive feedback. Their feedback prompted us to examine two parts of FSDesign more closely: how it generates competitive candidate algorithms and how it selects the final algorithm from the candidates it has evaluated. Based on this analysis, we made four revisions:

  1. Dataset-specific initialization. The LLM designer now constructs the initial algorithm graph G0G_0 from the training-data profile and the operator registry, replacing the fixed Pearson-based starting graph.
  2. Set-dependent redundancy control. We added two greedy scoring operators that update a feature's redundancy penalty as the selected set grows.
  3. Validation-based parameter calibration. The environment now tests alternative numerical settings for a proposed graph under a fixed budget while keeping its structure unchanged.
  4. Direct final selection. The reward judge continues to use validation metrics and result profiles to guide tree exploration. The final algorithm is selected from all evaluated tree nodes using a fixed score computed from the four internal validation metrics, replacing selection by accumulated path reward.

Motivation and evidence for the revisions

We used Scene as a pilot dataset to examine these design choices.

1. Dataset-specific initialization

We originally used the same Pearson root for every dataset. This choice is difficult to justify universally and can steer early search in an unhelpful direction when the root is poorly suited to the data. We therefore let the LLM design G0G_0 from the training-data profile and the operator registry. The Scene pilot showed only small, mixed changes in internal validation performance. We retained this change because it makes initialization part of dataset-specific algorithm design.

InitializationMacroAP ↑Jaccard ↑RL ↓HL ↓
Fixed Pearson root0.696920.500510.147490.11723
LLM-generated root0.693330.503090.148100.11725

2. Set-dependent redundancy control

We introduced two redundancy-aware ranking operators. A static penalty does not adapt as features are selected, so GREEDY_SCORE updates each candidate's penalty based on its pairwise redundancy with the features already selected. When several selected features jointly explain a candidate, pairwise comparisons can miss that redundancy, so GREEDY_CONDITIONAL_SCORE assesses the selected set as a whole. Across five Scene seeds, both operators improved the mean performance on all four held-out metrics over static ranking, with conditional greedy ranking performing best.

Ranking ruleMacroAP ↑Jaccard ↑RL ↓HL ↓
Static redundancy ranking0.49650.27310.25450.1564
Greedy redundancy ranking0.59080.37800.18780.1413
Conditional greedy redundancy ranking0.65480.45180.16480.1272

3. Validation-based parameter calibration

We added a calibration step after the LLM proposes an algorithm. Several attempts to improve the LLM's proposals yielded limited gains, so we instead tuned each graph's numerical parameters while keeping its structure fixed. Across five Scene runs, calibration improved all four internal validation metrics for 24 of 30 proposals relative to their original LLM settings.

Parameter setting (mean over 30 paired proposals)MacroAP ↑Jaccard ↑RL ↓HL ↓
Original LLM settings0.674070.471730.160730.12154
Calibrated settings0.704480.509300.145540.11469

4. Direct final selection

As the reviewer pointed out in Q4, accumulated binary rewards count wins and losses along a search path without capturing the magnitude of those changes. We therefore use rewards to guide exploration and select the final algorithm by directly comparing all evaluated candidates using a fixed validation score. Specifically, we select the algorithm with the highest average score across the four internal validation metrics, accounting for each metric's direction. Further discussion of this selection rule is provided in R4.

Overall performance after the revisions

Across seven datasets, the revised method improves 27 of the 28 reported dataset–metric results and matches the original result on Bibtex Hamming loss. These results evaluate the revised method as a whole. Bold values indicate the better result within each original–revised pair; both values are bold in the case of a tie.

MetricVersionBibtexBirdsEmotionsGenbaseMedicalSceneYeast
MacroAP ↑Original0.22210.29540.62060.89170.46730.65620.4207
Revised0.23850.32850.62900.91660.49550.71990.4316
Jaccard ↑Original0.22060.16320.44500.93000.58140.46810.4440
Revised0.23270.16820.46220.93640.64330.53330.4533
RL ↓Original0.36040.34560.23030.01340.09770.16690.2363
Revised0.34560.31920.22740.01160.09190.14090.2308
HL ↓Original0.01280.04630.22480.00610.01530.12350.2143
Revised0.01280.04580.21960.00570.01330.11090.2111

**We have updated the experimental results throughout the paper, including the main comparison tables, ablation study tables, and statistical significance analyses in the appendix. **

We address the reviewer's questions individually below.

R1: Comparison of reward strategies and LLM judge stability (W1 & Q1)

Q: Why use an LLM judge instead of a fixed reward aggregation rule?

A: We assess internal validation gains and whether selected features change as intended. For redundancy reduction, validation metrics gauge predictive gains, while the Result Profile shows changes in selected features' redundancy and label coverage. These diagnostics measure distinct properties, with no inherently good or bad direction. An LLM judge interprets them alongside validation metrics without fixed diagnostic weights or directions.

We compared our LLM Judge with four deterministic rules, changing only parent–candidate reward computation and keeping the search framework, evaluation protocol, and final selection rule fixed.

  • LLM Judge (FSDesign) jointly considers the four validation metrics and the Result Profiles. It assigns a positive reward only if both anonymous comparisons, presented in opposite orders, favor the candidate.
  • Equal Weight averages the four signed metric changes, counting increases in MacroAP and Jaccard and decreases in RL and HL as improvements.
  • Pareto Rank prefers the Pareto-dominant algorithm. If neither dominates, it compares the metrics in the fixed order MacroAP, Jaccard, RL, and HL.
  • Majority Vote gives each metric one vote and rewards the candidate if more metrics improve than worsen.
  • Signed Differences divides each signed metric change by the sum of the two algorithms’ absolute scores on that metric, then averages the four relative changes.

Time constraints limited evaluation to three datasets. We will report results for all datasets in the final version.

DatasetMethodMacroAP ↑Jaccard ↑RL ↓HL ↓Time (s/seed)
BirdsFSDesign0.3285±0.0466‾\underline{0.3285 \pm 0.0466}0.1682 ± 0.03120.3192 ± 0.02530.0458 ± 0.00321,751.98
Equal Weight0.3269 ± 0.03780.1524 ± 0.03220.3367 ± 0.04340.0468 ± 0.00191,970.48
Pareto Rank0.3274 ± 0.05010.1591 ± 0.01140.3380 ± 0.02850.0479 ± 0.00141,698.82
Majority Vote0.3475 ± 0.01060.1623±0.0212‾\underline{0.1623 \pm 0.0212}0.3213±0.0066‾\underline{0.3213 \pm 0.0066}0.0461±0.0018‾\underline{0.0461 \pm 0.0018}1,917.27
Signed Differences0.3092 ± 0.03690.1507 ± 0.03640.3578 ± 0.03410.0471 ± 0.00101,919.12
EmotionsFSDesign0.6290 ± 0.02410.4622 ± 0.01840.2274 ± 0.01680.2196 ± 0.01081,723.98
Equal Weight0.6014 ± 0.02150.4451 ± 0.01390.2388 ± 0.01430.2287 ± 0.00691,523.03
Pareto Rank0.6175 ± 0.02240.4477 ± 0.01080.2346 ± 0.01490.2246 ± 0.01001,742.60
Majority Vote0.6196 ± 0.01950.4506±0.0128‾\underline{0.4506 \pm 0.0128}0.2357 ± 0.01490.2228±0.0111‾\underline{0.2228 \pm 0.0111}1,775.44
Signed Differences0.6232±0.0378‾\underline{0.6232 \pm 0.0378}0.4487 ± 0.02520.2284±0.0245‾\underline{0.2284 \pm 0.0245}0.2240 ± 0.01571,758.64
SceneFSDesign0.7199 ± 0.01310.5333±0.0128‾\underline{0.5333 \pm 0.0128}0.1409 ± 0.00650.1109±0.0036‾\underline{0.1109 \pm 0.0036}1,841.43
Equal Weight0.7189±0.0093‾\underline{0.7189 \pm 0.0093}0.5342 ± 0.00820.1412±0.0048‾\underline{0.1412 \pm 0.0048}0.1108 ± 0.00112,039.93
Pareto Rank0.7141 ± 0.01270.5289 ± 0.00250.1448 ± 0.00550.1114 ± 0.00331,924.52
Majority Vote0.7166 ± 0.01790.5329 ± 0.01000.1413 ± 0.00550.1118 ± 0.00382,037.33
Signed Differences0.7172 ± 0.01730.5276 ± 0.02670.1450 ± 0.00620.1128 ± 0.00481,862.38

FSDesign achieves the best mean held-out result in 9 of the 12 dataset–metric comparisons.

LLM Judge Stability Experiment

We randomly sampled 44 judgments across seven datasets and repeated each five additional times using identical inputs. Of the 220 repeats, 200 matched the originals, yielding 90.9% consistency. All disagreements involved ambiguous comparisons where metric changes and result profiles showed mixed advantages and disadvantages, making a clear winner difficult to identify. These results show high judge stability.

Reviewer_ZV7POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FSDesign, an automatic feature selection algorithm design framework based on an LLM agent. This method models algorithm design as a sequential decision-making process, constructing an executable feature ranking algorithm by combining predefined atomic operators, and generating evaluation feedback by combining the dataset profile, downstream predictive metrics, and result profiles. Subsequently, LLM is used to judge the merits of candidate algorithms, and the algorithm is continuously improved through tree-based exploration and historical information.

Critical strengths

S1. The paper models feature selection algorithm design as a sequential decision-making process involving agent-environment interaction, and constructs a typed directed acyclic graph (DAG) through the combination of type-constrained atomic operators. Compared to selecting methods from a fixed algorithm pool, this design supports finer-grained algorithm structure adjustments, improving the flexibility of the algorithm design space.

S2. The paper proposes an evaluation feedback mechanism that combines result profiles with downstream prediction metrics, and utilizes LLM judgment, historical information, and tree-based exploration to guide algorithm iteration. This design considers the trade-offs between different evaluation metrics in multi-label feature selection, providing richer feedback information for automated algorithm design.

Critical weaknesses

W1. The core method lacks innovation, and the research positioning of sequential decision-making is unconvincing. Existing research combines feature selection components using hyperheuristic methods and utilizes LLM feedback and tree search iterative optimization algorithms. FSDesign mainly integrates atomic operator combination, LLM judgment, and conventional tree search, but actually only uses seven predefined operators, lacking new decision-making strategies or search mechanisms. The paper does not compare with traditional search methods under the same operator space, therefore it is difficult to prove that the proposed sequential decision-making framework has independent technical advantages.

[1] Ibrahim et al., New Feature Selection Paradigm Based on Hyper-Heuristic Technique, Applied Mathematical Modelling, 2021.

[2] Zheng et al., Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design, ICML 2025.

[3] Li et al., FeatureHospital: A Skill-Driven Multi-Agent Framework for Automated Algorithm Customization in Multi-View Multi-Label Feature Selection, arXiv:2608.16148, 2026.

W2. The LLM reward mechanism lacks reliability verification, and the accumulated reward design also has significant flaws. FSDesign relies on LLM to compare candidate algorithms with parent algorithms and simplifies the judgment result to +1 or -1, but fails to verify the consistency between these preferences and actual prediction performance. The final algorithm selects based on the accumulated reward along the search path, which may allow multiple small improvements to obtain higher scores than a single significant improvement. This reward ignores the magnitude of performance improvement and cannot guarantee a fair comparison between different search branches, weakening the rationality of the sequential decision-making process.

W3. Key ablation experiments contained confounding factors, and the results were insufficient to support the effectiveness of the core mechanism. The ablation experiments were conducted on only three datasets, and w/o Tree simultaneously altered both the search structure and the stopping mechanism. w/o History not only removed historical information but also removed evaluation evidence from the current parent algorithm, making it difficult to accurately attribute the contributions of each module. Furthermore, Appendix D.2 showed that the Iman–Davenport omnibus test for Ranking Loss failed (p=0.1376), and Nemenyi analysis did not demonstrate a significant advantage over all strong baselines.

Questions for authors

Q1. Compared to existing hyperheuristic feature selection and MCTS-AHD, what new technical contributions does FSDesign's sequential decision-making mechanism make? Can the authors directly compare it with random search, Bayesian optimization, and MCTS under the same seven atomic operators and the same evaluation budget?

Q2. Why use LLM judgment instead of explicit multi-objective optimization criteria?

Q3. Can the authors quantify the consistency between LLM judgments and Pareto dominance, as well as independent test performance, and demonstrate its superiority over fixed-weight, multi-objective ranking, or other deterministic evaluation methods?

Q4. If one search path contains two minor improvements, while another contains only one significant improvement, the accumulated reward will favor the former. How can the authors prove that this path-dependent reward correctly reflects the final algorithm quality, rather than simply favoring longer successful search paths?

Q5. FSDesign involves repeated cross-validation, multiple feature selection ratios, and multiple LLM calls for each candidate evaluation. Can the authors provide the complete end-to-end runtime and validate its performance advantage under the same computational budget and different downstream predictors?

11 comments

R0: Key Revisions to Our Method

We thank the reviewers for their thoughtful questions and constructive feedback. Their feedback prompted us to examine two parts of FSDesign more closely: how it generates competitive candidate algorithms and how it selects the final algorithm from the candidates it has evaluated. Based on this analysis, we made four revisions:

  1. Dataset-specific initialization. The LLM designer now constructs the initial algorithm graph G0G_0 from the training-data profile and the operator registry, replacing the fixed Pearson-based starting graph.
  2. Set-dependent redundancy control. We added two greedy scoring operators that update a feature's redundancy penalty as the selected set grows.
  3. Validation-based parameter calibration. The environment now tests alternative numerical settings for a proposed graph under a fixed budget while keeping its structure unchanged.
  4. Direct final selection. The reward judge continues to use validation metrics and result profiles to guide tree exploration. The final algorithm is selected from all evaluated tree nodes using a fixed score computed from the four internal validation metrics, replacing selection by accumulated path reward.

Motivation and evidence for the revisions

We used Scene as a pilot dataset to examine these design choices.

1. Dataset-specific initialization

We originally used the same Pearson root for every dataset. This choice is difficult to justify universally and can steer early search in an unhelpful direction when the root is poorly suited to the data. We therefore let the LLM design G0G_0 from the training-data profile and the operator registry. The Scene pilot showed only small, mixed changes in internal validation performance. We retained this change because it makes initialization part of dataset-specific algorithm design.

InitializationMacroAP ↑Jaccard ↑RL ↓HL ↓
Fixed Pearson root0.696920.500510.147490.11723
LLM-generated root0.693330.503090.148100.11725

2. Set-dependent redundancy control

We introduced two redundancy-aware ranking operators. A static penalty does not adapt as features are selected, so GREEDY_SCORE updates each candidate's penalty based on its pairwise redundancy with the features already selected. When several selected features jointly explain a candidate, pairwise comparisons can miss that redundancy, so GREEDY_CONDITIONAL_SCORE assesses the selected set as a whole. Across five Scene seeds, both operators improved the mean performance on all four held-out metrics over static ranking, with conditional greedy ranking performing best.

Ranking ruleMacroAP ↑Jaccard ↑RL ↓HL ↓
Static redundancy ranking0.49650.27310.25450.1564
Greedy redundancy ranking0.59080.37800.18780.1413
Conditional greedy redundancy ranking0.65480.45180.16480.1272

3. Validation-based parameter calibration

We added a calibration step after the LLM proposes an algorithm. Several attempts to improve the LLM's proposals yielded limited gains, so we instead tuned each graph's numerical parameters while keeping its structure fixed. Across five Scene runs, calibration improved all four internal validation metrics for 24 of 30 proposals relative to their original LLM settings.

Parameter setting (mean over 30 paired proposals)MacroAP ↑Jaccard ↑RL ↓HL ↓
Original LLM settings0.674070.471730.160730.12154
Calibrated settings0.704480.509300.145540.11469

4. Direct final selection

As the reviewer pointed out in W2 and Q4, accumulated binary rewards count wins and losses along a search path without capturing the magnitude of those changes. We therefore use rewards to guide exploration and select the final algorithm by directly comparing all evaluated candidates using a fixed validation score. Specifically, we select the algorithm with the highest average score across the four internal validation metrics, accounting for each metric's direction. Further discussion of this selection rule is provided in R6.

Overall performance after the revisions

Across seven datasets, the revised method improves 27 of the 28 reported dataset–metric results and matches the original result on Bibtex Hamming loss. These results evaluate the revised method as a whole. Bold values indicate the better result within each original–revised pair; both values are bold in the case of a tie.

MetricVersionBibtexBirdsEmotionsGenbaseMedicalSceneYeast
MacroAP ↑Original0.22210.29540.62060.89170.46730.65620.4207
Revised0.23850.32850.62900.91660.49550.71990.4316
Jaccard ↑Original0.22060.16320.44500.93000.58140.46810.4440
Revised0.23270.16820.46220.93640.64330.53330.4533
RL ↓Original0.36040.34560.23030.01340.09770.16690.2363
Revised0.34560.31920.22740.01160.09190.14090.2308
HL ↓Original0.01280.04630.22480.00610.01530.12350.2143
Revised0.01280.04580.21960.00570.01330.11090.2111

We have updated the experimental results throughout the paper, including the main comparison tables, ablation study tables, and statistical significance analyses in the appendix.

We address the reviewer's questions individually below.

R1: Comparison with other heuristic feature selection algorithms (W1 & Q1)

FSDesign differs from current methods in two respects:

  1. Designed for feature selection. Existing general-purpose automatic heuristic design frameworks, such as EoH, ReEvo, and MCTS-AHD, primarily focus on optimization problems with explicitly defined objectives, such as minimizing the total tour length in the Traveling Salesman Problem (TSP), rather than addressing the specific challenges of feature selection. In multi-label feature selection, different predictive metrics can favor different algorithms. FSDesign therefore guides revisions using these metrics alongside diagnostics of the selected features.
  2. A more controlled design space. FSDesign asks the LLM to compose predefined, typed operators into an algorithm graph. The shared format makes proposals easier for the environment to parse, validate, and evaluate consistently, while keeping the search focused on concrete algorithmic changes.
  3. FSDesign incorporates an iterative refinement mechanism compared to FeatureHospital. Specifically, while FeatureHospital [1] is also a multi-agent workflow for feature selection algorithm design, the key distinction lies in our integration of explicit algorithm evaluation and continuous iterative refinement.

To provide a direct empirical comparison, we additionally evaluated EoH [2] (ICML 2024), ReEvo [3] (NeurIPS 2024), MCTS-AHD [4] (ICML 2025), and HH [5] (Applied Mathematical Modelling, 2021). We followed the settings recommended in their original papers, adapting the methods to search the same design space as FSDesign. All LLM-based methods used the same model, gpt-5-mini. We controlled runtime budgets to keep the computational cost comparable to FSDesign; the measured time per seed is reported below. Due to time constraints, we have only completed evaluations on three datasets. We will include results for all datasets in the final version.

DatasetMethodMacroAP ↑Jaccard ↑RL ↓HL ↓Time (s/seed)
BirdsFSDesign0.3285 ± 0.04660.1682 ± 0.03120.3192 ± 0.02530.0458 ± 0.00321,751.98
EoH0.2808 ± 0.03470.1394 ± 0.05610.3463 ± 0.05260.0467 ± 0.00451,610.46
ReEvo0.2951±0.0209‾\underline{0.2951 \pm 0.0209}0.1620 ± 0.01190.3428±0.0281‾\underline{0.3428 \pm 0.0281}0.0467±0.0020‾\underline{0.0467 \pm 0.0020}1,499.08
MCTS-AHD0.2886 ± 0.03300.1648±0.0374‾\underline{0.1648 \pm 0.0374}0.3560 ± 0.04720.0468 ± 0.00371,765.39
HH0.2538 ± 0.07110.0929 ± 0.05050.3618 ± 0.07470.0498 ± 0.00331,752.00
EmotionsFSDesign0.6290±0.0241‾\underline{0.6290 \pm 0.0241}0.4622 ± 0.01840.2274 ± 0.01680.2196 ± 0.01081,723.98
EoH0.6219 ± 0.02870.4523 ± 0.01430.2334 ± 0.01670.2232 ± 0.00921,724.00
ReEvo0.6229 ± 0.01660.4568±0.0087‾\underline{0.4568 \pm 0.0087}0.2313 ± 0.01580.2219 ± 0.00701,784.95
MCTS-AHD0.6294 ± 0.03060.4542 ± 0.01360.2311±0.0148‾\underline{0.2311 \pm 0.0148}0.2202±0.0092‾\underline{0.2202 \pm 0.0092}1,688.53
HH0.6124 ± 0.01840.4281 ± 0.03460.2411 ± 0.01810.2276 ± 0.01271,724.00
SceneFSDesign0.7199 ± 0.01310.5333 ± 0.01280.1409 ± 0.00650.1109 ± 0.00361,841.43
EoH0.6494 ± 0.05600.4433 ± 0.05890.1766 ± 0.04020.1270 ± 0.00961,871.24
ReEvo0.7063±0.0216‾\underline{0.7063 \pm 0.0216}0.5212±0.0175‾\underline{0.5212 \pm 0.0175}0.1441±0.0095‾\underline{0.1441 \pm 0.0095}0.1136±0.0039‾\underline{0.1136 \pm 0.0039}2,102.12
MCTS-AHD0.6806 ± 0.05350.4869 ± 0.05880.1566 ± 0.02130.1195 ± 0.01191,756.48
HH0.6785 ± 0.02220.4900 ± 0.02440.1566 ± 0.00830.1196 ± 0.00531,841.46

Under the same budget, FSDesign achieves the best performance on nearly all metrics across the three datasets, with the only exception being MacroAP on Emotions, where it ranks second. These results demonstrate that FSDesign is more effective for feature selection than existing general-purpose algorithm design methods.

Reviewer_NnU5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FSDesign, which formulates automated feature selection algorithm design as sequential decision making between an evaluation environment and an LLM-based designer agent. Candidate feature-ranking algorithms are represented as graphs composed of atomic operators. The environment evaluates candidates using four multi-label prediction metrics and result profiles, while an LLM judge provides pairwise rewards. The designer uses evaluation feedback and interaction history to guide tree-based exploration. Across seven multi-label datasets, FSDesign achieves the best average results on MacroAP, Jaccard, ranking loss, and Hamming loss.

Critical strengths

(1) The paper addresses a relevant problem: constructing feature selection algorithms suited to individual datasets. The compositional search space allows the designer to modify individual operations and generate executable feature-ranking algorithms, providing greater flexibility than selecting from a fixed pool of methods. (2) The experimental results are encouraging. FSDesign achieves the best average performance across four metrics, and the ablations support the value of iterative refinement, evaluation feedback, and interaction history. The authors also separate algorithm design from held-out test evaluation.

Critical weaknesses

(1) The methodological contribution over existing LLM-based algorithm design methods remains unclear. FSDesign introduces an algorithm graph representation and combines evaluation feedback with history-guided search. However, iterative program generation and feedback-driven refinement have been explored in prior work, including EoH and ReEvo. While the ablations support iterative refinement and tree-based exploration, the evidence does not clearly establish the additional benefit of LLM-guided revisions over simpler search strategies within the same operator space. (2) The contribution of the LLM-based reward mechanism is not fully established. The authors use an LLM judge to compare candidates based on predictive metrics and result profiles. The ablation studies and case studies provide evidence that both sources of information are useful. However, they do not establish whether the LLM-based reward mechanism consistently produces better feature selection algorithms than a fixed aggregation of validation metrics under the same search procedure and evaluation budget. This comparison would help clarify the contribution of the proposed reward mechanism. (3) It is not clear why the final algorithm is chosen by adding up rewards along its search path. A small improvement and a large improvement both receive +1, so several small improvements could beat one large improvement even when the resulting algorithm performs worse. It would be helpful to compare this rule with choosing the final algorithm based on its validation results. (4) I'm not sure whether the baselines received enough tuning. Several use the same settings across datasets, and RLBMLFS runs for only two update iterations, while FSDesign repeatedly tests and adjusts its algorithms for each dataset. Could the baselines improve with more tuning or longer runs? This would help judge how much of the reported gain comes from the proposed method.

Questions for authors

Q1. How does FSDesign perform when the LLM judge is replaced by a fixed aggregation of validation metrics, while keeping the search procedure and evaluation budget unchanged? Q2. How does FSDesign compare with random search over the same valid algorithm graphs under the same number of candidate evaluations? Q3. What motivates selecting the final algorithm by accumulated reward rather than by its validation performance?

10 comments

R0: Key Revisions to Our Method

We thank the reviewers for their thoughtful questions and constructive feedback. Their feedback prompted us to examine two parts of FSDesign more closely: how it generates competitive candidate algorithms and how it selects the final algorithm from the candidates it has evaluated. Based on this analysis, we made four revisions:

  1. Dataset-specific initialization. The LLM designer now constructs the initial algorithm graph G0G_0 from the training-data profile and the operator registry, replacing the fixed Pearson-based starting graph.
  2. Set-dependent redundancy control. We added two greedy scoring operators that update a feature's redundancy penalty as the selected set grows.
  3. Validation-based parameter calibration. The environment now tests alternative numerical settings for a proposed graph under a fixed budget while keeping its structure unchanged.
  4. Direct final selection. The reward judge continues to use validation metrics and result profiles to guide tree exploration. The final algorithm is selected from all evaluated tree nodes using a fixed score computed from the four internal validation metrics, replacing selection by accumulated path reward.

Motivation and evidence for the revisions

We used Scene as a pilot dataset to examine these design choices.

1. Dataset-specific initialization

We originally used the same Pearson root for every dataset. This choice is difficult to justify universally and can steer early search in an unhelpful direction when the root is poorly suited to the data. We therefore let the LLM design G0G_0 from the training-data profile and the operator registry. The Scene pilot showed only small, mixed changes in internal validation performance. We retained this change because it makes initialization part of dataset-specific algorithm design.

InitializationMacroAP ↑Jaccard ↑RL ↓HL ↓
Fixed Pearson root0.696920.500510.147490.11723
LLM-generated root0.693330.503090.148100.11725

2. Set-dependent redundancy control

We introduced two redundancy-aware ranking operators. A static penalty does not adapt as features are selected, so GREEDY_SCORE updates each candidate's penalty based on its pairwise redundancy with the features already selected. When several selected features jointly explain a candidate, pairwise comparisons can miss that redundancy, so GREEDY_CONDITIONAL_SCORE assesses the selected set as a whole. Across five Scene seeds, both operators improved the mean performance on all four held-out metrics over static ranking, with conditional greedy ranking performing best.

Ranking ruleMacroAP ↑Jaccard ↑RL ↓HL ↓
Static redundancy ranking0.49650.27310.25450.1564
Greedy redundancy ranking0.59080.37800.18780.1413
Conditional greedy redundancy ranking0.65480.45180.16480.1272

3. Validation-based parameter calibration

We added a calibration step after the LLM proposes an algorithm. Several attempts to improve the LLM's proposals yielded limited gains, so we instead tuned each graph's numerical parameters while keeping its structure fixed. Across five Scene runs, calibration improved all four internal validation metrics for 24 of 30 proposals relative to their original LLM settings.

Parameter setting (mean over 30 paired proposals)MacroAP ↑Jaccard ↑RL ↓HL ↓
Original LLM settings0.674070.471730.160730.12154
Calibrated settings0.704480.509300.145540.11469

4. Direct final selection

As the reviewer pointed out in W3 and Q3, accumulated binary rewards count wins and losses along a search path without capturing the magnitude of those changes. We therefore use rewards to guide exploration and select the final algorithm by directly comparing all evaluated candidates using a fixed validation score. Specifically, we select the algorithm with the highest average score across the four internal validation metrics, accounting for each metric's direction. Further discussion of this selection rule is provided in R5.

**We have updated the experimental results throughout the paper, including the main comparison tables, ablation study tables, and statistical significance analyses in the appendix. **

We address the reviewer's questions individually below.

Overall performance after the revisions

Across seven datasets, the revised method improves 27 of the 28 reported dataset–metric results and matches the original result on Bibtex Hamming loss. These results evaluate the revised method as a whole. Bold values indicate the better result within each original–revised pair; both values are bold in the case of a tie.

MetricVersionBibtexBirdsEmotionsGenbaseMedicalSceneYeast
MacroAP ↑Original0.22210.29540.62060.89170.46730.65620.4207
Revised0.23850.32850.62900.91660.49550.71990.4316
Jaccard ↑Original0.22060.16320.44500.93000.58140.46810.4440
Revised0.23270.16820.46220.93640.64330.53330.4533
RL ↓Original0.36040.34560.23030.01340.09770.16690.2363
Revised0.34560.31920.22740.01160.09190.14090.2308
HL ↓Original0.01280.04630.22480.00610.01530.12350.2143
Revised0.01280.04580.21960.00570.01330.11090.2111

R1. Differences from prior algorithm-generation methods (W1)

FSDesign differs from EoH and ReEvo in two respects:

  1. Designed for feature selection. EoH and ReEvo are general frameworks for heuristic design, typically evaluated on tasks with an explicit objective, such as minimizing total tour length in the traveling salesman problem. In multi-label feature selection, different predictive metrics can favor different algorithms. FSDesign therefore guides revisions using these metrics alongside diagnostics of the selected features.
  2. A more controlled design space. FSDesign asks the LLM to compose predefined, typed operators into an algorithm graph. The shared format makes proposals easier for the environment to parse, validate, and evaluate consistently, while keeping the search focused on concrete algorithmic changes.

To provide a direct empirical comparison, we additionally evaluated EoH [1] (ICML 2024), ReEvo [2] (NeurIPS 2024), MCTS-AHD [3] (ICML 2025), and HH [4] (Applied Mathematical Modelling, 2021). We followed the settings recommended in their original papers, adapting the methods to search the same design space as FSDesign. All LLM-based methods used the same model, gpt-5-mini. We controlled runtime budgets to keep the computational cost comparable to FSDesign; the measured time per seed is reported below. Due to time constraints, we have only completed evaluations on three datasets. We will include results for all datasets in the final version.

DatasetMethodMacroAP ↑Jaccard ↑RL ↓HL ↓Time (s/seed)
BirdsFSDesign0.3285 ± 0.04660.1682 ± 0.03120.3192 ± 0.02530.0458 ± 0.00321,751.98
EoH0.2808 ± 0.03470.1394 ± 0.05610.3463 ± 0.05260.0467 ± 0.00451,610.46
ReEvo0.2951±0.0209‾\underline{0.2951 \pm 0.0209}0.1620 ± 0.01190.3428±0.0281‾\underline{0.3428 \pm 0.0281}0.0467±0.0020‾\underline{0.0467 \pm 0.0020}1,499.08
MCTS-AHD0.2886 ± 0.03300.1648±0.0374‾\underline{0.1648 \pm 0.0374}0.3560 ± 0.04720.0468 ± 0.00371,765.39
HH0.2538 ± 0.07110.0929 ± 0.05050.3618 ± 0.07470.0498 ± 0.00331,752.00
EmotionsFSDesign0.6290±0.0241‾\underline{0.6290 \pm 0.0241}0.4622 ± 0.01840.2274 ± 0.01680.2196 ± 0.01081,723.98
EoH0.6219 ± 0.02870.4523 ± 0.01430.2334 ± 0.01670.2232 ± 0.00921,724.00
ReEvo0.6229 ± 0.01660.4568±0.0087‾\underline{0.4568 \pm 0.0087}0.2313 ± 0.01580.2219 ± 0.00701,784.95
MCTS-AHD0.6294 ± 0.03060.4542 ± 0.01360.2311±0.0148‾\underline{0.2311 \pm 0.0148}0.2202±0.0092‾\underline{0.2202 \pm 0.0092}1,688.53
HH0.6124 ± 0.01840.4281 ± 0.03460.2411 ± 0.01810.2276 ± 0.01271,724.00
SceneFSDesign0.7199 ± 0.01310.5333 ± 0.01280.1409 ± 0.00650.1109 ± 0.00361,841.43
EoH0.6494 ± 0.05600.4433 ± 0.05890.1766 ± 0.04020.1270 ± 0.00961,871.24
ReEvo0.7063±0.0216‾\underline{0.7063 \pm 0.0216}0.5212±0.0175‾\underline{0.5212 \pm 0.0175}0.1441±0.0095‾\underline{0.1441 \pm 0.0095}0.1136±0.0039‾\underline{0.1136 \pm 0.0039}2,102.12
MCTS-AHD0.6806 ± 0.05350.4869 ± 0.05880.1566 ± 0.02130.1195 ± 0.01191,756.48
HH0.6785 ± 0.02220.4900 ± 0.02440.1566 ± 0.00830.1196 ± 0.00531,841.46

Under the same budget, FSDesign achieves the best performance on nearly all metrics across the three datasets, with the only exception being MacroAP on Emotions, where it ranks second. These results demonstrate that FSDesign is more effective for feature selection than existing general-purpose algorithm design methods.

Comments on the paper

1 comment

Thank you for your questions! We would like to provide further clarification on the remaining concerns.

1. For New Q1 (NQ1):

We designed several alternative search mechanisms to replace LLM-based algorithm design, which can be viewed as an ablation between the "environment" and the "LLM Agent." Specifically, we employed Random Search and Bayesian Optimization to explore the action space provided by the environment and compared their performance with our LLM-based approach (see R2 for detailed results). The experiments show that our LLM-based approach generally outperforms these search methods, suggesting that LLMs can effectively understand the environment and make better algorithmic design decisions.

Regarding the LLM-based evaluation mechanism, we compared our LLM reward evaluation with Equal Weight, Pareto Rank, Majority Vote, and Signed Differences (see R3). The results show that our LLM-based evaluation achieves the best performance on the largest number of metrics. Furthermore, we conducted a stability analysis of LLM judgments, obtaining a consistency rate of 90.9% across 220 repeated evaluations. These findings suggest that our LLM-based evaluation can not only make better evaluation decisions but also maintain a high level of stability.

2. For NQ2:

The newly introduced operators originated from an attempt during our exploration stage, and we also provided a corresponding case study in the second point of R0. However, we have not yet conducted more detailed controlled experiments to isolate their contributions.

Regarding the improvements introduced by parameter calibration, we can provide an additional ablation experiment. Specifically, we compared the performance of the final selected algorithms before and after parameter calibration. The results are presented below:

MetricVariantBibtexBirdsEmotionsGenbaseMedicalSceneYeastAverage
MacroAP ↑With calibration0.2385 ± 0.00580.3285 ± 0.04660.6290 ± 0.02410.9166 ± 0.04400.4955 ± 0.04590.7199 ± 0.01310.4316 ± 0.00560.5371
Raw LLM proposal0.2366 ± 0.00640.2876 ± 0.03580.5941 ± 0.06070.9190 ± 0.04820.4726 ± 0.04490.6902 ± 0.02000.4307 ± 0.00560.5187
Jaccard ↑With calibration0.2327 ± 0.00470.1682 ± 0.03120.4622 ± 0.01840.9364 ± 0.01860.6433 ± 0.02620.5333 ± 0.01280.4533 ± 0.00740.4899
Raw LLM proposal0.2309 ± 0.00530.1619 ± 0.01620.4130 ± 0.07350.9331 ± 0.01790.6150 ± 0.04830.5066 ± 0.02940.4524 ± 0.00740.4733
RL ↓With calibration0.3456 ± 0.01200.3192 ± 0.02530.2274 ± 0.01680.0116 ± 0.00490.0919 ± 0.02490.1409 ± 0.00650.2308 ± 0.00320.1953
Raw LLM proposal0.3456 ± 0.01230.3468 ± 0.02870.2574 ± 0.04730.0108 ± 0.00590.1103 ± 0.02820.1516 ± 0.01120.2336 ± 0.00620.2080
HL ↓With calibration0.0128 ± 0.00010.0458 ± 0.00320.2196 ± 0.01080.0057 ± 0.00130.0133 ± 0.00090.1109 ± 0.00360.2111 ± 0.00290.0884
Raw LLM proposal0.0128 ± 0.00010.0473 ± 0.00200.2350 ± 0.02530.0059 ± 0.00130.0147 ± 0.00220.1175 ± 0.00390.2115 ± 0.00260.0921

The results show that parameter calibration improves performance in most cases, demonstrating that the newly introduced calibration procedure provides meaningful benefits.

3. For NQ3:

We sincerely apologize for these inconsistencies. Due to the limited time available for revision, we updated only part of the descriptions and experimental tables in the manuscript, while some passages still reflect the original method. As a result, several inconsistencies and omissions remain. We are actively working to correct these issues and ensure that the manuscript consistently reflects the revised methodology. Thank you for bringing these issues to our attention.

We sincerely appreciate your continued attention to our work and your valuable questions and suggestions, especially given your busy schedule.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.