- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FSDESIGN, a framework for automated feature selection algorithm design formulated as sequential decision making. It represents candidate feature-ranking algorithms as computational graphs built from atomic operators. An LLM proposes and refines these algorithms based on dataset characteristics and previous interactions. An LLM judge then compares each candidate with the current base algorithm, and the resulting rewards guide a tree-based search process. The framework is evaluated on seven multi-label datasets against automated algorithm design methods and feature selection baselines. The results show the best average rank across four evaluation metrics. The paper also includes ablation studies, case analyses, cost estimates, and statistical tests.
Critical strengths
- FSDESIGN does not simply select from a fixed set of existing algorithms. Instead, it combines basic operators to build feature-ranking algorithms for different datasets. This is an interesting approach to automated feature selection.
- The experiments cover seven multi-label datasets and four evaluation metrics, with comparisons against several baselines. The authors also include ablation studies, LLM cost estimates, and statistical tests. These experiments provide a useful overall evaluation of the proposed method.
Critical weaknesses
- The reliability of the LLM-based reward mechanism is unclear. Reversing the presentation order may reduce positional bias, but the paper does not examine consistency across repeated calls or the correlation between judge rewards and independent test performance. Comparisons with simpler rules, such as Pareto dominance or weighted sums, would help establish whether the LLM judge is necessary. Also, the same negative reward is assigned to different outcomes, including worse candidates and inconsistent judgments, which may weaken the reward signal.
- It is unclear how much the improvements come from sequential exploration itself. FSDESIGN combines several components, but the experiments do not fully isolate their contributions. Comparisons with random search and simpler refinement strategies under the same search space and evaluation budget would help clarify the value of tree-based exploration.
- The evaluation covers only seven multi-label datasets, so generalization to other settings remains uncertain. Moreover, the omnibus test for Ranking Loss is not significant (p = 0.13761). The best average rank does not establish significant superiority over individual baselines, so the results should be interpreted with caution.
Questions for authors
- How consistent are the LLM judge's preferences across repeated calls? Do the assigned rewards correlate with independent outer-test performance? How does the judge compare with simpler multi-objective selection rules?
- Can the authors compare random search, linear refinement, tree-based exploration, and the full framework using the same operator space and candidate-evaluation budget?
- The final algorithm is selected by maximizing the sum of signed rewards along a path. How sensitive is this selection to path length and search depth? Could deeper paths receive an advantage simply because they accumulate more rewards?
- Since the omnibus test for Ranking Loss is not statistically significant, which pairwise differences against the main baselines are statistically supported?
7 comments
Overall performance after the revisions
Across seven datasets, the revised method improves 27 of the 28 reported dataset–metric results and matches the original result on Bibtex Hamming loss. These results evaluate the revised method as a whole. Bold values indicate the better result within each original–revised pair; both values are bold in the case of a tie.
| Metric | Version | Bibtex | Birds | Emotions | Genbase | Medical | Scene | Yeast |
|---|---|---|---|---|---|---|---|---|
| MacroAP ↑ | Original | 0.2221 | 0.2954 | 0.6206 | 0.8917 | 0.4673 | 0.6562 | 0.4207 |
| Revised | 0.2385 | 0.3285 | 0.6290 | 0.9166 | 0.4955 | 0.7199 | 0.4316 | |
| Jaccard ↑ | Original | 0.2206 | 0.1632 | 0.4450 | 0.9300 | 0.5814 | 0.4681 | 0.4440 |
| Revised | 0.2327 | 0.1682 | 0.4622 | 0.9364 | 0.6433 | 0.5333 | 0.4533 | |
| RL ↓ | Original | 0.3604 | 0.3456 | 0.2303 | 0.0134 | 0.0977 | 0.1669 | 0.2363 |
| Revised | 0.3456 | 0.3192 | 0.2274 | 0.0116 | 0.0919 | 0.1409 | 0.2308 | |
| HL ↓ | Original | 0.0128 | 0.0463 | 0.2248 | 0.0061 | 0.0153 | 0.1235 | 0.2143 |
| Revised | 0.0128 | 0.0458 | 0.2196 | 0.0057 | 0.0133 | 0.1109 | 0.2111 |
**We have updated the experimental results throughout the paper, including the main comparison tables, ablation study tables, and statistical significance analyses in the appendix. **
We address the reviewer's questions individually below.
R1: Comparison of reward strategies and LLM judge stability (W1 & Q1)
Q: Why use an LLM judge instead of a fixed reward aggregation rule?
A: We assess internal validation gains and whether selected features change as intended. For redundancy reduction, validation metrics gauge predictive gains, while the Result Profile shows changes in selected features' redundancy and label coverage. These diagnostics measure distinct properties, with no inherently good or bad direction. An LLM judge interprets them alongside validation metrics without fixed diagnostic weights or directions.
We compared our LLM Judge with four deterministic rules, changing only parent–candidate reward computation and keeping the search framework, evaluation protocol, and final selection rule fixed.
- LLM Judge (FSDesign) jointly considers the four validation metrics and the Result Profiles. It assigns a positive reward only if both anonymous comparisons, presented in opposite orders, favor the candidate.
- Equal Weight averages the four signed metric changes, counting increases in MacroAP and Jaccard and decreases in RL and HL as improvements.
- Pareto Rank prefers the Pareto-dominant algorithm. If neither dominates, it compares the metrics in the fixed order MacroAP, Jaccard, RL, and HL.
- Majority Vote gives each metric one vote and rewards the candidate if more metrics improve than worsen.
- Signed Differences divides each signed metric change by the sum of the two algorithms’ absolute scores on that metric, then averages the four relative changes.
Time constraints limited evaluation to three datasets. We will report results for all datasets in the final version.
| Dataset | Method | MacroAP ↑ | Jaccard ↑ | RL ↓ | HL ↓ | Time (s/seed) |
|---|---|---|---|---|---|---|
| Birds | FSDesign | 0.1682 ± 0.0312 | 0.3192 ± 0.0253 | 0.0458 ± 0.0032 | 1,751.98 | |
| Equal Weight | 0.3269 ± 0.0378 | 0.1524 ± 0.0322 | 0.3367 ± 0.0434 | 0.0468 ± 0.0019 | 1,970.48 | |
| Pareto Rank | 0.3274 ± 0.0501 | 0.1591 ± 0.0114 | 0.3380 ± 0.0285 | 0.0479 ± 0.0014 | 1,698.82 | |
| Majority Vote | 0.3475 ± 0.0106 | 1,917.27 | ||||
| Signed Differences | 0.3092 ± 0.0369 | 0.1507 ± 0.0364 | 0.3578 ± 0.0341 | 0.0471 ± 0.0010 | 1,919.12 | |
| Emotions | FSDesign | 0.6290 ± 0.0241 | 0.4622 ± 0.0184 | 0.2274 ± 0.0168 | 0.2196 ± 0.0108 | 1,723.98 |
| Equal Weight | 0.6014 ± 0.0215 | 0.4451 ± 0.0139 | 0.2388 ± 0.0143 | 0.2287 ± 0.0069 | 1,523.03 | |
| Pareto Rank | 0.6175 ± 0.0224 | 0.4477 ± 0.0108 | 0.2346 ± 0.0149 | 0.2246 ± 0.0100 | 1,742.60 | |
| Majority Vote | 0.6196 ± 0.0195 | 0.2357 ± 0.0149 | 1,775.44 | |||
| Signed Differences | 0.4487 ± 0.0252 | 0.2240 ± 0.0157 | 1,758.64 | |||
| Scene | FSDesign | 0.7199 ± 0.0131 | 0.1409 ± 0.0065 | 1,841.43 | ||
| Equal Weight | 0.5342 ± 0.0082 | 0.1108 ± 0.0011 | 2,039.93 | |||
| Pareto Rank | 0.7141 ± 0.0127 | 0.5289 ± 0.0025 | 0.1448 ± 0.0055 | 0.1114 ± 0.0033 | 1,924.52 | |
| Majority Vote | 0.7166 ± 0.0179 | 0.5329 ± 0.0100 | 0.1413 ± 0.0055 | 0.1118 ± 0.0038 | 2,037.33 | |
| Signed Differences | 0.7172 ± 0.0173 | 0.5276 ± 0.0267 | 0.1450 ± 0.0062 | 0.1128 ± 0.0048 | 1,862.38 |
FSDesign achieves the best mean held-out result in 9 of the 12 dataset–metric comparisons.
LLM Judge Stability Experiment
We randomly sampled 44 judgments across seven datasets and repeated each five additional times using identical inputs. Of the 220 repeats, 200 matched the originals, yielding 90.9% consistency. All disagreements involved ambiguous comparisons where metric changes and result profiles showed mixed advantages and disadvantages, making a clear winner difficult to identify. These results show high judge stability.
R0: Key Revisions to Our Method
We thank the reviewers for their thoughtful questions and constructive feedback. Their feedback prompted us to examine two parts of FSDesign more closely: how it generates competitive candidate algorithms and how it selects the final algorithm from the candidates it has evaluated. Based on this analysis, we made four revisions:
Motivation and evidence for the revisions
We used Scene as a pilot dataset to examine these design choices.
1. Dataset-specific initialization
We originally used the same Pearson root for every dataset. This choice is difficult to justify universally and can steer early search in an unhelpful direction when the root is poorly suited to the data. We therefore let the LLM design G0 from the training-data profile and the operator registry. The Scene pilot showed only small, mixed changes in internal validation performance. We retained this change because it makes initialization part of dataset-specific algorithm design.
2. Set-dependent redundancy control
We introduced two redundancy-aware ranking operators. A static penalty does not adapt as features are selected, so
GREEDY_SCOREupdates each candidate's penalty based on its pairwise redundancy with the features already selected. When several selected features jointly explain a candidate, pairwise comparisons can miss that redundancy, soGREEDY_CONDITIONAL_SCOREassesses the selected set as a whole. Across five Scene seeds, both operators improved the mean performance on all four held-out metrics over static ranking, with conditional greedy ranking performing best.3. Validation-based parameter calibration
We added a calibration step after the LLM proposes an algorithm. Several attempts to improve the LLM's proposals yielded limited gains, so we instead tuned each graph's numerical parameters while keeping its structure fixed. Across five Scene runs, calibration improved all four internal validation metrics for 24 of 30 proposals relative to their original LLM settings.
4. Direct final selection
As the reviewer pointed out in Q4, accumulated binary rewards count wins and losses along a search path without capturing the magnitude of those changes. We therefore use rewards to guide exploration and select the final algorithm by directly comparing all evaluated candidates using a fixed validation score. Specifically, we select the algorithm with the highest average score across the four internal validation metrics, accounting for each metric's direction. Further discussion of this selection rule is provided in R4.