- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper presents Feature Engineering with Self-evolving Trees, combining dualstream feature generation, semantic deduplication, and tree-guided iterative evolution to discover auditable features from raw text and images. FEST leads in 17 of 20 classifier-task combinations across brand classification, content authenticity detection, and stress detection, with a mean gain of 4.2 pp over the strongest baseline across five classifiers.
Critical strengths
The paper correctly identifies a genuine gap: existing automated feature engineering tools (AutoFeat, OpenFE, FeatLLM, LLM-FE) are restricted to tabular inputs and cannot operate on raw unstructured content (text/images), while domain experts in high-stakes settings (brand compliance, clinical care) need inspectable features. The positioning of FEST as a bridge between expert knowledge and automated feature engineering is clear and compelling.
Critical weaknesses
FEST's feature discovery pipeline depends on GPT-4o-mini for both semantic feature proposal and deterministic feature code generation. The paper does not analyze how results would change with open-source alternatives (e.g., Llama, Mistral), nor does it report the API cost per task. For a method targeting domains requiring transparency and auditability, this dependency is problematic.
Questions for authors
Since feature quality directly depends on the LLM's domain understanding, would FEST with a smaller open-source model (e.g., Llama-3-8B) achieve comparable results? Has the authors experimented with different LLM backbones?
2 comments
Official Comment by Authors
Dear Reviewer TyjF,
As the discussion period closes soon, we wanted to follow up on whether our response addresses your questions. A brief recap:
- Other LLM backbones: Table 6 replicates the full pipeline with the open-weight Kimi K2.5. FEST leads all baselines in Accuracy, Precision and F1 for all five classifiers (85.1–88.3% accuracy).
- API cost: Table 12 reports $0.10 per run, 86x cheaper and 4.4x faster than Felix.
We would be grateful for your feedback and are happy to clarify any remaining points.
Response to Reviewer TyjF
We thank the reviewer for recognizing the gap FEST addresses, namely that existing automated feature engineering tools are mostly restricted to tabular inputs, and for finding FEST's positioning as a bridge between expert knowledge and automated feature engineering clear and compelling.
Yes. Table 6 in Appendix E.3 ("Cross-Model Replication with Kimi K2.5") reports the complete FEST pipeline run with the open-weight reasoning model Kimi K2.5 (thinking) in place of GPT-4o-mini, with no algorithmic changes, on all five brand-text tasks.
Brand-averaged text-classification metrics (%), Kimi K2.5 (Table 6):
(Brand-averaged metrics, mean over Emirates, Adobe, Porsche, Louis Vuitton, Pizza Hut.)
FEST leads Felix in Accuracy, Precision and F1 on all five classifiers under Kimi K2.5. On specific classifiers Felix shows a high-recall, lower-precision skew. This confirms that FEST's performance gains hold across open-weight reasoning models and closed-source LLMs. Under GPT-4o-mini, FEST leads in 17 of 20 classifier-task combinations (Table 1).
Table 12 in Appendix L ("Runtime and Cost Analysis") reports cost on brand text classification, as mean ± std over 5 brands with GPT-4o-mini. Section 6.1 ("Task Performance", paragraph "Stability and efficiency") summarizes it.
FEST is 86x cheaper, 4.4x faster and uses 91x fewer tokens than Felix.