acceptodds
Under review as a conference paper at ICLR 2027

Bridging Expert Knowledge and Automated Feature Engineering via Self-Evolution

Abstract

In high-stakes settings such as brand compliance, clinical care, and content moderation, machine learning cannot be deployed as opaque oracles: practitioners must be able to inspect the features driving model decisions, and models must be able to leverage the expert documentation already governing these domains. This requires features discovered from raw text and images to be interpretable, discriminative, and aligned with what experts consider important. Existing methods fall short: they target tabular inputs, lack demonstrated expert alignment, and cannot operationalize qualitative criteria such as 'maintain professional tone' into more precise, features. To address these challenges, we present FEST (Feature Engineering with Self-evolving Trees), which combines dual-stream feature generation (semantic and deterministic), semantic deduplication, and tree-guided iterative evolution to discover features directly from unstructured data. FEST leads in 17 of 20 classifier-task combinations across brand classification (text and images), content authenticity detection, and stress detection, with a mean gain of 4.2 pp over the strongest baseline across five classifiers. An LLM-as-judge evaluation shows FEST achieves 60-80% coverage of expert-designed brand features at strict semantic-alignment thresholds, corroborated by a human expert study rating FEST features highly on relevance, clarity, and actionability. When seeded with expert guidelines, FEST refines qualitative criteria into precise, operational features, improving downstream accuracy by 6-12 pp on average across brands. To enable systematic evaluation of expert alignment in automated feature engineering, we release BrandGuide, the first dataset pairing expert-designed features with 1M+ assets across 2683 brands. By grounding automated feature engineering in expert knowledge, FEST opens a practical pathway for deploying interpretable ML in domains that demand human oversight and accountability.

Primary area:
interpretability and explainable AI
Keywords:
Self Evolving AI, Interpretable AI, Feature Engineering, Knowledge Engineering, LLMs, Human-Centered AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TyjFOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents Feature Engineering with Self-evolving Trees, combining dualstream feature generation, semantic deduplication, and tree-guided iterative evolution to discover auditable features from raw text and images. FEST leads in 17 of 20 classifier-task combinations across brand classification, content authenticity detection, and stress detection, with a mean gain of 4.2 pp over the strongest baseline across five classifiers.

Critical strengths

The paper correctly identifies a genuine gap: existing automated feature engineering tools (AutoFeat, OpenFE, FeatLLM, LLM-FE) are restricted to tabular inputs and cannot operate on raw unstructured content (text/images), while domain experts in high-stakes settings (brand compliance, clinical care) need inspectable features. The positioning of FEST as a bridge between expert knowledge and automated feature engineering is clear and compelling.

Critical weaknesses

FEST's feature discovery pipeline depends on GPT-4o-mini for both semantic feature proposal and deterministic feature code generation. The paper does not analyze how results would change with open-source alternatives (e.g., Llama, Mistral), nor does it report the API cost per task. For a method targeting domains requiring transparency and auditability, this dependency is problematic.

Questions for authors

Since feature quality directly depends on the LLM's domain understanding, would FEST with a smaller open-source model (e.g., Llama-3-8B) achieve comparable results? Has the authors experimented with different LLM backbones?

2 comments
AuthorsOpenReview ↗

Response to Reviewer TyjF

We thank the reviewer for recognizing the gap FEST addresses, namely that existing automated feature engineering tools are mostly restricted to tabular inputs, and for finding FEST's positioning as a bridge between expert knowledge and automated feature engineering clear and compelling.

The paper does not analyze how results would change with open-source alternatives

Have the authors experimented with different LLM backbones?

Yes. Table 6 in Appendix E.3 ("Cross-Model Replication with Kimi K2.5") reports the complete FEST pipeline run with the open-weight reasoning model Kimi K2.5 (thinking) in place of GPT-4o-mini, with no algorithmic changes, on all five brand-text tasks.

Brand-averaged text-classification metrics (%), Kimi K2.5 (Table 6):

ClassifierMetricZero-ShotFew-ShotFelixFEST
DTAcc / Rec / Prec / F167.9 / 71.3 / 69.6 / 68.775.0 / 78.8 / 74.3 / 76.180.0 / 86.2 / 76.8 / 81.285.1 / 85.7 / 84.9 / 85.2
MLPAcc / Rec / Prec / F168.1 / 71.2 / 70.1 / 69.075.2 / 77.4 / 75.4 / 75.983.1 / 88.5 / 80.1 / 84.085.1 / 84.3 / 85.9 / 85.1
LRAcc / Rec / Prec / F166.9 / 68.4 / 70.1 / 67.474.0 / 75.0 / 74.3 / 74.583.0 / 78.4 / 86.2 / 82.087.6 / 84.6 / 90.1 / 87.2
RFAcc / Rec / Prec / F168.2 / 72.7 / 69.6 / 69.375.2 / 78.1 / 75.0 / 76.085.1 / 89.7 / 82.8 / 85.888.3 / 86.1 / 90.0 / 87.9
XGBAcc / Rec / Prec / F168.5 / 71.7 / 70.3 / 69.375.3 / 78.3 / 75.1 / 76.282.7 / 89.1 / 80.3 / 84.385.6 / 84.5 / 86.3 / 85.4

(Brand-averaged metrics, mean over Emirates, Adobe, Porsche, Louis Vuitton, Pizza Hut.)

FEST leads Felix in Accuracy, Precision and F1 on all five classifiers under Kimi K2.5. On specific classifiers Felix shows a high-recall, lower-precision skew. This confirms that FEST's performance gains hold across open-weight reasoning models and closed-source LLMs. Under GPT-4o-mini, FEST leads in 17 of 20 classifier-task combinations (Table 1).

API cost per task

Table 12 in Appendix L ("Runtime and Cost Analysis") reports cost on brand text classification, as mean ± std over 5 brands with GPT-4o-mini. Section 6.1 ("Task Performance", paragraph "Stability and efficiency") summarizes it.

MetricFESTFelix
Cost per run (USD)$0.10 ± 0.03$8.62 ± 4.64
Runtime (minutes)15.9 ± 3.269.4 ± 35.9
Total tokens75K ± 20K6.8M ± 3.5M

FEST is 86x cheaper, 4.4x faster and uses 91x fewer tokens than Felix.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer TyjF,

As the discussion period closes soon, we wanted to follow up on whether our response addresses your questions. A brief recap:

  1. Other LLM backbones: Table 6 replicates the full pipeline with the open-weight Kimi K2.5. FEST leads all baselines in Accuracy, Precision and F1 for all five classifiers (85.1–88.3% accuracy).
  2. API cost: Table 12 reports $0.10 per run, 86x cheaper and 4.4x faster than Felix.

We would be grateful for your feedback and are happy to clarify any remaining points.

Reviewer_AGyDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FEST, an LLM-based framework that automatically discovers and refines interpretable features from unstructured data through dual-stream feature generation and tree-guided self-evolution. Extensive experiments demonstrate the effectiveness of the method that authors proposed

Critical strengths
  1. Clear Presentation: The paper is well-written and clearly organized, making the proposed framework easy to understand.
  2. Practical Relevance: The paper addresses an interesting, application-oriented problem with clear practical value in real-world scenarios.
  3. Comprehensive Experiments: The experimental evaluation is relatively comprehensive, covering multiple tasks, classifiers, and ablation studies to demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. Limited Comparison with SOTA LLMs: Although the paper compares FEST with GPT-4o-mini for direct classification, I suggest including more recent and widely used LLMs/VLMs to better demonstrate its effectiveness against strong end-to-end classification baselines.
  2. Limited OOD Generalization: Since the feature selection process heavily relies on the training data distribution, the selected features may not generalize well under distribution shifts. I suggest conducting additional out-of-distribution (OOD) experiments to evaluate the robustness and generalizability of FEST.
  3. Insufficient Dataset Quality Validation: Although the authors conduct human evaluation, more details about the dataset construction process should be provided to improve transparency and reproducibility. Additionally, more comprehensive quality checks, particularly regarding the alignment between expert guidelines and collected assets, would help establish the reliability of the proposed dataset.
  4. Minor: the text in Figure 1 should be larger
Questions for authors

see in weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer AGyD (1/2)

We thank the reviewer for finding the paper well written and clearly organized, the problem of clear practical value, and the experimental evaluation relatively comprehensive across tasks, classifiers and ablations. We address each point below.

Limited Comparison with SOTA LLMs: Although the paper compares FEST with GPT-4o-mini for direct classification, I suggest including more recent and widely used LLMs/VLMs

Table 6 in Appendix E.3 ("Cross-Model Replication with Kimi K2.5") reports the complete FEST pipeline with the open-weight reasoning model Kimi K2.5 (thinking) in place of GPT-4o-mini, with no algorithmic changes, on all five brand-text tasks. The full Accuracy / Recall / Precision / F1 table across all five classifiers is reproduced in our response to Reviewer TyjF.

FEST leads Felix and all feature-generation baselines in Accuracy, Precision and F1 on all five classifiers (accuracy: DT 85.1, MLP 85.1, LR 87.6, RF 88.3, XGB 85.6). Felix trades precision for recall on some classifiers, and FEST still has the higher F1 on all five. FEST's gains therefore hold under both a closed-source model (GPT-4o-mini) and an open-weight reasoning model (Kimi K2.5).

For direct end-to-end classification, where the LLM predicts the label and no features are extracted, Table 4 in Appendix E.1 ("Direct End-to-End Text Classification") reports GPT-4o-mini. We repeated the protocol with Kimi K2.5:

BackboneDirect Zero-ShotDirect Few-ShotFEST (RF)
GPT-4o-mini51.372.985.1
Kimi K2.565.373.788.3

(Brand-averaged accuracy (%), mean over Emirates, Adobe, Porsche, Louis Vuitton, Pizza Hut.)

Per-brand direct classification accuracy (Zero-Shot / Few-Shot):

  • GPT-4o-mini: Emirates 43.2 / 58.4, Adobe 50.8 / 67.6, Porsche 51.6 / 70.4, Louis Vuitton 52.4 / 86.0, Pizza Hut 58.5 / 82.3.
  • Kimi K2.5: Emirates 57.6 / 56.4, Adobe 73.2 / 81.2, Porsche 66.0 / 66.4, Louis Vuitton 75.2 / 88.4, Pizza Hut 54.3 / 76.2.

FEST exceeds direct classification by 23.0 pp (zero-shot) and 14.6 pp (few-shot) on Kimi K2.5, and by 33.8 pp and 12.2 pp on GPT-4o-mini. The stronger LLM backbone improves both direct classification and FEST, and FEST stays ahead on each.

more details about the dataset construction process should be provided to improve transparency and reproducibility, more comprehensive quality checks

Appendix U ("BrandGuide Dataset Details") documents BrandGuide's construction and quality control:

  • U.1 Collection Pipeline:

    1. Acquisition of 3,466 candidate guidelines (1963–2025) with metadata (year, region, language, sector).
    2. Temporal filtering to 2014–2025, leaving 2,736.
    3. Structured extraction of design specifications (color codes, typography hierarchies, logo clearance rules, usage constraints).
    4. Retrieval of visual assets per brand.
    5. Manual verification.
  • U.2 Quality Assurance: 53 non-aligned documents removed, leaving 2,683. A stratified 10% sample (n = 300) was checked by two independent annotators, with a senior annotator adjudicating disagreements (Table 16):

    FieldModel–HumanHuman–Human
    Sector classification269/300 (89.7%)283/300 (94.3%)
    Official website URL278/300 (92.7%)298/300 (99.3%)

    The pipeline also applies specification-format validation, resolution filtering with manual review of ambiguous cases, duplicate and near-duplicate removal, and metadata cross-referencing against source documents.

  • U.3 Extended Statistics: 80 sectors, 103 regions, 28 languages, with distributions by year (Table 17).

Minor: the text in Figure 1 should be larger

Thank you. We shall enlarge the text in Figure 1 in the revised version of the paper.

AuthorsOpenReview ↗

Response to Reviewer AGyD (2/2)

Limited OOD Generalization: Since the feature selection process heavily relies on the training data distribution, the selected features may not generalize well under distribution shifts.

The content-authenticity task (Section 5.1) asks whether a story was written by a human or an LLM. Zhou et al. (2024) provide two versions of this dataset: GPT-GC, with GPT-generated stories, and Llama-GC, with Llama-generated stories. We train FEST on one and test on the other in both directions, leading to distribution shift.

Test setTrained on same content generator (in-domain)Trained on other generator (OOD)Change
GPT-GC91.287.6 (trained on Llama-GC)−3.6
Llama-GC80.477.6 (trained on GPT-GC)−2.8

(FEST accuracy %, GPT-5.4 mini backbone, DT classifier.)

On each test set, features learned from the other content generator stay within 3.6 pp of features learned in-domain. The comparison holds the test set fixed, so the difference reflects only the shift in training distribution.

On brand classification, FEST's learned features remain discriminative whether off-brand negatives are competitor posts (Table 2) or synthetically generated with matched topics and a generic writing style (79.8–91.2%, Table 14 in Appendix N, "Synthetic Off-Brand Robustness").

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer AGyD,

As the discussion period closes soon, we wanted to follow up on whether our response addresses your concerns. A brief recap:

  1. Recent LLMs: With Kimi K2.5 as backbone, FEST leads all baselines in Accuracy, Precision and F1 for all five classifiers (Table 6). With the same backbone, FEST reaches 88.3% against 65.3% and 73.7% for direct zero- and few-shot classification.
  2. OOD generalization: On content authenticity, FEST trained on one LLM generator and tested on the other stays within 3.6 pp of in-domain training, in both directions.
  3. Dataset quality: Appendix U documents the BrandGuide pipeline and a two-annotator verification of 300 brands (Table 16).

We would be grateful for your feedback and are happy to clarify any remaining points.

Reviewer_mqJ1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FEST, an iterative automated feature engineering framework designed for unstructured data. The primary objective of FEST is to generate features that are interpretable, discriminative, and aligned with domain expert knowledge. FEST consists of three main components: Dual-stream Feature Generation, Semantic Deduplication, and Tree-guided Feature Evolution. The framework supports two modes: one that automatically discovers features without predefined expert features, and another that initializes the feature bank with existing expert guidelines and subsequently refines and augments them. This paper addresses an important research problem: not only whether LLMs can generate effective features, but also whether an automated feature discovery process can produce features that are predictive, human-inspectable, and aligned with expert-defined criteria.

Critical strengths
  1. FEST uses expert guidelines to initialize, refine, and expand the feature bank. The improved predictive performance suggests that data-driven refinement can effectively enhance expert-defined features.

  2. FEST iteratively generates new features and removes less informative ones. This mechanism enables continuous refinement and helps retain discriminative features.

Critical weaknesses
  1. FEST relies mainly on classification accuracy and Gini-based feature importance, without explicitly optimizing expert alignment. Important expert-defined features may therefore be removed if they contribute little to prediction, potentially undermining the preservation of expert knowledge.

  2. The ablations compare semantic and deterministic features but do not isolate the effects of deduplication, iterative pruning, and contrastive generation. It remains unclear whether the gains come from self-evolution or simply a larger candidate feature pool.

  3. Expert alignment is evaluated on few brands, with only one human expert per brand and no inter-rater agreement analysis. Moreover, semantic feature values rely on a black-box LLM, so readable feature descriptions do not guarantee an interpretable prediction process.

Questions for authors
  1. How does FEST ensure that important expert-defined features are preserved when they have low predictive importance?

  2. Could the authors provide additional ablation studies to clarify the individual contributions of semantic deduplication, iterative pruning, and contrastive feature generation?

  3. How do the authors ensure reliable expert alignment and interpretability, given the limited human evaluation and reliance on black-box LLMs for semantic feature scoring?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mqJ1 (1/2)

We thank the reviewer for the detailed review and for recognizing that FEST uses expert guidelines to initialize, refine and expand the feature bank, and that iterative generation and pruning keep discriminative features.

How does FEST ensure that important expert-defined features are preserved when they have low predictive importance?

When a practitioner mandates retaining expert-defined features despite their low predictive importance, FEST supports protected expert seeds (Appendix Q, "Future Work"). These seeds are exempt from importance-based pruning but can still be refined. We evaluated expert-seeded FEST with and without protection on the same splits:

BrandExpert seedsDTLRRFAvg
LGUnprotected79.776.381.879.2
Protected76.780.182.679.8
AdobeUnprotected87.290.490.089.2
Protected90.091.292.091.1
PorscheUnprotected91.293.294.492.9
Protected92.093.294.493.2
AverageUnprotected86.086.688.787.1
Protected86.288.289.788.0

(Test accuracy %. Both modes use GPT-4o as the backbone)

With protection, every expert guideline is retained, and average accuracy is 88.0% against 87.1% without protection. Practitioners can therefore keep every guideline they require, including those with low predictive value (e.g., organizational policies, abstract brand philosophy, platform meta-guidelines in Appendix J), without reducing average accuracy.

Ablations

We isolate each design choice and report the metric it is meant to affect.

Semantic (SE) and deterministic (DE) streams. SE+DE wins 11 of 14 DT/RF combinations across 7 tasks, and DE-only is always the weakest (Table 8 in Appendix F.1).

Multi-iteration loop. Replacing iterative evolution with a single iteration lowers DT accuracy on brand text classification by 5.4 pp on average across the five brands in the paper (Emirates, Adobe, Porsche, Louis Vuitton, Pizza Hut; Section 5.1). Candidate discovery and refinement across iterations therefore contribute to FEST's performance.

Contrastive generation. Without a paired off-brand sample, the LLM proposes generic properties of on-brand posts, such as topic or overall quality. With a same-sector contrast, it discovers the finer differences that distinguish the brand from its competitors (Appendix D). Removing the contrast therefore lowers both accuracy, by 4.6 pp DT accuracy on brand text classification on average across the same five brands, and GPT-4o-judged coverage at threshold ≥7, from 80.0% to 53.3% for Adobe and from 60.0% to 50.0% for Porsche.

Semantic deduplication. Each iteration generates about 300 candidate features, many of them paraphrases of the same concept ("uses informal language", "casual tone", "conversational style"). Clustering consolidates them into about 30 canonical features (Appendix S.2). Without clustering, DT accuracy on brand text classification drops by only 3.2 pp on average across the same five brands, since the tree can still find discriminative features among the raw candidates. Expert coverage drops more sharply in absence of semantic consolidation, from 80.0% to 40.0% for Adobe, from 69.2% to 53.8% for LG and from 60.0% to 50.0% for Porsche, as idiosyncratic phrasings are retained. Further, each iteration also uses 7.6x more tokens and runs 6.6x slower, since every sample is scored against all candidate features.

It remains unclear whether the gains come from self-evolution or simply a larger candidate feature pool.

Felix (Malberg et al., 2024) proposes thousands of semantic candidate features and still ends up selecting hundreds of them for prediction. In contrast, FEST consolidates about 300 candidates per iteration into 30 clusters and retains about 15 semantic plus 5–10 deterministic features (Appendix S.2). Despite its smaller feature bank, FEST achieves higher task accuracy and expert coverage (DT, Table 1):

MethodBrand TextBrand ImagesContent AuthenticityStress
Felix75.8667.8281.6075.60
FEST81.7078.3291.2078.00

Expert coverage (%) at threshold ≥7 (Table 10):

MethodAdobeLGPorsche
Felix53.353.80.0
FEST80.069.260.0

FEST's predictive gains stem from how it builds the feature bank:

  1. Iterative pruning. FEST removes features with low decision-tree importance across iterations (Appendix I).
  2. Iterative semantic consolidation. FEST merges paraphrased features into canonical concepts across iterations, instead of keeping idiosyncratic LLM phrasing (Appendix I).
  3. Deterministic features. FEST adds executable features, such as sentence-length variance or edge density, alongside semantic ones (Table 8, Appendix F.1).
AuthorsOpenReview ↗

Response to Reviewer mqJ1 (2/2)

How do the authors ensure reliable expert alignment and interpretability, given the limited human evaluation and reliance on black-box LLMs for semantic feature scoring?

FEST's expert alignment is measured by two separate instruments: (A) coverage of expert-authored guidelines, and (B) an expert evaluation by brand practitioners.

(A) Coverage of Expert Guidelines (Section 6.2 "Expert Validation"; Appendix H "LLM-as-Judge Expert Coverage").

Under the GPT-4o judge at threshold ≥7, FEST covers 80.0 / 69.2 / 60.0% of guidelines for Adobe / LG / Porsche, against 53.3 / 53.8 / 0.0% for Felix (Table 10).

We validate the LLM-as-judge coverage protocol along three lines: task formulation, multi-judge agreement, and direct judge-human agreement.

1. Task formulation (semantic similarity): The judge performs a reference-anchored semantic comparison: given a fixed expert guideline, is a discovered feature semantically the same? Unlike unreferenced open-ended quality scoring (where LLM-as-judge can be unreliable), this task is structurally equivalent to natural language inference (NLI) or semantic textual similarity (STS), a regime where LLM evaluations show substantially higher reliability and human alignment.

2. Second-judge cross-check (Kimi K2.5) (Appendix H.3): We repeated the full coverage evaluation across all 1,520 feature-guideline pairs (the complete set across Adobe, LG and Porsche, with zero sampling) using a second independent judge model, Kimi K2.5. Continuous 0–10 scores from GPT-4o and Kimi K2.5 correlate strongly (Pearson r = 0.87, Spearman ρ = 0.87). There are zero pairs where one judge scores a strong match (≥7) and the other scores unrelated (≤2), so the two judges never give opposite verdicts on the same pair, and the coverage scores are not specific to GPT-4o.

3. Direct judge-human agreement study (Appendix H.4): We conducted a blinded human agreement study on a pool of 100 feature-guideline pairs stratified by GPT-4o score, covering both FEST and Felix, so that the sample spans the full score distribution. Raters see two blinded, neutrally labeled statements, with no indication of which is the guideline and which is the feature and no LLM score shown. They rate semantic similarity from 0 to 10 using the same rubric and scale as the LLM judge. In each of 19 independent rating sessions, a rater evaluated 20 randomly sampled pairs from the pool (380 total ratings). Human ratings correlate with GPT-4o at Pearson r = 0.64 and Spearman ρ = 0.64, a moderate-to-strong agreement for this kind of semantic-similarity task.

(B) Expert evaluation by brand practitioners (Table 11 in Appendix K, "Expert Human Study").

This evaluation measures whether practitioners find the features meaningful and usable on their own assets. It requires brand-specific institutional knowledge, which a larger pool of generic, non-specialist annotators cannot supply (Appendix K.1). Experts rated each feature alongside up to 10 assets exhibiting it (Appendix K.2). Refined and newly discovered features were interleaved in one blinded pool.

BrandRelevanceClarityActionability
Zomato (images)4.204.334.13
Adobe (images)4.043.913.80

Moreover, semantic feature values rely on a black-box LLM, so readable feature descriptions do not guarantee an interpretable prediction process.

First, in the expert evaluation (Appendix K.2), brand experts saw each FEST feature together with up to 10 assets that scored highest on it, so they judged both the description and the content it identifies. They rated the features 3.80–4.33 out of 5 on relevance, clarity and actionability (Table 11).

Next, to test whether semantic feature values depend on the LLM, we encoded the same FEST feature bank on the same test samples with GPT-4o-mini and with Kimi K2.5. Each feature value is thresholded at 0.5 to a present/absent label, and we report the fraction of labels on which the two models agree:

BrandAgreement (%)
Emirates86.2
Adobe88.5
Porsche86.5
Louis Vuitton87.6
Pizza Hut87.6
Average87.3

(Brand text classification.)

Two independent LLMs agree on whether a semantic feature is present in 87.3% of cases on average, so the semantic encodings are consistent across LLMs. This indicates that the semantic value is driven by the feature description itself, and not by the internals of either model. These values are fed to a decision tree that predicts through transparent decision paths, so the prediction process is interpretable.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer mqJ1,

As the discussion period closes soon, we wanted to follow up on whether our response addresses your concerns. A brief recap:

  1. Preservation of expert features: With protected expert seeds, every guideline is retained, and average accuracy is 88.0% against 87.1% without protection.
  2. Ablations: On brand text classification, using a single iteration, removing contrastive generation or removing clustering lowers DT accuracy by 5.4, 4.6 and 3.2 pp. The latter two also lower expert coverage.
  3. Reliability of expert alignment: The coverage judge agrees with a second LLM judge (r = 0.87, 1,520 pairs) and with blinded human ratings (r = 0.64, 380 ratings).
  4. LLM-scored features: Two independent LLMs agree on semantic feature values in 87.3% of cases.

We would be grateful for your feedback and are happy to clarify any remaining points.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers for their feedback. The reviewers recognize that FEST addresses a gap left by automated feature engineering methods that mainly target tabular inputs (TyjF), its positioning as a bridge between expert knowledge and automated feature engineering (TyjF), its clear practical value (AGyD), and its use of expert guidelines to initialize, refine and expand the feature bank (mqJ1).

FEST summary. FEST discovers interpretable features from raw text and images. In each iteration, it generates semantic features (natural-language predicates scored by an LLM) and deterministic features (executable Python functions) from contrastive pairs. It then consolidates paraphrased candidates by clustering and summarization, and prunes features with low decision-tree importance. The same engine discovers features from scratch or refines existing expert guidelines.

Three-tier evaluation. We evaluate FEST at three levels: whether its features are discriminative (task performance), whether they match criteria written by domain experts (expert alignment), and whether practitioners find them meaningful and usable (expert evaluation).

Tier 1: Task performance.

  • GPT-4o-mini backbone. FEST leads in 17 of 20 classifier-task combinations across brand classification (text and images), content authenticity and stress detection (Table 1), while being 86x cheaper and 4.4x faster than Felix (Table 12).
  • Kimi K2.5 backbone. With the open-weight reasoning model Kimi K2.5, FEST leads all baselines in Accuracy, Precision and F1 for all five classifiers (Table 6).
  • Direct end-to-end classification (AGyD). FEST outperforms direct zero- and few-shot classification with the same backbone (brand text accuracy, %):
BackboneDirect zero-shotDirect few-shotFEST (RF)
GPT-4o-mini51.372.985.1
Kimi K2.565.373.788.3

Tier 2: Expert alignment. Expert brand guidelines are the specifications practitioners use to enforce brand consistency, and compliance platforms ingest them as ground truth (Section 6.2). FEST discovers features from data without access to these guidelines, so coverage of expert guidelines tests whether FEST finds what domain practitioners consider important for their task.

  • Coverage (GPT-4o judge). At a strict threshold (≥7/10), FEST's features cover 80.0%, 69.2% and 60.0% of guidelines for Adobe, LG and Porsche, compared with 53.3%, 53.8% and 0.0% for Felix (Table 10).
  • Second-judge agreement. Kimi K2.5 rescored all 1,520 feature-guideline pairs. Its scores correlate with GPT-4o at r = 0.87, and the two judges never give opposite verdicts on the same pair (Appendix H.3).
  • Judge-human agreement. In a blinded study with 380 ratings over 100 stratified pairs, human ratings correlate with GPT-4o at r = 0.64 (Appendix H.4).

Tier 3: Expert evaluation. Brand practitioners rated FEST features, each shown with up to 10 assets exhibiting it, at 3.80–4.33 out of 5 on relevance, clarity and actionability (Table 11).

Operationalizing expert knowledge. When seeded with expert guidelines, FEST refines and augments them, improving average accuracy over the expert features as-is by 11.7, 7.9 and 6.2 pp for Adobe, LG and Porsche (Table 9).

New results.

  • Ablations (mqJ1). Removing the multi-iteration loop, contrastive generation or semantic consolidation lowers DT accuracy by 5.4, 4.6 and 3.2 pp, respectively. Removing contrast or consolidation also lowers expert coverage, for example from 80.0% to 53.3% and to 40.0% for Adobe. Semantic and deterministic features are complementary (Table 8).
  • Protected expert seeds (mqJ1). Practitioners can retain every expert guideline, with 88.0% average accuracy against 87.1% without protection.
  • Encoding agreement (mqJ1). GPT-4o-mini and Kimi K2.5 agree on semantic feature values in 87.3% of cases.
  • Direct classification with Kimi K2.5 (AGyD). FEST outperforms direct zero- and few-shot classification with the same backbone (88.3% vs 65.3% and 73.7%).
  • OOD generalization (AGyD). On content authenticity, training on stories from one LLM generator and testing on the other stays within 3.6 pp of in-domain training, in both directions.

Together, the task benchmarks, expert-guideline coverage and practitioner evaluations show that FEST discovers discriminative features that correspond to expert criteria and are usable on real assets. We respectfully request that the reviewers take these empirical benchmarks, expert alignment results and human studies into account in their assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.