acceptodds
Under review as a conference paper at ICLR 2027

EvoSkillRec: Skill-Genome Evolution for Recommender Architecture Discovery

10 views

Abstract

Modern recommender systems advance not only by scaling data and parameters, but also by encoding task-specific inductive biases through architecture, including sparse feature interactions for click-through rate (CTR) prediction, temporal attention for sequential recommendation, and expert routing for multi-task learning. However, these biases are typically human expert designed or searched within predefined operator spaces. Although Recent LLM-driven code evolution expands this space, unconstrained edits often produce invalid or ineffective architectures, underuse established architecture design knowledge, and fail to preserve successful innovations for reuse. We introduce evoskillrec, a promotion-and-reuse framework for cumulative recommender architecture evolution. It first decomposes recommenders into atomic executable skills and represents architectures as typed skill genomes, with each skill equipped with input–output types, semantic annotations, and implementation code. We then evolve models with different tasks through two coupled spaces: a constrained skill–space that mutates, recombines, specializes, and reuses validated skills, and an open-ended code–space in which LLM planners and synthesizers invent new skill modules using prior evolution traces and accumulated experience. An autoresearch controller evaluates candidates, diagnoses failures, retrieves relevant skills, promotes validated innovations into the skill library, and adaptively allocates the proposal budget between the two spaces. Extensive experiments on CTR prediction, multi-task learning, and multi-domain learning, including resource-constrained co-optimization of predictive quality and model FLOPs utilization in generative recommendation, consistently demonstrate the effectiveness of our proposed evoskillrec.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Recommender Systems, Neural Architecture Search, Automated Machine Learning, Evolutionary Search, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_12pROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EvoSkillRec (Skill-Genome Evolution for Recommender Architecture Discovery), which explores how to leverage large language models (LLMs) to automatically discover architectures for recommender systems, while simultaneously distilling effective modules generated during the search process into reusable knowledge. This framework effectively addresses several key challenges: the low efficiency of traditional manual architecture design, the restricted search spaces in existing AutoML approaches, and the lack of knowledge reuse in LLM-driven code evolution.

Critical strengths
  1. EvoSkillRec formulates recommender self-evolution as cumulative skill-genome growth, decomposing models into validated, reusable architectural units and coupling type-checked skill-space mutation with open-ended code-space invention. Successful innovations are continuously distilled back into the skill library, enabling persistent knowledge accumulation and reuse across evolution.
  2. The paper covers CTR, MTL, MDL, and industrial-scale generative recommendation, while also analyzing performance, search dynamics, component contributions, hardware efficiency, and LLM dependency. The experimental design is comprehensive and thorough.
Critical weaknesses
  1. The proposed framework abstracts recommendation modules as "skills." However, it remains unclear exactly what constitutes a skill that is "atomic, independent, and reusable." Furthermore, in my view, this approach relies primarily on heuristic decomposition and semantic descriptions, lacking formal criteria.
  2. The description of the methodology requires further strengthening, including the management and scheduling of the Tier-2 skill library, as well as the interactions, stochasticity, and evolution among the candidate frameworks.
Questions for authors
  1. What types of model deficiencies are each of the four genetic operations (addition, replacement, crossover, and specialization) within the skill space of EvoSkillRec designed to address? Furthermore, how does the type-checking mechanism employed prior to these operations ensure that the resulting architecture is guaranteed to be valid?
  2. How should the five weights in the retrieval ranking formula for skill-space operations (Eq. 4) be set? Do these weights change dynamically depending on the type of task?
  3. What modifications would be required to extend the skill-genome framework to other recommendation paradigms, such as sequential and graph-based recommendation? Would the framework remain efficient and effective in these settings?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 12pR [1/2]

Thank you for recognizing how type-checked skill edits and open-ended code search progressively build a reusable library and highlighting evaluations across CTR, MTL, MDL, and industrial-scale generative recommendation. We address the weaknesses and questions you raised in turn.

W1. What makes a skill atomic, independent, and reusable?

  • Model decomposition follows the source code. We start with the three stages common to most recommender models: embedding, predictive modeling, and loss computation. Within each stage, we follow the source code to identify functional blocks such as feature interactions, DIN attention, and MLPs.
  • Human review. We use an LLM to help inspect the code and suggest where to split it. We review those suggestions against the source implementation and decide the boundary of each skill.
  • Validation by reconstruction. We reassemble each source model from its extracted skills, following the original data flow. We then check whether the reassembled model reproduces the performance of the original implementation.

Through the three steps above, we confirmed that our skill construction is robust, effective, independent, and reusable.

W2. Tier-2 management and scheduling, candidate interactions, and stochasticity

Thank you for highlighting these points. We have made targeted clarifications in Sections 4.1.3 and 4.3 and added an Amazon worked example in Appendix B.6.

  • Tier-2 management and scheduling. Section 4.1.3 (p. 5, lines 252–262) now specifies that promotion occurs at the end of a generation and that promoted skills become available for retrieval in the next generation. Admission requires validation and portability checks and survival of the host candidate. Tier-2 skills use the same compatibility checks and retrieval ranking as Tier-1 skills.
  • Candidate interactions and evolution. Section 4.3 (p. 7, lines 331–340) now explicitly states that candidates from both spaces compete for the same survivor slots. Survivors become the next parents, while promoted skills and search memory carry discoveries and feedback into subsequent generations. The adaptive budget rule in Section 4.2.3 allocates proposal slots between the two spaces.
  • Stochasticity. Section 4.3 (p. 7, lines 331–340) now identifies LLM generation and model training as sources of stochasticity and refers to the training seeds and LLM settings in Appendices D.1 and D.3.
  • Worked example. Appendix B.6 (pp. 23–24, lines 1239–1271) gives a concrete example that follows a SAR-Net-initialized Amazon search from initialization through proposal generation, validation-based selection, and skill registration to final prediction.

Q1. Model deficiencies addressed by each operator and type-checking guarantees

OperatorDeficiency and exampleStatic preconditions
AdditionMissing capability. Add target-aware attention to supply a new way to model user interests.The required input keys are available directly or through one upstream module, and the fusion layer accepts the output.
ReplacementA component's inductive bias is unsuitable. Swap mean pooling for target-aware attention.Input and output are compatible, and the output key is preserved.
CrossoverComplementary capabilities occur in separate candidates. Combine a DeepFM feature-interaction branch with a DIN interest branch.The branches expose compatible fusion points, and the combined graph remains typed and acyclic.
SpecializationAn existing mechanism needs task-specific adaptation. Widen or narrow a branch.Task or scenario metadata is available, and the edit changes hyperparameters while preserving skill identity and wiring.
AuthorsOpenReview ↗

Reply to Reviewer 12pR [2/2]

Q2. How the five retrieval weights are set

Thank you for the question. We set the retrieval coefficients in Eq. (4) by hand and use the fixed values across task types and evolution rounds. The current task determines which skills are eligible, while the available inputs, parent genome, and diagnosis affect the scores of eligible skills. The same coefficients can therefore produce different rankings across tasks and search states. The following are our rules for setting different weights:

  1. Task compatibility: We exclude task-incompatible skills before ranking. Task compatibility contributes zero points to every eligible skill.
  2. Input and composition compatibility: A feasible input binding contributes 8 points. Matches between the parent’s skills and the candidate’s declared upstream and downstream skills contribute 5 and 3 points, respectively.
  3. Textual relevance: Each diagnosis token that matches the skill-card text contributes 4 points. We check for matches in the skill description, inductive bias, and failure-related metadata.
  4. Inductive-bias alignment: Predefined rules match the inductive bias described in the skill card to structural properties of the current genome. Each rule whose condition is met contributes its fixed weight of 8 or 3 points, and we sum these contributions.
  5. Validation evidence: For a promoted skill with a recorded best validation AUC aia_i, this term contributes clip⁡(2ai,0,4)\operatorname{clip}(2a_i,0,4) points. The term is zero when no such record is available.

Q3. Support for other recommendation paradigms (e.g., sequential and graph-based recommendation)?

Our framework is designed to support diverse recommendation architectures, including sequential and graph-based recommendation.

  • Sequential recommendation: Our framework already supports sequential architectures in CTR prediction, with 19 sequence-related skills included in our skill library (Table 6, Appendix A.3, p. 18, line 943). Moreover, our generative recommendation experiments also cover modern sequential models, using RankMixer as the base model and established sequential architectures such as HyFormer and LONGER as baselines (Appendix C.3, p. 27, lines 1408–1450).
  • Graph recommendation: We have conducted additional experiments on graph-based recommendation, further validating the generalizability and effectiveness of our framework across different recommendation architectures. We extended EvoSkillRec to MovieLens-100K, using six initial graph models (LightGCN, NGCF, GCN, GraphSAGE, SGC, and APPNP) and 11 executable skills. We conducted 15 evolution rounds, evaluating 90 evolved candidates (60 skill-space and 30 code-space). All models were trained from scratch for up to 20 epochs with validation-based early stopping.

Table 1. Evolution statistics.

ComponentCount
Initial models / skills6 / 11
Evolution rounds15
Skill-space / code-space candidates60 / 30
Newly generated and validated skills30
Promoted skills15
Candidates reusing generated skills43
Final searchable library26 (11 initial + 15 promoted)
Total registered skills41

Code-space evolution used GPT-6-astra for architecture planning and code synthesis (70 LLM calls, 35 attempts, 5 validation failures). Promoted skills were available for subsequent reuse.

Table 2. Performance.

MetricNGCF (Best Initial)EvoSkillRecGain
Val. NDCG@200.060720.06772+11.53%
Test NDCG@200.044780.04718+5.36%
Test Recall@200.120470.12793+6.19%

The best evolved model (round 14) combines four LLM-generated propagation modules with concatenation-based aggregation. Selection used validation data exclusively; testing was performed only after evolution.

These results demonstrate EvoSkillRec's generalizability to graph recommendation and its ability to generate, promote, and reuse novel architectural modules beyond a fixed skill library. Our code has also been updated at https://anonymous.4open.science/r/EvoSkillRec-BF55 in GraphEvo.

Thank you very much for your responses to my questions. Through your replies, I have gained a better understanding of the motivation and significance of this work. I should also mention that I am not an expert in this field; however, your responses have effectively addressed my concerns. Consequently, I am considering increasing my score to 3.

Dear Reviewer 12pR,

Thank you very much for your thoughtful and constructive feedback. We sincerely appreciate your careful attention to the methodological details and your suggestions on extending the evaluation to broader recommendation paradigms and benchmarks.

On a personal note, I believe that integrating recommendation research with modern LLMs and agentic workflows is an exciting and important direction, extending beyond the RecSys community. Your insightful comments have further inspired me to think more deeply about this direction in my future research.

Thank you again for this valuable academic exchange and your encouragement.

Best regards,

Authors from Submission 19488

Reviewer_QsSQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper address the laborious process of architecture design in recommender system by proposing a framework searching for the architecture, in which modules are searched to form a computation graph, and each module are either from crystalizing existing works or through LLM powered synthesizing, coupled with a LLM powered code search for new implementations. Both parts are jointly evolved with LLM. Experiments demonstrate the performance and effectiveness of the proposed method.

Critical strengths

On Informativeness

The general idea of applying graph based architecture search and moduel based evolution with the help of LLM is helpful for building better recommender system.

Conceptually, model to skill compiler is an important component as this crystallize proven prior works. The dual search of crystallized method (skills) and related code is an empirically sounding method.

The study of effects of LLM backbone also provide insights into these methods.

Critical weaknesses

On Soundness

Model to skill compiler should be an important component as this crystallize proven prior works. However, even as Appendix A shows the skill cards, it is not clear how exactly they are done, and how sounding such crystallization are. and sounding they are.

It's hard to assess what the evolved solution (pipeline) look like, even with looking at the appendix. This makes assessing this work a hard job.

On Presentation Figure 1 is hard to parse. The writing for methods unnecessarily focus on abstraction that are seemly less informative for understanding (formalization for formalization's seek) or involving implementation details at coding level that are better placed appendix. Meanwhile important details regarding how the proposed evolution works are missing. This makes the understanding of this work a substantial effort.

On Informativeness:

The novelty is limited. The proposed method can be viewed as NAS for recommender system with some modules forming a computation graph. Each module is either crystalized by LLM from prior works or evolved by LLM, also coupled with code search. However, at least judging with the soundness issues mentioned above, it does not provide much novelty from prior works.

For the proposed method, it's unclear how significant the performance gain is. It's not clear what a simpler alternative or a simple extension of LLM powered search based works (e.g. OpenEvolve) with sufficient runs would archive similar performance. Similarly, the performance gains are marginal, and are just single run point estimate as the author admits. Thus it is hard to assess the marginal gains compared to prior works, especially LLM powered methods. As such, I'm not sure wether the proposed complex setup is justified.

Maybe what limit assessing the novelty most is the limited coverage of generative recommenders, which represents recent developments for recommendation system, as building blocks. Through out paper, Kim et al., 2025 is is mentioned only in the intro and OneRec (Zhou et al., 2025) is only mentioned for RQ4. A thorough study of such most recent advances is crucial to show the novelty and the informativeness of the proposed method.

Questions for authors

None

8 comments
AuthorsOpenReview ↗

Reply to Reviewer QsSQ [1/2]

Thank you for recognizing the potential of LLM-guided architecture search for recommendation. We appreciate your assessment of the compiler's reuse of established components, combined skill and code search, and LLM backbone comparisons. We address each identified weakness below.

W1. Robustness of the model-to-skill compiler

  1. Model decomposition follows the source code. We identify three stages common to most recommender models: embedding, predictive modeling, and loss computation, then extract functional blocks such as feature interactions, DIN attention, and MLPs from each stage's source code.
  2. Human review. We use an LLM to help inspect the code and suggest where to split it. We review those suggestions against the source implementation and decide the boundary of each skill.
  3. Validation by reconstruction. We reassemble each source model from its extracted skills, following the original data flow. We then check whether the reassembled model reproduces the performance of the original implementation.

Through the three steps, we confirmed that our skill construction is robust, effective, independent, and reusable.

W2. What do the evolved pipelines look like?

Following your advice, we added diagrams comparing original and evolved architectures in different tasks.

BenchmarkChanges in the evolved pipelineRevised figure
CTR–MLDeepFM evolves into a model with 20 branch outputs feeding a feature-conditioned factorization fusion module and two AutoInt--SENET bridges tapping intermediate features from both paths.Fig. 7, p. 40, lines 2157–2159
CTR–BeautyThe evolved DeepFM adds contextual field reparameterization to selected branches. A branch-token Transformer fuses 29 branch outputs, including seven CrossNet/AFM pairs.Fig. 8, p. 41, lines 2173–2183
CTR–BooksFour CrossNet/AFM pairs, low-rank cross-attention, and kernel-memory branches augment the linear, FM, and MLP paths. Branch--field coattention fuses 18 branch outputs.Fig. 9, p. 41, lines 2197–2211
MTL–CensusShared-Bottom's evolved pipeline has 12 field and 6 vector transformations. Covariance shrinkage and bilinear interactions refine fields; AITM transfers information between the two task towers for income and marital status.Fig. 10, p. 42, lines 2237–2238
MDL–MLThe reconstructed candidate uses an EPNet gate and prediction MLP. It adds two field adapters for domain contrast and token prototypes, plus one vector adapter for domain cross-basis adaptation.Fig. 11, p. 42, lines 2257–2264
MDL–AmazonSeven field adapters and two vector adapters, including a shared/private splitter and sparse delta router, feed a SAR-Net expert mixer.Fig. 12, p. 43, lines 2292–2304
QK-VideoThe reconstructed pipeline follows high MFU architecture design and increase rankmixer layers and experts and Hadamard cross mixer and Low-rank task router.Fig. 13, p. 44, lines 2353–2365

W3. Figure 1 is hard to parse

  • Split overview. Following your suggestion, we separated the original Fig. 1 into 3 subfigures so that each diagram has fewer elements to read at normal zoom.
  • Fig. 1 — Motivation. The figure shows how our methods differ from other methods.
  • Fig. 2 — Recommender modules as skills. The diagram shows how the compiler extracts and validates recommender modules and represents them as skills (p. 4, lines 178–181).
  • Fig. 3 — Dual-space evolution. The diagram shows how both search spaces propose candidates, how survivors seed the next round, and how validated code-space modules return to the library for reuse (p. 5, lines 232–235).

W4. What happens in an actual evolution run?

We thank the reviewer for the advice. We agree that abstract descriptions alone may not clearly explain how our method works. Therefore, we have added a concrete step-by-step example in Appendix B.6 (pp. 23–24, lines 1239–1270), illustrating how the evolutionary process operates on the Amazon dataset, including the specific evolutionary steps and operations.

1. Seed initialization: Construct a SAR-Net-initialized skill genome connecting embeddings, the expert mixer, and prediction components.

2. Dual-space proposal: Generate six skill-space and two code-space candidates, including component edits and domain-aware field normalization.

3. Evaluation and selection: Train valid candidates from scratch and select up to four promising parents (best R0: 0.6786 validation AUC).

4. Skill registration: Promote validated, portable modules to Tier-2 as reusable skills with recorded interfaces and implementations.

5. Iterative evolution: Shift budgets toward code space upon stagnation, with improvements from PLE expansion (R1: 0.6794), M3oE (R9: 0.6902), and SAR-Net (R12: 0.7021).

6. Final output: Return the best genome (R21: shared/domain-private feature splitting; 0.7034 validation AUC, 0.7033 test AUC) and the expanded library after 25 rounds.

AuthorsOpenReview ↗

Reply to Reviewer QsSQ [2/2]

W5. Novelty compared to modular NAS and LLM code search

Our method fundamentally differs from prior approaches by addressing the limitations of both NAS and LLM-based coding:

  • NAS methods can search for architectures but are largely limited to recombining predefined modules.
  • LLM-based coding methods can create new modules but struggle to reuse accumulated knowledge from recommender architectures. Without an explicit evolutionary mechanism, they also lack support for stable, continuous evolution and knowledge reuse across rounds.
  • Our method bridges these two approaches by representing recommender architectures as reusable skills and introducing a dual-space evolutionary mechanism. This enables both knowledge reuse and sustained multi-round evolution, overcoming the limitations of both paradigms (see the added Fig. 1, p. 2, lines 63–66).

W6. Runs with multiple random seeds and significance

Following your suggestion, we repeated the complete runs for both EvoSkillRec and OpenEvolve on each of the six benchmarks using three seeds.

Individual test AUC (OpenEvolve / EvoSkillRec)

BenchmarkSeed 12022Seed 13407Seed 15678
CTR–MovieLens0.75213 / 0.752230.72486 / 0.728670.73842 / 0.77584
CTR–Beauty0.67604 / 0.684560.64287 / 0.658920.66053 / 0.71018
CTR–Books0.60876 / 0.670870.66542 / 0.625740.64139 / 0.71593
MTL–Census0.97226 / 0.974830.96083 / 0.966970.96537 / 0.98276
MDL–Amazon0.69032 / 0.703340.66328 / 0.669880.65634 / 0.73682
MDL–MovieLens0.77305 / 0.793270.74947 / 0.772630.76672 / 0.81386

Mean ± sample SD and 95% Welch confidence intervals for the mean difference (EvoSkillRec − OpenEvolve)

BenchmarkOpenEvolveEvoSkillRecΔAUC [95% CI]
CTR–MovieLens0.73847 ± 0.013640.75225 ± 0.02359+0.01378 [−0.03453, +0.06209]
CTR–Beauty0.65981 ± 0.016600.68455 ± 0.02563+0.02474 [−0.02761, +0.07709]
CTR–Books0.63852 ± 0.028440.67085 ± 0.04510+0.03232 [−0.05977, +0.12442]
MTL–Census0.96615 ± 0.005760.97485 ± 0.00790+0.00870 [−0.00756, +0.02496]
MDL–Amazon0.66998 ± 0.017950.70335 ± 0.03347+0.03337 [−0.03562, +0.10235]
MDL–MovieLens0.76308 ± 0.012200.79325 ± 0.02062+0.03017 [−0.01200, +0.07235]
  • Across all six benchmarks, EvoSkillRec consistently achieves substantial mean improvements (up to +0.03337 AUC), including a +0.01378 average gain on CTR-ML over three full-search runs.
  • EvoSkillRec outperforms OpenEvolve in 17 of 18 paired runs, demonstrating strong robustness against the stochasticity of LLM generation and model training.
  • EvoSkillRec reliably guides evolutionary search toward higher-performing architectures across tasks, confirming the practical significance of skill-guided evolution.

W7. Inclusion of generative recommender models

We would like to clarify that existing generative recommendation methods can be broadly divided into generative retrieval (e.g., OneRec and TIGER) and generative ranking (e.g., HSTU and RankMixer).

  • Generative ranking: We have already conducted extensive experiments on generative ranking in RQ4, focusing on multi-objective optimization involving MFU, a key problem for current generative recommendation. To make this clearer, we have revised the RQ4 title from “Multi-Objective Evolution” to “Generative Ranking via Multi-Objective Evolution” (see p. 9, line 459).
  • Generative retrieval: Our current framework does not support generative retrieval because it typically involves two separate stages: tokenization to generate semantic IDs (SIDs) and Transformer-based prediction. In contrast, our framework relies on an end-to-end evolutionary process, where outcome feedback guides evaluation, diagnosis, and architectural evolution. This makes it difficult to directly extend our current approach to a two-stage pipeline. We will explore this extension in future work.
AuthorsOpenReview ↗

Follow-up on our response to your EvoSkillRec review

Dear Reviewer QsSQ,

Thank you for your detailed and constructive review. Our two replies respond point by point to your comments on the compiler, the evolved architectures, and the presentation of the search procedure. They also discuss the contribution relative to prior work, the comparison with OpenEvolve across three seeds, and the scope of the generative recommendation evaluation.

If your schedule permits, we would appreciate your feedback on whether our responses address the issues you raised. We would welcome any further questions in the discussion thread and would be glad to elaborate wherever helpful.

Best regards,

Authors from Submission 19488

Reviewer_pHnrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoSkillRec, a framework for Recommender NAS through skill-genome evolution. The authors identify limitations in conventional NAS, where search space is rather limited and predefined, as well as LLM-based code evolution methods, where unconstrained code edits often produce invalid or low-value architectures and successful modifications are not systematically preserved for reuse. To address these problems, the authors propose to decompose existing recommendation architectures into reusable computational modules called skills organized in a structured skill library. They perform NAS in two complementary spaces, skill space, where new architectures are composed from existing skills using genetic operators, and code space, where new architectural components are evolved with LLMs. Successful mutants will be promoted to skill library for future generation reuse. The authors evaluate EvoSkillRec on six recommendation benchmarks covering CTR prediction, multi-task learning, and multi-domain learning. They have also conducted extensive ablation studies on the contributions of the two search spaces and skill reuse, analyze the evolution trajectories, different LLM backbone comparisons, and demonstrate accuracy-efficiency co-optimization on a large-scale recommendation dataset. The results outperform manually designed and NAS baselines.

Critical strengths

1. Combines existing architectural knowledge with free architecture exploration

I appreciate the motivation and design behind this work, which attempts to address the limitations of both conventional NAS and LLM-based code evolution. In particular, I think the idea of combining existing architectural knowledge with the ability to invent new modules is interesting and practically meaningful. Their framework provides a relatively systematic approach to extracting reusable components from existing recommendation architectures, bookkeeping them in a structured library, and using them to guide subsequent NAS. Another point I like about this work is that they consider the compatibility and validation of the evolved modules, this effectively deduplicates the search space while retaining expressiveness. Brought together, they provide an interesting approach for NAS, which lies in the middle of classical NAS and LLM-based code generation.

2. Extensive experimental evaluation on different RecSys benchmarks and search mechanisms

The authors conducted a relatively comprehensive experimental evaluation of the proposed framework, which I think is one of the stronger aspects of this paper. This includes experiments across six recommendation benchmarks covering CTR prediction, multi-task learning, and multi-domain learning, and compare against both manually designed and NAS baselines. The evolution trajectory analysis and candidate statistics provide additional interpretation of how the two search mechanisms contribute throughout the evolution.

Critical weaknesses

1. The novelty of the skill-genome representation is not clear.

My main concern is that the proposed skill-genome representation seems closely related to existing modular NAS methods. Representing neural networks as computational graphs of reusable modules is not new, and the proposed genetic operators are also already introduced. I understand that the main contribution may lie in combining these components with LLM-based module generation and reuse. However, I am not fully convinced that the explicit skill-genome representation is necessary. I would appreciate a clearer discussion/justification of the novelty of this method proposed.

2. The skill promotion mechanism may not reliably identify useful skills.

As described in Appendix B.5, a newly generated skill is promoted when its host candidate survives selection. However, this does not necessarily mean that the skill itself contributes positively to the model's performance. A skill could be promoted simply because it happens to appear in a successful architecture or because of the combination of a few different skills appearning in the network together. Since accumulating useful architectural knowledge is a central motivation of this work, I think it would be helpful to evaluate the marginal contributions of promoted skills, rather than relying solely on the performance of their host architectures. Looking into pair-wise skill contributions will also be a good interpretation experiment to show.

3. The statistical reliability of the experimental results is unclear.

The experiments appear to be conducted with a single random seed, and no confidence intervals or standard deviations are reported. Some of the improvements are also quite small. For example, EvoSkillRec improves over OpenEvolve by only 1e-4 AUC on CTR-MovieLens. Given the stochastic nature of LLM-based evolutionary search, I think repeated experiments would be important to establish the reliability of these improvements. I acknowledge that modern RecSys have relatively sensitive benchmarks, and 1e-4 improvement on certain benchmarks might be substantial, but evaluations, especially NAS search on those architectures, still experience noise on the same level of magnitude. Having repeated experiments and confidence intervals will be necessary for verifying whether this proposed method is of practical significance.

4. Minor comments on presentation.

The text in Figure 1 is too small and unreadable at normal zoom levels. I would suggest increasing the font size or simplifying the figure (or stratifying the abstraction levels in different figures) to improve readability.

Questions for authors

I do not have any additional questions at this point. My main concerns are summarized in the Critical Weaknesses section, and I would appreciate the authors’ responses to those points. I am open to raising my score if the authors can adequately address these concerns during the rebuttal.

9 comments
AuthorsOpenReview ↗

Reply to Reviewer pHnr [1/2]

Thank you for recognizing EvoSkillRec's combination of reusable recommender modules and open-ended architecture exploration, and highlighting evaluations across six benchmarks and analyses of both search spaces. We address each identified weakness below.

W1. Novelty compared to modular NAS methods

  • Open-ended search. Existing modular NAS methods operate within a fixed library of predefined modules, limiting search to their recombination. In contrast, EvoSkillRec enables open-ended architectural evolution, where an LLM autonomously discovers and implements new modules to address diagnosed model weaknesses, rather than merely rearranging existing components (see the added Fig. 1, p. 2, lines 54–66).
  • Why a skill genome? Our novelty lies in our use of skill in an architectural sense. Enabling LLMs to understand and evolve modular architectures is challenging, as they inherently lack knowledge of model inductive biases and how to guide architectural evolution. Therefore, an explicit mechanism is needed. We introduce models as skills, bridging the gap between LLM reasoning and architectural understanding. Furthermore, we adopt a genome-based mechanism to guide the evolution process effectively. These mechanisms are absent from existing methods and constitute a key novelty of EvoSkillRec.

W2. Skill promotion mechanism discussion

Following your advice, we conducted two more experiments:

Exp 1: Measure the marginal validation AUC of each of the 186 promoted skills

We tested them in two contexts.

  • Evolved host. We subtracted the AUC after removing the skill from the AUC of the intact host.
  • Compatible original model. We subtracted the original model’s AUC from its AUC after inserting the skill.
  • Mean across contexts. We gave the two differences equal weight. The four examples below show positive contributions in both contexts.
SkillOriginal modelHost ΔAUCOriginal ΔAUCMean ΔAUC
AutoInt–SENet bridgeDeepFM+0.0006+0.0004+0.0005
Contextual field reparameterizerDeepFM+0.0036+0.0024+0.0030
Field covariance shrinkageShared-Bottom+0.0004+0.0008+0.0006
Flat domain moment normalizerSAR-Net+0.0040+0.0024+0.0032

Results across all promoted skills:

BenchmarkOriginally promotedPositive meanNonpositive mean
CTR–MovieLens433112
CTR–Beauty16124
CTR–Books19145
MTL–Census18135
MDL–Amazon453213
MDL–MovieLens453411
Total18613650

136 of 186 skills (73.1%) achieved a positive mean improvement, suggesting that most generated skills consistently contribute to performance gains, while only a small fraction show negative effects. We therefore conducted Exp 2.

Exp 2: Rerun the evolution using strictly positive mean validation ΔAUC

The rerun evolution results are shown as follows:

BenchmarkOriginal test AUC (our paper)Alternative test AUCΔAUC
CTR–MovieLens0.75220.7519−0.0003
CTR–Beauty0.68450.6838−0.0007
CTR–Books0.67080.6702−0.0006
MTL–Census0.97480.9747−0.0001
MDL–Amazon0.70330.7025−0.0008
MDL–MovieLens0.79320.7931−0.0001

We found that stricter or more complicated promotion mechanisms actually lead to worse performance. We believe this is because they limit exploration diversity, preventing the model from discovering more promising solutions. This further supports the effectiveness of our current evolutionary strategy.

AuthorsOpenReview ↗

Reply to Reviewer pHnr [2/2]

W3. Statistical reliability

Following your suggestion, we repeated the complete runs for both EvoSkillRec and OpenEvolve on each of the six benchmarks using three seeds.

Individual test AUC (OpenEvolve / EvoSkillRec)

BenchmarkSeed 12022Seed 13407Seed 15678
CTR–MovieLens0.75213 / 0.752230.72486 / 0.728670.73842 / 0.77584
CTR–Beauty0.67604 / 0.684560.64287 / 0.658920.66053 / 0.71018
CTR–Books0.60876 / 0.670870.66542 / 0.625740.64139 / 0.71593
MTL–Census0.97226 / 0.974830.96083 / 0.966970.96537 / 0.98276
MDL–Amazon0.69032 / 0.703340.66328 / 0.669880.65634 / 0.73682
MDL–MovieLens0.77305 / 0.793270.74947 / 0.772630.76672 / 0.81386

Mean ± sample SD and 95% Welch confidence intervals for the mean difference (EvoSkillRec − OpenEvolve)

BenchmarkOpenEvolveEvoSkillRecΔAUC [95% CI]
CTR–MovieLens0.73847 ± 0.013640.75225 ± 0.02359+0.01378 [−0.03453, +0.06209]
CTR–Beauty0.65981 ± 0.016600.68455 ± 0.02563+0.02474 [−0.02761, +0.07709]
CTR–Books0.63852 ± 0.028440.67085 ± 0.04510+0.03232 [−0.05977, +0.12442]
MTL–Census0.96615 ± 0.005760.97485 ± 0.00790+0.00870 [−0.00756, +0.02496]
MDL–Amazon0.66998 ± 0.017950.70335 ± 0.03347+0.03337 [−0.03562, +0.10235]
MDL–MovieLens0.76308 ± 0.012200.79325 ± 0.02062+0.03017 [−0.01200, +0.07235]
  • Across all six benchmarks, EvoSkillRec consistently achieves substantial mean improvements (up to +0.03337 AUC), including a +0.01378 average gain on CTR-ML over three full-search runs.
  • EvoSkillRec outperforms OpenEvolve in 17 of 18 paired runs, demonstrating strong robustness against the stochasticity of LLM generation and model training.
  • EvoSkillRec reliably guides evolutionary search toward higher-performing architectures across tasks, confirming the practical significance of skill-guided evolution.

W4. Readability of the overview

  • Split overview. Following your suggestion, we separated the original Fig. 1 so that each diagram has fewer elements to read at normal zoom.
  • Fig. 1 — Motivation. The figure shows how our methods differ from other methods.
  • Fig. 2 — Recommender modules as skills. The diagram shows how the compiler extracts and validates recommender modules and represents them as skills (p. 4, lines 178–181).
  • Fig. 3 — Dual-space evolution. The diagram shows how both search spaces propose candidates, how survivors seed the next round, and how validated code-space modules return to the library for reuse (p. 5, lines 232–235).
AuthorsOpenReview ↗

Follow-up on our response to your EvoSkillRec review

Dear Reviewer pHnr,

Thank you for taking the time to review our submission. We have replied to each point in your review across two comments, covering the role of the skill genome, the experiments on skill promotion, the full searches repeated across three random seeds, and the revised overview figures.

If you have a chance to read our replies, we would be grateful to hear whether they address the concerns you raised. We would be happy to continue the discussion and clarify any points that remain open.

Best regards,

Authors from Submission 19488

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We thank all reviewers for their careful reading and constructive feedback. Their suggestions helped us make the method easier to follow and guided our additional experiments.

Manuscript references are to v2.

  • Motivation Explained · pHnr W1, QsSQ W5
    The new motivation figure shows how code-space inventions expand the skill library.
    Sec. 1, Fig. 1 (p. 2, lines 54–66)

  • Enhanced Visualization · pHnr W4, QsSQ W3
    Following your suggestions, we split the overview into separate diagrams for compilation and evolution.
    Figs. 2–3 (pp. 4–5)

  • Compiler Explanation · QsSQ W1, 12pR W1
    We clarified how the compiler extracts typed modules and reconstructs the source model.
    Sec. 4.1.2 (p. 5, lines 243–250) and App. A.2 (p. 17)

  • Explanation of Tier-2 Scheduling and Selection · 12pR W2
    Promoted skills become available in the next round, and candidates from both spaces share survivor slots.
    Secs. 4.1.3 and 4.3 (pp. 5 and 7)

  • Added Amazon Evolution Walkthrough · QsSQ W4, 12pR W2
    The Amazon example follows a SAR-Net seed through candidate selection and skill promotion.
    App. B.6 (pp. 23–24, lines 1239–1270)

  • Added Evolved Architecture Diagrams · QsSQ W2
    The diagrams compare initial and evolved architectures across six benchmarks and QK-Video.
    Apps. E.2.1 and E.4, Figs. 7–13 (pp. 40–44)

  • Skill Reuse Analysis · pHnr W1, QsSQ W5
    We traced reuse from round 3 to round 6 on CTR–MovieLens and measured its gains by ablation.
    Fig. 4 and Table 2 (pp. 8–9)

  • Skill Promotion Experiments · pHnr W2
    Of 186 promoted skills, 136 had a positive mean effect on validation AUC across two contexts.
    Reply to pHnr (W2)

  • Experiments Across Three Random Seeds · pHnr W3, QsSQ W6
    Mean test AUC favored EvoSkillRec on all six benchmarks, with every Welch 95% confidence interval including zero.
    Replies to pHnr (W3) and QsSQ (W6)

  • Inclusion of GenRec and GraphRec · QsSQ W7, 12pR Q3
    EvoSkillRec evolved RankMixer on QK-Video using skills from 15 ranking architectures and raised MovieLens-100K test NDCG@20 from NGCF’s 0.04478 to 0.04718.
    GenRec in RQ4 and Table 4 (pp. 9–10) and App. C.3 (p. 27) · GraphRec in the reply to 12pR (Q3)

We would be glad to answer any further questions about the revision or the additional experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.