- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EvoSkillRec (Skill-Genome Evolution for Recommender Architecture Discovery), which explores how to leverage large language models (LLMs) to automatically discover architectures for recommender systems, while simultaneously distilling effective modules generated during the search process into reusable knowledge. This framework effectively addresses several key challenges: the low efficiency of traditional manual architecture design, the restricted search spaces in existing AutoML approaches, and the lack of knowledge reuse in LLM-driven code evolution.
Critical strengths
- EvoSkillRec formulates recommender self-evolution as cumulative skill-genome growth, decomposing models into validated, reusable architectural units and coupling type-checked skill-space mutation with open-ended code-space invention. Successful innovations are continuously distilled back into the skill library, enabling persistent knowledge accumulation and reuse across evolution.
- The paper covers CTR, MTL, MDL, and industrial-scale generative recommendation, while also analyzing performance, search dynamics, component contributions, hardware efficiency, and LLM dependency. The experimental design is comprehensive and thorough.
Critical weaknesses
- The proposed framework abstracts recommendation modules as "skills." However, it remains unclear exactly what constitutes a skill that is "atomic, independent, and reusable." Furthermore, in my view, this approach relies primarily on heuristic decomposition and semantic descriptions, lacking formal criteria.
- The description of the methodology requires further strengthening, including the management and scheduling of the Tier-2 skill library, as well as the interactions, stochasticity, and evolution among the candidate frameworks.
Questions for authors
- What types of model deficiencies are each of the four genetic operations (addition, replacement, crossover, and specialization) within the skill space of EvoSkillRec designed to address? Furthermore, how does the type-checking mechanism employed prior to these operations ensure that the resulting architecture is guaranteed to be valid?
- How should the five weights in the retrieval ranking formula for skill-space operations (Eq. 4) be set? Do these weights change dynamically depending on the type of task?
- What modifications would be required to extend the skill-genome framework to other recommendation paradigms, such as sequential and graph-based recommendation? Would the framework remain efficient and effective in these settings?
4 comments
Reply to Reviewer 12pR [2/2]
Q2. How the five retrieval weights are set
Thank you for the question. We set the retrieval coefficients in Eq. (4) by hand and use the fixed values across task types and evolution rounds. The current task determines which skills are eligible, while the available inputs, parent genome, and diagnosis affect the scores of eligible skills. The same coefficients can therefore produce different rankings across tasks and search states. The following are our rules for setting different weights:
- Task compatibility: We exclude task-incompatible skills before ranking. Task compatibility contributes zero points to every eligible skill.
- Input and composition compatibility: A feasible input binding contributes 8 points. Matches between the parent’s skills and the candidate’s declared upstream and downstream skills contribute 5 and 3 points, respectively.
- Textual relevance: Each diagnosis token that matches the skill-card text contributes 4 points. We check for matches in the skill description, inductive bias, and failure-related metadata.
- Inductive-bias alignment: Predefined rules match the inductive bias described in the skill card to structural properties of the current genome. Each rule whose condition is met contributes its fixed weight of 8 or 3 points, and we sum these contributions.
- Validation evidence: For a promoted skill with a recorded best validation AUC , this term contributes points. The term is zero when no such record is available.
Q3. Support for other recommendation paradigms (e.g., sequential and graph-based recommendation)?
Our framework is designed to support diverse recommendation architectures, including sequential and graph-based recommendation.
- Sequential recommendation: Our framework already supports sequential architectures in CTR prediction, with 19 sequence-related skills included in our skill library (Table 6, Appendix A.3, p. 18, line 943). Moreover, our generative recommendation experiments also cover modern sequential models, using RankMixer as the base model and established sequential architectures such as HyFormer and LONGER as baselines (Appendix C.3, p. 27, lines 1408–1450).
- Graph recommendation: We have conducted additional experiments on graph-based recommendation, further validating the generalizability and effectiveness of our framework across different recommendation architectures. We extended EvoSkillRec to MovieLens-100K, using six initial graph models (LightGCN, NGCF, GCN, GraphSAGE, SGC, and APPNP) and 11 executable skills. We conducted 15 evolution rounds, evaluating 90 evolved candidates (60 skill-space and 30 code-space). All models were trained from scratch for up to 20 epochs with validation-based early stopping.
Table 1. Evolution statistics.
| Component | Count |
|---|---|
| Initial models / skills | 6 / 11 |
| Evolution rounds | 15 |
| Skill-space / code-space candidates | 60 / 30 |
| Newly generated and validated skills | 30 |
| Promoted skills | 15 |
| Candidates reusing generated skills | 43 |
| Final searchable library | 26 (11 initial + 15 promoted) |
| Total registered skills | 41 |
Code-space evolution used GPT-6-astra for architecture planning and code synthesis (70 LLM calls, 35 attempts, 5 validation failures). Promoted skills were available for subsequent reuse.
Table 2. Performance.
| Metric | NGCF (Best Initial) | EvoSkillRec | Gain |
|---|---|---|---|
| Val. NDCG@20 | 0.06072 | 0.06772 | +11.53% |
| Test NDCG@20 | 0.04478 | 0.04718 | +5.36% |
| Test Recall@20 | 0.12047 | 0.12793 | +6.19% |
The best evolved model (round 14) combines four LLM-generated propagation modules with concatenation-based aggregation. Selection used validation data exclusively; testing was performed only after evolution.
These results demonstrate EvoSkillRec's generalizability to graph recommendation and its ability to generate, promote, and reuse novel architectural modules beyond a fixed skill library. Our code has also been updated at https://anonymous.4open.science/r/EvoSkillRec-BF55 in GraphEvo.
Thank you very much for your responses to my questions. Through your replies, I have gained a better understanding of the motivation and significance of this work. I should also mention that I am not an expert in this field; however, your responses have effectively addressed my concerns. Consequently, I am considering increasing my score to 3.
Dear Reviewer 12pR,
Thank you very much for your thoughtful and constructive feedback. We sincerely appreciate your careful attention to the methodological details and your suggestions on extending the evaluation to broader recommendation paradigms and benchmarks.
On a personal note, I believe that integrating recommendation research with modern LLMs and agentic workflows is an exciting and important direction, extending beyond the RecSys community. Your insightful comments have further inspired me to think more deeply about this direction in my future research.
Thank you again for this valuable academic exchange and your encouragement.
Best regards,
Authors from Submission 19488
Reply to Reviewer 12pR [1/2]
Thank you for recognizing how type-checked skill edits and open-ended code search progressively build a reusable library and highlighting evaluations across CTR, MTL, MDL, and industrial-scale generative recommendation. We address the weaknesses and questions you raised in turn.
W1. What makes a skill atomic, independent, and reusable?
Through the three steps above, we confirmed that our skill construction is robust, effective, independent, and reusable.
W2. Tier-2 management and scheduling, candidate interactions, and stochasticity
Thank you for highlighting these points. We have made targeted clarifications in Sections 4.1.3 and 4.3 and added an Amazon worked example in Appendix B.6.
Q1. Model deficiencies addressed by each operator and type-checking guarantees