- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
An evolutionary approach to improving RL algorithms using a 2-phased, age-layered, LLM-based mutations.
Critical strengths
The phased approach is practical and can be used in other use-cases beyond RL improvements. The insights library and interpretability of the algorithms are useful side effect of the approach. A meta analysis of the types of changes that result in improvements, might even lead to better general purpose RL algorithms.
Critical weaknesses
None
Questions for authors
The role of the LLM and the choice of LLMs in the algorithm is not addressed. What LLM was used in the experiments. What is the effect of the choice of LLM on the process? How were the LLMs prompted?
Some minor suggestions: Line 65: Add citation for UCB Line 68: Consider Age-layering in evolutionary algorithms as related prior work (e.g., https://nn.cs.utexas.edu/?shahrzad:gecco16). Line 142: How are hyper parameters such as K=5 decided? Line 146: Doesn't remaining below a threshold imply the component requires more mutations? Line 152: Define Insight Library here, or in the intro paragraph for this section to avoid a forward reference Line 159: In stochastic systems, the insight library might end up being noisy. How do you deal with that?
3 comments
Response (2/2): Citations, freezing rule and noise control in the insight library
Response 4 (Minor 1, 2, 3, 5): Citations, related work and definitions.
Conclusion: all four have been addressed in the revision.
| Comment | Change |
|---|---|
| UCB (l.65) | citation added at first mention (Auer et al., 2002) |
| Age-layering (l.68) | Related Work now discusses ALPS (Hornby, 2006) and Shahrzad et al. (2016); PPE is related in spirit, but layers candidates by training budget (50K → 200K → 1M steps) rather than by genotypic age |
| K = 5 (l.142) | K is the number of slots in the component interface, not a tuned value; standard model-free algorithms (DQN, Rainbow, DDPG/TD3, A2C/PPO, SAC) fill exactly these slots (Section 3.1, Appendix B) |
| Insight library (l.152) | now defined at the start of Section 3.2, before its first use |
Location (blue in the revised PDF). UCB: original p.2, l.65 → revised p.2, l.65. Age-layering: Related Work, original p.3, l.109 → revised p.3, l.112; new references Hornby (2006) and Shahrzad et al. (2016) are shown in blue in the References (p.11, p.13). K = 5: original p.3, l.118 and l.142 → revised Section 3.1, p.3, l.123. Insight library: original forward reference p.3, l.152 (definition l.159) → revised definition p.3, l.140, reference p.4, l.187.
Response 5 (Minor 4): Does a below-threshold gain mean a component needs more mutations rather than freezing?
Conclusion: freezing does not remove a component from the search; it moves the component's budget to Phase 2, where the component stays fully mutable together with the others.
A component is frozen after two consecutive batches with < 2% relative gain, i.e., once single-component edits in an otherwise fixed program stop paying off; the remaining potential then usually lies in interactions, which only joint edits can exploit. Phase 1 is bounded to 50–120 of the 240 candidates. Section 3.2 now explains this.
Location (blue in the revised PDF). Original: p.3, l.146. Revised: p.3, l.157.
Response 6 (Minor 6): How is noise in the insight library controlled in the stochastic setting?
Conclusion: an entry reaches the prompt only after aggregation over many programs and confirmation at higher fidelity, so a single noisy run cannot create one.
| Mechanism | How it limits noise |
|---|---|
| Aggregation | an entry summarizes all programs in a batch that carry a mechanism (mean fitness change, count, direction consistency), including failures |
| L1 confirmation | validated only after at least two wins at L1 (200K steps, 2 seeds); refuted after two consecutive L1 losses and then shown as a direction to avoid |
| Bounded injection | only the top-8 entries by evidence enter the prompt; near-duplicates are merged |
Section 3.2 now describes this lifecycle.
Location (blue in the revised PDF). Original: p.3, l.159. Revised: p.3, l.142.
The revision adds the requested citations, definitions, prompt details and noise-control lifecycle, and keeps the LLM's proposal role separate from objective evaluation and selection.
Thank you
Dear Reviewer Lw8A,
Thank you for reading our responses and for raising your score. Your questions on the role and choice of the LLM, the prompts and the insight library made the paper clearer and more reproducible, and all of the corresponding changes are in the revised PDF.
Best regards, The Authors
Response (1/2): Role, choice and prompting of the LLM
We sincerely thank the reviewer for the positive assessment, for the concrete suggestions (including the pointer to age-layered evolution), and for recognizing the practical phased design, the usefulness of the insight library, and the interpretability of the evolved algorithms. We answer the questions in order; all changes are marked in blue in the revised PDF.
Response 1 (Q1): What role does the LLM play?
Conclusion: the LLM plays three roles, all on the proposal side: it writes candidate code, reviews each evaluated candidate, and curates the insight library. It never assigns scores; every number in the paper comes from actual RL training and evaluation.
Selection and promotion through the L0/L1/L2 funnel use only RL returns. Section 3.2 and Appendix V now describe this.
Location (blue in the revised PDF). Original: Section 3.2 (p.3). Revised: Section 3.2, new paragraph (p.3, l.138); new Appendix V, "Role" (p.34).
Response 2 (Q2): Which LLM was used, and how does its choice affect the process?
Conclusion: all programs were generated by gemini-3.1-pro-preview; with gpt-5.5 the final fitness is lower, but the framework still clearly improves the baseline and rediscovers the same key mechanism.
We re-ran the full method twice on dm-cheetah-run with gpt-5.5 in all three roles, with the same seed program, candidate budget, evaluation protocol and search seeds as the gemini runs (baseline L2 fitness 0.302):
Because the LLM only proposes, its choice affects search efficiency but cannot inflate reported performance. The model is named in the AI-use statement and Appendix D, and the swap experiment is in Appendix V.
Location (blue in the revised PDF). Original: AI use statement (p.10, l.492). Revised: AI use statement (p.10, l.536); Appendix D, "Evolution budget" (p.15, l.805); new Appendix V, "Model" and "Choice of LLM" (p.34, l.1818).
Response 3 (Q2): How is the LLM prompted?
Conclusion: each mutation request uses a fixed two-part template, and the full templates are now in Appendix V (Listings 2–3) to support reproducibility.
.backward()inside a component); how candidates are evaluated; accumulated insights (Response 6)The templates will also be released with the code.
Location (blue in the revised PDF). Revised: new Appendix V, "Prompts" and Listings 2–3 (p.34–35).