acceptodds
Under review as a conference paper at ICLR 2027

RLDISCOVER: LLM-DRIVEN CO-EVOLUTION OF REINFORCEMENT LEARNING ALGORITHMS

Abstract

LLM-guided program evolution has enabled discoveries in mathematics and computational optimization, raising the prospect of reinforcement learning (RL) algorithms that self-evolve to improve how agents learn. However, realizing this prospect faces two obstacles. Joint search over coupled algorithmic components is difficult to scale: simultaneous changes can disrupt learning, while isolated changes overlook their dependencies. Evaluating candidate algorithms also requires costly training, with fitness remaining uncertain across random seeds. We introduce RLDiscover, a framework for the self-evolution of model-free deep RL algorithms. Progressive Co-Evolution advances from targeted component edits to joint evolution, while Progressive Probabilistic Evaluation balances search breadth and evaluation fidelity through staged training and repeated evaluation.Experiments across SAC, PPO, and DQN on four benchmark suites show substantial improvements in mean return, with per-family median gains of 32%–84% and a peak return ratio of approximately 363× over a near-zero baseline. These gains include transitions from failed learning to successful task completion, and improvements persist when evolution starts from stronger open-source implementations. On measured SAC locomotion runs, evaluation uses approximately onefifteenth the estimated compute required to fully evaluate the same candidate pool.Remarkably, independent searches repeatedly discover interpretable combinations of adaptive robust losses, progress-dependent value targets, and running statistics,with selected programs transferring to unseen tasks. These findings point toward a broader role for self-evolution in AI: discovering interpretable algorithms that improve how agents learn.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, LLM-driven code evolution, automated algorithm design, evolutionary search

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Lw8AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

An evolutionary approach to improving RL algorithms using a 2-phased, age-layered, LLM-based mutations.

Critical strengths

The phased approach is practical and can be used in other use-cases beyond RL improvements. The insights library and interpretability of the algorithms are useful side effect of the approach. A meta analysis of the types of changes that result in improvements, might even lead to better general purpose RL algorithms.

Critical weaknesses

None

Questions for authors

The role of the LLM and the choice of LLMs in the algorithm is not addressed. What LLM was used in the experiments. What is the effect of the choice of LLM on the process? How were the LLMs prompted?

Some minor suggestions: Line 65: Add citation for UCB Line 68: Consider Age-layering in evolutionary algorithms as related prior work (e.g., https://nn.cs.utexas.edu/?shahrzad:gecco16). Line 142: How are hyper parameters such as K=5 decided? Line 146: Doesn't remaining below a threshold imply the component requires more mutations? Line 152: Define Insight Library here, or in the intro paragraph for this section to avoid a forward reference Line 159: In stochastic systems, the insight library might end up being noisy. How do you deal with that?

3 comments
AuthorsOpenReview ↗

Response (1/2): Role, choice and prompting of the LLM

We sincerely thank the reviewer for the positive assessment, for the concrete suggestions (including the pointer to age-layered evolution), and for recognizing the practical phased design, the usefulness of the insight library, and the interpretability of the evolved algorithms. We answer the questions in order; all changes are marked in blue in the revised PDF.

Response 1 (Q1): What role does the LLM play?

Conclusion: the LLM plays three roles, all on the proposal side: it writes candidate code, reviews each evaluated candidate, and curates the insight library. It never assigns scores; every number in the paper comes from actual RL training and evaluation.

RoleWhat the LLM does
Mutation operatorwrites candidate component code
Code reviewerafter each evaluation, writes a short structured review (changes relative to the parent, likely errors, key features) that later prompts include
Insight curatormaintains the insight library (Response 6)

Selection and promotion through the L0/L1/L2 funnel use only RL returns. Section 3.2 and Appendix V now describe this.

Location (blue in the revised PDF). Original: Section 3.2 (p.3). Revised: Section 3.2, new paragraph (p.3, l.138); new Appendix V, "Role" (p.34).

Response 2 (Q2): Which LLM was used, and how does its choice affect the process?

Conclusion: all programs were generated by gemini-3.1-pro-preview; with gpt-5.5 the final fitness is lower, but the framework still clearly improves the baseline and rediscovers the same key mechanism.

We re-ran the full method twice on dm-cheetah-run with gpt-5.5 in all three roles, with the same seed program, candidate budget, evaluation protocol and search seeds as the gemini runs (baseline L2 fitness 0.302):

LLMRun 1Run 2Gain over baselineCritic loss discovered
gemini-3.1-pro-preview0.6620.712+119% / +136%adaptive robust loss (as in 16/17 SAC champions)
gpt-5.50.4950.611+64% / +102%pseudo-Huber/log-cosh mixture; clipped-MSE/robust mixture

Because the LLM only proposes, its choice affects search efficiency but cannot inflate reported performance. The model is named in the AI-use statement and Appendix D, and the swap experiment is in Appendix V.

Location (blue in the revised PDF). Original: AI use statement (p.10, l.492). Revised: AI use statement (p.10, l.536); Appendix D, "Evolution budget" (p.15, l.805); new Appendix V, "Model" and "Choice of LLM" (p.34, l.1818).

Response 3 (Q2): How is the LLM prompted?

Conclusion: each mutation request uses a fixed two-part template, and the full templates are now in Appendix V (Listings 2–3) to support reproducibility.

PartContent
System prompttask and environment description; specification of the evolvable function(s) (signature, inputs and outputs, constraints, default implementation; Phase 1 edits only the selected component, Phase 2 any subset); common coding mistakes that crash training (e.g., calling .backward() inside a component); how candidates are evaluated; accumulated insights (Response 6)
User promptevolution history (parent and inspiration programs with code and scores), followed by the mutation instruction

The templates will also be released with the code.

Location (blue in the revised PDF). Revised: new Appendix V, "Prompts" and Listings 2–3 (p.34–35).

AuthorsOpenReview ↗

Response (2/2): Citations, freezing rule and noise control in the insight library

Response 4 (Minor 1, 2, 3, 5): Citations, related work and definitions.

Conclusion: all four have been addressed in the revision.

CommentChange
UCB (l.65)citation added at first mention (Auer et al., 2002)
Age-layering (l.68)Related Work now discusses ALPS (Hornby, 2006) and Shahrzad et al. (2016); PPE is related in spirit, but layers candidates by training budget (50K → 200K → 1M steps) rather than by genotypic age
K = 5 (l.142)K is the number of slots in the component interface, not a tuned value; standard model-free algorithms (DQN, Rainbow, DDPG/TD3, A2C/PPO, SAC) fill exactly these slots (Section 3.1, Appendix B)
Insight library (l.152)now defined at the start of Section 3.2, before its first use

Location (blue in the revised PDF). UCB: original p.2, l.65 → revised p.2, l.65. Age-layering: Related Work, original p.3, l.109 → revised p.3, l.112; new references Hornby (2006) and Shahrzad et al. (2016) are shown in blue in the References (p.11, p.13). K = 5: original p.3, l.118 and l.142 → revised Section 3.1, p.3, l.123. Insight library: original forward reference p.3, l.152 (definition l.159) → revised definition p.3, l.140, reference p.4, l.187.

Response 5 (Minor 4): Does a below-threshold gain mean a component needs more mutations rather than freezing?

Conclusion: freezing does not remove a component from the search; it moves the component's budget to Phase 2, where the component stays fully mutable together with the others.

A component is frozen after two consecutive batches with < 2% relative gain, i.e., once single-component edits in an otherwise fixed program stop paying off; the remaining potential then usually lies in interactions, which only joint edits can exploit. Phase 1 is bounded to 50–120 of the 240 candidates. Section 3.2 now explains this.

Location (blue in the revised PDF). Original: p.3, l.146. Revised: p.3, l.157.

Response 6 (Minor 6): How is noise in the insight library controlled in the stochastic setting?

Conclusion: an entry reaches the prompt only after aggregation over many programs and confirmation at higher fidelity, so a single noisy run cannot create one.

MechanismHow it limits noise
Aggregationan entry summarizes all programs in a batch that carry a mechanism (mean fitness change, count, direction consistency), including failures
L1 confirmationvalidated only after at least two wins at L1 (200K steps, 2 seeds); refuted after two consecutive L1 losses and then shown as a direction to avoid
Bounded injectiononly the top-8 entries by evidence enter the prompt; near-duplicates are merged

Section 3.2 now describes this lifecycle.

Location (blue in the revised PDF). Original: p.3, l.159. Revised: p.3, l.142.

The revision adds the requested citations, definitions, prompt details and noise-control lifecycle, and keeps the LLM's proposal role separate from objective evaluation and selection.

Dear Reviewer Lw8A,

Thank you for reading our responses and for raising your score. Your questions on the role and choice of the LLM, the prompts and the insight library made the paper clearer and more reproducible, and all of the corresponding changes are in the revised PDF.

Best regards, The Authors

Reviewer_PSrpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RLDiscover, an LLM-driven framework for modifying reinforcement learning algorithms through executable code evolution. Progressive Co-Evolution first searches individual algorithmic components and then allows joint modifications, while Progressive Probabilistic Evaluation allocates training resources through three evaluation levels.

Critical strengths

The framework is evaluated across several algorithm families and substantially different control domains, making the study more informative than a demonstration on a single algorithm or benchmark. The stronger SAC initializations, independent re-evaluations of selected programs, and audits of executable code are useful checks on whether the gains reflect improvements to the starting implementation. The transfer results are also encouraging.

Critical weaknesses
  1. The framework is evaluated across several algorithm families and substantially different control domains, making the study more informative than a demonstration on a single algorithm or benchmark. The stronger SAC initializations, independent re-evaluations of selected programs, and audits of executable code are useful checks on whether the gains reflect improvements to the starting implementation.

  2. The experiments do not isolate the contribution of PPE. The single-fidelity variants also remove component decomposition and Phase 1, while L2-only search outperforms the full method on both PPO tasks.

  3. The claim of discovering reusable algorithmic principles exceeds the current evidence. Evolved programs combine task-specific observation processing and reward shaping with changes to learning rules, without isolating their contributions. Recurring modifications and transfer results from four selected programs are suggestive, but do not establish the proposed explanations concerning robust losses, Q-blending, or generalization.

I would be willing to raise my rating if the authors adequately address these concerns.

Questions for authors
  1. Under the same LLM proposal process and evaluation budget, how does PPE compare with successive halving or Hyperband when component decomposition and Phase 1 are kept unchanged?
6 comments
AuthorsOpenReview ↗

Response (1/3): Controlled comparison of PPE with successive halving

We sincerely thank the reviewer for the constructive review, for recognizing the breadth of our evaluation across algorithm families and control domains and the value of the stronger SAC initializations, re-evaluations, code audits and transfer results, and especially for the precise experimental request in Q1, which we have run. All changes are marked in blue in the revised PDF.

Response 1 (W1): The duplicated paragraph. The first paragraph under Critical Weaknesses repeats the Critical Strengths paragraph, so we read it as a strength and focus on the two substantive concerns below.

Response 2 (W2, Q1): Does PPE itself beat successive halving under the same LLM proposals and budget, with decomposition and Phase 1 unchanged?

Conclusion: it does; with everything else identical, PPE beats successive halving on 4 of 6 paired runs, ties on 1, and is better on average (+0.025 normalized fitness).

Only the L0→L1 promotion rule differs: successive halving promotes the top ⌈n/η⌉ (η = 5) of each L0 batch by L0 fitness. Seed program, decomposition, Phase 1, Phase 2, insight library, LLM, candidate budget, L1 quota and search seeds are identical. The comparison is now in Section 5 and Appendix S. Final L2 fitness of each run's champion:

TaskSearch seedPPE (full method)Successive halving
dm-cheetah-run420.6620.662
dm-cheetah-run1230.7120.701
dm-cheetah-run20260.6810.652
gym-HalfCheetah-v5420.8030.730
gym-HalfCheetah-v51230.7090.802
gym-HalfCheetah-v520260.7760.646

We chose successive halving over Hyperband because it maps directly onto PPE's fixed fidelity levels; one evolution run takes about four days of GPU time.

Location (blue in the revised PDF). Original: Section 5 (p.8, l.412 to p.9). Revised: Section 5, new paragraph "Isolating PPE" (p.10, l.499); first contribution (p.2, l.73–76); new Appendix S (Table 20, p.32); added references Jamieson & Talwalkar (2016), Li et al. (2018, 2020) in blue in the References (p.11–12).

AuthorsOpenReview ↗

Response (2/3): Component isolation separates learning rules from task-specific changes

Response 3 (W3): Do the gains come from reusable learning rules or from task-specific observation processing and reward shaping?

Conclusion: component isolation separates the two; in locomotion the gain comes from a learning rule (the evolved critic loss alone recovers 85–162% of it), while in Meta-World it comes from task-specific observation features and reward shaping.

We evaluated four whole-function splices of 8 own-seed SAC champions at L2 on fresh seeds (Appendix T): LR-only (champion with observation encoding and reward shaping reverted), Shape-only (seed plus the champion's observation encoding and reward shaping), Critic-only (seed plus the champion's critic loss), Target-only (seed plus progressive Q-blending). Share of the full gain retained:

TaskLR-onlyShape-onlyCritic-onlyTarget-only
dm-cheetah-run112%−44%119%9%
dm-walker-run288%−77%162%−30%
gym-HalfCheetah-v577%9%89%−5%
gym-Walker2d-v571%16%85%−2%
mw-push-v325%96%15%41%
mw-peg-insert-side-v3−75%126%−14%19%
md-100envs19%36%−26%−70%
Tasks improved over baseline6 / 75 / 75 / 73 / 7
Mean gain retained73.8%23.1%61.2%−5.5%

The critic loss is the mechanism found in 16/17 SAC and 5/5 PPO champions. Progressive Q-blending helps only in combination with other components. Section 4.3 now says so.

Location (blue in the revised PDF). Original: Section 4.3, "Robust critic loss" and "Interpreting the discovered mechanisms" (p.7, l.328 to p.8, l.390); Appendix I, "Q-annealing" (p.19). Revised: Section 4.3 (p.8, l.413–423); Appendix I; new Appendix T (Table 21, p.33).

AuthorsOpenReview ↗

Response (3/3): Mechanism-level transfer of the evolved critic losses

Response 4 (W3): Does transfer of four selected programs show that the discovered mechanisms are reusable?

Conclusion: tested directly, the mechanism transfers; transplanted alone into the unmodified baseline, the evolved critic losses improve 31 of 36 task pairs (27 of the 32 on unseen tasks) and outperform the standard Huber loss on average.

(a) Whole-program transfer (Figure 2b, Appendix H): three of four champions improve 6–7 of 8 unseen tasks; the walker2d program, which relies on hard-coded step thresholds, improves 2 of 8. (b) Mechanism-level transfer (new, Appendix U): only the critic loss of four champions is transplanted into the seed program and evaluated on all nine locomotion tasks on fresh seeds, with standard Huber as a control. Change relative to the baseline:

Target taskStandard Huberfrom walker-walkfrom walker-runfrom Humanoidfrom Walker2d
dm-cheetah-run+93.7%+81.2%+85.6%+92.8%+25.9%
dm-walker-walk+23.2%+23.3%*+22.4%+9.9%+14.3%
dm-walker-run+28.9%+36.1%+26.5%*+46.7%+20.9%
dm-quadruped-walk+3.1%+2.0%+1.7%−55.7%+3.4%
gym-HalfCheetah-v5+106.5%+89.2%+139.8%+111.7%+104.8%
gym-Hopper-v5−25.8%+15.8%−29.0%+58.0%+86.1%
gym-Walker2d-v5+22.2%+44.7%+53.9%+30.5%+42.8%*
gym-Humanoid-v5+2.6%+14.6%−14.2%+11.1%*+27.2%
gym-Ant-v5+5.8%−11.9%+32.9%+36.3%−36.6%
Tasks improved8 / 98 / 97 / 98 / 98 / 9
Mean gain+28.9%+32.8%+35.5%+37.9%+32.1%

(* source task.) The advantage over standard Huber is largest where Huber struggles (Hopper, Humanoid). We also renamed "Principles for generalizable programs" to "Observations on transfer".

Location (blue in the revised PDF). Original: whole-program transfer in Figure 2b (p.7, l.339) and Appendix H (p.18, l.948); Section 4.3, "Principles for generalizable programs" (p.8, l.401). Revised: Section 4.3 (p.8, l.413–419); "Observations on transfer" (p.9, l.435); abstract (p.1, l.28–29); new Appendix U, which describes the mechanism-level transfer and its relation to Appendix H (p.33, l.1758; Table 22, p.34).

These controls isolate the contribution of PPE and show that the evolved critic loss is a reusable mechanism. We are happy to provide any further details.

Reviewer_sFQaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a framework named RLDiscover, a framework for the self-evolution of model-free DRL algorithms. Progressive Co-Evolution advances from targeted component edits to joint evolution, while Progressive Probabilistic Evaluation balances search breadth and evaluation fidelity through staged training and repeated evaluation. The experiments show its claimed improvements across algorithms and benchmarks.

Critical strengths

This work advances automated RL algorithm evolution by searching for executable learning rules with a limited evaluation.

Critical weaknesses
  1. With n=3 seeds and one evolution run per cell, noise-level gains on several tasks, selection seeds reused at re-evaluation, a ×363 built on a near-zero baseline solved on 1–2 of 3 seeds, and a 15× measured against a strawman its own compute-matched L2-only control refutes, the headline claims exceed their evidence.
  2. The small gains over strong implementations, scope for implicit hyperparameter tuning, and screening bias toward early-training speedups leave it unclear whether the method discovers reusable algorithms or simply performs expensive per-task tuning.
Questions for authors
  1. Appendix D covers re-evaluations for the DMControl and Gym locomotion cells, but not Ant, Meta-World, or MetaDrive. Are the mw-push and mw-pick-place entries independent re-evaluations or pipeline selection scores?
  2. How are the task-specific normalization bounds chosen, and were they fixed before search?
5 comments
AuthorsOpenReview ↗

Response (1/3): Fresh-seed re-evaluation and 27 independent evolution runs

We sincerely thank the reviewer for the careful and detailed review, for the precise questions on seeds, selection and normalization, and for recognizing that RLDiscover advances automated RL algorithm evolution by searching for executable learning rules under limited evaluation. These comments led us to run several new experiments. They confirm the central results and are summarized below; all changes are marked in blue in the revised PDF.

Response 1 (W1): Are the re-evaluations independent of selection, given that they reused the selection seeds {0, 42, 123}?

Conclusion: re-trained on fresh seeds never used for selection, the main results hold, with 16/17 own-init SAC, 4/5 PPO and 4/4 DQN tasks still improving.

We re-trained every champion and its baseline in all 35 rows of Table 1 on fresh seeds {1, 2, 3} (10 seeds for MountainCar and six bimodal cells), with the same hardware and protocol:

FamilyRows improved (fresh seeds)Median gain (fresh seeds)Median gain (original table)
SAC, own seed16 / 17+42.2%+39.8%
SAC, open-source seed5 / 9+0.5%+2.6%
PPO4 / 5+18.5%+83.6%
DQN4 / 4+23.5%+32.1%

Several SAC gains are even larger on fresh seeds (HalfCheetah +120%, drawer-open +116%, Hopper +59%). The fresh-seed numbers are now the primary result (Table 1b), and Appendix Q lists all 35 rows with per-seed returns.

Location (blue in the revised PDF). Original: Table 1 (p.6) and Appendix D, "Provenance of the reported numbers" (p.15, l.780). Revised: abstract (p.1, l.21–25); second contribution (p.2, l.77–81); results paragraph (p.5, l.254–255) and new Table 1b (p.7, l.324); Table 1 footer (p.6, l.322); Appendix D provenance (p.16, l.843); new Appendix Q with all 35 rows and per-seed returns (Tables 17–18, p.30–31).

Response 2 (W1): With three seeds and one search per cell, could the gains be evaluation noise or a lucky search?

Conclusion: the gains are neither evaluation noise nor search luck; they hold on additional fresh seeds, and all 27 independent repetitions of the full method improve on their baseline.

Evaluation noise: with the fresh seeds of Response 1, each cell now has 6 seeds (10 for bimodal cells). Search variance: our ablation study contains three independent evolution runs for each of 9 task–algorithm pairs (normalized fitness; Appendix R):

AlgorithmTaskBaselineRun 1Run 2Run 3
SACdm-cheetah-run0.3020.7120.6810.662
SACdm-walker-walk0.6800.9690.9620.970
SACdm-walker-run0.4880.7350.5150.726
SACgym-HalfCheetah-v50.3400.7090.7760.803
SACgym-Ant-v50.4670.7660.6660.506
PPOgym-Hopper-v50.4800.8800.8160.851
PPOgym-HalfCheetah-v50.1150.2750.1700.158
DQNLunarLander-v30.2230.7430.7290.784
DQNMountainCar-v00.8360.9030.9160.906

Three seeds or runs is the common protocol in LLM-driven program evolution (ShinkaEvolve, Lange et al., 2025; EvoX, Liu et al., 2026).

Location (blue in the revised PDF). Original: Appendix A, Limitations (p.12). Revised: results paragraph (p.5, l.254–255); Limitations, fourth item (p.13, l.693); Appendix Q, "Number of seeds" (p.29, l.1549); new Appendix R with the 27 runs (Table 19, p.32).

AuthorsOpenReview ↗

Response (2/3): Revised headline claims; gains are a learning rule, not tuning

Response 3 (W1): Do the ×363 and 15× claims overstate the results, given the near-zero baseline and the strawman comparison?

Conclusion: we have rewritten both claims in the abstract and contributions; the underlying results remain strong, as the mw-push gain holds on every fresh seed and, at matched compute, the funnel matches or beats full-fidelity search on SAC.

ClaimRevised statement and evidence
×363 (mw-push)Fresh seeds: champion 3173 vs. baseline 28; every seed far above the baseline (4248 / 4763 / 509, the weakest 18× the baseline). The ratio is no longer used in the abstract or contributions.
15× efficiencyCompute-matched: at the same GPU budget the funnel screens ~230 candidates, L2-only search 13, and the funnel leads by 10.7% on the SAC four-task mean.

Location (blue in the revised PDF). Original: abstract (p.1, l.21–27); first contribution (p.2, l.73–76); footnote 1 (p.2); Section 4.3, "Cross-domain generality" (p.8, l.390). Revised: abstract (p.1, l.21–27); first contribution (p.2, l.73–76); footnote 1 (p.2, l.105); Section 4.3, "Cross-domain generality" (p.8, l.423); Appendix Q (p.29–31).

Response 4 (W2): Are the gains reusable algorithmic changes or implicit per-task tuning?

Conclusion: they are a change of learning objective, not tuning; in locomotion, transplanting only the evolved critic loss into the unmodified baseline recovers 85–162% of the full gain, and this loss transfers to unseen tasks.

No hyperparameter is tuned: network size, optimizer, learning rates, batch size and training steps are fixed per algorithm family, are not exposed by the component interface, and are identical for each champion and its baseline (now stated in Appendix D). Gain retained by splices of the champions on fresh seeds (Appendix T):

TaskCritic loss onlyLearning-rule changes only
dm-cheetah-run119%112%
dm-walker-run162%288%
gym-HalfCheetah-v589%77%
gym-Walker2d-v585%71%

Transplanted alone, the evolved critic losses improve the baseline on 31 of 36 task pairs, including 27 of the 32 pairs on tasks they were not evolved on (Appendix U). In Meta-World, evolution instead finds task-relevant observation features and reward shaping. From strong open-source SAC implementations, evolution still finds double-digit gains on fresh seeds (Hopper +20.9%, HalfCheetah +15.3%, Ant +14.4%).

Location (blue in the revised PDF). Original: Section 4.2 (p.5, l.266); Section 4.3 (p.8, l.385); second and third contributions (p.2, l.77–82). Revised: Appendix D, new paragraph "No per-task tuning" (p.16, l.823); Section 4.2 (p.7, l.348); Section 4.3 (p.8, l.413–419); contributions (p.2, l.77–86); new Appendix T (Table 21, p.33).

AuthorsOpenReview ↗

Response (3/3): Selection scores and normalization bounds

Response 5 (Q1): Are the mw-push, mw-pick-place, Ant, Meta-World and MetaDrive numbers independent re-evaluations or selection scores?

Conclusion: they are effectively selection scores, and the fresh-seed results of Response 1 now replace them as the independent evaluation.

They come from separate L2 re-runs of the champions, but with the selection seeds {0, 42, 123}; since training is deterministic given the seed, these re-runs reproduce the selection scores exactly. The same applies to the DMControl and Gym re-evaluations. Appendix D now states this explicitly.

Location (blue in the revised PDF). Original: Appendix D, "Provenance of the reported numbers" (p.15, l.780). Revised: same paragraph (p.16, l.843).

Response 6 (Q2): How were the normalization bounds chosen, and were they fixed before search?

Conclusion: they are fixed per-task constants set before search, and they cannot change which program is selected.

SuiteBound
DMControl1000 (suite maximum)
GymnasiumHalfCheetah 15000, Hopper 4000, Walker2d 6000, Ant 8000, Humanoid 7000
Meta-World4500
MetaDrive350
Classic controlCartPole 500, LunarLander 300; Acrobot −80 and MountainCar −110 (negative rewards, mapped linearly from 6× these targets)

Within a task, fitness is a fixed monotone function of return and selection is a within-task ranking on unclipped fitness, so the bounds do not affect the ranking; all Improvement values use raw returns. Appendix F now includes this.

Location (blue in the revised PDF). Original: Table 6 (p.18, l.918); Appendix F (p.17, l.891). Revised: Appendix F, new paragraph "Normalization bounds" (p.18, l.955).

The new experiments strengthen the evidence for our main claims, and every result is reported in full in the revised paper. We would be glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary

We thank the AC and the reviewers for their hard work. We are pleased that the reviewers have recognized our core contributions and provided constructive feedback that has helped us improve our work. Below, we provide a brief summary of our core contributions, the key concerns addressed during the rebuttal, and the final status of each reviewer.


1. Core Contribution

RLDiscover uses LLMs to automatically evolve executable code for model-free RL algorithms (SAC, PPO, DQN) via Progressive Co-Evolution (PCE) and Progressive Probabilistic Evaluation (PPE).

Across diverse control domains, it delivers substantial, interpretable gains on fresh seeds and independently rediscovers a transferable, error-adaptive robust value loss that outperforms standard Huber loss.


2. Key Concerns Addressed

(All in blue in the revised PDF)

Rigorous Generalization & Stability (sFQa, PSrp): Full re-training on fresh held-out seeds and repeated independent evolution runs confirm that our gains are robust and reproducible across SAC, PPO, DQN, and strong open-source baselines, alongside calibrated headline claims. (Table 1b, Appendices Q–R)

Verified PPE Advantage (PSrp): Under an identical budget and setup, controlled isolation confirms that PPE consistently outperforms representative multi-fidelity baselines (e.g., Successive Halving). (Section 5, Appendix S)

Genuine, Transferable Algorithmic Discovery (sFQa, PSrp): With zero hyperparameter tuning, the ablation study proves that gains come from reusable learning rules rather than per-task tuning: the evolved critic loss alone recovers locomotion gains, recurs across all three algorithm families, transfers reliably to unseen tasks, and outperforms standard Huber loss. (Appendices T–U)


3. Reviewer Status

  • Reviewer Lw8A (Rating: 2 → 3): Acknowledged our rebuttal and raised the score.

  • Reviewer PSrp (Rating: 2): The reviewer indicated a willingness to raise the score if concerns were addressed; our rebuttal resolved the issues initially raised. Subsequently, the reviewer posed a new question regarding the extent to which this finding generalizes as a principle for RL algorithm design. We have provided a detailed response, but there has been no further feedback yet.

  • Reviewer sFQa (Rating: 2): All empirical/statistical concerns were fully resolved with new experiments (Nov 13) and a polite reminder (Nov 16), but the reviewer has not yet responded.


We kindly ask the AC to consider these substantial empirical updates in the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.