acceptodds
Under review as a conference paper at ICLR 2027

R-Split: Population-Level Reasoning over Reward Mechanisms for Automated Reward Design with LLMs

Abstract

Reward functions strongly affect how reinforcement learning (RL) agents learn, but designing them for complex control tasks requires substantial expert effort. Large language models (LLMs) can generate and improve reward code. However, existing methods typically apply LLM reasoning to individual candidates, which limits their use of population-wide information for candidate comparison and search planning. We propose R-Split, which analyzes candidate reward designs, compares them across the population, makes evolution decisions, and generates new reward code. R-Split summarizes each evaluated reward function as a structured set of reward mechanisms: textual descriptions of how its components shape learning, grounded in reward code and training-process data. A similarity filter removes candidates with highly similar mechanism descriptions, reducing repeated exploration. We evaluated our approach against multiple baselines on ten low-level control tasks, using a shared PPO evaluation pipeline and five independent runs for each selected reward. R-Split achieves competitive performance across both locomotion and manipulation tasks. These results show that comparing reward mechanisms across a population can help improve LLM-based reward design.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Reward Design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aZiMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LLM-based automatic reward design for reinforcement learning, focusing on how information across a candidate population can guide the next generation of search. R-Split's Analyzer converts reward code and training-process data into structured mechanism records, separating mechanism descriptions, performance assessments, and numerical implementation details. The Planner compares the current population, selects parents, and specifies refinement, combination, or exploration directions; the Generator implements these directions as reward programs. The system also retains candidates by prioritizing task-score AUC while filtering similar mechanism descriptions. The authors compare against Eureka, REvolve, RF-Agent, R*, and CARD on ten Isaac Gym/Bi-DexHands control tasks and retrain the selected rewards with five policy-training seeds. The results show competitive performance, while ablations on four tasks investigate the Planner, structured representation, mechanism input, and diversity filtering.

Critical strengths
  1. The division of responsibilities among the Analyzer, Planner, and Generator connects high-level design comparison with concrete implementation: the Planner can compare mechanisms and their observed evidence, while the Generator handles formulas and numerical details. This allows the search to use design information from weaker overall candidates instead of refining only the current best candidate.

  2. The experiments use common task environments and PPO configurations, and the selected rewards are subsequently evaluated through five policy-training runs from scratch. R-Split has the highest mean AUC on five of the ten tasks in Table 1.

Critical weaknesses
  1. The paper's central object is a reward-search algorithm, but repeated evaluation mainly measures policy-training randomness conditional on a selected reward rather than variation in the complete search. One complete search is reported per task, while five seeds retrain the same selected reward. The results therefore leave open how reliably the algorithm discovers rewards of similar quality and how reproducible the baseline advantages and ablation differences are.

  2. The ablations support the complete system's design, but important control conditions remain underspecified, limiting attribution of the gains to the planning module and structured mechanism records.

  3. Search selects rewards by AUC over a fixed training horizon, whereas final evaluation uses task-specific horizons. Because accumulated performance and candidate rankings can change with the horizon, alignment between the selection criterion and the final-evaluation objective has not been validated.

Questions for authors
  1. Are independent complete-search repetitions available on representative tasks, rather than only retraining the same selected reward?

  2. Table 8 reports aggregate token usage across ten tasks only for R-Split and Eureka. Could the authors report per-task LLM call counts and token usage for one complete reward search by the other main baselines, particularly RF-Agent and R*, under the reported search settings? The 4.65-fold token ratio relative to Eureka alone is insufficient to assess the performance-cost tradeoff with the main baselines.

  3. In w/o Planner, how are parents selected, and which parent code, Analyzer diagnoses, training curves, and history remain available to the Generator? How are the Planner's refinement or combination instructions replaced, or are rewards generated from scratch in every generation?

  4. In w/o Structure, what text is embedded for diversity filtering? Is the combined description-and-assessment text used directly, or are mechanism descriptions extracted first?

  5. The paper selects rewards by task-score AUC. Have the authors observed candidates with higher AUC but lower final success rates? If such a tradeoff occurs, how is the better reward defined, and why is AUC prioritized for selection?

  6. AUC and candidate rankings depend on the training horizon. Search uses 3,000 PPO epochs, whereas final evaluation uses 500 for Ant, 5,000 for AllegroHand, and 6,000 for Bi-DexHands. Does the reward selected at 3,000 epochs still outperform other high-scoring candidates at the corresponding final-evaluation horizon? Could changes in candidate rankings be checked on representative tasks?

2 comments
AuthorsOpenReview ↗

Response to Reviewer aZiM

Thank you for the time and effort you devoted to reviewing our paper and for your detailed and thought-provoking suggestions. Your attention to search stability, computational cost, and evaluation methodology is helpful for improving the completeness and readability of our experimental report. We have carefully considered these comments and added complete-search repetitions, per-task cost accounting, and relevant control results in the revision. We also indicate the corresponding appendix locations for ease of reference. Revised passages in the manuscript are shown in blue.

The complete-search repetitions in Q1, request counts and tokens in Q2, and the selection-criterion and extended-training experiments in Q5–Q6 are consolidated in the common response: see Section 1 / Appendix G.1, Section 3 / Appendix G.3, Section 2 / Appendix G.2, and Section 4 / Appendix G.4, respectively. Below, we further clarify the ablation implementations in Q3–Q4, the rationale for AUC-based selection in Q5, and the training-horizon settings in Q6.

Q3: Information available to w/o Planner and its generation procedure

The original w/o Planner condition uses top-1 single-parent selection by AUC and generates 16 variants with a fixed optimization instruction. The Generator still receives the parent code, the Analyzer's implementation diagnostics, training curves, and sampled numerical values. Only the best valid candidate in the current generation is analyzed at each generation, and diversity filtering is disabled. Thus, this variant implements single-parent elite optimization, with corresponding changes to the parent scope, Analyzer coverage, and filtering settings. We have added these implementation details to Ablation Studies in the main text and Appendix A.4.

Q4: Input to diversity filtering in w/o Structure

The w/o Structure condition directly feeds the entire text mixing mechanism descriptions and assessments into the embedding model, without re-extracting pure mechanism descriptions. It uses the same embedding model and threshold rules. It changes both the Planner representation and the input to diversity filtering, and therefore tests the behavior of the system after removing the per-mechanism structured representation, rather than merely changing the text format. We have explicitly documented this treatment in Ablation Studies in the main text and Appendix A.4.

Q5: Rationale for using AUC

We believe that a good reward function should possess a degree of robustness and guide policy learning continuously and stably. We therefore select candidates using AUC over the entire training process, considering both learning speed and sustained performance and reducing reliance on a high score at a single point in time. A higher AUC does not necessarily coincide with the highest terminal training score. Our selection reflects an emphasis on performance throughout learning, while we also report maximum task score to present different evaluation dimensions.

The supplementary control results support this choice: for R-Split, selection by AUC produces a lower standard deviation of evaluation AUC across the five seeds on seven of the ten tasks, with ties on two others, compared with selection by maximum task score. Mean AUC is also higher on eight tasks and tied on two. These results show that the rewards selected by AUC exhibit more stable learning performance overall in this multi-seed evaluation, consistent with our design objective of reward robustness.

Q6: Search and final-training horizons

The settings in Table 6 come from the following sources: the 3000 PPO epochs used during search and the task-specific final evaluation horizons follow the defaults in Eureka and the public code; the number of parallel environments (num_envs) for each task follows the settings in the R* paper, allowing 16 candidates to be trained concurrently on two RTX 4090 GPUs. Within each task, all compared methods use the same num_envs and training budget. The configuration sources and rationale have been added to Appendix A.2 (Task-Wise Training Settings).

Complete numerical results and curves for all seven methods and ten tasks have been added to Appendix G.4 of the revised manuscript. With extended training, the relative performance of the methods changes on some tasks, but R-Split remains competitive overall, achieving the highest mean AUC on five of the ten tasks. These results further support the effectiveness of the selected rewards under longer training budgets.

Thank you for your detailed response, which helps me understand this paper better.

Reviewer_5rTDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes R-Split, an LLM-based population method for reward design with three roles. The Analyzer summarizes each evaluated reward as a set of structured reward mechanism records (description, assessment, and implementation detail) from the reward code and training-process data. The Planner reads the AUC and mechanism records of the whole population, selects parents, and writes instructions for the next generation, and the Generator produces new reward code from these instructions. A filter based on text embeddings of the mechanism descriptions removes similar designs. Experiments on ten tasks from Isaac Gym and Bi-DexHands compare R-Split with Eureka, REvolve, RF-Agent, R*, and CARD, with GPT-5-mini and the same PPO evaluation pipeline for all methods. R-Split achieves the highest AUC on five of the ten tasks, and the paper also reports a manual check of the Analyzer outputs, statistics on Planner parent selection, and ablations.

Critical strengths
  1. The problem is relevant. Most LLM-based reward design methods refine candidates one at a time, and using information from the whole population to compare candidates and plan the search is a reasonable direction.
  2. The experimental details are thorough and support reproducibility. The search stage and the final evaluation are separated, the selected rewards are retrained with five seeds, and the appendix provides the full prompts, task settings, token counts, and unedited generated code.
  3. The paper includes several analyses beyond the main results, including a manual check and counterfactual tests of the Analyzer, statistics on Planner parent selection, statistics on the diversity filter, and ablations.
Critical weaknesses
  • The novelty is limited, and the difference from prior work is not made clear. The introduction presents population-level parent selection by an LLM as the main difference from existing methods, but LMEA and OPRO already let the LLM read multiple candidate solutions to select parents or generate offspring, and ReEvo uses LLM reflections that compare parents and accumulate experience to guide evolution, which is close to the roles of the Analyzer and the Planner. The reward reflection in Eureka and the trajectory analysis in Auto MC-Reward already use training-process information to revise rewards, and the selective combination of multiple parents is close to the module-level crossover in R*, except that the LLM rather than a rule decides how to combine them. ReEvo and R* are cited, but these differences are not discussed. What seems new is mainly the mechanism-level record and the split of information between the Planner and the Generator, and the paper should state its contribution relative to these works more clearly.
  • The fairness of the comparison is unclear. CARD evaluates only five rewards under its original protocol, while the other methods evaluate 80. R* and CARD are reimplemented from their papers and REvolve is used without human feedback, and since CARD reaches an AUC close to 0 on AllegroHand and SwingCup, it is hard to tell whether these implementations are faithful. R-Split selects its final reward by AUC, which is also the main metric, while Eureka selects by maximum task score. R-Split also uses 4.65 times the tokens of Eureka, but there is no comparison under equal cost, the cost and wall-clock time of the other baselines are not reported, and all experiments use GPT-5-mini only.
  • The results are not very convincing. Each method runs a single search per task with a fixed seed and LLM temperature 1.0, so the five final seeds only capture PPO variance for the selected reward and not the randomness of the search. R-Split has the best AUC on five of the ten tasks, the gaps to baselines on tasks such as AllegroHand, Pen, and Scissors are around one standard deviation, and no statistical test is reported. The expert rewards also almost fail on DoorCloseOutward, Kettle, and SwingCup (maximum task score 0.01 to 0.05), and the AllegroHand expert has zero standard deviation over five seeds. This is unusual for human-designed dense rewards and suggests that the modified numbers of parallel environments and training horizons in Table 6 change the task difficulty, which makes the results hard to relate to prior work.
  • The analysis and ablations do not support the main claim. There is no ablation that keeps the same Analyzer and Generator and uses rule-based selection such as top-k or tournament selection, and it is not clear what w/o Planner does, so the effect of parent selection by the LLM is not isolated. The ablations cover four tasks with one search each and are mostly shown as curves, and w/o Mechanism Input removes several inputs at once, as the authors note. Table 3 shows that the Planner sometimes selects lower-half parents but not whether this helps, and Figure 4 is a single example. The evidence for the Analyzer is also limited. The manual check covers one task, the counterfactual test uses one call per condition, the similarity of repeated descriptions of the same reward (0.899) overlaps with that of different rewards (0.811), the 0.80 threshold is close to the mean similarity of different rewards, and the drop in similarity in Table 5 follows directly from the filtering rule.
Questions for authors
  1. What is the main novelty compared with methods where the LLM selects parents or reflects on them, such as LMEA, OPRO, and ReEvo, and with the reward reflection in Eureka and the trajectory analysis in Auto MC-Reward? Can the authors compare with top-k or tournament selection using the same Analyzer and Generator? How exactly does w/o Planner produce the next generation?
  2. Why does CARD evaluate only five rewards? Can the comparison be repeated with the same evaluation budget and the same selection criterion for all methods? Can the reimplementations of R* and CARD reproduce the results reported in their original papers on shared tasks?
  3. Can the authors run several independent searches with different seeds for each method, and report the variance of the search outcome and statistical tests?
  4. Why do the expert rewards almost fail on DoorCloseOutward, Kettle, and SwingCup, and why is the AllegroHand expert standard deviation zero? How much do the changed numbers of parallel environments and training horizons affect each method, and do the conclusions hold under the standard settings?
  5. Can the authors report token usage and wall-clock time for all methods, and compare with Eureka and other baselines under the same token budget? Do the conclusions hold with other LLMs?
  6. How was the 0.80 threshold chosen? Since different rewards have a mean similarity of about 0.81, does the filter remove designs that are actually different? Can the manual check of the Analyzer be extended to more tasks with independent annotators?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 5rTD — Part 1/2

We sincerely thank you for your careful review and detailed suggestions regarding the positioning of the method, experimental design, and interpretation of the results. Your comments helped us revisit the key aspects of the paper that require further explanation and provided clear direction for our revision. We greatly value these suggestions and have added experiments and analyses and refined the presentation accordingly, so that the rationale for our design and the experimental results are clearer. Revised passages in the manuscript are shown in blue.

You and the other reviewers have independently highlighted key issues concerning comparison fairness, search stability, and computational cost. These insightful and specific comments have provided valuable guidance for improving our experiments. We consolidate the shared experiments raised in Q2–Q5 in the common response: the common selection criteria in Q2 are covered in Section 2 and Appendix G.2; the search repetitions in Q3 in Section 1 and Appendix G.1; the Expert updates and extended training in Q4 in Section 4, Appendix G.4, and Table 1 and Figure 2 in the main text; and the cost and timing statistics in Q5 in Section 3 and Appendix G.3. Below, we separately clarify the method and ablations, baseline protocols, configuration choices, controls with other LLMs, and threshold settings.

Q1: Contribution positioning and ablation definitions

LMEA, OPRO, EoH, and ReEvo provide important foundations for LLM-based search and reflection. Building on this foundation, our work focuses on reward-mechanism representations and their role in planning across candidates.

Our design focuses on organizing executed reward code and training feedback into mechanism-level records, separating numerical details such as specific weights from the planning representation, and comparing the functions and relationships of mechanisms across candidates. The Generator then combines code and numerical diagnostics to implement the design intent. Relative to Eureka and Auto MC-Reward, the distinction lies in organizing feedback at the mechanism level and using it for planning across rewards; relative to REvolve, it lies in the mechanism representation and the division of information between planning and implementation. R* already separates structure and parameter optimization; our work focuses on mechanism-semantic representations and their role in cross-candidate planning, with our contribution centered on this representation and division of information.

The original w/o Planner condition uses top-1 single-parent selection by AUC and generates 16 variants with a fixed optimization instruction. The Generator still receives the parent code, the Analyzer's implementation diagnostics, training curves, and sampled numerical values. Only the best valid candidate in the current generation is analyzed at each generation, and diversity filtering is disabled. Thus, this variant implements single-parent elite optimization, with corresponding changes to the parent scope, Analyzer coverage, and filtering settings. We have added these implementation details to Ablation Studies in the main text and Appendix A.4.

We recognize the value of your suggested control that retains the same input information and replaces only the selector with tournament/top-k selection, which would help further distinguish the effects of the individual design choices. We apologize that, due to the computational budget and response time available in this round, we have not completed this strict control and can currently provide only the existing ablation results and the implementation explanation above. We sincerely apologize for being unable to provide sufficient supplementary experiments in this round to address this concern fully.

The parent-selection statistics and mechanism examples illustrate planning behavior, while performance is compared through evaluation and ablation results.

The w/o Structure condition directly feeds the entire text mixing mechanism descriptions and assessments into the embedding model, without re-extracting pure mechanism descriptions. It uses the same embedding model and threshold rules. It changes both the Planner representation and the input to diversity filtering, and therefore tests the behavior of the system after removing the per-mechanism structured representation, rather than merely changing the text format. We have explicitly documented this treatment in Ablation Studies in the main text and Appendix A.4.

AuthorsOpenReview ↗

Response to Reviewer 5rTD — Part 2/2

CARD's budget and reproduction settings in Q2

We construct the baselines according to the algorithmic procedures described in their original papers and the available public implementations, and evaluate them within our common training pipeline. Specifically, R* and CARD are implemented from their published descriptions; CARD retains five sequential rounds of optimizing a single reward, whereas population-based methods typically evaluate 5×16 candidates. For REvolve, we use the version without human feedback provided in RF-Agent's public code repository; this implementation source is described in Appendix D (Baseline Implementation Details). We understand your concern about comparison fairness. We apologize that, given the computational budget and time available for this rebuttal, we prioritized the supplementary experiments described above and cannot currently provide strictly budget-matched comparisons with sufficient independent repetitions.

Expert rewards and configuration choices in Q4

The settings in Table 6 come from the following sources: the 3000 PPO epochs used during search and the task-specific final evaluation horizons follow the defaults in Eureka and the public code; the number of parallel environments (num_envs) for each task follows the settings in the R* paper, allowing 16 candidates to be trained concurrently on two RTX 4090 GPUs. Within each task, all compared methods use the same num_envs and training budget. The configuration sources and rationale have been added to Appendix A.2 (Task-Wise Training Settings).

Expert directly uses the original human-designed rewards in the public code; we have made no modifications to the reward code. We understand that your concern is the effect of changing num_envs on the experimental comparison. This setting is applied uniformly to all methods within each task. Expert and automatically generated rewards use the same num_envs, PPO configuration, number of training epochs, and evaluation seeds, thereby maintaining an equal training budget for final policy evaluation. Our claim concerns the relative performance of the reward-design methods under this common setting and equal policy-training budget; the uniform adjustment of num_envs does not change this basis of comparison. We have also included Expert in the extended-training evaluation with the same budget, with results in Appendix G.4.

Equal-token budgets and controls with other LLMs in Q5

We apologize that, given the time and computational budget available for this rebuttal, we have not completed equal-token-budget comparisons. The current supplementary experiments use the same LLM, and we have not provided comparison results using other LLMs.

Q6: Similarity threshold and mechanism analysis

The value 0.80 is an empirical initial threshold, based on preliminary embedding computations in which semantically similar modules generally had similarities above 0.8; it is not a strict classification boundary for mechanism equivalence. The actual filter uses the mechanism-description text of rewards and relaxes the threshold by 0.02 when too few candidates remain. The mean final threshold over the 30 filtering events recorded in the paper is 0.85. Text-similarity distributions can overlap, and filtering is based on similarity between textual representations. The coverage of the existing manual audit and the number of counterfactual repetitions are limited. We have not added cross-task independent annotations, threshold-sensitivity experiments, or repeated counterfactual experiments; the existing analyses illustrate the mechanism representation and filtering behavior.

Reviewer_YC7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes R-Split, an LLM-driven evolutionary framework for automated reward design in RL. It has three LLM roles. An Analyzer turns each evaluated reward program, together with its training-process data, into a structured record of "reward mechanisms" (description, assessment and implementation detail; Eq. 1–2). A Planner sees the AUC, mechanism descriptions/assessments and decision feedback for the whole population (up to 2N individuals) and jointly emits N evolution decisions, each with 0–2 parents and an instruction (Eq. 3). A Generator writes the new reward code (Eq. 4). An elite-preserving diversity filter ranks candidates by AUC and discards those whose concatenated mechanism descriptions exceed a cosine-similarity threshold (initial value 0.80, relaxed as needed). Search uses the task-score AUC as the fitness signal.

The method is evaluated on ten Isaac Gym / Bi-DexHands tasks (AllegroHand, Ant and eight ShadowHand bimanual tasks) against Eureka, feedback-free REvolve, RF-Agent, reimplemented R* and CARD, and the Eureka expert rewards. All methods use GPT-5-mini and a budget of 5 iterations × 16 candidates. Each method's selected reward is retrained with five seeds (Table 1, Figure 2). On the AUC metric R-Split has the best mean on 5 of 10 tasks (AllegroHand, BottleCap, Kettle, Pen, Scissors) and is close to the best on several others. Supporting analyses include:

  • a manual audit of Analyzer descriptions on one task (Table 2);
  • small counterfactual grounding checks (Appendix C) and a repeatability study (Appendix B);
  • parent-rank statistics for the Planner (Table 3) and a case study (Figure 4);
  • ablations on four tasks (Figure 5, Table 4) and similarity statistics before and after filtering (Table 5).

R-Split uses about 4.65× Eureka's LLM tokens (Table 8). The paper describes its own results as "competitive". The evidence supports this, but not consistent superiority.

Critical strengths
  1. The evaluation protocol is careful and transparent in several respects. All methods share task environments, PPO configurations, the LLM backbone (GPT-5-mini) and a nominal 80-candidate budget. Search-stage and final-evaluation training are separated, with selected rewards retrained from scratch on seeds 0–4 (Appendix A.1–A.3, Tables 6–7). Baselines are described in detail, including where reimplementations depart from the originals (Appendix D). Exact LLM token counts are reported, and they are unfavorable to the proposed method (Table 8). Full prompts and unedited generated reward programs are included (Appendices E–F). This makes the method largely reproducible from the text.

  2. The paper goes beyond headline scores and probes its intermediate representation and the Planner's behavior:

    • a manual audit of 289 mechanism entries (Table 2);
    • a controlled 2×2 grounding intervention whose limitations are reported honestly (Appendix C notes that the Analyzer wrongly attributes a learning acceleration to bonus activation);
    • a repeatability study with same-task and cross-task controls (Table 9);
    • parent-rank statistics showing that the Planner is not a greedy elite selector (Table 3).

    The authors are generally careful not to over-interpret these analyses (e.g., Section 5.3 and Appendix B.2 state what the checks do not show).

  3. The empirical results are reasonably strong on harder dexterous tasks. On AllegroHand, ShadowHandKettle, ShadowHandPen and ShadowHandBottleCap, R-Split has clearly higher mean AUC and maximum score than Eureka, with lower variance (Table 1, Figure 2). The ablations (Figure 5, Table 4) suggest that both the Planner step and the Planner-facing mechanism input contribute, most visibly on AllegroHand.

Critical weaknesses
  1. Search-level variance is not measured, so the main comparison rests on one search run per method and task. Reward search is run once per method and task with a fixed training seed (42), and LLM sampling uses temperature 1.0 (Section 5.1, Appendix A.4). The five seeds in Table 1 measure only the variance of final-policy training for one already-selected reward, not the variance of the search procedure, which is the object being compared. With six stochastic LLM-driven search methods, a single search outcome per cell can easily reverse rankings. This is especially concerning given:

    • the very large spreads in some cells (e.g., AllegroHand RF-Agent 29.15±17.17; DoorClose R-Split 3.75±1.41);
    • the near-ties (DoorClose, Scissors, Ant, Over).

    The ablations in Figure 5 and Table 4 have the same issue. Repeated searches, with aggregate statistics such as normalized scores, rank distributions or confidence intervals across tasks, would be needed to support claims about the method rather than about one search outcome.

  2. The comparison is not fully matched in selection criterion or compute.

    • Selection criterion. R-Split selects the reward with the largest search-stage AUC, while Eureka selects by maximum task score (Appendix D). AUC is then one of the two reported metrics, which structurally favors R-Split in Table 1(a).
    • Compute. R-Split uses 4.65× Eureka's tokens (Table 8). Token counts for REvolve, RF-Agent, R* and CARD are not reported. No baseline is given a matched LLM budget, for example more refinement samples or an equal-token Eureka or REvolve.
    • Iteration structure. CARD follows a sequential five-iteration protocol, apparently evaluating far fewer candidates than the 80 used by the others. This makes its near-zero results on AllegroHand and SwingCup hard to interpret.
    • Fidelity to original settings. R* and CARD are paper-based reimplementations. Table 6 uses reduced parallel-environment counts on several tasks (e.g., 128 envs on Kettle and Scissors) and a 3,000-epoch search horizon versus a 500-epoch final horizon on Ant. The expert rewards score near zero on DoorClose, Kettle and SwingCup, and far below all automated methods on AllegroHand (Table 1). This suggests the shared pipeline departs substantially from the original benchmark settings. That undermines comparability with published results and may affect methods differently.
  3. The evidence that population-level mechanism reasoning, the paper's central claim, is what drives the gains is limited.

    • Ablation scope. Ablations cover four tasks with a single search each and are reported mainly as curves (Figure 5). Only one ablation has numbers (Table 4). Several results are mixed: w/o Structure matches or exceeds the full system on Kettle and Over, and w/o Div. Filter is comparable on Over and SwingCup.
    • Underspecified ablation. w/o Planner is not specified: the paper does not say how parents and instructions are chosen when there is no planning step (Section 5.4, Appendix A.4).
    • Missing comparisons. Two natural controls are absent: the same Analyzer/Generator pipeline with a conventional selector (e.g., fitness-proportional or tournament parent selection over mechanism records), and a Planner that sees per-candidate records but plans slots independently. Without them it is unclear whether the gains come from joint population reasoning or simply from richer per-candidate feedback (training-process data and code-grounded analysis), which prior work such as CARD and Eureka's reward reflection already uses in other forms.
    • Descriptive analyses. Table 3 and Figure 4 describe what the Planner does but do not show that its choices are better than alternatives. Table 5 is largely tautological: filtering by similarity necessarily lowers similarity, and it does not show that filtering improves search.
  4. The novelty over existing LLM-based evolutionary program search is incremental. Jointly evolving natural-language descriptions with code (EoH), reflection-guided evolution (ReEvo), population-based LLM reward evolution (REvolve) and structure-level reward evolution (R*) are all close relatives (Section 2). The new elements are a structured per-mechanism record, a population-wide Planner prompt and an embedding-based diversity filter, and they are mostly prompt-engineering choices. Given the modest and partly unreliable empirical margins (Weaknesses 1–2), the added insight for the community is limited.

Minor: Algorithm 1 and Eq. 4 index the generated code inconsistently (c^t_k vs c^{t+1}_k), and the order of Select and the Planner call in Algorithm 1 is hard to follow. The prompts hard-code "consecutive_successes" as the objective (Listings 3 and 7), although the logged score differs across tasks (Table 7). The paper also does not say whether code will be released.

Questions for authors
  1. Can you rerun the full search (not only the final policy training) with several independent search seeds for R-Split and at least the strongest baselines (Eureka, RF-Agent, REvolve), and report across-search variability and an aggregate cross-task statistic? Whether the Table 1 rankings survive this is the main factor in my assessment.
  2. How do the results change when all methods select their final reward by the same criterion (e.g., search-stage AUC), and when a baseline such as Eureka or REvolve is given an LLM token budget matched to R-Split's 18.07M tokens? Please also report token usage for REvolve, RF-Agent, R* and CARD.
  3. How exactly does w/o Planner generate the next population (parent selection, instructions, number of parents)? Have you compared the full system with a variant that keeps the Analyzer records but replaces the Planner with a standard selection operator? This would isolate the contribution of population-level reasoning.
  4. Why do the expert rewards score near zero on DoorCloseOutward, Kettle and SwingCup, and far below every automated method on AllegroHand? Are the parallel-environment counts and horizons in Table 6 the benchmark defaults, and if not, why were they changed? This determines how far the results generalize beyond this particular pipeline.
2 comments
AuthorsOpenReview ↗

Response to Reviewer YC7p — Part 1/2

We sincerely thank you for taking the time to review our paper carefully and for providing specific and insightful comments. Your attention to complete-search repetitions, comparison settings, and methodological contributions has provided valuable guidance for improving our experiments and presentation. We have carefully considered each of your concerns and added experiments, result summaries, and explanations of the settings. We hope these additions will address your concerns more fully. Revised passages in the manuscript are shown in blue.

You and the other reviewers have independently highlighted key issues concerning search stability, comparison settings, and computational cost. These thoughtful and pertinent suggestions have provided important guidance for improving our experiments. We consolidate the complete-search repetitions, common selection criteria, cost statistics, Expert updates, and extended-training results raised in your Q1, Q2, and Q4 in the common response: Q1 is addressed in Section 1 and Appendix G.1; the selection criteria and cost statistics in Q2 are addressed in Sections 2–3 and Appendices G.2–G.3; the Expert updates and extended-training results in Q4 are addressed in Section 4, Appendix G.4, and Table 1 and Figure 2 in the main text. Below, we further address the baseline protocols and budget comparisons in Q2, the method and ablations in Q3, and the configuration choices in Q4.

Baseline protocols and budget comparisons in Q2

We construct the baselines according to the algorithmic procedures described in their original papers and the available public implementations, and evaluate them within our common training pipeline. Specifically, R* and CARD are implemented from their published descriptions; CARD retains five sequential rounds of optimizing a single reward, whereas population-based methods typically evaluate 5×16 candidates. For REvolve, we use the version without human feedback provided in RF-Agent's public code repository; this implementation source is described in Appendix D (Baseline Implementation Details). We understand your concern about comparison fairness. We apologize that, given the computational budget and time available for this rebuttal, we prioritized the supplementary experiments described above and have not completed the equal-token-budget comparison you requested.

Q3 and W3–W4: Methodological contributions and planning ablations

LMEA, OPRO, EoH, and ReEvo provide important foundations for LLM-based search and reflection. Building on this foundation, our work focuses on reward-mechanism representations and their role in planning across candidates.

Our design focuses on organizing executed reward code and training feedback into mechanism-level records, separating numerical details such as specific weights from the planning representation, and comparing the functions and relationships of mechanisms across candidates. The Generator then combines code and numerical diagnostics to implement the design intent. Relative to Eureka and Auto MC-Reward, the distinction lies in organizing feedback at the mechanism level and using it for planning across rewards; relative to REvolve, it lies in the mechanism representation and the division of information between planning and implementation. R* already separates structure and parameter optimization; our work focuses on mechanism-semantic representations and their role in cross-candidate planning, with our contribution centered on this representation and division of information.

The original w/o Planner condition uses top-1 single-parent selection by AUC and generates 16 variants with a fixed optimization instruction. The Generator still receives the parent code, the Analyzer's implementation diagnostics, training curves, and sampled numerical values. Only the best valid candidate in the current generation is analyzed at each generation, and diversity filtering is disabled. Thus, this variant implements single-parent elite optimization, with corresponding changes to the parent scope, Analyzer coverage, and filtering settings. We have added these implementation details to Ablation Studies in the main text and Appendix A.4.

AuthorsOpenReview ↗

Response to Reviewer YC7p — Part 2/2

Regarding Table 3 and Figure 4, we intend to explain the Planner's role together with the performance results in Section 5.4 (Ablation Studies), rather than relying only on behavioral statistics and examples. In Figure 5, w/o Planner, which uses greedy optimization of the single top-AUC parent, shows relatively consistent performance degradation, especially on AllegroHand and SwingCup, as well as late-stage regression on Over. The full method's performance relative to this greedy-optimization control provides empirical evidence for planning based on population information. We therefore believe that the existing ablation offers convincing support for the Planner's effectiveness. The parent-selection statistics and mechanism examples in Section 5.3 (Planner Analysis) further explain how this planning uses information across candidates.

We also agree that selection methods such as tournament and top-k, as you suggest, are worth comparing. We apologize that, given the computational budget and time available for this rebuttal, we could not additionally complete a control that preserves the same Analyzer/Generator inputs and changes only the selector. We have added the implementation details of w/o Planner to Section 5.4 and Appendix A.4 to clarify the settings of the existing greedy-optimization control.

Expert rewards and training configurations in Q4

The settings in Table 6 come from the following sources: the 3000 PPO epochs used during search and the task-specific final evaluation horizons follow the defaults in Eureka and the public code; the number of parallel environments (num_envs) for each task follows the settings in the R* paper, allowing 16 candidates to be trained concurrently on two RTX 4090 GPUs. Within each task, all compared methods use the same num_envs and training budget. The configuration sources and rationale have been added to Appendix A.2 (Task-Wise Training Settings).

Expert directly uses the original human-designed rewards in the public code; we have made no modifications to the reward code. We understand that your concern is the effect of changing num_envs on the experimental comparison. This setting is applied uniformly to all methods within each task. Expert and automatically generated rewards use the same num_envs, PPO configuration, number of training epochs, and evaluation seeds, thereby maintaining an equal training budget for final policy evaluation. Our claim concerns the relative performance of the reward-design methods under this common setting and equal policy-training budget; the uniform adjustment of num_envs does not change this basis of comparison. We have also included Expert in the extended-training evaluation with the same budget, with results in Appendix G.4.

Other comments: notation, prompts, and code release

We have aligned the generation indices in Algorithm 1 and Eq. 4 and clarified the order of candidate reading, planning, generation, and selection. The actual prompts explain the target metric for each task; the example wording consecutive_successes does not mean that all tasks optimize the same literal notion of consecutive successes. We have clarified this task-dependent meaning in Appendix A.1, with score definitions in Table 7. We will release the implementation code to support reproducibility.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers: Additional Experiments and Revisions — Part 1/2

We thank all reviewers for taking the time to carefully review our paper and for providing specific and insightful comments. We have added complete-search repetitions, common selection criteria, computational-cost statistics, and extended-training experiments. We summarize these additions here and address specific questions separately in our responses to each reviewer. Revised passages in the manuscript are shown in blue.

1. Complete-search repetitions (Appendix G.1)

The original table reports five-seed PPO evaluations after reward selection. To examine randomness in the search process, we added two complete searches for four methods (R-Split, Eureka, RF-Agent, and R*) on six tasks (AllegroHand, BottleCap, DoorClose, DoorOpen, Kettle, and Over), using search seeds 43 and 44. The reward selected from each search is then evaluated with seeds 0–4. The table below summarizes the original search and the two additional searches, using the mean performance across the five policy seeds from each search as the statistical unit. Complete results and curves have been added to Appendix G.1. In these repeated-search summaries, R-Split achieves the highest mean AUC on four of the six tasks, indicating that it remains competitive overall with the additional search repetitions. Apart from changing the search seed from 42 to 43 and 44, all other experimental settings remain the same.

We sincerely apologize that these repetitions do not yet cover all ten tasks. Given the time and computational budget available during this rebuttal period, we can currently provide complete-search repetitions for the six tasks listed above.

The complete-search summary is shown below. It includes the original search and the two additional searches (excluding seed45), consistent with the overlaid plots. For each search, we first average the metrics over five policy seeds, then report the mean ± standard deviation across the three searches. The highest mean in each row is bolded. Max Task Score is the highest task score reached within each policy-training run, rather than its terminal score. For the plots, the five evaluation seeds are first averaged within each search, then the mean and ±1 standard deviation across the three searches are shown. The EMA coefficient is 0.99 and smoothing is used only for plotting; tables are computed from unsmoothed scores.

AUC (×10³)

TaskR-SplitEurekaRF-AgentR*
AllegroHand28.29±11.7826.95±7.8826.32±8.1426.70±10.37
BottleCap2.92±1.913.85±0.831.12±1.472.59±1.18
DoorClose3.31±0.482.95±0.680.86±0.151.42±1.46
DoorOpen4.42±1.304.65±0.543.35±2.063.01±1.41
Kettle3.83±0.501.68±1.073.14±1.133.02±0.18
Over4.26±0.193.29±0.503.53±1.032.72±1.91

Maximum task score (Max Task Score)

TaskR-SplitEurekaRF-AgentR*
AllegroHand9.59±3.379.39±2.259.50±1.799.29±2.81
BottleCap0.67±0.310.84±0.160.39±0.350.75±0.21
DoorClose0.92±0.080.90±0.080.56±0.120.48±0.31
DoorOpen0.92±0.110.93±0.090.86±0.100.72±0.28
Kettle0.98±0.020.60±0.250.92±0.080.93±0.01
Over0.89±0.020.83±0.030.85±0.070.59±0.40

2. Common selection criteria (Appendix G.2)

Using the original search histories, we have reselected rewards for R-Split by the highest task score during search and for Eureka by search AUC, and evaluated them with seeds 0–4. Selection uses only search-stage data and does not access the final evaluation results.

Per-task AUC, maximum task score, and learning curves have been added to Appendix G.2. This control compares only R-Split and Eureka, keeping the original search unchanged and modifying only the final selection rule. Policies are trained from scratch, with final training budgets following Table 6.

The supplementary control results support this choice: for R-Split, selection by AUC produces a lower standard deviation of evaluation AUC across the five seeds on seven of the ten tasks, with ties on two others, compared with selection by maximum task score. Mean AUC is also higher on eight tasks and tied on two. These results show that the rewards selected by AUC exhibit more stable learning performance overall in this multi-seed evaluation, consistent with our design objective of reward robustness.

3. Tokens, request counts, and search time (Appendix G.3)

Per-task token usage and LLM request counts for the additional repetitions have been added to Appendix G.3, covering six tasks and two searches per method per task, or 12 searches per method. Total token usage is 21,221,144 for R-Split, 4,506,487 for Eureka, 18,264,385 for RF-Agent, and 4,023,613 for R*.

AuthorsOpenReview ↗

Response to All Reviewers: Additional Experiments and Revisions — Part 2/2

Token totals include text-embedding usage; detailed statistics are provided in Appendix G.3.

Appendix G.3 also reports search times for dedicated-GPU runs: both searches on AllegroHand and the first search on BottleCap, using the same subset of repetitions for all methods. The table reports the actual time required to complete reward search, excluding the final five-seed policy evaluation.

Given that these searches typically take several hours, we believe that additional LLM expenditure should be considered alongside the performance gains of the resulting rewards. In the complete-search repetition results, R-Split achieves the highest mean AUC on four of six tasks and remains competitive overall. This indicates the practical value of investing additional LLM computation in mechanism analysis and planning when reward search involves time-consuming policy training and evaluation.

4. Extended training and Expert updates (Appendix G.4; Table 1 and Figure 2 in the main text)

We have further included all seven methods (six automated reward methods and Expert) on all ten tasks in the extended-training evaluation, using seeds 0–4 for each reward. The total budget is 5500 epochs for Ant, 10000 for AllegroHand, and 11000 for the other tasks. Training resumes with the original rewards and training configurations. Complete curves and explanations of trajectory stitching have been added to Appendix G.4 of the revised manuscript. We extend the rewards selected in the original main experiments, rather than those reselected in G.2; both AUC and maximum task score cover the full training interval. We apologize that, due to the time and computational budget available in this round, we prioritized continuation from saved checkpoints rather than uninterrupted training from scratch to the longer budget. Model and optimizer states are restored, but the local epoch counter and learning-rate scheduling are reinitialized according to the configuration, so resumed training is not equivalent to uninterrupted training. Details of checkpoint restoration, trajectory stitching, and plotting means and standard deviations across five seeds have been added to Appendix G.4.

Complete numerical results and curves for all seven methods and ten tasks have been added to Appendix G.4 of the revised manuscript. With extended training, the relative performance of the methods changes on some tasks, but R-Split remains competitive overall, achieving the highest mean AUC on five of the ten tasks. These results further support the effectiveness of the selected rewards under longer training budgets.

Our check confirmed that the original five AllegroHand Expert evaluations mistakenly used the same seed=42 rather than five different seeds. We have completed corrected evaluations with seeds 0–4 and updated the AllegroHand Expert results in Table 1 and Figure 2 of the main paper (AUC: 1.61±1.94 ×10³; Max Task Score: 0.94±0.88).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.