- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies LLM-based automatic reward design for reinforcement learning, focusing on how information across a candidate population can guide the next generation of search. R-Split's Analyzer converts reward code and training-process data into structured mechanism records, separating mechanism descriptions, performance assessments, and numerical implementation details. The Planner compares the current population, selects parents, and specifies refinement, combination, or exploration directions; the Generator implements these directions as reward programs. The system also retains candidates by prioritizing task-score AUC while filtering similar mechanism descriptions. The authors compare against Eureka, REvolve, RF-Agent, R*, and CARD on ten Isaac Gym/Bi-DexHands control tasks and retrain the selected rewards with five policy-training seeds. The results show competitive performance, while ablations on four tasks investigate the Planner, structured representation, mechanism input, and diversity filtering.
Critical strengths
-
The division of responsibilities among the Analyzer, Planner, and Generator connects high-level design comparison with concrete implementation: the Planner can compare mechanisms and their observed evidence, while the Generator handles formulas and numerical details. This allows the search to use design information from weaker overall candidates instead of refining only the current best candidate.
-
The experiments use common task environments and PPO configurations, and the selected rewards are subsequently evaluated through five policy-training runs from scratch. R-Split has the highest mean AUC on five of the ten tasks in Table 1.
Critical weaknesses
-
The paper's central object is a reward-search algorithm, but repeated evaluation mainly measures policy-training randomness conditional on a selected reward rather than variation in the complete search. One complete search is reported per task, while five seeds retrain the same selected reward. The results therefore leave open how reliably the algorithm discovers rewards of similar quality and how reproducible the baseline advantages and ablation differences are.
-
The ablations support the complete system's design, but important control conditions remain underspecified, limiting attribution of the gains to the planning module and structured mechanism records.
-
Search selects rewards by AUC over a fixed training horizon, whereas final evaluation uses task-specific horizons. Because accumulated performance and candidate rankings can change with the horizon, alignment between the selection criterion and the final-evaluation objective has not been validated.
Questions for authors
-
Are independent complete-search repetitions available on representative tasks, rather than only retraining the same selected reward?
-
Table 8 reports aggregate token usage across ten tasks only for R-Split and Eureka. Could the authors report per-task LLM call counts and token usage for one complete reward search by the other main baselines, particularly RF-Agent and R*, under the reported search settings? The 4.65-fold token ratio relative to Eureka alone is insufficient to assess the performance-cost tradeoff with the main baselines.
-
In w/o Planner, how are parents selected, and which parent code, Analyzer diagnoses, training curves, and history remain available to the Generator? How are the Planner's refinement or combination instructions replaced, or are rewards generated from scratch in every generation?
-
In w/o Structure, what text is embedded for diversity filtering? Is the combined description-and-assessment text used directly, or are mechanism descriptions extracted first?
-
The paper selects rewards by task-score AUC. Have the authors observed candidates with higher AUC but lower final success rates? If such a tradeoff occurs, how is the better reward defined, and why is AUC prioritized for selection?
-
AUC and candidate rankings depend on the training horizon. Search uses 3,000 PPO epochs, whereas final evaluation uses 500 for Ant, 5,000 for AllegroHand, and 6,000 for Bi-DexHands. Does the reward selected at 3,000 epochs still outperform other high-scoring candidates at the corresponding final-evaluation horizon? Could changes in candidate rankings be checked on representative tasks?
2 comments
Thank you for your detailed response, which helps me understand this paper better.
Response to Reviewer aZiM
Thank you for the time and effort you devoted to reviewing our paper and for your detailed and thought-provoking suggestions. Your attention to search stability, computational cost, and evaluation methodology is helpful for improving the completeness and readability of our experimental report. We have carefully considered these comments and added complete-search repetitions, per-task cost accounting, and relevant control results in the revision. We also indicate the corresponding appendix locations for ease of reference. Revised passages in the manuscript are shown in blue.
The complete-search repetitions in Q1, request counts and tokens in Q2, and the selection-criterion and extended-training experiments in Q5–Q6 are consolidated in the common response: see Section 1 / Appendix G.1, Section 3 / Appendix G.3, Section 2 / Appendix G.2, and Section 4 / Appendix G.4, respectively. Below, we further clarify the ablation implementations in Q3–Q4, the rationale for AUC-based selection in Q5, and the training-horizon settings in Q6.
Q3: Information available to w/o Planner and its generation procedure
The original w/o Planner condition uses top-1 single-parent selection by AUC and generates 16 variants with a fixed optimization instruction. The Generator still receives the parent code, the Analyzer's implementation diagnostics, training curves, and sampled numerical values. Only the best valid candidate in the current generation is analyzed at each generation, and diversity filtering is disabled. Thus, this variant implements single-parent elite optimization, with corresponding changes to the parent scope, Analyzer coverage, and filtering settings. We have added these implementation details to Ablation Studies in the main text and Appendix A.4.
Q4: Input to diversity filtering in w/o Structure
The w/o Structure condition directly feeds the entire text mixing mechanism descriptions and assessments into the embedding model, without re-extracting pure mechanism descriptions. It uses the same embedding model and threshold rules. It changes both the Planner representation and the input to diversity filtering, and therefore tests the behavior of the system after removing the per-mechanism structured representation, rather than merely changing the text format. We have explicitly documented this treatment in Ablation Studies in the main text and Appendix A.4.
Q5: Rationale for using AUC
We believe that a good reward function should possess a degree of robustness and guide policy learning continuously and stably. We therefore select candidates using AUC over the entire training process, considering both learning speed and sustained performance and reducing reliance on a high score at a single point in time. A higher AUC does not necessarily coincide with the highest terminal training score. Our selection reflects an emphasis on performance throughout learning, while we also report maximum task score to present different evaluation dimensions.
The supplementary control results support this choice: for R-Split, selection by AUC produces a lower standard deviation of evaluation AUC across the five seeds on seven of the ten tasks, with ties on two others, compared with selection by maximum task score. Mean AUC is also higher on eight tasks and tied on two. These results show that the rewards selected by AUC exhibit more stable learning performance overall in this multi-seed evaluation, consistent with our design objective of reward robustness.
Q6: Search and final-training horizons
The settings in Table 6 come from the following sources: the 3000 PPO epochs used during search and the task-specific final evaluation horizons follow the defaults in Eureka and the public code; the number of parallel environments (num_envs) for each task follows the settings in the R* paper, allowing 16 candidates to be trained concurrently on two RTX 4090 GPUs. Within each task, all compared methods use the same num_envs and training budget. The configuration sources and rationale have been added to Appendix A.2 (Task-Wise Training Settings).
Complete numerical results and curves for all seven methods and ten tasks have been added to Appendix G.4 of the revised manuscript. With extended training, the relative performance of the methods changes on some tasks, but R-Split remains competitive overall, achieving the highest mean AUC on five of the ten tasks. These results further support the effectiveness of the selected rewards under longer training budgets.