acceptodds
Under review as a conference paper at ICLR 2027

A Bird's-Eye View of Iterative Reward Design

Abstract

Designing effective reward functions in RL typically requires substantial expertise and trial and error. Recent work automates this process with LLM-based systems that generate and iteratively improve reward code using policy feedback. However, these methods are often hard to compare because they differ in implementation details, feedback assumptions, and evaluation environments. To address this, we introduce a **B**enchmark for **I**terative **R**eward **D**esign (**BIRD**) that expresses existing methods in a unified configuration and evaluation space. This lets us compare algorithms directly, ablate individual design choices, and prototype new components under matched feedback conditions and policy-training budgets. Across MuJoCo, Meta-World, Assistax, and HumanoidBench, we identify a small set of simple design choices that consistently improve performance. Combining these choices yields significantly better performance than the evaluated methods from prior work. Our results highlight the strength of simple baselines and motivate further study of when additional algorithmic complexity improves iterative reward design.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning, Reward Learning, Robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RF4kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark for iterative reward design, using a unified framework to compare existing methods and identify their design choices. Through extensive ablation experiments, the authors identify useful components and combine them into two new algorithms that outperform the evaluated baselines.

Critical strengths
  1. The paper provides a unified perspective on existing iterative reward design methods and investigates the effects of their design choices through extensive experiments.
  2. The paper develops two simple and effective reward design algorithms, demonstrating that combining relatively simple components can yield strong performance.
Critical weaknesses
  1. The effectiveness of some design choices may depend on the capabilities of the underlying LLMs. Although the paper acknowledges this limitation, it does not experimentally assess how model selection affects its conclusions.
  2. Some design choices may interact and become more effective when combined. However, the baseline-specific ablations and sequential hill-climb do not systematically disentangle these interactions or establish the contribution of each component to the final algorithms.
  3. Although aggregate resource usage is reported, detailed comparisons of runtime and token consumption across methods and design choices are missing. Efficiency is an important dimension for assessing the practical value of reward design methods.
Questions for authors
  1. How sensitive are the observed effects of design choices to the selection of generator and evaluator models? Would the influence of design choices vary with the selected model?
  2. Could the authors summarize the total policy-training budget, runtime, and token consumption of the hill-climbing procedure used to develop ERA-U to better evaluation the design choices?
2 comments

Thank you for the positive review! We address your questions and identified weaknesses below.

Weaknesses

  1. See Question 1 below.

"Some design choices may interact and become more effective when combined"

Yes, we agree. As much as possible, we try to capture this in our ablations by changing explicitly linked choices together (e.g., design thought + the parent's thought, or per-component output + per-component feedback, which likely have complementary effects). These are the 7 of 30 marked ‡ in Table 5. We use the original papers as a guide to this, but the combinatorial complexity (300+ keys is ~45,000 pairs alone) makes this infeasible to measure in full.

"do not systematically disentangle these interactions or establish the contribution of each component to the final algorithms"

This is fair; we don't run a systematic ablation. The climb does include a few direct tests (Table 10): design thought and elite crossover each helped alone (+0.13, +0.15), but not more together (+0.08). There may be tools from experimental design (e.g., fractional factorial screening) that make this affordable, which could be interesting future work.

"detailed comparisons of runtime and token consumption across methods and design choices are missing"

Agreed. See Question 2 below.

Questions

"How sensitive are the observed effects of design choices to the selection of generator and evaluator models?"

Because of compute budget restrictions, we were constrained to running our experiments on Claude models (Appendix D). However, we suspect from other papers that stronger models generally perform better (e.g., fewer coding bugs, better visual reasoning). For further reference, see the model comparisons in Eureka (GPT-3.5 vs GPT-4, appendix), CARD (GPT-3.5 vs GPT-4) and Text2Reward (open-source models, appendix).

"Would the influence of design choices vary with the selected model?"

Likely for some: checking for coding errors, for example, matters most with weaker models (Section 2). The closest evidence we know is RF-Agent, which compares methods under GPT-4o-mini and GPT-4o and reports that its advantage holds under both.

"summarize the total policy-training budget, runtime, and token consumption of the hill-climbing procedure used to develop ERA-U"

We agree this is good to report! Another reviewer also requested this, so we'll add a per-method table of model calls, tokens, environment steps and wall-clock time. For the climb itself (Table 10): 24 configurations, about 430 searches and about 7 billion environment steps, roughly a fifth of the MT10 comparison in Section 4.1. We'll add these totals to Appendix I.5.

Thanks for your response. I look forward to seeing the revisions to the manuscript and your future work. I will maintain my current score.

Reviewer_YMRqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BIRD (Benchmark for Iterative Reward Design), a unified framework for implementing, comparing, and developing LLM-based iterative reward design (IRD) methods for reinforcement learning. The central motivation is that existing methods are difficult to compare because they differ in implementation details, feedback signals, policy-training procedures, and evaluation environments.

BIRD represents IRD methods as configurations of a common generate–verify–train–evaluate–select loop. The authors implement 14 published methods and evaluate six existing methods across 21 tasks from MuJoCo, Meta-World, Assistax, and HumanoidBench under a matched experimental protocol. They further conduct ablations of 30 design choices, finding that most do not yield consistent improvements, whereas several relatively simple choices—such as retaining the best policy checkpoint, summarizing feedback, using task-success-based fitness, and providing a longer generation history—are beneficial.

Building on these findings, the authors develop ERA-U and ERA-S, unsupervised and supervised IRD variants derived from RDA. Both achieve strong results in the evaluated benchmark suites and rank first and second, respectively, in the pooled Bradley–Terry ranking. The authors also investigate the relationship between benchmark performance and intended behavior through a human evaluation involving two author raters. Their results expose substantial discrepancies between reference-reward scores and naturalistic task execution, particularly in humanoid locomotion and assistive robotics.

The paper's main contribution is therefore a unified empirical framework for studying IRD, together with evidence that simple design choices and systematic configuration search can outperform more elaborate existing methods under a controlled evaluation protocol. Its additional contribution is an empirical demonstration that higher benchmark scores do not necessarily imply better behavioral alignment.

Critical strengths
  1. A unified and practically useful evaluation framework. BIRD evaluates multiple IRD methods under a common experimental protocol, facilitating systematic comparison, reproducibility, and future research.
  2. Valuable negative empirical findings. The ablations show that many intuitively reasonable design choices do not consistently improve performance, helping clarify empirical assumptions underlying IRD method design.
  3. Attention to the discrepancy between reward metrics and behavioral quality. By combining human evaluation with reference-reward measurements, the paper demonstrates that a single automated metric may not adequately capture the quality of learned behaviors.
  4. Cross-task evaluation with additional generalization evidence. The experiments cover multiple simulated environments and task types. The supplementary material also reports rankings after excluding development tasks, providing additional evidence regarding generalization beyond the tasks used for method development.
Critical weaknesses
  1. Limitations in attributing the effects of individual design choices. Some ablations change multiple design factors simultaneously, making it difficult to attribute observed performance differences to a single factor. Moreover, selecting a limited subset of ablations manually from a much larger configuration space restricts the coverage of the conclusions.
  2. Limited evidence from the human evaluation. The study uses two author raters. Although the evaluation includes hundreds of video annotations, the limited number of raters and the lack of a sufficiently explicit operational definition of naturalness may affect the reliability and generalizability of the findings.
  3. Incomplete analysis of computational cost. The paper reports model-call counts, token usage, and some computational-cost information, but lacks a comprehensive cost–performance analysis. Differences in resource budgets across methods also warrant caution when interpreting efficiency comparisons.
Questions for authors
  1. For ERA-U and ERA-S, could the authors clarify the development–evaluation task split and configuration-search procedure, and further characterize the robustness of the results when development tasks are excluded? In particular, how does ERA-S's inheritance of ERA-U's configuration affect the interpretation of their comparative performance?
  2. Could the authors include additional independent raters and provide a more explicit, operational naturalness-rating rubric to assess the reliability of the human-evaluation results?
  3. Could the authors compare methods under more consistent environment-interaction or total-compute budgets and report a more comprehensive cost–performance trade-off?
2 comments

Thank you for the positive review! We address the weaknesses and questions below.

Weaknesses

"Some ablations change multiple design factors simultaneously"

True for 7 of the 30, marked ‡ in Table 5. Those settings are coupled (e.g., per-component output needs per-component feedback), so we report the effect of the group. The other 23 are single-key changes, including all 4 that clearly help (1, 16, 18, 19).

"selecting a limited subset of ablations manually from a much larger configuration space restricts the coverage of the conclusions"

Agreed, and the limit is compute. Each ablation runs on up to 3 baselines × 9 tasks × 5 seeds, so covering all 300+ keys isn't feasible. To our knowledge this is already a much larger ablation than prior IRD papers report. We prioritized choices that published methods present as contributions, plus our own proposals. Many of the remaining keys are sub-settings of a single mechanism (e.g., a threshold) rather than standalone design choices.

2, 3. See Questions 2 and 3 below.

Questions

"clarify the development–evaluation task split and configuration-search procedure"

We hill-climbed ERA-U on 4 MT10 tasks (pick-place, push, peg-insert-side, drawer-open): they were quick to run and the only MT10 tasks not immediately saturated (the others scored high within a single iteration).

  • Development–evaluation split: developed on those 4, evaluated on all 21. The other 17 were never used in the climb.
  • Configuration-search procedure: start from RDA, change one design choice at a time, compare by mean task success on the 4 tasks. 24 configurations in total (Appendix I.5, Table 10).

"characterize the robustness of the results when development tasks are excluded"

We were also worried about this, so we re-ran the ranking without them. The results (Appendix K, Figure 17) confirm that ERA-S and ERA-U still rank 1st and 2nd overall even with the development tasks excluded.

"how does ERA-S's inheritance of ERA-U's configuration affect the interpretation of their comparative performance?"

This is a great question! ERA-S needed no hill-climbing of its own: it is ERA-U with one key flipped (the ranking signal). So a search aimed at ERA-S can likely do much better. As for the comparison, it is perhaps unsurprising that the supervised method comes out ahead overall: a stronger supervision signal makes for a better IRD loop. The one exception is HumanoidBench, where ERA-U ranks above ERA-S; this may be because the reference reward there is itself hackable (Section 4.4).

"include additional independent raters"

Fair point, the study is small, as we acknowledge in Section 4.4 and Appendix D. That said, behaviors like these are commonly reported in the specification gaming literature (e.g., Krakovna et al., 2020), usually with no human study at all. Of course, we agree that adding more raters would strengthen the case, and so we're looking into adding outside raters.

"a more explicit, operational naturalness-rating rubric"

Good call. Raters answered 2 questions on 5-point scales: "how well is the task achieved?" and "is the task approached in the way you would expect?" We'll add the exact wording and scale to Appendix J.

"assess the reliability of the human-evaluation results"

Table 12 reports within- and between-rater reliability (between raters, ρ = 0.83 on completion and 0.54 on naturalness). Our main claim is clearest on HumanoidBench: 70 of 73 naturalness ratings were the minimum score, all 73 given with high confidence.

"compare methods under more consistent environment-interaction or total-compute budgets"

Thanks for raising this; we should have been clearer. Every method except CARD already runs at the same budget: 8 candidates × 5 iterations = 40 policy trainings per search, with the same steps and learner per benchmark. CARD uses about a tenth of the steps by design, and Figure 12 compares it at equal steps. We'll say so in Section 4.1.

"report a more comprehensive cost–performance trade-off"

Agreed, this was thin. We'll add a per-method table of model calls, tokens, environment steps and wall-clock time, plus a cost-versus-score plot.

Thanks for your feedback. I have no additional questions about this paper.

Reviewer_jgEGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a benchmark for iterative reward design (IDR), structuring the design space of IDR into 5 dimensions. They evaluate 6 IDR methods on 4 benchmarks, and show that only 4 of 30 investigated reward design choices meaningfully improve downstream policy performance. Based on this insight, they propose their own IRD flavor and generally outperform/match existing methods on the chosen benchmarks and hold out tasks, as measured by Elo scores. The paper provides useful insights on the (un-)naturalness of the high-performing behaviors on HumanoidBench and Assistax, showing that while many IRD methods achieve high completion, they do mostly not score highly in naturalness (rated by 2 authors).

Critical strengths
  • The paper is clearly written and the figures are well designed
  • I foresee that the benchmark/design space is useful basis for future work
  • The authors compare 6 methods on 21 tasks across four benchmark suites and ablate 30 design choices. That only four of them consistently help is a useful result for practitioners and a good indicator for researchers in the field of IDR.
Critical weaknesses
  • While the author's transparency is commendable, they themselves admit that the user study is small and the annotators are from the author team. While the core claims of the paper do not rest on the study, section 4.4 is still listed as a contribution, so I would ask the authors to consider increasing the pool of annotators (outside the author team) and report familiarity with the tasks.
Questions for authors
  • Does the paper report the performance compared to policies trained on the reference reward directly anywhere? If not, I think this would be a valuable comparison, as IRD's premise (afaik) was to iteratively design rewards that would be better (learnability or faithfulness) than manually engineered ones. In this spirit, it would also be good to report if the reward hacking artifacts (like flailing arms) also occur for the shipped default reward.
  • As the authors concede, task information might have already leaked into VLMs during pre-training, since these environments are open source. A task similar in spirit to the Hopper backflip of Christiano et al. (2017), for which no reference reward exists and the objective is only defined by human judgement would be a good test for the unsupervised recipe and the VLM judge. Human annotation is required to test completion and naturalness again.
2 comments

Thank you for the positive review! We answer questions and address the weaknesses you identified below.

Weaknesses

"the user study is small and the annotators are from the author team"

Fair point, the study is small, as we acknowledge in Section 4.4 and Appendix D. That said, behaviors like these are commonly reported in the specification gaming literature (e.g., Krakovna et al., 2020), usually with no human study at all. Of course, we agree that adding more raters would strengthen the case, and so we're looking into adding outside raters. On familiarity: both raters are authors and know the tasks well (Appendix J).

Questions

"Does the paper report the performance compared to policies trained on the reference reward directly anywhere?"

No, and we agree this is a good comparison. We will add it as an appendix section in the final version. A preliminary check on Gym MuJoCo, with our learner and budget (PPO, 1M steps, 3 seeds): training on the shipped reward goes 84 m on HalfCheetah, and the average policy from every IRD method goes farther (101–231 m). Eureka reports beating human-written rewards on 83% of its tasks, and ERA ranks above Eureka in our comparison, so we expect a similar picture.

"report if the reward hacking artifacts (like flailing arms) also occur for the shipped default reward"

We do see odd behavior under the shipped reward too (Section 4.4, Figure 8): we wrote 2 scripted policies that hop backward or rock in place, and they earn comparable reference reward without walking. An Assistax policy also scores well while pushing on the patient’s arm with over 3 kN.

"task information might have already leaked into VLMs during pre-training"

We were concerned about this too and list it as a limitation in Appendix D. While we don't report this, we did run some informal studies on homebrew versions of MuJoCo tasks to test robustness to leakage. We didn't see any noticeable difference (at least on the simple tasks we tested), so we chose not to report anything.

"A task similar in spirit to the Hopper backflip of Christiano et al. (2017), for which no reference reward exists"

Good idea! If accepted, we'll add an appendix section on the Hopper backflip.

Reviewer_jgEGOpenReview ↗

Response to authors' rebuttal

Thank you for clarifying my questions. I am looking forward to the final version of this paper. I will maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.