- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a benchmark for iterative reward design, using a unified framework to compare existing methods and identify their design choices. Through extensive ablation experiments, the authors identify useful components and combine them into two new algorithms that outperform the evaluated baselines.
Critical strengths
- The paper provides a unified perspective on existing iterative reward design methods and investigates the effects of their design choices through extensive experiments.
- The paper develops two simple and effective reward design algorithms, demonstrating that combining relatively simple components can yield strong performance.
Critical weaknesses
- The effectiveness of some design choices may depend on the capabilities of the underlying LLMs. Although the paper acknowledges this limitation, it does not experimentally assess how model selection affects its conclusions.
- Some design choices may interact and become more effective when combined. However, the baseline-specific ablations and sequential hill-climb do not systematically disentangle these interactions or establish the contribution of each component to the final algorithms.
- Although aggregate resource usage is reported, detailed comparisons of runtime and token consumption across methods and design choices are missing. Efficiency is an important dimension for assessing the practical value of reward design methods.
Questions for authors
- How sensitive are the observed effects of design choices to the selection of generator and evaluator models? Would the influence of design choices vary with the selected model?
- Could the authors summarize the total policy-training budget, runtime, and token consumption of the hill-climbing procedure used to develop ERA-U to better evaluation the design choices?
2 comments
Thanks for your response. I look forward to seeing the revisions to the manuscript and your future work. I will maintain my current score.
Thank you for the positive review! We address your questions and identified weaknesses below.
Weaknesses
See Question 1 below.
Yes, we agree. As much as possible, we try to capture this in our ablations by changing explicitly linked choices together (e.g., design thought + the parent's thought, or per-component output + per-component feedback, which likely have complementary effects). These are the 7 of 30 marked ‡ in Table 5. We use the original papers as a guide to this, but the combinatorial complexity (300+ keys is ~45,000 pairs alone) makes this infeasible to measure in full.
This is fair; we don't run a systematic ablation. The climb does include a few direct tests (Table 10): design thought and elite crossover each helped alone (+0.13, +0.15), but not more together (+0.08). There may be tools from experimental design (e.g., fractional factorial screening) that make this affordable, which could be interesting future work.
Agreed. See Question 2 below.
Questions
Because of compute budget restrictions, we were constrained to running our experiments on Claude models (Appendix D). However, we suspect from other papers that stronger models generally perform better (e.g., fewer coding bugs, better visual reasoning). For further reference, see the model comparisons in Eureka (GPT-3.5 vs GPT-4, appendix), CARD (GPT-3.5 vs GPT-4) and Text2Reward (open-source models, appendix).
Likely for some: checking for coding errors, for example, matters most with weaker models (Section 2). The closest evidence we know is RF-Agent, which compares methods under GPT-4o-mini and GPT-4o and reports that its advantage holds under both.
We agree this is good to report! Another reviewer also requested this, so we'll add a per-method table of model calls, tokens, environment steps and wall-clock time. For the climb itself (Table 10): 24 configurations, about 430 searches and about 7 billion environment steps, roughly a fifth of the MT10 comparison in Section 4.1. We'll add these totals to Appendix I.5.