- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Reward evaluation often requires many complete training runs, creating a major bottleneck for large scale reward optimization. To address the question, this paper proposes Fast Reward Evaluation, which uses partially observed learning curves to assess the potential of candidate reward functions and guide training resource allocation. Therefore, Candidate reward functions with lower estimated performance and uncertainty receive less training budget. The method combines a Gaussian learning curve surrogate with Optimal Computing Budget Allocation (OCBA) to formalize this allocation process mathematically.
Critical strengths
- The paper addresses an important computational bottleneck in reward design. Furthermore, the method can be applied to other domains such as agentic rl.
- Applying OCBA in training resource allocation of reward evaluation is a reasonable and useful method.
- The experiments cover classical sparse-reward benchmarks, automatic reward generation tasks, and an LLM reasoning benchmark, supporting the method’s effectiveness across different domains.
Critical weaknesses
- Important implementation details are insufficiently explained. The paper mainly focuses on mathematical assumptions and theoretical results, but the practical estimation and updating of the surrogate statistics remain unclear. For example, the procedure for estimating the surrogate variances is not adequately specified, despite their central role in uncertainty quantification and budget allocation.
- The experiments are limited. Although the experiments cover several domains, they do not clearly establish whether the method remains effective on more challenging tasks that require long exploration before substantial performance improvements emerge. Such learning dynamics may violate the surrogate assumptions and cause promising reward candidates to receive insufficient training budget. Meanwhile, the advantage of Reward-Adaptive OCBA over standard OCBA is not clearly established.
Questions for authors
- How are the surrogate variances estimated and updated in practice?
- Can the method handle temporary plateaus followed by substantial improvements, which happen in lots of tasks?
3 comments
Thank you for the clarification. I will keep my original score because the theoretical analysis still oversimplifies learning dynamics in practical RL settings, limiting the method’s practical applicability.
Thank you for your follow-up. We appreciate your thoughtful concerns about the complexity of learning dynamics in practical RL settings.
We would like to clarify an important distinction between the assumptions used for theoretical analysis and the practical applicability of the allocation method. Our goal is not to predict the entire future learning curve, but to make more effective training-budget allocation decisions using the information available at each allocation round. The local learning-curve approximation provides a tractable framework for analyzing the allocation mechanism, rather than imposing a global assumption on how RL training must evolve.
In practice, Reward-Adaptive OCBA operates on observed training performance. At each allocation round, the performance estimates and empirical variances are updated using newly collected training segments, allowing subsequent allocation decisions to adapt to the observed learning progress. Therefore, the practical implementation does not require an accurate global model of the learning dynamics or an explicit prediction of each candidate's final performance.
Importantly, the effectiveness of this allocation strategy is also evaluated empirically across sparse-reward control, automated reward design, and LLM post-training. The additional six-candidate GSM8K experiment further demonstrates improvements over standard OCBA and uniform allocation under the same training budget. These results provide evidence that the proposed method can improve allocation efficiency in practical settings where policy performance evolves throughout training.
We acknowledge that abrupt changes following prolonged plateaus remain challenging, as discussed in Appendix J.2. Nevertheless, we believe that the simplified theoretical model and the demonstrated empirical utility should be evaluated as distinct aspects of the work.
We hope this clarification helps address your concern, and we thank you again for your valuable feedback.
We thank the reviewer for their insightful feedback and helpful comments.
1. Important implementation details are insufficiently explained. How are the surrogate variances estimated and updated in practice?
Thank you for pointing out that the implementation details needed clarification. In the implementation, we summarize each training segment by averaging the final 10% of its logged task scores, using at least five observations. At each allocation round, we compute the arithmetic mean and empirical variance of all segment scores collected for each candidate since warm-up, with the number of segment scores as the variance divisor. When a candidate receives additional training, we append its new segment score and update both statistics. With only the warm-up segment available, the empirical variance is zero, so we apply a floor of 10−8. The allocator uses this variance to quantify the observed variability across training segments. The Gaussian surrogate provides the local model used in the theoretical analysis.
2. The experiments are limited. Although the experiments cover several domains, they do not clearly establish whether the method remains effective on more challenging tasks that require long exploration before substantial performance improvements emerge. Can the method handle temporary plateaus followed by substantial improvements, which happen in lots of tasks?
Our experiments cover sparse-reward control, automated reward design, and LLM post-training. We agree that delayed improvements pose a distinct challenge. As discussed in Appendix J.2, the local approximation can break down when training dynamics change abruptly. If the observed return stays flat, a candidate may receive too little training to reach a later improvement within the available budget.
To further assess our method with a more complex set of reward candidates, we added a six-candidate GSM8K experiment in Appendix I.5. The candidates combine answer correctness with different forms of process feedback from two frozen reward models. Under the same training budget, Reward-Adaptive OCBA achieves higher mean held-out test accuracy than both standard OCBA and Uniform.
Standard OCBA assumes that repeated evaluations of a candidate come from a fixed performance distribution. In our setting, additional training updates the candidate's policy and can change its performance. Our analysis accounts for this evolution through a local learning-curve model. In the empirical comparison, both OCBA methods use the same updated score and variance estimates. Standard OCBA uses each candidate's estimated performance gap to the current leader. Reward-Adaptive OCBA uses the two leading candidates to define a shared selection boundary and weights each candidate by its variability and distance from that boundary. Across several experiments in the paper, Reward-Adaptive OCBA outperforms standard OCBA under the same training budget.