- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates reward-rate optimization for MLE agents in RL, where actions have different execution times. The proposed RPG estimates the policy's reward rate from historical samples and uses it to penalize actions according to their duration. The paper provide convergence results in a contextual bandit setting. The method is then evaluated on MLE-Bench Lite tasks and NanoGPT and report improvements over vanilla RL under fixed execution-time budgets.
Critical strengths
- The paper studies a relevant problem for MLE agents, where action execution time can vary.
- The empirical results are extensive and the method is simple enough to be included in standard RL training procedures
Critical weaknesses
- The theoretical results are established in a setting and with assumptions that are not established for the proposed algorithm. More precisely, Thm. 1 assumes controlled approximation error for the greedy reward rate while Thm. 2 assumes exact rate estimation and expected action values. The algorithm instead uses posterior-based estimator and PPO.
- The proposed experimental results do not isolate the effect of the proposed policy gradient objective.
Questions for authors
- Could you isolate the effects of the modified PPO reward and that of the buffer-selection?
- How does RPG compare with RL using the same time-aware buffer selection and a fixed coefficient for the time penalty?
- Thm. 1 assumes a bounded approximation error for the greedy reward rate. Since the optimlal poilcies are known in the synthetic bandit experiments, can you report the estimation error of the NIW and show that the condition in Thm. 1 is satisfied?
5 comments
Thank you for the clarifications and additional experiments. While these address some of my empirical concerns, I remain concerned about the limited applicability of the theoretical results to the proposed algorithm. I will maintain my original score.
We thank the reviewer for the subsequent response. We would like to kindly follow up for further clarification.
Our motivation for the theoretical analysis is not to justify each design choice of the proposed algorithm. Rather, it is to conduct analysis on what is unique about reward-rate RL, which is suggested by Thm.2 that even in the best case scenario (with exact expectation and zero estimation error), we would have a coupling issue if we use vanilla policy gradient and thus necessitates the use of natural policy gradient. For this reason, we believe our theoretical analysis is a meaningful contribution to performing policy-gradient-style updates in the reward-rate RL setting, and this observation holds for all reward-rate RL bandit settings. The observations from theory then shed light on designing a practical algorithm on MLE-Bench and NanoGPT.
If possible, would the reviewer kindly inform us what kind of theoretical results would be considered as a strong contribution? We look forward to your feedback! Thank you once again!
We thank the reviewer for the constructive feedback. Please see our responses to each of the raised concerns below.
Weakness 1
We would like to kindly highlight that our motivation for the theoretical analysis is to study the training dynamics of reward-rate RL by separating the problem of finite-sample estimation from training. In particular, we believe having an accurate finite-sample reward-rate estimate is a problem of estimating a ratio of expectation of sum of random variables over time, which is not central to studying what is unique about the reward-rate RL training dynamics.
For this reason, our analysis is focused on answering these questions under exact expected values: (Q1) under what conditions can we obtain an approximate (or exact) value of optimal ρ∗ through iterating the policy, since ρ∗ is central to finding the problem's solution; (Q2) even with exact estimation, what is unique about reward-rate RL's training.
To that end, Thm.1 answers (Q1) by showing that the iteration on the greedy reward rate converges near ρ∗ when γ=0 and to exactly ρ∗ when γ=0. To answer (Q2), Thm.2 further analyzes the training dynamics when that estimation error is removed (γ=0), which informs us that reward-rate RL has its unique coupling issue even with exact estimation, and that issue is addressed through natural policy gradient.
The posterior-based estimator and PPO is an extension of our theoretical finding into practice. The posterior estimate approximates the greedy reward-rate by fitting on greedy samples ranked by r−ρδ from the buffer (direct extension of finding from Thm. 1 that greedy reward rate iteration can approximate ρ∗), and PPO is a practical method for first-order approximation of natural policy gradient (extension of finding from Thm. 2 that we need natural policy gradient for de-coupling).
Weakness 2 and Question 1
We answer both concerns together since the questions are asking for an additional ablation experiment. Please see our additional ablation below, where we isolated the effect of reward-rate on MLE-Bench by using the same buffer rule as vanilla RL's by selecting samples keyed by reward only. The only difference between vanilla and ablation is the reward signal, and the evaluation is completely the same.
Question 2
We have conducted additional experiment by using scalarized reward r−αδ with α tuned through a grid search for α∈(0.001,0.05,0.1). Our results are shown below. It is worth noting that the grid search process is very computationally expensive, and RPG doesn't require hyperparameter search.
Question 3
We thank the reviewer for the advice. We provide the result below by calculating the ratio ρ~k+1/ρ(uρk). For the assumption to satisfy, we need 1/(1+γ)≤ρ~k+1/ρ(uρk)≤1+γ,, and the table below shows us that this is consistently the case, where we take the median across environments and arms for each context setting. We also show visualizations of the ratio in our revised paper in Appendix D.3 and edited the main text to reflect this new information.