- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies reward shaping for LLM-generated programs in robotic environments. The paper introduces Ground Process and online memory to shape the reward. The Grounder maps the reward signals into four sources: Reach, Interact, Goal, and Stable. The mapping provides a finer signal for training. Online memories record visit frequency and encourage exploration. The proposed method, ReLoop, on Meta World and ManiSkill 2. The evaluation results show that ReLoop improves policy learning and training efficiency. The ablation studies support the design of Online memories.
Critical strengths
In robotic learning, previous paradigms try to learn directly from the reward function using SAC or PPO. This paper introduces an in-training reward-function modification method to show there is room to construct a finer training signal and improve the LLM-generated reward function. The LLM-generated reward function may be imperfect and hard for humans to modify. The proposed method can make the LLM-generated reward function more training-efficient and help it converge to better performance.
Critical weaknesses
The largest problem with this paper is soundness. The proposed method does not compare with any known baselines. This makes the results hard to evaluate. The paper evaluates only some of the design choices. The justification for the design choice is unclear: the loss function introduces five coefficients, and each component should be evaluated carefully.
Questions for authors
- Your supplementary contains much more detail; it is better to put it in the Appendix and use hyperlinks for better understanding.
- In Table 2, are FrozenProram, Matched, and ReLoop all from your method and its ablations? What is the performance of other open-source baselines like Eureka [1]?
- Do the selected tasks have a reference reward function in the environment? If so, I'd like to see the learning curve for the oracle reward function. If possible, please share the original prompt used to generate the reward function. This information tells us how granular the reward function is. Your experiments suggest the robotic reward needs to be more fine-grained. If you prompt the reward function to be more fine-grained, will ReLoop still improve performance?
- The ablation studies are not enough to justify all designs. For example, you introduce five coefficients in the modified loss function; in a new environment, how should I tune these hyperparameters? Also, two key designs are Ground Process and Online Memory; your ablation studies justify Online Memory, but they do not include Ground Process.
- As introduced in the related work, your method is closely related to reward shaping and exploration. You should compare it to some of the related methods.
- The predicate and the discretization are not fully discussed in the paper.
[1] Ma, Y. J., et al. (2023). Eureka: Human-Level Reward Design via Coding Large Language Models. arXiv:2310.12931.