- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Model-Free Umbrella Reinforcement Learning to address sparse-reward RL problems. The method extends Umbrella RL by replacing analytical dynamics with one-step simulator transitions. It combines state-distribution entropy regularization, DualDICE-based occupancy density estimation. Experiments on Multi Valley Mountain Car and StandUp show performance comparable to model-based URL and substantially better than standard baselines.
Critical strengths
The paper provides a mathematically grounded extension of Umbrella RL using discrete-time occupancy estimation. The interpretation is simple and illustration with the toy task is straightforward.
Critical weaknesses
The evaluation focuses on simple tasks, how is the performance on more complicated deep RL control suite is unclear. It is questionable on the extension of the framework to high-dimensional tasks, given that the method is quite complicated and require multiple optimization schemes. The baseline comparison is also limited, how about ask Astra to write a reward function and iterate? These simple tasks should be solvable in this way and basic RL algo.
Questions for authors
- how is performance to high-dim control problems with this approach?
Response to Reviewer LVPn
1. Simple tasks, limited baselines, and a reward designed by a language model plus basic RL.
Thank you for these notes. We study hard problems with a sparse reward, state traps and no terminal state, where MF URL learns directly from the sparse reward. Reward shaping, whether written by hand or by a language model, is environment dependent and needs task knowledge and iteration for every new task; SAC with our Multi Valley Mountain Car shaping still reaches only 0.013 (Table 10). The revised paper compares against more baselines at equal training queries (Sec. 5.2, Table 1): PPO and A2C with the sparse reward reach a return of 0 on both tasks, while MF URL solves both. Designed rewards are represented by PPO with shaped rewards, which also solves both tasks, and SAC with shaped rewards (Table 10). We added planning methods with the same simulator access, Grid VI and fitted Q-iteration (App. A.13). A reward written and refined by a language model is another form of reward design; we could not run it in the discussion period.
Q1. Performance on high-dimensional control problems.
Thank you for this question. Shared queries lower the cost per iteration by a factor of 3.85, and the queries of one iteration can run in parallel (App. A.8). For larger state spaces we are extending MF URL with adaptive proposals (App. D), and on StandUp the first one reaches a return above 0.9 within 50,000 iterations, where the uniform proposal is still near 0 (Fig. 12). Our plan is to show that each proposal works on simple environments and then move step by step to high-dimensional tasks (App. D.5).