- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies how to improve the data efficiency of online reinforcement learning using historical data that does not contain rewards. The authors propose Unsupervised Generative Replay (UGR), which relabels exploration data using a random reward function and pre-trains a reusable diffusion model. In downstream tasks, UGR learns a task-relevant utility model to guide the diffusion model in generating synthetic data to enhance the replay buffer. Experimental results show that UGR outperforms baseline methods on multiple continuous control tasks.
Critical strengths
- This paper studies how to improve the data efficiency of online reinforcement learning using historical exploration data without reward, which is a problem of practical significance in itself. UGR separates the pre-training of generative models from the adaptation to downstream tasks, allowing the pre-trained diffusion model to be reused across different tasks.
- The experimental results are good. The author evaluated UGR on 12 continuous control tasks, and it outperformed existing baseline methods in most tasks, demonstrating the potential of using historical exploration data to enhance online RL.
Critical weaknesses
- I don't quite understand why Random reward priors are used. The author uses a random reward function to relabel historical exploration data and trains a diffusion model to learn the joint distribution of transition and reward. However, random rewards do not change the original transition distribution, so I don't quite understand what additional information these random rewards can provide for downstream tasks. Why not directly pre-train the diffusion model on transitions without rewards, and then adapt it through task-specific guidance?
- I have some concerns about theoretical analysis. I think the support overlap assumption in the theoretical analysis may not hold. Theorem 1 assumes that the target distribution is absolutely continuous with respect to the source distribution, i.e., . However, the rewards in the source distribution come from randomly sampled reward functions, while the rewards for downstream tasks are determined by a fixed function . If the random rewards follow a continuous distribution given a transition, then the target distribution is concentrated at , and the probability of this set under the source distribution could be zero. Therefore, I have doubts about the validity of this assumption in practical methods and am unclear about how it would affect subsequent density-ratio estimation.
- The author lacks a thorough explanation of the design of density-ratio guidance. In the article, by estimating the density ratio between the online replay distribution and the pre-trained source distribution, the diffusion model is guided to generate data that is close to the current online replay distribution. However, I don't understand why matching the current online replay distribution can generate more valuable training data. The transitions in the current online buffer do not necessarily have high value. The authors need to further explain the relationship between this distribution matching and policy improvement.
- The experiments need to further distinguish the contributions of each component. UGR not only used additional unsupervised exploratory data but also combined it with random reward labeling, diffusion model pre-training, and task-specific guidance. Although the paper compares with methods like SynthER and PGR, these methods did not use the same historical exploration data, so the performance improvement may partly come from the additional data. The authors should further compare generative methods using the same exploration data and analyze the independent contributions of each component.
Questions for authors
- Why do we need random reward priors? See weakness 1.
- How is the support overlap assumption in Theorem 1 ensured? See weakness 2.
- How can we ensure that the reward in the generated data is accurate? The diffusion model jointly generates , but the generated reward may not necessarily align with the true downstream reward function. Especially in cases where the guidance model can only be trained with limited online data, such errors may affect subsequent Q-learning. Has the author evaluated the reward accuracy of the generated data and the impact of reward estimation error on policy performance?
- Why can matching the current online replay distribution improve strategy performance? See weakness 3.
- How is the actual computational efficiency of UGR? The author emphasizes that UGR can reuse pre-trained diffusion models, avoiding the need to retrain for each downstream task. However, the downstream phase still requires training multiple classifiers and utility models, and generating data through iterative diffusion sampling. Can the authors compare the actual training time and computational overhead of different methods?