acceptodds
Under review as a conference paper at ICLR 2027

Unsupervised Generative Replay for Efficient Reinforcement Learning

Abstract

Leveraging the diffusion model to augment the replay buffer has become a useful paradigm to enhance the efficiency of reinforcement learning (RL). Can we pre-train a diffusion model from unlabeled exploratory data such that it can be adapted to new downstream tasks to enrich the RL buffer, even when the downstream reward function cannot be queried on these pre-collected data? In this work, we present an Unsupervised Generative Replay (UGR), a reusable generative replay framework for this setting. UGR pre-trains a diffusion model on exploratory transitions relabeled with diverse reward priors to serve as unsupervised generative priors. When adapting to a specific task, we only train a lightweight guidance model to steer data generation towards the target task. Experiments across complex control tasks on DeepMind Control Suite reveal that UGR effectively improves online data efficiency under a fixed interaction budget. These results suggest that reusing unsupervised generative priors is a promising way to improve online RL across downstream tasks.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, data augmentation, diffusion model, unsupervised RL data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eawqOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies how to improve the data efficiency of online reinforcement learning using historical data that does not contain rewards. The authors propose Unsupervised Generative Replay (UGR), which relabels exploration data using a random reward function and pre-trains a reusable diffusion model. In downstream tasks, UGR learns a task-relevant utility model to guide the diffusion model in generating synthetic data to enhance the replay buffer. Experimental results show that UGR outperforms baseline methods on multiple continuous control tasks.

Critical strengths
  1. This paper studies how to improve the data efficiency of online reinforcement learning using historical exploration data without reward, which is a problem of practical significance in itself. UGR separates the pre-training of generative models from the adaptation to downstream tasks, allowing the pre-trained diffusion model to be reused across different tasks.
  2. The experimental results are good. The author evaluated UGR on 12 continuous control tasks, and it outperformed existing baseline methods in most tasks, demonstrating the potential of using historical exploration data to enhance online RL.
Critical weaknesses
  1. I don't quite understand why Random reward priors are used. The author uses a random reward function to relabel historical exploration data and trains a diffusion model to learn the joint distribution of transition and reward. However, random rewards do not change the original transition distribution, so I don't quite understand what additional information these random rewards can provide for downstream tasks. Why not directly pre-train the diffusion model on transitions without rewards, and then adapt it through task-specific guidance?
  2. I have some concerns about theoretical analysis. I think the support overlap assumption in the theoretical analysis may not hold. Theorem 1 assumes that the target distribution is absolutely continuous with respect to the source distribution, i.e., qω≪pSq_\omega \ll p_S. However, the rewards in the source distribution come from randomly sampled reward functions, while the rewards for downstream tasks are determined by a fixed function RωR_\omega. If the random rewards follow a continuous distribution given a transition, then the target distribution is concentrated at r=Rω(z)r = R_\omega(z), and the probability of this set under the source distribution could be zero. Therefore, I have doubts about the validity of this assumption in practical methods and am unclear about how it would affect subsequent density-ratio estimation.
  3. The author lacks a thorough explanation of the design of density-ratio guidance. In the article, by estimating the density ratio between the online replay distribution and the pre-trained source distribution, the diffusion model is guided to generate data that is close to the current online replay distribution. However, I don't understand why matching the current online replay distribution can generate more valuable training data. The transitions in the current online buffer do not necessarily have high value. The authors need to further explain the relationship between this distribution matching and policy improvement.
  4. The experiments need to further distinguish the contributions of each component. UGR not only used additional unsupervised exploratory data but also combined it with random reward labeling, diffusion model pre-training, and task-specific guidance. Although the paper compares with methods like SynthER and PGR, these methods did not use the same historical exploration data, so the performance improvement may partly come from the additional data. The authors should further compare generative methods using the same exploration data and analyze the independent contributions of each component.
Questions for authors
  1. Why do we need random reward priors? See weakness 1.
  2. How is the support overlap assumption in Theorem 1 ensured? See weakness 2.
  3. How can we ensure that the reward in the generated data is accurate? The diffusion model jointly generates (s,a,r,s′)(s,a,r,s'), but the generated reward may not necessarily align with the true downstream reward function. Especially in cases where the guidance model can only be trained with limited online data, such errors may affect subsequent Q-learning. Has the author evaluated the reward accuracy of the generated data and the impact of reward estimation error on policy performance?
  4. Why can matching the current online replay distribution improve strategy performance? See weakness 3.
  5. How is the actual computational efficiency of UGR? The author emphasizes that UGR can reuse pre-trained diffusion models, avoiding the need to retrain for each downstream task. However, the downstream phase still requires training multiple classifiers and utility models, and generating data through iterative diffusion sampling. Can the authors compare the actual training time and computational overhead of different methods?
Reviewer_4mqcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes unsupervised generative replay (UGR), a novel reusable generative replay framework for improving the efficiency of reinforcement learning. To be specific, UGR uses two types of random reward functions (linear and MLP) to annotate the unlabeled data. It then trains a diffusion model based on these data. For downstream tasks, UGR trains a lightweight utility model to guide diffusion sampling with a contrastive learning loss and a noisy-dependent cycle regularization term. Experiment results on multiple benchmarks show that UGR outperforms the baseline methods.

Critical strengths
  1. The method section is clearly presented and easy to follow.

  2. The overall experiments are comprehensive and demonstrate the effectiveness of the proposed UGR.

Critical weaknesses
  1. Some technical details are not explained clearly, which raises concerns about reproducibility. Please refer to the Question For Authors section for details.

  2. The paper lacks quantitative analysis of the generated data. Specifically, it does not measure the error between the reward labels of the generated transitions and the true task rewards.

Questions for authors
  1. The peoposed UGR uses a mixture of random linear rewards and random MLP rewards (Section 3.2 and Appendix C.3). However, it does not specify how the two types are mixed during training.

  2. There are multiple hyperparameters in the paper. The author conducted a sensitivity analysis of some hyperparameters, but lacks detailed instructions on how to select hyperparameters during training.

  3. What is the actual difference between the reward labels of the generated transitions and the actual task rewards? Please provide some quantitative analysis.

Reviewer_AF2SOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

UGR pretrains a diffusion model on reward-free exploratory transitions relabeled with random linear and MLP reward functions, giving a “source” distribution over (s, a, r, s'). During online RL it trains a classifier-based density-ratio utility model, using CEP-style contrastive energy prediction plus a "noisy-time cycle regularizer." This model guides sampling of synthetic transitions that augment a REDQ agent’s buffer. Evaluations are primarily on state-based DMC.

Critical strengths
  1. The setting is well motivated. I think the question of how to use reward-free exploration data when the downstream reward cannot be queried is an interesting one.
  2. The gains on are reasonably large relative to PGR/SynthER, and the ablations give a good sense of sensitivity.
Critical weaknesses
  1. I am a bit concerned that UGR/Relabel-REDQ use a large pretraining dataset, whereas vanilla REDQ, SynthER and PGR use only online data. There are some obvious baselines which are missing here: SynthER/PGR with their diffusion also pretrained on the same exploratory data, RLPD-style mixing of the relabeled exploratory data, offline-to-online baselines with a learned reward model, etc.
Questions for authors
  1. Do SynthER and PGR with diffusion pretrained on the same exploratory data, and RLPD-style use of that data, reach comparable performance?
  2. Why not use the noisy classifier's gradient directly as guidance? What does f_{psi} add?
  3. How were \beta and the other settings tuned for baselines versus UGR, and what are the results with a single shared \beta?
  4. How accurate are the generated rewards and dynamics, for example one-step model error on synthetic samples?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.