acceptodds
Under review as a conference paper at ICLR 2027

PADGen: Policy-Adaptive Data Generation for Offline Reinforcement Learning

Abstract

Data augmentation has been widely used to improve offline reinforcement learning (RL) by expanding the available training data. However, existing methods largely follow a static generation paradigm, where augmented data remain fixed throughout policy optimization, overlooking that data utility changes as the policy evolves. Consequently, statically generated data may become less useful as training progresses and fail to accommodate the evolving data needs of the policy, thereby limiting further policy improvement. To address this limitation,we propose a policy-adaptive data generation framework (PADGen), which consists of a generator, selector, and policy. The generator generates data based on the utility feedback from both the selector and the policy. The selector evaluates and selects generated data according to their utility for policy optimization. After each training iteration, the policy provides optimization feedback, while the selector returns the utility evaluation of the generated data. The generator then leverages these feedback signals to adaptively generate new data for the next iteration, forming a closed-loop data generation and policy optimization process. This design enables the generated data to be continuously adjusted according to the current policy, improving their utility throughout policy optimization. Extensive experiments across multiple offline RL algorithms and benchmark datasets demonstrate that PADGen consistently outperforms existing data augmentation methods. Our code is available at https://anonymous.4open.science/r/PADGen-ABCD.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sqHWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PADGen augments offline RL data during policy training. It keeps a pretrained TGCVG generator fixed and changes which trajectory segments provide its generation contexts, using feedback from policy fitting loss and a data selector. The selector compares state changes and state-action patterns with high-return reference segments, then samples data for policy training. The contribution is this feedback driven generation and selection loop.

The evaluation covers nine D4RL locomotion datasets with DT, IQL, and TD3+BC, with five-seed results, component ablations, and experiments mixing in random data. Results improve on medium and medium-replay datasets, but benefits are less consistent on medium-expert data. The authors report these failures, including a TD3+BC decrease from 89.9 to 42.1 on HalfCheetah-medium-expert.

Critical strengths
  1. There are meaningful gains across different learners. Table 1 reports the highest mean return in 14 of 18 settings over five seeds, with average improvements of 4.1, 2.1, and 6.3 points over the strongest reported augmentation baseline for IQL, TD3+BC, and DT, respectively. The DT gains on HalfCheetah-medium and Hopper-medium are substantial, supporting the practical value of the combined method.

  2. The degraded-data results identify a useful application. With 75% random data, IQL reaches 90.4 +/- 2.4, compared with 71.3 +/- 14.3 without augmentation and 69.7 +/- 9.9 with selection alone (Table 2). This supports combining generation with selection when the available data contain many poor trajectories.

Critical weaknesses
  1. The experiments do not adequately isolate policy adaptation. Section 5.3 does not establish matched generated-transition counts or policy-update budgets between one-shot and repeated generation, while -Feedback removes both selector and policy feedback. Appendix A.1 does include λg=0\lambda_g=0, but Figure 4 does not identify its environment or dataset, IQL shows little final-performance separation, and this policy-specific control is not reported across the main evaluation.

  2. Important sampling details are missing. Section 5.1 describes fgatef_{\mathrm{gate}} as linear and fdf_d as distance based without specifying their formulas, coefficients, or scaling. Equation 6 can produce negative scores, yet the transformation into valid sampling probabilities and the construction of the high-return reference set are unspecified.

  3. The novelty claim is overstated. The claim to be the first framework adapting generation to an evolving policy (lines 94–95) is too broad: MOPO already regenerates model rollouts under the current policy during optimization (Yu et al., 2020, Algorithm 2).

Questions for authors
  1. What are the exact definitions and coefficients of fgatef_{\mathrm{gate}} and fdf_d, how are scores transformed into nonnegative sampling probabilities, and how is the high-return reference set constructed?
  2. What generation schedule, rollout length, total generated-transition count, policy-update count, and original/synthetic data ratio are used, and which quantities are matched across PADGen, TGCVG, and the ablations?
  3. Which task is shown in Figure 4, and can you report the λg=0\lambda_g=0 variant across the main tasks while retaining selector feedback and matching sample and optimization budgets?
5 comments
AuthorsOpenReview ↗

Response to Reviewer sqHW(Q1 and Q2)

Q1: Section 5.3 does not establish matched generated-transition counts or policy-update budgets between one-shot and repeated generation, while -Feedback removes both selector and policy feedback.

R1: Thanks for your comment. In the section of the main results(5.2) and the ablation study(5.3), the one-shot generation methods, such as TGCVG, generate a fixed package before training, whereas PADGen generates data periodically using feedback-adjusted contexts. However, the policy-update count and generated quantities for both methods are identical to ensure a fair comparison.

Regarding the -Feedback ablation, the generator in PADGen receives two complementary feedback signals: policy feedback, and selector feedback, which evaluates the quality of generated data. Together, they form the feedback mechanism that guides data generation. Therefore, the original -Feedback ablation removes both signals to evaluate the effectiveness of the overall feedback mechanism, rather than the individual contribution of policy feedback.

Nevertheless, we agree that isolating the contribution of policy feedback is also important. We have therefore conducted additional experiments across multiple challenging settings by removing only policy feedback, while retaining selector feedback. The results are shown below, where 25% and 50% denote the proportions of random data mixed into the original HalfCheetah-Medium-Replay dataset.

MethodHalfCheetah 50%HalfCheetah 75%AntMaze-Large-Play
-Feedback37.2 ± 0.933.6 ± 1.469.0 ± 4.0
PADGen40.4 ± 0.738.3 ± 2.678.3 ± 9.5

These results demonstrate the importance of policy feedback in guiding data generation and improving policy performance in most cases.

Q2: Which task is shown in Figure 4?

R2: Thank you for your comment. Figure 4 presents the results on Hopper-Medium-Replay. The comparison between PADGen and the two settings with (λg=0\lambda_g = 0) is shown below.

MethodIQLTD3+BC
λg=0\lambda_g=0102.3 ± 0.980.4 ± 2.2
PADGen102.3 ± 0.785.0 ± 3.7

We observe that setting (λg=0\lambda_g = 0) has little impact on PADGen under IQL, but leads to a performance drop under TD3+BC. This observation is consistent with our discussion in Section 5.5 (Limitations): PADGen tends to provide greater benefits in challenging settings, while its improvements are less pronounced when the data quality is already high or the offline RL algorithm achieves strong performance without augmentation.

AuthorsOpenReview ↗

Response to Reviewer sqHW(Q3 and Q4)

Q3: Equation 6 can produce negative scores. What are the exact definitions and coefficients of fgatef_{\mathrm{gate}} and fdf_d, how are scores transformed into nonnegative sampling probabilities, and how is the high-return reference set constructed?

R3: As we discussed in Section 5.1, we instantiate fd(⋅)f_d(\cdot) in Eq. 5 with a simple distance-based formulation, and implement fgate(⋅)f_{\mathrm{gate}}(\cdot) in Eq. 1 as a predefined linear function with fixed coefficients.

For fdf_d, it is defined as:

[f_h(\mathbf{s});\mathbf{a}], [f_h(\mathbf{s}^e);\mathbf{a}^e] )=c\delta_a+(1-c)\delta_s, \qquad c=\exp\left(-\delta_s\right),$$ where $(\mathbf{s}^{e},\mathbf{a}^{e})$ is the reference state-action pair whose state representation is nearest to $f_h(\mathbf{s})$, and $\delta_s$ and $\delta_a$ are the normalized state and action distances. For $f_{\mathrm{gate}}$, it is defined as : $$w_t^j=f_{\mathrm{gate}}\left(s_t^s+\lambda_g s_t^p,w_t^{j-1}\right)=a w_t^{j-1}+ b \left(s_t^s+\lambda_g s_t^p\right), \qquad \lambda_g=0.5 ,$$ where $a$ and $b$ are the coefficients, and we fixed $a = 0.8$ and $b = 0.2$ in all the experiments. To ensure that $w_t^j$ is non-negative, we apply softmax normalization to $w_t^j$ in our implementation. Therefore, negative scores produced by Equation 6 do not affect the validity of the resulting sampling. $j$ indexes generation rounds, $t$ denotes the time step, $s_t^s$ and $s_t^p$ are its selection and policy feedback scores, respectively. Regarding the "high-return reference set", it consists of samples from the top 10% of the dataset in terms of return. **Q4**: The claim to be the first framework adapting generation to an evolving policy (lines 94--95) is too broad: MOPO already regenerates model rollouts under the current policy during optimization (Yu et al., 2020, Algorithm 2). **R4**: Thank you for your comment. However, PADGen differs from MOPO in two key aspects. First, MOPO is a model-based offline RL algorithm, whereas PADGen is a data augmentation framework designed to improve the performance of offline RL algorithms. In principle, the data generated by PADGen could also be used to train MOPO. Second, MOPO generates model rollouts using the current policy, whereas PADGen explicitly leverages policy training feedback, together with selector feedback, to guide the generator toward producing high-quality data that better meets the policy's evolving learning needs. Thus, while MOPO generates data by following the evolving policy, PADGen actively adapts data generation based on feedback from policy learning to better support the policy's evolving training needs.
AuthorsOpenReview ↗

Response to Reviewer sqHW

Dear Reviewer sqHW,

We have provided detailed responses to your concerns many days ago, we hope these responses have adequately addressed your concerns. As the discussion phase is coming to an end, we sincerely request your further responses.

If we have resolved your issues, please consider raising your score to the positive side. If you have any further questions, please feel free to share them with us! Any valuable feedback is crucial for improving our work.

Best regards,

The Authors

Reviewer_V8s8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the offline reinforcement learning (RL) problem, and proposes a framework for data augmentation called PADGen. The authors start by introducing the observation that changing the data distribution half way through an ordinary offline RL training could change the progress significantly, to motivate an adaptive paradigm for data augmentation, that takes feedback from the policy, and decides the training distribution for the next step. Then the paper introduces the PADGen paradigm. In particular, the selector takes in policy loss, and a similarity score to successful trajectories, and guides the generator to generate augmented data for the training in the next step. The generator, as well as the learning algorithm, could be any off-the-shelf algorithm. Finally, the paper conducts comprehensive experiments to verify the effectiveness of the proposed method.

Critical strengths

The experiment section is comprehensive. In particular, it includes three well-known learning algorithms as static data augmentation methods. I also appreciate that the authors are being honest about its limited improvements when the offline dataset contains more expert trajectories.

Critical weaknesses

My main criticism is that the problem is not well motivated. At a high level, the proposed method seems to be selecting the trajectories that have higher similarity to expert trajectories. It is not clear why this is favorable for all offline RL algorithms, and it is not clear that this is a major bottleneck to existing offline RL algorithms.

Questions for authors
  1. In figure 1, why are (a) and (b) different tasks with different interventions instead of same task with different interventions

  2. Why is not being adaptive a major bottleneck for existing augmentation methods for offline RL?

  3. Why should we favor trajectories that are closer to the expert trajectories regardless of the base algorithms?

  4. The selector seems to be only considering the similarity. Why is policy loss only present in generator instead of in selector?

4 comments
AuthorsOpenReview ↗

Response to Reviewer V8s8(Q1 and Q2)

Q1: The problem is not well motivated. In figure 1, why are (a) and (b) different tasks with different interventions instead of same task with different interventions?

R1: Thank you for your insightful comment. Our PADGen aims to adaptively generate training data according to the evolving needs of the policy at different stages of offline RL training. Our introduction is intended to motivate this central idea. However, we acknowledge that our example may not have effectively conveyed this motivation. We have updated the example in the introduction to better illustrate this. Specifically, we construct three subsets, HH, MM, and LL, by randomly sampling from Hopper-Medium,high-return trajectories in Hopper-Medium-Replay, and low-return trajectories in Hopper-Medium-Replay, respectively, with average trajectory returns satisfying RH>RM>RLR_H > R_M > R_L. We train policies on each subset for 50K steps. Then, we separately continue training the policies obtained from HH, MM, and LL using each of the two subsets (HH and MM) in the next 50K steps. We denote the six resulting configurations as HH, HM, MH, MM, LH, and LM, and compare them with a policy trained on the fixed mixture H∪M∪LH \cup M \cup L throughout the entire training process. The results are shown below:

ConfigurationData: 0–50KData: 50K–100KScore at 50KScore at 100K
HHHH45.05 ± 1.0857.22 ± 8.00
HMHM45.05 ± 1.0883.56 ± 15.05
MHMH28.46 ± 3.0855.34 ± 5.61
MMMM28.46 ± 3.0876.27 ± 9.84
LHLH20.71 ± 3.9358.24 ± 7.09
LMLM20.71 ± 3.9364.86 ± 11.46
Fixed mixtureH∪M∪LH\cup M\cup LH∪M∪LH\cup M\cup L40.20 ± 2.4359.00 ± 7.21

we observe that (1) training on the fixed mixture of all available data does not achieve the best performance, suggesting that keeping the data remaining static during training may offer limited benefits; (2) training on HH achieves the best performance in the first 50K steps, whereas continuing training on MM during the subsequent 50K steps yields the best overall performance, suggesting that data utility varies across training stages. Consequently, existing data generation methods that generate synthetic data in a one-shot manner without adapting to the policy's evolving data preferences may limit further policy improvement.

Q2: Why is not being adaptive a major bottleneck for existing augmentation methods for offline RL?

R2: Thank you for your question. As discussed in R1, training the policy in a static manner, i.e., using a fixed data, yields substantially lower performance than dynamically adjusting the training data (e.g., (HM) and (MM)). However, existing data augmentation methods typically follow a similar static training paradigm, where additional data are generated in a one-shot manner and then mixed with the original data to train the policy. These observations suggest that such static data augmentation strategies may not adequately accommodate the evolving data needs of the policy during training, motivating our policy-adaptive data generation.

AuthorsOpenReview ↗

Response to Reviewer V8s8(Q3 and Q4)

Q3: Why should we favor trajectories that are closer to the expert trajectories regardless of the base algorithms?

R3: We would like to clarify that PADGen does not solely favor trajectories that are closer to expert trajectories. As shown in Eq. (1), PADGen considers two factors when selecting seed data for generation: (1) the current learning status of the offline RL policy and (2) data quality, reflected by the similarity to expert trajectories. The former helps ensure that the generated data align with the evolving needs of the policy, while the latter helps maintain data quality and reduces the risk of misleading policy optimization.

Q4: The selector seems to be only considering the similarity. Why is policy loss only present in the generator instead of in the selector?

R4: Thank you for your question. We would like to clarify that the selector and policy loss serve different purposes in PADGen. The policy loss, together with feedback from the selector, guides the generator to produce high-quality data that meet the evolving needs of the policy. Here, the selector also serves as a data quality evaluator, providing quality scores to guide the generator. However, the generated data may not always meet these expectations. Therefore, the selector further filters the generated data based on their similarity to expert data to maintain data quality. Regarding why the selector only considers similarity to expert data, our experiments show that incorporating additional policy-related information into the selector leads to performance degradation:

GeneratorSelectorPerformance
sss^ssss^s78.9±6.978.9\pm6.9
sss^ssss^s and sps^p67.3±4.067.3\pm4.0
sss^s and sps^psss^s and sps^p68.0±5.268.0\pm5.2
sss^s and sps^psss^s85.0±3.7\mathbf{85.0}\pm3.7

Here, sss^s denotes the similarity with the expert data, while sps^p denotes the policy feedback (loss). We hypothesize that this is because policy feedback has already been incorporated into the generation process, and introducing it again during selection may overemphasize the current policy's preferences, potentially reducing data diversity and reinforcing existing policy biases.

AuthorsOpenReview ↗

Response to Reviewer V8s8

Dear Reviewer V8s8,

We have provided detailed responses to your concerns many days ago, we hope these responses have adequately addressed your concerns. As the discussion phase is coming to an end, we sincerely request your further responses.

If we have resolved your issues, please consider raising your score to the positive side. If you have any further questions, please feel free to share them with us! Any valuable feedback is crucial for improving our work.

Best regards,

The Authors

AuthorsOpenReview ↗

Response to Reviewer V8s8

Dear Reviewer V8s8,

Thank you for your time and constructive feedback. We provided detailed responses to your concerns several days ago and hope that they have adequately addressed your questions. As the discussion period is drawing to a close, we would greatly appreciate any further feedback you may have.

If our responses have addressed your concerns, we would be grateful if you would consider raising your score to the positive side accordingly. If any questions remain, please feel free to let us know. Your feedback is valuable and will help us improve our work.

Best regards,

The Authors

Reviewer_YZfKOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces, PADGen, a data augmentation framework for offline RL that generates data based on the ulitiy of feedback from a "selector" (which selects which data to augment) and the agent's current policy (which evolves throughout training). Experiments on D4RL benchmark tasks show that PADGen achieves larger average return than existing data augmentation baselines.

Critical strengths
  1. The research topic---can we improve offline RL performance by adapting data augmentation throughout policy optimization---is a good topic in my opinion. The introduction does a good job recognizing this gap in the literature.
  2. Empirical design is largely solid. The choice of benchmarks baselines are reasonable to me, and the ablation experiments were useful in showing how PADGen behaves with low-quality data.
  3. I appreciate the limitations section because it is very up-front with a limitation observed empirically. While I believe the reasoning for the observed performance degrading is not quite right (more on that in the Weaknesses section), it is very useful to me as a reader to have this information. It helps me better understand the algorithm.
Critical weaknesses

The results in Table 1 do not convincingly show that PADGen yields larger returns than baselines. In almost every experiment, PADGen's return +/- 1 stdev overlaps with the return +/- stdev of at least one baseline. The abstract states that "PADGen consistently outperforms existing data augmentation methods," and I do not think the experiments support this central claim. This the primary reason I vote reject.

It's unclear to me that adaptive data selection is an actual problem in the introduction example nor the main experiments. While I like this research question and do believe adaptive augmentation should be useful, the story in the introduction does not make a convincing argument that it should be useful. The intro experiment concludes

continued training on the original high-return data leads to performance degradation, whereas introducing randomly sampled data improves performance

but this observation may have nothing to do with adaptive data sampling. The new random data may be improving performance simply because it's increasing state-action coverage, which we know to be important in offline RL. What if you just trained the agent on all 20k samples for 1M steps instead? My hunch is that this approach---which uses static data and does NOT use adaptive data selection---would performance just as well. In this case, adaptive data selection is not the problem; the problem is simply not having enough data (e.g. not enough coverage, diversity, etc).

The paper does not illustrate or motivate at what kind of data we should be giving the agent as we adapt data selection. If adaptive data selection is important, the introduction should hint at some signal we should be using to adapt. Much later in the paper, it is revealed that PADGen seeks to generate high-quality data, where "quality refers to compatibility with the high-return reference patterns." In other words, try to generate data that resembles high-return trajectories. There are a few methods out there that already do this, like GuDA [1] and GTA [2] (and maybe more if we check out the related work in these papers). GTA is cited, but not included as a baseline, and I'm not sure why. I make these comments because (1) it would be useful to the reader if the paper motivated the data selection signal early on, and (2) other data augmentation methods already try to generate high-return augmented data, so the method itself is not particularly novel.

References

  1. Corrado et al. "Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning." RLC 2024. https://arxiv.org/abs/2310.18247
  2. Lee et al. GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning. Neurips 2024. https://arxiv.org/abs/2405.16907
Questions for authors
  1. While I do believe adaptive augmentation should be useful, I don't think paper in its current form makes a strong case for why it should be useful. Could the authors provide a carefully designed experiment that clearly illustrates its importance? This would be great motivation in the introduction. If the introduction experiment already shows this, can you explain how?

  2. Why did you choose these baselines specifically? Do any of these baselines try to generate high-quality augmented data like PADGen?

  3. Can you clarify why the results in Table 1 support the claim that "PADGen consistently outperforms existing data augmentation methods"?

6 comments
AuthorsOpenReview ↗

Response to Reviewer YZfK(Q1)

Q1: The results in Table 1 do not convincingly show that PADGen yields larger returns than baselines.

R1: Thank you for your comment. We acknowledge that although PADGen achieves substantial improvements in certain settings, such as HalfCheetah-Medium (DT) and Hopper-Medium (DT), its performance does overlap with that of the baselines in some other settings. However, we would like to clarify: (1) These results are reported on HalfCheetah, Hopper, and Walker2d, which are generally considered less challenging benchmark environments. We conduct experiments on the environment primarily to follow the experimental settings of prior studies [1, 2, 3], thereby enabling direct and fair comparisons with existing methods. (2) As discussed in Section 5.5 (Limitations), PADGen does not exhibit particularly pronounced advantages on datasets containing abundant high-quality trajectories. Instead, its advantages become more evident in challenging settings where the available data are of relatively low quality or contain a substantial proportion of low-quality trajectories. For example, as demonstrated in Section 5.4, PADGen achieves the highest mean return for both DT and IQL across all three random-data mixture ratios (25%, 50%, and 75%, the percentages indicate the proportion of random data.). To further validate this observation, we have extended these experiments to HalfCheetah and Walker2d with different proportions of random data. The results are presented below.

LearnerDatasetMethod25% random50% random75% random
DTHalfCheetah-medium-replayTGCVG49.6 ± 3.146.3 ± 8.740.6 ± 5.5
DTHalfCheetah-medium-replayPADGen50.9 ± 0.452.5 ± 1.046.0 ± 3.9
DTHopper-medium-replayTGCVG63.0 ± 11.350.0 ± 12.246.4 ± 8.8
DTHopper-medium-replayPADGen82.0 ± 1.970.0 ± 3.062.8 ± 2.9
DTWalker2d-medium-replayTGCVG64.6 ± 2.865.0 ± 7.066.1 ± 5.0
DTWalker2d-medium-replayPADGen65.7 ± 8.767.6 ± 6.869.8 ± 6.1
IQLHalfCheetah-medium-replayTGCVG37.0 ± 1.133.7 ± 0.827.0 ± 1.0
IQLHalfCheetah-medium-replayPADGen39.2 ± 1.640.4 ± 0.738.3 ± 2.6
IQLHopper-medium-replayTGCVG18.5 ± 7.76.8 ± 5.014.0 ± 15.8
IQLHopper-medium-replayPADGen91.2 ± 7.790.8 ± 2.790.4 ± 2.4
IQLWalker2d-medium-replayTGCVG38.3 ± 5.838.4 ± 9.036.3 ± 14.4
IQLWalker2d-medium-replayPADGen49.0 ± 1.061.0 ± 10.355.5 ± 4.2

We additionally conduct experiments on Maze and AntMaze, the challenge environments require long-horizon navigation under sparse constraints. The results below show the effectiveness of PADGen(all experiments conducted based on IQL).

DatasetBaselineSERGTATGCVGPADGen
Maze2D-umaze37.4 ± 2.839.0 ± 2.341.7 ± 1.453.8 ± 7.591.8 ± 3.3
Maze2D-medium32.8 ± 1.534.3 ± 2.537.8 ± 1.748.4 ± 3.069.7 ± 12.4
Maze2D-large59.0 ± 9.261.7 ± 4.576.6 ± 4.7104.0 ± 12.997.9 ± 11.3
AntMaze-umaze58.8 ± 8.917.1 ± 6.566.5 ± 6.941.2 ± 4.769.0 ± 4.9
AntMaze-medium-play78.1 ± 3.441.0 ± 20.681.9 ± 4.283.2 ± 2.190.9 ± 1.2
AntMaze-large-play40.6 ± 8.837.5 ± 6.544.4 ± 4.755.2 ± 6.778.3 ± 9.5
AntMaze-umaze-diverse50.4 ± 17.423.9 ± 11.857.9 ± 9.557.2 ± 4.438.7 ± 7.5
AntMaze-medium-diverse65.5 ± 9.540.9 ± 14.278.1 ± 7.984.6 ± 4.392.0 ± 0.9
AntMaze-large-diverse45.8 ± 6.337.5 ± 8.447.8 ± 6.757.2 ± 4.372.5 ± 4.6
Average52.037.059.265.077.9

In these case, we observe that PADGen demonstrates a more pronounced advantage.

AuthorsOpenReview ↗

Response to Reviewer YZfK(Q2 and Q3)

Q2: It's unclear to me that adaptive data selection is an actual problem in the introduction example nor the main experiments. Could the authors provide a carefully designed experiment that clearly illustrates its importance? 

R2: Thank you for your insightful comment. PADGen aims to adaptively generate training data according to the evolving needs of the policy at different stages of offline RL training, rather than relying on conventional one-shot data generation. Both our introduction and experiments are intended to motivate and validate this central idea. However, we acknowledge that your concern is well-founded: the motivating example in the original introduction does not adequately demonstrate the importance of policy-adaptive data generation. Following your suggestion, we have conducted a new experiment to better illustrate this and visualized the motivating example in the introduction. Specifically, we construct three subsets, HH, MM, and LL, by randomly sampling from Hopper-Medium,high-return trajectories in Hopper-Medium-Replay, and low-return trajectories in Hopper-Medium-Replay, respectively, with average trajectory returns satisfying RH>RM>RLR_H > R_M > R_L. We train policies on each subset for 50K steps. Then, we separately continue training the policies obtained from HH, MM, and LL using each of the two subsets (HH and MM) in the next 50K steps. We denote the six resulting configurations as HH, HM, MH, MM, LH, and LM, and compare them with a policy trained on the fixed mixture H∪M∪LH \cup M \cup L throughout the entire training process. The results are shown below:

ConfigurationData: 0–50KData: 50K–100KScore at 50KScore at 100K
HHHH45.05 ± 1.0857.22 ± 8.00
HMHM45.05 ± 1.0883.56 ± 15.05
MHMH28.46 ± 3.0855.34 ± 5.61
MMMM28.46 ± 3.0876.27 ± 9.84
LHLH20.71 ± 3.9358.24 ± 7.09
LMLM20.71 ± 3.9364.86 ± 11.46
Fixed mixtureH∪M∪LH\cup M\cup LH∪M∪LH\cup M\cup L40.20 ± 2.4359.00 ± 7.21

We can observe that (1) training on the fixed mixture of all available data does not achieve the best performance, suggesting that keeping the data remaining static during training may offer limited benefits; (2) training on HH achieves the best performance in the first 50K steps, whereas continuing training on MM during the subsequent 50K steps yields the best overall performance, suggesting that data utility varies across training stages. Consequently, existing data generation methods that generate synthetic data in a one-shot manner without adapting to the policy's evolving data preferences may limit further policy improvement.

Q3: The paper does not illustrate or motivate at what kind of data we should be giving the agent as we adapt data selection.

R3: We would like to clarify that PADGen aims to dynamically generate data according to the evolving needs of the policy, rather than merely selecting existing data. As discussed in Section 4.1, PADGen considers both data quality and policy needs, rather than focusing solely on high-quality data. Specifically, PADGen first selects seed samples for augmentation based on feedback from the policy and scores provided by the selector, and then synthesizes new training data from these samples. Here, the policy feedback reflects the policy's current learning needs, while the selector primarily evaluates data quality.

Regarding which types of data should be selected or generated, our experiments suggest that the answer varies across environments and different stages of policy training. For example, in the Hopper environment, the policy tends to benefit more from high-quality data during the early stages of training, whereas its preference shifts toward other types of data in later stages.

AuthorsOpenReview ↗

Response to Reviewer YZfK(Q4)

Q4: Why did you choose these baselines specifically? Do any of these baselines try to generate high-quality augmented data like PADGen?

R4: To the best of our knowledge, many recent offline RL data augmentation methods focus on generating high-quality data. Among our baselines, DiffStitch and TGCVG are both designed for this purpose. Specifically, DiffStitch generates high-return trajectories by using diffusion models to stitch low-return trajectories with high-return ones, while TGCVG generates high-quality trajectories through conservative value-guided policy learning and a learned dynamics model. We include SER and TATU as baselines following DiffStitch, in which SER represents methods that conduct transition synthesis, TATU represents methods that conduct model-based rollouts. We select DiffStitch and TGCVG because they represent two recent approaches to high-quality data generation: diffusion-based trajectory stitching and conservative value-guided trajectory generation, respectively. Following your suggestion, we also compare our method with GTA, the results are presented in below, we can observe that PADGen achieves better performance than GTA.

Method + AlgorithmHalfCheetah-mediumHopper-mediumWalker2d-mediumHalfCheetah-medium-replayHopper-medium-replayWalker2d-medium-replayAverage
GTA + IQL55.1 ± 0.252.3 ± 6.471.1 ± 10.947.1 ± 0.885.5 ± 9.974.4 ± 6.064.3
PADGen + IQL67.2 ± 0.480.4 ± 6.988.9 ± 0.459.5 ± 0.6102.3 ± 0.796.0 ± 1.282.4
GTA + TD3+BC58.7 ± 0.561.2 ± 3.389.6 ± 2.349.3 ± 0.577.2 ± 6.890.3 ± 2.671.1
PADGen + TD3+BC67.6 ± 0.284.0 ± 1.885.9 ± 0.459.0 ± 0.285.0 ± 3.793.5 ± 0.879.1
GTA + DT43.4 ± 1.157.9 ± 6.168.5 ± 6.338.3 ± 3.052.4 ± 5.050.9 ± 6.151.9
PADGen + DT62.5 ± 0.285.0 ± 1.769.6 ± 0.835.7 ± 10.491.6 ± 4.167.2 ± 3.768.6

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Program Chairs, Area Chair, and Reviewers,

We would like to express our sincere gratitude to all reviewers for their time, careful reading, and constructive feedback. The discussion has helped us substantially improve the motivation, empirical focus, and presentation of the paper. We are particularly encouraged that Reviewer YZfK found the revised motivating experiment convincing and updated the assessment, and that Reviewer sqHW found the clarification of matched generation and policy-update budgets and the additional ablations useful and updated the assessment. We are also grateful to Reviewer V8s8 for the thoughtful questions on motivation and selector design, which prompted the new controlled experiment and the additional selection–feedback analysis. Although the discussion period drew to a close before we received further comments from Reviewer V8s8, we respectfully believe that these responses and the corresponding revisions address the substantive questions raised.

The reviewers recognized the importance of the research question, the solid empirical design, the low-quality-data analysis, and the comprehensive experimental evaluation. The main responses and revisions are summarized below. All changes are marked in blue in the revised paper:

  1. We strengthened the paper's motivation with a new controlled experiment. We replaced the original motivating example with a same-task experiment using different data subsets across consecutive 50K training stages. The results show that a fixed mixture does not achieve the best overall performance, while the most effective second-stage data differ from those that are most effective initially. This provides direct evidence that data utility evolves during policy optimization.

  2. We refined the paper's main performance claims. We revised the abstract, introduction, and conclusion to emphasize PADGen's effectiveness across offline RL benchmarks and its particularly pronounced advantages in challenging settings with substantial low-quality data. The revised claims are now more closely aligned with the empirical findings.

  3. We expanded the main empirical comparisons by adding GTA. GTA is now included as a baseline for IQL, TD3+BC, and DT on the MuJoCo benchmarks. This directly addresses the question of whether methods designed to generate high-return augmented data are adequately represented in our evaluation.

  4. We extended the low-quality-data study across tasks and learners. The random-data mixture experiments now cover HalfCheetah, Hopper, and Walker2d with both DT and IQL under three random-data proportions. These experiments show that PADGen's advantage becomes more pronounced as the proportion of low-quality data increases.

  5. We added new navigation and manipulation experiments, together with additional ablation analyses. The new experiments in Appendix A.3 evaluate PADGen on Maze2D, AntMaze, and Adroit. We further added experiments that decouple selection and feedback signals, isolate the role of the feedback pathway, and analyze the individual feedback signals on AntMaze-large-play.

  6. We clarified the method and evaluation protocol. We clarified the respective roles of the selector and policy feedback, the reason for using policy loss in generation rather than candidate selection, the matched generation and policy-update budgets, and the task shown in Figure 4. We further added an experiment showing that incorporating policy scores into candidate selection reduces performance.

We sincerely thank all reviewers again for their valuable comments and suggestions, which have significantly improved the clarity, rigor, and empirical grounding of our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.