- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PADGen augments offline RL data during policy training. It keeps a pretrained TGCVG generator fixed and changes which trajectory segments provide its generation contexts, using feedback from policy fitting loss and a data selector. The selector compares state changes and state-action patterns with high-return reference segments, then samples data for policy training. The contribution is this feedback driven generation and selection loop.
The evaluation covers nine D4RL locomotion datasets with DT, IQL, and TD3+BC, with five-seed results, component ablations, and experiments mixing in random data. Results improve on medium and medium-replay datasets, but benefits are less consistent on medium-expert data. The authors report these failures, including a TD3+BC decrease from 89.9 to 42.1 on HalfCheetah-medium-expert.
Critical strengths
-
There are meaningful gains across different learners. Table 1 reports the highest mean return in 14 of 18 settings over five seeds, with average improvements of 4.1, 2.1, and 6.3 points over the strongest reported augmentation baseline for IQL, TD3+BC, and DT, respectively. The DT gains on HalfCheetah-medium and Hopper-medium are substantial, supporting the practical value of the combined method.
-
The degraded-data results identify a useful application. With 75% random data, IQL reaches 90.4 +/- 2.4, compared with 71.3 +/- 14.3 without augmentation and 69.7 +/- 9.9 with selection alone (Table 2). This supports combining generation with selection when the available data contain many poor trajectories.
Critical weaknesses
-
The experiments do not adequately isolate policy adaptation. Section 5.3 does not establish matched generated-transition counts or policy-update budgets between one-shot and repeated generation, while -Feedback removes both selector and policy feedback. Appendix A.1 does include , but Figure 4 does not identify its environment or dataset, IQL shows little final-performance separation, and this policy-specific control is not reported across the main evaluation.
-
Important sampling details are missing. Section 5.1 describes as linear and as distance based without specifying their formulas, coefficients, or scaling. Equation 6 can produce negative scores, yet the transformation into valid sampling probabilities and the construction of the high-return reference set are unspecified.
-
The novelty claim is overstated. The claim to be the first framework adapting generation to an evolving policy (lines 94–95) is too broad: MOPO already regenerates model rollouts under the current policy during optimization (Yu et al., 2020, Algorithm 2).
Questions for authors
- What are the exact definitions and coefficients of and , how are scores transformed into nonnegative sampling probabilities, and how is the high-return reference set constructed?
- What generation schedule, rollout length, total generated-transition count, policy-update count, and original/synthetic data ratio are used, and which quantities are matched across PADGen, TGCVG, and the ablations?
- Which task is shown in Figure 4, and can you report the variant across the main tasks while retaining selector feedback and matching sample and optimization budgets?
5 comments
Response to Reviewer sqHW(Q3 and Q4)
Q3: Equation 6 can produce negative scores. What are the exact definitions and coefficients of and , how are scores transformed into nonnegative sampling probabilities, and how is the high-return reference set constructed?
R3: As we discussed in Section 5.1, we instantiate in Eq. 5 with a simple distance-based formulation, and implement in Eq. 1 as a predefined linear function with fixed coefficients.
For , it is defined as:
[f_h(\mathbf{s});\mathbf{a}], [f_h(\mathbf{s}^e);\mathbf{a}^e] )=c\delta_a+(1-c)\delta_s, \qquad c=\exp\left(-\delta_s\right),$$ where $(\mathbf{s}^{e},\mathbf{a}^{e})$ is the reference state-action pair whose state representation is nearest to $f_h(\mathbf{s})$, and $\delta_s$ and $\delta_a$ are the normalized state and action distances. For $f_{\mathrm{gate}}$, it is defined as : $$w_t^j=f_{\mathrm{gate}}\left(s_t^s+\lambda_g s_t^p,w_t^{j-1}\right)=a w_t^{j-1}+ b \left(s_t^s+\lambda_g s_t^p\right), \qquad \lambda_g=0.5 ,$$ where $a$ and $b$ are the coefficients, and we fixed $a = 0.8$ and $b = 0.2$ in all the experiments. To ensure that $w_t^j$ is non-negative, we apply softmax normalization to $w_t^j$ in our implementation. Therefore, negative scores produced by Equation 6 do not affect the validity of the resulting sampling. $j$ indexes generation rounds, $t$ denotes the time step, $s_t^s$ and $s_t^p$ are its selection and policy feedback scores, respectively. Regarding the "high-return reference set", it consists of samples from the top 10% of the dataset in terms of return. **Q4**: The claim to be the first framework adapting generation to an evolving policy (lines 94--95) is too broad: MOPO already regenerates model rollouts under the current policy during optimization (Yu et al., 2020, Algorithm 2). **R4**: Thank you for your comment. However, PADGen differs from MOPO in two key aspects. First, MOPO is a model-based offline RL algorithm, whereas PADGen is a data augmentation framework designed to improve the performance of offline RL algorithms. In principle, the data generated by PADGen could also be used to train MOPO. Second, MOPO generates model rollouts using the current policy, whereas PADGen explicitly leverages policy training feedback, together with selector feedback, to guide the generator toward producing high-quality data that better meets the policy's evolving learning needs. Thus, while MOPO generates data by following the evolving policy, PADGen actively adapts data generation based on feedback from policy learning to better support the policy's evolving training needs.Response to Reviewer sqHW
Dear Reviewer sqHW,
We have provided detailed responses to your concerns many days ago, we hope these responses have adequately addressed your concerns. As the discussion phase is coming to an end, we sincerely request your further responses.
If we have resolved your issues, please consider raising your score to the positive side. If you have any further questions, please feel free to share them with us! Any valuable feedback is crucial for improving our work.
Best regards,
The Authors
Response to Reviewer sqHW(Q1 and Q2)
Q1: Section 5.3 does not establish matched generated-transition counts or policy-update budgets between one-shot and repeated generation, while -Feedback removes both selector and policy feedback.
R1: Thanks for your comment. In the section of the main results(5.2) and the ablation study(5.3), the one-shot generation methods, such as TGCVG, generate a fixed package before training, whereas PADGen generates data periodically using feedback-adjusted contexts. However, the policy-update count and generated quantities for both methods are identical to ensure a fair comparison.
Regarding the -Feedback ablation, the generator in PADGen receives two complementary feedback signals: policy feedback, and selector feedback, which evaluates the quality of generated data. Together, they form the feedback mechanism that guides data generation. Therefore, the original -Feedback ablation removes both signals to evaluate the effectiveness of the overall feedback mechanism, rather than the individual contribution of policy feedback.
Nevertheless, we agree that isolating the contribution of policy feedback is also important. We have therefore conducted additional experiments across multiple challenging settings by removing only policy feedback, while retaining selector feedback. The results are shown below, where 25% and 50% denote the proportions of random data mixed into the original HalfCheetah-Medium-Replay dataset.
These results demonstrate the importance of policy feedback in guiding data generation and improving policy performance in most cases.
Q2: Which task is shown in Figure 4?
R2: Thank you for your comment. Figure 4 presents the results on Hopper-Medium-Replay. The comparison between PADGen and the two settings with (λg=0) is shown below.
We observe that setting (λg=0) has little impact on PADGen under IQL, but leads to a performance drop under TD3+BC. This observation is consistent with our discussion in Section 5.5 (Limitations): PADGen tends to provide greater benefits in challenging settings, while its improvements are less pronounced when the data quality is already high or the offline RL algorithm achieves strong performance without augmentation.