- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a target trajectory generator based on the diffusion model by identifying the potential problem of reinforcement redundancy when using her for learning. It also employs contrastive learning to optimize the strategy, significantly improving the task performance and sample efficiency.
Critical strengths
This paper is well-written with clear logic, and the experimental process is thorough. It provides the corresponding code and tests the proposed GEAR method in real-world scenarios. The success rate and sample efficiency have significantly improved.
Critical weaknesses
My main concern regarding this article stems from the novelty of the method and its comparison with existing methods.
- Although the method design of GEAR is quite reasonable and the logic is correct, the contrast strategy learning and the data augmentation method based on diffusion models that it uses are already existing methods. Therefore, the components of GEAR belong to a combination of existing methods, and the authors did not mention any differences from existing works [1, 2, 3].
[1] Contrastive learning as goal-conditioned reinforcement learning. NIPS, 2022.
[2] Gta: Generative trajectory augmentation with guidance for offline reinforcement learning. NIPS, 2024.
[3] Goal-Conditioned Data Augmentation for Offline Reinforcement Learning. TMLR, 2025.
-
The author states that HER will repeatedly sample the same trajectory, leading to the reinforcement of repetitive behavior. However, there is no actual experimental verification, so it is unknown whether the description is accurate.
-
The data augmentation methods discussed in the article and the comparison with the variants of HER are too outdated, making it difficult to determine their contributions.
-
The ablation experiments were insufficient, for instance, in the optimization function, the hyperparameter was not properly set.
Questions for authors
-
The author has insufficiently conducted a review of relevant literature. Could the author provide more novel baselines to demonstrate their contributions? For example, the related work on HER, the related work on data augmentation, and the related work on generating diverse actions.
-
If I haven't misunderstood, GEAR can also be compared with model-based goal-conditioned data augmentation methods, because they both essentially involve generating high-quality data using different approaches. So, why didn't the authors compare it with related works such as Go-plan or synthetic data methods [1, 2]?
[1] Looking backward: Retrospective backward synthesis for goal-conditioned gflownets. ICLR, 2025.
[2] Temporal distance-aware transition augmentation for offline model-based reinforcement learning. ICML, 2025.
-
GEAR employs diffusion to generate successful trajectories, and then enhances the diversity of actions. But how can we ensure the quality of the generated trajectories?
-
Has the author considered using other generation models to generate successful trajectories, such as the Generative Flow Networks (GFlowNets)? Because I believe that diffusion not only involves a large amount of computation, but also the generated quality may not be better than that of flows.
-
Since the author employed a contrast optimization strategy, can the author present the comparison results of JaxGCRL against CRL and TD-InfoNCE on the standard?
JaxGCRL: Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Research. ICLR, 2025.
CRL: Contrastive learning as goal-conditioned reinforcement learning. NIPS, 2022.
TD-InfoNCE: Contrastive difference predictive coding. ICLR, 2024.
6 comments
To clarify the remaining implementation and literature details on W1 and Q1-Q4 beyond our response above, we provide the following additions, which we will incorporate into the revised manuscript:
Q3: Generator Architecture, Inputs, and Exact Usage
Embedding-space DDPMs (Sec. 3.1, App. C.4)
Both and output a goal vector conditioned on the current state , trained on within-trajectory pairs matching with the goal representation of a later state (). Generated goals enter only the training-time critic and value-gap computations (Eqs. 4, 6-11) and are not used for multi-step dynamics rollouts.
W1, W3, Q1 & Q2: Remaining Clarifications (CRL, GoPlan, W3, Diverse Actions)
Contrast with CRL and GoPlan, recent baselines, and action diversity (Sec. 5, L456-475)
Beyond the augmentation contrast already detailed above for [3-6]: (i) unlike contrastive GCRL (CRL (Eysenbach et al., 2022), TD-InfoNCE [1], JaxGCRL [2]) that fits occupancy similarity via InfoNCE, GEAR retains soft Bellman backups (Eqs. 7-10) and contrasts dual-critic values (); (ii) unlike GoPlan (Wang et al., 2024a), which plans model-based rollouts with a learned dynamics model, GEAR is model-free; and (iii) on W3 and Q1, Sec. 5 and our experiments already include 2023-2025 diffusion/generative baselines (Decision Diffuser, CEP, SynthER, CPQL, QVPO, PGR), while action diversity is driven by the SAC entropy bonus (Eqs. 7-8) and conditional MI (Eq. 13). We will expand Sec. 5 to cover [1-6].
Q3 & Q4: Reverse-Sampling Cost and Entropy Evidence
Per-goal latency and entropy curve (App. C.4, Fig. 4)
Sub-sampling reverse steps takes ms per goal during training actor updates (App. C.4), whereas at test time only is executed. Fig. 4 also shows redundant-goal entropy increasing over training alongside Table 4's runtime comparison.
References
[1] Contrastive difference predictive coding. ICLR, 2024.
[2] Accelerating goal-conditioned reinforcement learning algorithms and research. ICLR, 2025.
[3] GTA: Generative trajectory augmentation with guidance for offline reinforcement learning. NeurIPS, 2024.
[4] Goal-conditioned data augmentation for offline reinforcement learning. TMLR, 2025.
[5] Looking backward: Retrospective backward synthesis for goal-conditioned GFlowNets. ICLR, 2025.
[6] Temporal distance-aware transition augmentation for offline model-based reinforcement learning. ICML, 2025.
Thank you for the author's response. Firstly, I encourage the author to present the new results and modifications in the article, such as the curve graph comparing JaxGCRl with CRL and TD-InFoNCE in the JaxGCRl environment. Secondly, I am still concerned about the novelty. Besides contrastive loss optimization, diffusion has already been applied in GCRL, for example, Learning to Reach Goals via Diffusion, ICML 2024. Generative trajectory stitching through diffusion composition, NIPS 2025. Finally, has the author considered other generative models to generate trajectories? For example, flow matching and drift models? Perhaps their computational cost is not high and the quality might be quite good.
We thank Reviewer fSFF for the insightful comments. We will include the discussion in the revision.
For Q1 Q2: We will cite CRL [NIPS2022], Gta [NIPS2024], GODA [TMLR2025], GoPlan [TMLR2024], Retrospective GFlowNets [ICLR25], and Temporal Distance-Aware Augmentation [ICML25] in Sec. 5. They differ in two ways:
To verify gains come from this closed-loop design rather than the D+/D− split alone, we tested three non-generative variants on the identical D+/D− split on Habitat Office (5 seeds):
Three results verify this:
The MI term Iϕ(a;g+∣s) (weighted by β in Eq. 13, L316; implemented with a conditional InfoNCE bound over M goals sampled from Gψ+, App. C.6, L1432) encourages goal-dependent actions, and Thm. 2 (L324; App. A.3, L841) shows that adding it preserves an ascent direction for Jgap under the stated alignment condition. Table 2 (w/o MI, β=0) ablates β across all 3 scenes, showing drops of −2.1% (
Residential), −2.6% (Office), and −2.6% (Commercial). For β∈{0,0.05,0.10,0.20} (search space in App. C.2 Table 10, L1253) on Office, GEAR reaches 76.3%, 77.5%, 78.9% (β=0.10), and 77.6%.For Q3: GEAR generates single goal states g+,g−∈G (not rollouts), avoiding compounding error. Anchoring to D+,D− via LDDPM (Eq. 3, L204) encourages valid goals: on Office, 89.3% of g+∼Gψ+(st) are reachable with 0.84 efficiency (vs. 53.7%,0.56 for Gψ−). Tracking confusion CQ=Prg−[Qθ(s,a∣g−)>τ] and entropy H(Gψ−) across training shows CQ drops (0.42→0.12 on Office; 0.47→0.16 on Cheetah-Run) while H(Gψ−) rises (1.8→3.9; 1.3→3.4).
For Q4: Unlike GFlowNets (for discrete DAGs), diffusion models continuous goals in the encoded feature space of high-dimensional visual observations (RGB-D in Habitat, App. C.3). Replacing diffusion with simpler goal sampling (w/o Diff, Table 2) drops success by -16.5% to -25.2%. Cost is O(2Nrev) (Nrev=8) in training, and the generators are not needed at deployment (only πϕ(a∣s,g) runs). In Table 4 (L432--435), training takes 816 min, close to QVPO (774m) and DD (786m).
Under matched budgets, we compare GEAR with all three: