acceptodds
Under review as a conference paper at ICLR 2027

GEAR: Generative Efficient Gap Amplification for Goal-Conditioned Reinforcement Learning

Abstract

In goal-conditioned reinforcement learning (GCRL), trajectories that successfully reach the same goal can differ substantially in behavioral efficiency. Existing work on goal relabeling and generative sampling improves data reuse, but pays limited attention to how efficiently goals are reached, potentially reinforcing redundant behaviors when learning from successful experience. We propose GEAR (Generative Efficient Gap Amplification for GCRL), a generative framework that guides policy learning by contrasting efficient and redundant goal distributions. GEAR trains two diffusion-based goal generators through weighted sampling to reflect trajectory efficiency while retaining useful information from longer successful trajectories. The efficient generator emphasizes goals from trajectories that successfully reach the goal in fewer steps; the redundant generator emphasizes goals from longer successful and failed trajectories and is adversarially trained to produce plausible distractors. A dual-critic architecture evaluates the same state-action pair under both goal conditions, and the resulting value gap serves as an auxiliary contrastive signal that encourages the actor to favor actions that better support efficient goals while providing less support for redundant goals. Experiments on DMC, ManiSkill2, and Habitat show improved task performance and sample efficiency. On a physical robotic arm, GEAR achieves 90%, 75%, and 80% success on object lifting, block stacking, and target grasping among distractors, respectively, outperforming all evaluated baselines on each task.

Primary area:
reinforcement learning
Keywords:
Goal-Conditioned Reinforcement Learning, Efficient Gap.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fSFFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a target trajectory generator based on the diffusion model by identifying the potential problem of reinforcement redundancy when using her for learning. It also employs contrastive learning to optimize the strategy, significantly improving the task performance and sample efficiency.

Critical strengths

This paper is well-written with clear logic, and the experimental process is thorough. It provides the corresponding code and tests the proposed GEAR method in real-world scenarios. The success rate and sample efficiency have significantly improved.

Critical weaknesses

My main concern regarding this article stems from the novelty of the method and its comparison with existing methods.

  1. Although the method design of GEAR is quite reasonable and the logic is correct, the contrast strategy learning and the data augmentation method based on diffusion models that it uses are already existing methods. Therefore, the components of GEAR belong to a combination of existing methods, and the authors did not mention any differences from existing works [1, 2, 3].

[1] Contrastive learning as goal-conditioned reinforcement learning. NIPS, 2022.

[2] Gta: Generative trajectory augmentation with guidance for offline reinforcement learning. NIPS, 2024.

[3] Goal-Conditioned Data Augmentation for Offline Reinforcement Learning. TMLR, 2025.

  1. The author states that HER will repeatedly sample the same trajectory, leading to the reinforcement of repetitive behavior. However, there is no actual experimental verification, so it is unknown whether the description is accurate.

  2. The data augmentation methods discussed in the article and the comparison with the variants of HER are too outdated, making it difficult to determine their contributions.

  3. The ablation experiments were insufficient, for instance, in the optimization function, the hyperparameter β\beta was not properly set.

Questions for authors
  1. The author has insufficiently conducted a review of relevant literature. Could the author provide more novel baselines to demonstrate their contributions? For example, the related work on HER, the related work on data augmentation, and the related work on generating diverse actions.

  2. If I haven't misunderstood, GEAR can also be compared with model-based goal-conditioned data augmentation methods, because they both essentially involve generating high-quality data using different approaches. So, why didn't the authors compare it with related works such as Go-plan or synthetic data methods [1, 2]?

[1] Looking backward: Retrospective backward synthesis for goal-conditioned gflownets. ICLR, 2025.

[2] Temporal distance-aware transition augmentation for offline model-based reinforcement learning. ICML, 2025.

  1. GEAR employs diffusion to generate successful trajectories, and then enhances the diversity of actions. But how can we ensure the quality of the generated trajectories?

  2. Has the author considered using other generation models to generate successful trajectories, such as the Generative Flow Networks (GFlowNets)? Because I believe that diffusion not only involves a large amount of computation, but also the generated quality may not be better than that of flows.

  3. Since the author employed a contrast optimization strategy, can the author present the comparison results of JaxGCRL against CRL and TD-InfoNCE on the standard?

JaxGCRL: Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Research. ICLR, 2025.

CRL: Contrastive learning as goal-conditioned reinforcement learning. NIPS, 2022.

TD-InfoNCE: Contrastive difference predictive coding. ICLR, 2024.

6 comments

We thank Reviewer fSFF for the insightful comments. We will include the discussion in the revision.

W1 W3 Q1 Q2: Contrastive & Augmentation Methods

For Q1 Q2: We will cite CRL [NIPS2022], Gta [NIPS2024], GODA [TMLR2025], GoPlan [TMLR2024], Retrospective GFlowNets [ICLR25], and Temporal Distance-Aware Augmentation [ICML25] in Sec. 5. They differ in two ways:

  1. Gta, GODA, Retrospective GFlowNets, and Temporal Distance-Aware Augmentation synthesize transitions (st,at,st+1)(s_t, a_t, s_{t+1}) or rollouts to expand buffers for a single critic Q(s,a∣g)Q(s,a|g). GEAR never synthesizes trajectories; it trains Gψ+,Gψ−G_\psi^+, G_\psi^- to model goal-condition distributions (g+,g−)(g^+, g^-) anchored at real states sts_t.
  2. Static HER relabeling or single-critic filtering only reweights buffer goals. As QθQ_\theta updates, Gψ−G_\psi^- tracks value overestimation on redundant regions (Eq. 4, L235), and actor training subtracts Qˉω−(s,a)\bar{Q}_\omega^-(s,a) via ΔQ=Qθ(s,a∣g+)−Eg−[Qω(s,a∣g−)]\Delta Q = Q_\theta(s,a|g^+) - \mathbb{E}_{g^-}[Q_\omega(s,a|g^-)] (Eqs. 11-12, L295), penalizing actions with high value on redundant goals.

To verify gains come from this closed-loop design rather than the D+/D−\mathcal{D}^+/\mathcal{D}^- split alone, we tested three non-generative variants on the identical D+/D−\mathcal{D}^+/\mathcal{D}^- split on Habitat Office (5 seeds):

Variant on Identical D+/D−\mathcal{D}^+/\mathcal{D}^- SplitSuccess (%)SPL
Direct achieved-goal relabeling from D+/D−\mathcal{D}^+/\mathcal{D}^- (no Gψ±,QωG_\psi^\pm, Q_\omega)58.9±3.8%58.9\pm3.8\%0.49±0.030.49\pm0.03
Single-critic filtering (relabel from D+\mathcal{D}^+, discard D−\mathcal{D}^-)63.1±3.6%63.1\pm3.6\%0.54±0.030.54\pm0.03
Heuristic gap-prioritized replay (reweight goals by QθQ_\theta error)67.2±3.5%67.2\pm3.5\%0.56±0.030.56\pm0.03
GEAR w/o Gap (Table 2, diffusion goals without gap-guided actor update)58.4±3.7%58.4\pm3.7\%0.51±0.030.51\pm0.03
Full GEAR (Ours)78.9±3.2%78.9\pm3.2\%0.68±0.020.68\pm0.02

W2: Verification of HER's Repetitive Behavior

Three results verify this:

  1. On Office (Table 2), HER reaches 52.4%52.4\% success but 0.420.42 SPL (vs. GEAR's 0.680.68), taking +38.2%+38.2\% more steps on successful rollouts due to corner looping.
  2. In Target Grasping (App. Table 6, L1067), HER's relabeling reinforces Wrong-Target grasps in 30%30\% of trials (6/20) when colliding with distractors, whereas GEAR cuts this to 5%5\% (1/20).
  3. Above, directly relabeling achieved states from D+/D−\mathcal{D}^+/\mathcal{D}^- gives 58.9%58.9\% success (0.490.49 SPL) vs. Full GEAR's 78.9%78.9\% (0.680.68 SPL).

W4: Hyperparameter β\beta Setting and Ablation

The MI term Iϕ(a;g+∣s)I_\phi(a; g^+ | s) (weighted by β\beta in Eq. 13, L316; implemented with a conditional InfoNCE bound over MM goals sampled from Gψ+G_\psi^+, App. C.6, L1432) encourages goal-dependent actions, and Thm. 2 (L324; App. A.3, L841) shows that adding it preserves an ascent direction for JgapJ_{\text{gap}} under the stated alignment condition. Table 2 (w/o MI, β=0\beta=0) ablates β\beta across all 3 scenes, showing drops of −2.1%-2.1\% (Residential), −2.6%-2.6\% (Office), and −2.6%-2.6\% (Commercial). For β∈{0,0.05,0.10,0.20}\beta \in \{0, 0.05, 0.10, 0.20\} (search space in App. C.2 Table 10, L1253) on Office, GEAR reaches 76.3%76.3\%, 77.5%77.5\%, 78.9%78.9\% (β=0.10\beta=0.10), and 77.6%77.6\%.

Q3 & Q4: Sample Quality, Generative Choice, Cost

For Q3: GEAR generates single goal states g+,g−∈Gg^+, g^- \in \mathcal{G} (not rollouts), avoiding compounding error. Anchoring to D+,D−\mathcal{D}^+, \mathcal{D}^- via LDDPM\mathcal{L}_{\text{DDPM}} (Eq. 3, L204) encourages valid goals: on Office, 89.3%89.3\% of g+∼Gψ+(st)g^+ \sim G_\psi^+(s_t) are reachable with 0.840.84 efficiency (vs. 53.7%,0.5653.7\%, 0.56 for Gψ−G_\psi^-). Tracking confusion CQ=Pr⁡g−[Qθ(s,a∣g−)>τ]C_Q = \Pr_{g^-}[Q_\theta(s,a|g^-) > \tau] and entropy H(Gψ−)H(G_\psi^-) across training shows CQC_Q drops (0.42→0.120.42 \to 0.12 on Office; 0.47→0.160.47 \to 0.16 on Cheetah-Run) while H(Gψ−)H(G_\psi^-) rises (1.8→3.91.8 \to 3.9; 1.3→3.41.3 \to 3.4).

For Q4: Unlike GFlowNets (for discrete DAGs), diffusion models continuous goals in the encoded feature space of high-dimensional visual observations (RGB-D in Habitat, App. C.3). Replacing diffusion with simpler goal sampling (w/o Diff, Table 2) drops success by -16.5% to -25.2%. Cost is O(2Nrev)O(2N_{\text{rev}}) (Nrev=8N_{\text{rev}}=8) in training, and the generators are not needed at deployment (only πϕ(a∣s,g)\pi_\phi(a|s,g) runs). In Table 4 (L432--435), training takes 816 min, close to QVPO (774m) and DD (786m).

Q5: Comparison with JaxGCRL, CRL &TD-InfoNCE

Under matched budgets, we compare GEAR with all three:

MethodCheetah-RunAcrobot-SwingupPickCubeHabitat Office (Success% / SPL)
CRL (NeurIPS'22)638.2±29.3638.2\pm29.3112.4±28.6112.4\pm28.668.4±3.8%68.4\pm3.8\%61.4±3.9%61.4\pm3.9\% / 0.50±0.030.50\pm0.03
TD-InfoNCE (ICLR'24)651.5±25.2651.5\pm25.2137.8±30.5137.8\pm30.573.0±3.4%73.0\pm3.4\%62.2±3.7%62.2\pm3.7\% / 0.55±0.040.55\pm0.04
JaxGCRL (ICLR'25)645.8±27.0645.8\pm27.0128.6±29.1128.6\pm29.170.2±3.6%70.2\pm3.6\%63.0±3.8%63.0\pm3.8\% / 0.52±0.030.52\pm0.03
GEAR680.1±27.5680.1\pm27.5192.6±28.3192.6\pm28.386.4±2.9%86.4\pm2.9\%78.9±3.2%78.9\pm3.2\% / 0.68±0.020.68\pm0.02

To clarify the remaining implementation and literature details on W1 and Q1-Q4 beyond our response above, we provide the following additions, which we will incorporate into the revised manuscript:

Q3: Generator Architecture, Inputs, and Exact Usage

Embedding-space DDPMs (Sec. 3.1, App. C.4)

Both Gψ+G_\psi^+ and Gψ−G_\psi^- output a goal vector conditioned on the current state sts_t, trained on within-trajectory pairs matching sts_t with the goal representation of a later state sjs_j (j>tj>t). Generated goals enter only the training-time critic and value-gap computations (Eqs. 4, 6-11) and are not used for multi-step dynamics rollouts.

W1, W3, Q1 & Q2: Remaining Clarifications (CRL, GoPlan, W3, Diverse Actions)

Contrast with CRL and GoPlan, recent baselines, and action diversity (Sec. 5, L456-475)

Beyond the augmentation contrast already detailed above for [3-6]: (i) unlike contrastive GCRL (CRL (Eysenbach et al., 2022), TD-InfoNCE [1], JaxGCRL [2]) that fits occupancy similarity via InfoNCE, GEAR retains soft Bellman backups (Eqs. 7-10) and contrasts dual-critic values (ΔQ\Delta Q); (ii) unlike GoPlan (Wang et al., 2024a), which plans model-based rollouts with a learned dynamics model, GEAR is model-free; and (iii) on W3 and Q1, Sec. 5 and our experiments already include 2023-2025 diffusion/generative baselines (Decision Diffuser, CEP, SynthER, CPQL, QVPO, PGR), while action diversity is driven by the SAC entropy bonus (Eqs. 7-8) and conditional MI Iϕ(a;g+∣s)I_\phi(a;g^+|s) (Eq. 13). We will expand Sec. 5 to cover [1-6].

Q3 & Q4: Reverse-Sampling Cost and Entropy Evidence

Per-goal latency and entropy curve (App. C.4, Fig. 4)

Sub-sampling Nrev=8N_{\text{rev}}=8 reverse steps takes ≈1.2\approx 1.2 ms per goal during training actor updates (App. C.4), whereas at test time only πϕ(a∣s,g)\pi_\phi(a|s,g) is executed. Fig. 4 also shows redundant-goal entropy increasing over training alongside Table 4's runtime comparison.

References

[1] Contrastive difference predictive coding. ICLR, 2024.

[2] Accelerating goal-conditioned reinforcement learning algorithms and research. ICLR, 2025.

[3] GTA: Generative trajectory augmentation with guidance for offline reinforcement learning. NeurIPS, 2024.

[4] Goal-conditioned data augmentation for offline reinforcement learning. TMLR, 2025.

[5] Looking backward: Retrospective backward synthesis for goal-conditioned GFlowNets. ICLR, 2025.

[6] Temporal distance-aware transition augmentation for offline model-based reinforcement learning. ICML, 2025.

Thank you for the author's response. Firstly, I encourage the author to present the new results and modifications in the article, such as the curve graph comparing JaxGCRl with CRL and TD-InFoNCE in the JaxGCRl environment. Secondly, I am still concerned about the novelty. Besides contrastive loss optimization, diffusion has already been applied in GCRL, for example, Learning to Reach Goals via Diffusion, ICML 2024. Generative trajectory stitching through diffusion composition, NIPS 2025. Finally, has the author considered other generative models to generate trajectories? For example, flow matching and drift models? Perhaps their computational cost is not high and the quality might be quite good.

Reviewer_5yceOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GEAR, a goal-conditioned reinforcement learning approach for continuous control tasks, evaluated on DeepMind Control Suite (DMC), ManiSkill2, Habitat and real robot hardware. The core claimed goal is to improve robust goal reaching and reliable long-horizon execution for embodied agents. However, the paper heavily relies on normalized cumulative return as the primary reported metric, which is a formalized continuous reward signal rather than task success. This choice obscures the core experimental objective: whether the agent reliably completes the target task. The paper’s conclusions are weakened by this mismatch between reported metrics and the real-world goal of goal-conditioned robotics.

Critical strengths
  1. The work covers a reasonable set of environments, including low-dimensional DMC simulation, manipulation simulation (ManiSkill2), visual navigation (Habitat), plus real hardware robot experiments, which is a plus for embodied RL.
  2. The algorithmic design is clearly presented, and ablation studies on core components are provided.
  3. Some qualitative rollout videos are included to illustrate agent behavior.
Critical weaknesses
  1. Over-reliance on formalized continuous return metric, decoupled from true task completion The paper’s main reported performance is the normalized cumulative reward/return from DMC. DMC reward is a smooth, continuous distance-based shaping reward, designed to guide optimization, not to directly report task success or failure. A high return only means the agent stays close to the goal state for some timesteps, but it does not equal successfully accomplishing the task. This creates a critical gap: the paper’s stated purpose is goal reaching, yet it rarely reports binary success rate (whether the agent actually reaches and stays at the goal). Smooth shaped reward can be inflated by partial progress; an agent may obtain decent return by getting near the goal and oscillating, without ever reliably completing the task. This formalized continuous metric hides failure modes such as near-miss, unstable hovering, temporary goal proximity and then drifting away. The reader cannot tell from reported scores how often the agent actually finishes the target task.
  2. Cannot translate reported score into failure probability; missing human baseline reference The paper does not map continuous return values to interpretable task failure probability. We cannot know, for any given reported score, what fraction of episodes end with task failure. Moreover, there is no comparison to human performance on the same tasks. Without binary success statistics and human reference, it is impossible to judge how far GEAR is from solving the practical task, or whether the gain over baselines is meaningful from an application perspective. Numerical improvements in normalized return may be statistically significant but practically trivial in terms of task completion.
Questions for authors
  1. Your core claim is improved goal reaching. Why is binary episode success rate (goal achieved within tolerance for required duration) not your primary metric across DMC, ManiSkill2 and Habitat? Please report success rate for all tasks, not only continuous shaped return.
  2. Can you convert your continuous return values into interpretable per-task failure probability? How much absolute improvement in task success does GEAR bring over baselines, not just reward return?
  3. Please provide disaggregated results split by task difficulty (easy / medium / hard). Is the performance gain concentrated only on simple tasks?
  4. Could you add human baseline performance on a subset of environments, to quantify the remaining performance gap between GEAR and human-level task completion?
  5. Please provide failure-mode statistics: what fraction of failed episodes correspond to goal drift, transient near-goal oscillation, timeout, or collision?
4 comments

We thank Reviewer 5yce for the constructive suggestions on evaluation metrics.

W1 & Q1: Binary Episode Success Rate Across Benchmarks

Across our 4 benchmark suites (17 tasks total), binary episode success rate is already the primary metric for 11 of the 17 tasks: ManiSkill2 (LiftCube, PickCube, PickYCB, StackCube, TurnFaucet in Fig. 3, L399), Habitat (Residential, Office, Commercial in Table 2, L401--408), and all Real-Robot tasks (Lift, Stack, Grasp in Tables 3 & 5).

For DMC: Table 1 reports episode return to match prior DMC baselines (DD, QVPO). However, as specified in Sec. 4.1 (L358) and App. C.3 (L1292--1305), all 6 DMC tasks use sparse binary goal-tolerance rewards rt=I(∥ϕ(st)−g∥2≤ϵg)r_t = \mathbb{I}(\|\phi(s_t) - g\|_2 \le \epsilon_g), and GEAR tracks binary indicator cτ∈{0,1}c_\tau \in \{0,1\} in Eq. (1) (L177--178). Extracting binary episode success (cτ=1c_\tau = 1, staying within ϵg\epsilon_g for ≥10\ge 10 consecutive steps under App. C.3 tolerances) from logged Table 1 rollouts gives:

DMC TaskHERRIGDDQVPOGEAR (Ours)Failure (1−Succ1 - \text{Succ})
Acrobot-Swingup24.0±5.2%24.0 \pm 5.2\%46.8±4.9%46.8 \pm 4.9\%54.2±4.6%54.2 \pm 4.6\%58.6±4.5%58.6 \pm 4.5\%74.2±4.1%74.2 \pm 4.1\%25.8%25.8\% (vs. QVPO 41.4%41.4\%)
Cartpole-Swingup84.5±3.1%84.5 \pm 3.1\%87.2±2.8%87.2 \pm 2.8\%88.0±2.9%88.0 \pm 2.9\%89.4±2.6%89.4 \pm 2.6\%93.8±2.1%93.8 \pm 2.1\%6.2%6.2\% (vs. QVPO 10.6%10.6\%)
Cheetah-Run64.2±3.8%64.2 \pm 3.8\%71.6±3.5%71.6 \pm 3.5\%76.0±3.4%76.0 \pm 3.4\%78.5±3.2%78.5 \pm 3.2\%88.4±2.7%88.4 \pm 2.7\%11.6%11.6\% (vs. QVPO 21.5%21.5\%)
Finger-Spin83.0±3.6%83.0 \pm 3.6\%86.4±2.9%86.4 \pm 2.9\%87.5±3.1%87.5 \pm 3.1\%88.8±2.8%88.8 \pm 2.8\%94.0±1.8%94.0 \pm 1.8\%6.0%6.0\% (vs. QVPO 11.2%11.2\%)
Reacher-Hard91.2±2.4%91.2 \pm 2.4\%93.5±2.1%93.5 \pm 2.1\%94.0±2.0%94.0 \pm 2.0\%94.8±1.9%94.8 \pm 1.9\%97.6±1.2%97.6 \pm 1.2\%2.4%2.4\% (vs. QVPO 5.2%5.2\%)
Walker-Walk89.6±2.2%89.6 \pm 2.2\%92.8±2.0%92.8 \pm 2.0\%93.4±1.9%93.4 \pm 1.9\%94.6±1.7%94.6 \pm 1.7\%97.2±1.3%97.2 \pm 1.3\%2.8%2.8\% (vs. QVPO 5.4%5.4\%)

W2 & Q2: Per-Task Failure Probability (1−Succ1 - \text{Succ})

Converting success into failure probability Pfail=1−SuccP_{\text{fail}} = 1 - \text{Succ} shows large reductions over QVPO:

  • On DMC (6-task mean), GEAR achieves 90.9%90.9\% success (Pfail=9.1%P_{\text{fail}} = 9.1\%) vs. QVPO's 84.1%84.1\% (15.9%15.9\%), cutting failure probability by 42.8%42.8\%.
  • On ManiSkill2 (Fig. 3, 4M steps), PickCube is 86.4%86.4\% vs. 74.2%74.2\% (Pfail:25.8%→13.6%P_{\text{fail}}: 25.8\% \to 13.6\%); StackCube is 70.5%70.5\% vs. 18.2%18.2\% (81.8%→29.5%81.8\% \to 29.5\%); TurnFaucet is 82.0%82.0\% vs. 68.5%68.5\% (31.5%→18.0%31.5\% \to 18.0\%).
  • On Habitat (Table 2), Residential PfailP_{\text{fail}} drops 42.8%→38.3%42.8\% \to 38.3\%; Office 31.6%→21.1%31.6\% \to 21.1\%; Commercial 39.1%→35.2%39.1\% \to 35.2\% (cutting failures by 10.0%10.0\%--33.2%33.2\% and raising SPL on Office by +0.10+0.10, 0.58→0.680.58 \to 0.68).
  • On Real-Robot (Table 3 & App. Table 5), GEAR reaches 81.7%81.7\% mean success (Pfail=18.3%P_{\text{fail}} = 18.3\%) vs. QVPO's 73.3%73.3\% (26.7%26.7\%).

Q3: Results by Task Difficulty (Easy / Medium / Hard)

Across tasks: On DMC, gains over QVPO grow from easier (Reacher-Hard: +2.8%+2.8\%) to hard underactuated tasks (Acrobot-Swingup: +15.6%+15.6\%, Cheetah-Run: +9.9%+9.9\%). On Real-Robot (Table 3), gains grow from Lift (+5%+5\%) to Stack (+10%+10\%) and distractor Grasp (+10%+10\%).

Within-task (Habitat Office split by geodesic distance):

  • Easy (<3m< 3\text{m}): GEAR 91.5%91.5\% vs. QVPO 86.8%86.8\% (+4.7%+4.7\%, Pfail:13.2%→8.5%P_{\text{fail}}: 13.2\% \to 8.5\%).
  • Medium (33--7m7\text{m}): GEAR 79.4%79.4\% vs. QVPO 68.2%68.2\% (+11.2%+11.2\%, Pfail:31.8%→20.6%P_{\text{fail}}: 31.8\% \to 20.6\%).
  • Hard (>7m> 7\text{m}, multi-room): GEAR 65.8%65.8\% vs. QVPO 50.2%50.2\% (+15.6%+15.6\%, Pfail:49.8%→34.2%P_{\text{fail}}: 49.8\% \to 34.2\%), showing ΔQ\Delta Q prevents detour compounding over long horizons.

Q4: Human and Oracle Reference Baselines

  • Habitat Geodesic Oracle (App. C.3 protocol, L1340--1342): 100%100\% Success, 1.001.00 SPL; GEAR reaches 78.9%/0.6878.9\% / 0.68 SPL on Office vs. QVPO's 68.4%/0.5868.4\% / 0.58.
  • ManiSkill2 Oracle: 98%/94%/90%98\% / 94\% / 90\% on PickCube / StackCube / TurnFaucet vs. GEAR's 86.4%/70.5%/82.0%86.4\% / 70.5\% / 82.0\%.
  • Real-Robot Human Teleop (20 trials/task): 100%/95%/95%100\% / 95\% / 95\% (96.7%96.7\% mean) vs. GEAR's 90%/75%/80%90\% / 75\% / 80\% (81.7%81.7\% mean).

Q5: Failure-Mode Statistics

For Real-Robot Grasp with distractors (App. Table 6, L1067--1079): Over 20 trials, HER (45%45\% succ) fails 11 times: 6 (30%30\%) Wrong-Target distractions and 5 (25%25\%) drops. GEAR (80%80\% succ) cuts Wrong-Target distraction to 1 (5%5\%), with 3 (15%15\%) execution drops/slips, showing Qˉω−\bar{Q}_\omega^- suppresses attraction to distractors.

For Habitat Office (500 episodes): Splitting PfailP_{\text{fail}} into Collision, Timeout/Looping, and Near-Goal Drift:

  • HER (47.6%47.6\% fail): 16.2%16.2\% / 22.8%22.8\% / 8.6%8.6\%
  • QVPO (31.6%31.6\% fail): 11.8%11.8\% / 13.6%13.6\% / 6.2%6.2\%
  • GEAR (21.1%21.1\% fail): 7.4%7.4\% / 8.2%8.2\% / 5.5%5.5\%, cutting looping timeouts by 64.0%64.0\% vs. HER.

To supplement our earlier response with details requested in Q1 and Q3-Q5:

Q1: Exact Per-Benchmark Success Definitions

Habitat (App. C.3)

Binary success requires executing a STOP action within 0.20.2 m of the goal under a 1,000-step cap.

ManiSkill2 (Fig. 3, App. C.3)

We report binary task success rate under 500-step episode limits.

Real Robot (App. B.1)

Object Lifting requires lifting the target object and holding it stably; Block Stacking requires the placed block to stay stable after release; Target Grasping requires grasping the specified target rather than a distractor.

DMC oscillation check (App. C.3)

In our binary extraction from Table 1, requiring the sparse-reward goal-proximity condition (App. C.3) to hold for ≥10\ge 10 consecutive steps rules out transient near-goal oscillation or hovering.

Q3-Q5: Additional Paper Evidence and Revision Plan

Extra difficulty evidence (Q3)

In addition to the splits above, App. B.3 (Fig. 7) reports 8 additional DMC tasks where the performance gap widens on harder control tasks such as humanoid-run.

Revision additions (Q4, Q5)

We will include the DMC binary success table, the Habitat geodesic-difficulty split and failure breakdown, and the oracle/human reference baselines in the revised appendix.

Dear Reviewer 5yce,

Thank you again for your helpful feedback. We are writing to let you know that we have uploaded the revised manuscript (with all updates highlighted in red), incorporating the requested metrics and analyses into Section 4.2 and Appendix B.6:

  • Table 10 (App. B.6): Binary episode success rates and failure probabilities (PfailP_{\text{fail}}) across all 6 DMC tasks, alongside PfailP_{\text{fail}} conversions across ManiSkill2, Habitat, and Real-Robot tasks (W1, W2, Q1, Q2).
  • Tables 11-12 (App. B.6): Habitat Office difficulty splits by geodesic distance (Table 11; Q3) and Oracle / Human teleoperation reference baselines across suites (Table 12; W2, Q4).
  • Table 13 (App. B.6): 500-episode failure-mode breakdown into Collision, Timeout/Looping, and Near-Goal Drift on Habitat Office (W1, Q5).

We would appreciate it if you could take a look at these additions in the updated PDF when convenient, and please let us know if any questions remain.

Thank you for your constructive feedback so far. We would be happy to clarify any remaining questions.

Reviewer_6XTFOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors proposed GEAR for goal-conditioned reinforcement learning, which trains two diffusion models to generate goals from more efficient and more redundant experience. Two critics evaluate the same action under these different goals. Their Q-value difference is used to guide the policy. The paper reports improvements on DMC, ManiSkill2, Habitat, and three real robot tasks.

Critical strengths
  1. Reaching a goal is not the same as reaching it efficiently. Learning from long and indirect trajectories can be a real problem in goal-conditioned RL.
  2. Using two goal generators and a Q-value gap gives the policy a way to compare different goal conditions during training.
  3. The paper includes several benchmarks, component ablations, a Q-gap and entropy plot, and a small real robot evaluation.
Critical weaknesses
  1. The baseline comparison is not strong enough. The paper compares with HER, RIG, DD, and QVPO. However, it misses direct contrastive goal-conditioned RL methods such as CRL (Contrastive Learning as Goal-Conditioned Reinforcement Learning). This is an important omission because contrastive goal learning is central to the proposed method.

  2. The definition of efficient experience is not fully convincing. The two generators mainly use trajectory length and success or failure to decide what to learn. But a longer successful trajectory may simply start farther away or face a harder task. It is not necessarily a redundant trajectory. The paper does not clearly show that the generated goals represent true differences in behavior efficiency.

  3. The Q-value gap is not clearly linked to efficient behavior. Eq. (6) compares two different goals using two different critics. A larger gap could come from goal difficulty or critic errors. It does not necessarily mean that the action reaches the requested goal in fewer steps.

  4. The experiments do not explain enough about why GEAR works. Figure 4 already shows average Q-gap and goal entropy during training. The appendix also includes some real robot failure counts. These are useful. But the paper does not visualize the generated goal distributions or the two critics separately. It also does not show how the gap changes actual actions or whether the learned paths become shorter. Most evidence still comes from final performance tables.

Questions for authors
  1. Why should the gap between two critics under different goals indicate better behavior toward the requested goal? How do you rule out differences in goal difficulty or critic error? Also, why does the negative generator use one critic in Eq. (4) while the actor uses two in Eq. (11)?
  2. Could you explain the theoretical idea with a small example? Please show two paths to the same goal and explain when maximizing the Q-gap selects the shorter path and when it might fail.
  3. Could you compare against direct contrastive goal-conditioned RL methods such as CRL? Please use matched training and environment budgets.
  4. What goals do the two diffusion models generate? Could you visualize their samples and distributions from the same starting states?
  5. Figure 4 only plots the average gap. Could you plot the two Q-values separately during training.
  6. Where does gap guidance improve actual decisions? Could you show trajectories from the same start and goal for GEAR and GEAR without gap guidance? Include action-value heatmaps.
6 comments

We thank Reviewer 6XTF for the constructive feedback.

W1Q3: Contrastive GCRL Baselines

Contrastive GCRL methods like CRL [NIPS2022](cited in Sec.1, L034) and TD-InfoNCE [Zheng et al., 2024] learn state-goal similarity via trajectory-level InfoNCE over buffer future states. They do not generate counterfactual goal distributions at sts_t or form an action-level dual-critic gap ΔQ\Delta Q to penalize detours. Under the protocol of Tables 1-2 (5 seeds), GEAR outperforms both:

MethodCheetah-RunAcrobot-SwingupHabitat Office (Success%/SPL)
HER610.8±36.1610.8\pm 36.178.7±34.078.7\pm 34.052.4±2.8%52.4\pm 2.8\%/0.42±0.030.42\pm0.03
CRL (NIPS'22)638.2±29.3638.2 \pm 29.3112.4±28.6112.4\pm28.661.4±3.9%61.4\pm3.9\%/0.50±0.030.50 \pm 0.03
TD-InfoNCE (ICLR'24)651.5±25.2651.5\pm25.2137.8±30.5137.8\pm30.562.2±3.7%62.2 \pm 3.7\%/0.55±0.040.55 \pm 0.04
QVPO663.2±28.1663.2\pm28.1162.9±34.5162.9\pm34.568.4±3.1%68.4\pm3.1\%/0.58±0.030.58\pm 0.03
GEAR680.1±27.5680.1\pm27.5192.6±28.3192.6\pm28.378.9±3.2%78.9\pm3.2\%/0.68±0.020.68\pm 0.02

W2Q4: Tmax⁡T_{\max} and Generated Goals Gψ+,Gψ−G_\psi^+, G_\psi^-

For Tmax⁡T_{\max}: Partitioning rollouts by a horizon/return cutoff follows standard RL practice (GCSL [Ghosh et al., 2021], RvS [Emmons et al., 2022], DT [Chen et al., 2021]). Here Tmax⁡T_{\max} is the reference step count in our soft sigmoid weight (Eq. 1), defining coarse pools D+,D−\mathcal{D}^+, \mathcal{D}^- for conditional distribution learning rather than per-step labels. For Tmax⁡/H∈{0.20,0.30,0.40}T_{\max}/H \in \{0.20, 0.30, 0.40\} on Habitat Office, GEAR reaches 70.9%70.9\%, 78.9%78.9\%, and 64.9%64.9\% success, and our default soft logistic split gives 78.9%78.9\% vs. 77.6%77.6\% for a hard cutoff (72.4%72.4\% vs. 71.3%71.3\% under noise p=0.1p=0.1).

For longer successful rollouts: First, training pairs anchor sts_t with a later state g=ϕ(sj)g = \phi(s_j) via ρ(⋅∣st,τ)\rho(\cdot|s_t,\tau) (Sec. 3.1, L189; App. C.4, L1354), and the generator conditions on the current state sts_t rather than the start state, capturing step-wise goal-reaching progress rather than start-to-end distance. Second, Eq. (1) (L177--183, L065) sets wτ+=cτ[δw+(1−2δw) σ(Tmax⁡−T(τ)h)]w_\tau^+ = c_\tau\big[\delta_w + (1-2\delta_w)\,\sigma\big(\frac{T_{\max}-T(\tau)}{h}\big)\big] with δw∈(0,1/2)\delta_w \in (0, 1/2), so any successful rollout (cτ=1c_\tau=1) from a distant start keeps wτ+≥δw>0w_\tau^+ \ge \delta_w > 0 in D+\mathcal{D}^+; only failed rollouts (cτ=0c_\tau=0) get wτ+=0w_\tau^+=0 (and wτ−=1w_\tau^-=1).

For Q4: From the same anchor sts_t across 500 Office states, Gψ+(st)G_\psi^+(s_t) generates forward shortest-path waypoints (89.3%89.3\% reachable; efficiency 0.840.84; diversity 0.420.42), while Gψ−(st)G_\psi^-(s_t) generates backtracking/corner states (53.7%53.7\% reachable; efficiency 0.560.56; diversity 0.250.25). Pool overlap is 3.5%3.5\% and sample overlap is 1.7%1.7\%.

W3 Q1 Q2: Critic Setup and Q-Gap Mechanism

For ΔQ\Delta Q: Both critics share the identical reward function and γ\gamma under soft Bellman backups (Eqs. 7--8, L277--279), keeping their values on a common scale. In Eqs. (11)--(12) (L295--304), target g+g^+ is fixed while g−∼Gψ−(s)g^- \sim G_\psi^-(s) is marginalized in the greedy rule: Qˉω−(s,a)=Eg−[Qω(s,a∣g−)]\bar{Q}_\omega^-(s,a) = \mathbb{E}_{g^-}[Q_\omega(s,a|g^-)]. Thus ΔQ=Qθ(s,a∣g+)−Qˉω−(s,a)\Delta Q = Q_\theta(s,a|g^+) - \bar{Q}_\omega^-(s,a) penalizes actions broadly valuable for redundant goals from ss.

For Q1: As noted in Sec. 3.1 (L239--241), in Eq. (4) (L235--236), Ladv−\mathcal{L}_{\text{adv}}^- tests if g−g^- looks like a high-value goal under positive critic QθQ_\theta, giving a single scale so Gψ−G_\psi^- targets hard negatives that QθQ_\theta overestimates. For actor updates (Eq. 11), separate critics QθQ_\theta (on D+\mathcal{D}^+) and QωQ_\omega (on D−\mathcal{D}^-) avoid value interference.

For Q2: At state ss with target g+g^+, let direct a1a_1 and detour a2a_2 have (Qθ,Qˉω−)=(10,2)(Q_\theta, \bar{Q}_\omega^-) = (10, 2) and (8,7)(8, 7); ΔQ(a1)=8>ΔQ(a2)=1\Delta Q(a_1)=8 > \Delta Q(a_2)=1 widens the margin (2→72 \to 7). If QθQ_\theta overestimates a2a_2 from buffer bias, e.g., (8.0,2.0)(8.0, 2.0) vs. (8.5,7.0)(8.5, 7.0), a single critic picks a2a_2 (8.5>8.08.5 > 8.0), whereas ΔQ\Delta Q flips the choice whenever Qˉω−(a2)−Qˉω−(a1)>Qθ(a2∣g+)−Qθ(a1∣g+)\bar{Q}_\omega^-(a_2) - \bar{Q}_\omega^-(a_1) > Q_\theta(a_2|g^+) - Q_\theta(a_1|g^+), selecting a1a_1 (6.0>1.56.0 > 1.5).

W4 Q5 Q6: Separate Q-Trajectories and Action Selection

For W4: On Office (0k →\to 250k →\to 500k steps), QθQ_\theta rises (1.4→5.9→7.81.4 \to 5.9 \to 7.8), while Qˉω−\bar{Q}_\omega^- peaks then drops as loops vanish (1.3→3.6→2.11.3 \to 3.6 \to 2.1), widening ΔQ\Delta Q (0.1→2.3→5.70.1 \to 2.3 \to 5.7).

For Q5 Q6: Across 500 Office pairs, ΔQ\Delta Q correlates with SPL (ρ=0.43,p<10−4\rho = 0.43, p < 10^{-4}) and steps (ρ=−0.39\rho = -0.39). From identical (s0,g)(s_0, g), removing ΔQ\Delta Q (w/o Gap, Table 2) drops success (78.9%→58.4%78.9\% \to 58.4\%) and SPL (0.68→0.510.68 \to 0.51). At junctions, w/o Gap gives similar QθQ_\theta to entering vs. wall-following (24.6%24.6\% longer paths; 30%30\% vs. 5%5\% robot wrong grasps, App. Table 6). Heatmaps show Qˉω−\bar{Q}_\omega^- concentrates on corners/dead-ends, so subtracting Qˉω−\bar{Q}_\omega^- suppresses looping.

To supplement our earlier response with details, we clarify the following points on Q1, Q2, and Q4-Q6.

Q1 & Q2: Critic-Dependent Gap, Behavioral Specificity, and Failure Conditions

Critic-dependent signal and reward grounding (Q1)

As emphasized in Sec. 3.2, ΔQ\Delta Q is a critic-dependent auxiliary signal in the actor objective (Eqs. 11, 14), not a task-independent definition of trajectory efficiency. Both critics remain grounded in task rewards via soft Bellman backups with the shared discount γ\gamma (Eqs. 7-8), and the w/o Dual ablation in Table 2 confirms that separating QθQ_\theta and QωQ_\omega is needed to keep the value contrast reliable rather than reflecting arbitrary critic noise.

Distribution-level contrast and when ΔQ\Delta Q may fail (Q2)

In Eq. (12), g−g^- is not a per-sample negative action label, but a training-only contrast marginalized over Gψ−(s)G_\psi^-(s) using 20 sampled redundant goals per actor update (App. C.4). Subtracting Qˉω−(s,a)\bar{Q}_\omega^-(s,a) discounts actions that are also broadly valuable under redundant or failed goal distributions, favoring actions with stronger behavioral specificity toward g+g^+. Maximizing ΔQ\Delta Q favors shorter paths when QθQ_\theta ranks goal-reaching progress above Qˉω−(s,a)\bar{Q}_\omega^-(s,a), whereas it can mis-rank actions when critic estimates are noisy early in training or when an efficient action passes through states where redundant rollouts retain high Qˉω−(s,a)\bar{Q}_\omega^-(s,a). Scaling Ladv−\mathcal{L}_{\text{adv}}^- in Eq. (5) by the curriculum weight βt\beta_t (Table 10) mitigates this by strengthening adversarial separation only as the critics stabilize (Secs. 3.1-3.3, Thm. 1).

Q4-Q6: Real-Robot Task Breakdown and Revision Additions

Per-task effect of removing ΔQ\Delta Q (Q4, Q5)

Consistent with w/o Gap in Table 2, removing the ΔQ\Delta Q actor update lowers real-robot Lift / Stack / Grasp success from 18/2018/20, 15/2015/20, 16/2016/20 to 15/2015/20, 11/2011/20, 10/2010/20 (Table 5).

Planned additions (Q6)

In the revision we will include separate QθQ_\theta and Qˉω−\bar{Q}_\omega^- training curves, goal samples from identical start states, and same-start, same-goal trajectories with action-value heatmaps for GEAR and w/o Gap.

Reference:

[1] Contrastive learning as goal-conditioned reinforcement learning. NeurIPS, 2022.

[2] Contrastive difference predictive coding. ICLR, 2024.

[3] Learning to reach goals via iterated supervised learning. ICLR, 2021.

[4] RvS: What is essential for offline RL via supervised learning? ICLR, 2022.

[5] Decision Transformer: Reinforcement learning via sequence modeling. NeurIPS, 2021.

Dear Reviewer 6XTF,

Thank you again for your detailed and constructive review. We are writing to let you know that we have uploaded the revised manuscript (with all updates highlighted in red), incorporating the specific analyses and visualizations you requested:

  • Appendix A.3 (Sec. 3.2): Critic-scale consistency, the two-path worked example, and boundary conditions of ΔQ\Delta Q (Q1, Q2).
  • Appendix B.5 (Table 9): Matched-budget comparisons against CRL, TD-InfoNCE, and JaxGCRL (W1, Q3).
  • Appendix B.7 (Figs. 9-11): Separate Qθ/Qˉω−Q_\theta / \bar{Q}_\omega^- training trajectories (Fig. 9; Q5), generated goal distributions from the same anchor state (Fig. 10; Q4), and same-start, same-goal trajectories with action-value heatmaps (Fig. 11; W4, Q6), alongside Tmax/HT_{\text{max}}/H sensitivity (W2).

We would be grateful if you could take a look at these additions in the updated PDF when convenient, and please let us know if any questions remain.

Comments on the paper

2 comments

We thank the Reviewers 6XTF, 5yce, and fSFF for their constructive feedback. We have uploaded the revised manuscript (all additions highlighted in red), addressing every weakness and question raised across all review rounds. Below is a summary of the revisions:

  1. Theory & Cutoff Clarification (6XTF: W2, W3, Q1, Q2)
  1. Sec. 3.1: Clarifies soft-cutoff TmaxT_{\text{max}} practice and δw>0\delta_w > 0 coverage for distant starts (6XTF: W2).
  2. Sec. 3.2 & App. A.3: Adds critic-scale consistency, why Eq. (4) uses QθQ_\theta while Eq. (11) separates Qθ,QωQ_\theta, Q_\omega, two-path worked example, and boundary conditions of ΔQ\Delta Q (6XTF: W3, Q1, Q2).
  1. Contrastive GCRL & Literature Expansion (6XTF: W1, Q3; fSFF: W1, W3, Q1, Q2, Q5, Follow-up Q1-Q2)
  1. Sec. 4.2 & App. B.5 (Table 9, Fig. 8): Adds matched-budget comparisons against CRL, TD-InfoNCE, and JaxGCRL across suites (Table 9; 6XTF: W1, Q3; fSFF: Q5), plus 0-150M learning curves on JaxGCRL Ant (Fig. 8; fSFF: Follow-up Q1).
  2. Sec. 5 & Bibliography: Contrasts GEAR with diffusion GCRL (Merlin, CompDiffuser), contrastive GCRL (CRL, TD-InfoNCE, JaxGCRL), and recent augmentation methods (GTA, GODA, Retrospective GFlowNets, TempDATA), alongside existing 2023-2025 baselines and action-diversity mechanisms (6XTF: W1; fSFF: W1, W3, Q1, Q2, Follow-up Q2).
  1. Binary Success, Failure Modes & Reference Baselines (5yce: W1, W2, Q1-Q5)
  1. Sec. 4.2 & App. B.6 (Tables 10-13): Extracts DMC binary episode success and PfailP_{\text{fail}} across all 6 DMC tasks and converts ManiSkill2, Habitat, and Real-Robot metrics to PfailP_{\text{fail}} (Table 10; 5yce: W1, W2, Q1, Q2).
  2. App. B.6 (Tables 11-13): Adds Habitat Office geodesic difficulty splits (Table 11; 5yce: Q3), Geodesic Oracle, ManiSkill2 Oracle, and Real-Robot Human Teleoperation baselines (Table 12; 5yce: W2, Q4), and 500-episode failure-mode breakdowns into Collision, Timeout/Looping, and Near-Goal Drift (Table 13; 5yce: W1, Q5).
  1. Closed-Loop Ablations, Diagnostics & Generative Backbones (6XTF: W2, W4, Q4-Q6; fSFF: W1, W2, W4, Q3, Q4, Follow-up Q3)
  1. Sec. 4.2 & App. B.7 (Table 14): Adds identical D+/D−\mathcal{D}^+/\mathcal{D}^- split ablations confirming gains beyond static relabeling (fSFF: W1, W2).
  2. App. B.7 (Figs. 9-11): Adds Tmax/HT_{\text{max}}/H and β\beta sensitivity (6XTF: W2; fSFF: W4), separate Qθ/Qˉω−Q_\theta / \bar{Q}_\omega^- trajectories (Fig. 9; 6XTF: W4, Q5), generated-goal distributions and reachability/entropy/confusion CQC_Q diagnostics (Fig. 10; 6XTF: W4, Q4; fSFF: Q3, Q4), and same-start action-value heatmaps and path-length analyses (Fig. 11; 6XTF: W4, Q6).
  3. App. C.4 (Table 17): Adds the 4-step conditional Flow Matching ablation on Habitat Office and curriculum/noise-regularization analysis (fSFF: Q4, Follow-up Q3).

All newly cited works are included in the updated bibliography. We thank the reviewers for helping strengthen the paper.

Dear Reviewers,

As the reviewer–author discussion phase is approaching its deadline (November 18, AoE), I would kindly ask reviewers who have not yet provided an update to review the authors' responses and share whether their concerns and questions have been adequately addressed.

Your feedback would be greatly appreciated and will help us reach a well-informed decision.

Thank you for your time and valuable contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.