- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces NEFD, a Near-to-Far Self-Distillation method for offline goal-conditioned reinforcement learning. The approach employs a teacher policy conditioned on privileged near-future states along with the distant goal, providing dense action-level and representation-level supervision to a student that observes only the distant goal. The distillation is modulated by a goal-progress signal estimated from a learned value function. The method achieves strong empirical gains across 29 OGBench datasets, particularly in long-horizon navigation and multi-step manipulation, where distant goals provide weak guidance for current decisions.
Critical strengths
-
Clarity of presentation: The method is clearly described, with precise notation, a concise architecture overview, and an end-to-end algorithm. The roles of the teacher/student conditions, the loss terms, and the weighting scheme are easy to follow.
-
Experimental rigor and validation: Extensive evaluation is conducted on 29 OGBench datasets spanning locomotion and manipulation, in both state and visual modalities, with results averaged over multiple seeds following benchmark protocols. The ablations are thorough, separately isolating (i) the role of privileged information (OPD-Large vs. OPD-Privilege), (ii) action versus representation distillation, and (iii) goal-progress weighting; additional analyses examine performance as a function of goal distance and the predictability of the near future from student features.
Critical weaknesses
1.Missing comparisons and discussions to several recent and strong baselines on OGBench: [1] Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL [2] Transitive RL: Value Learning via Divide and Conquer [3] QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL These are highly relevant to long-horizon GCRL and could affect SOTA claims.
- Limited reporting on computational cost (training time and inference time, memory). Distilling both actions and representations with dual condition runs presumably increases runtime; fairness of compute vs baselines is not discussed.
Questions for authors
-
The near-future state does not cross trajectories, so its role in stitching is unclear. For an in-trajectory goal , the teacher sees ; for a random goal, it sees (Appendix A.4). Progress weighting can still score whether a local step moves toward an off-trajectory goal, but the teacher never sees a stitched path. Please report how often is truncated on short stitch trajectories, and note that the gain over LAVL is small on the hardest stitch tasks ( vs.\ on Antmaze-giant-stitch; vs.\ on Humanoidmaze-giant-stitch).
-
How sensitive is NEFD to the near-future horizon H, the temperature β, and the distillation weight λrep? Could you provide a sensitivity analysis and default recommendations across domains?
-
How robust is goal-progress weighting to value-function misestimation? For example, what happens when you intentionally perturb V or evaluate weighting with earlier/later snapshots?
-
Could you elaborate on potential degeneracies from sharing parameters between teacher and student? Did you observe failure cases where the model collapses to teacher ≈ student predictions regardless of s_{i+H}, and how did you mitigate them?
6 comments
Response to Reviewer RqgZ. (Part2)
Questions
Q1: The near-future state does not cross trajectories, so its role in stitching is unclear. For an in-trajectory goal , the teacher sees ; for a random goal, it sees (Appendix A.4). Progress weighting can still score whether a local step moves toward an off-trajectory goal, but the teacher never sees a stitched path. Please report how often is truncated on short stitch trajectories, and note that the gain over LAVL is small on the hardest stitch tasks (83 vs.\ 82 on Antmaze-giant-stitch; 83 vs.\ 80 on Humanoidmaze-giant-stitch).
The Teacher's near-future states are indeed sampled within the same trajectory. However, the distant goal can be sampled from other trajectories, allowing the Teacher to receive cross-trajectory goal conditioning. Trajectory stitching does not require the Teacher to observe a complete stitched path. Similar to HIQL, SAW, and LAVL, NEFD uses a goal-conditioned value function trained through TD learning, which propagates goal-conditioned value information through transitions across the offline dataset, allowing behaviors from different trajectories to be stitch when reaching distant goals.
NEFD does not aim to introduce a new stitching mechanism. Instead, as discussed in Section 3, it addresses the weak supervision for current decisions under distant goals. The observed near-future state provides the Teacher with a more direct reference for current decision and is also used to estimate progress toward the distant goal, which determines the weight of the corresponding teacher guidance. These roles remain consistent across stitch, navigate, and manipulation tasks, rather than being specific to trajectory stitching.
To further examine the contribution of our proposed components to trajectory stitching, we extend the distillation and goal progress weighting ablations in Tables 3 and 4 to stitch envionments. The results are presented below.
| Dataset | QRL | HIQL | w/o | w/o | w/o | w/o act weight | w/o rep weight | NEFD |
|---|---|---|---|---|---|---|---|---|
| Antmaze-gaint-stitch-v0 | 0±0 | 2±2 | 0±0 | 77±5 | 52±5 | 74±2 | 61±4 | 83±3 |
| Humanoid-giant-stitch-v0 | 0±0 | 3±2 | 1±0 | 65±1 | 45±3 | 68±2 | 57±2 | 83±2 |
We acknowledge that the gains over LAVL are modest on the hardest stitch tasks, including Antmaze-giant-stitch (83% vs. 82%) and Humanoidmaze-giant-stitch (83% vs. 80%). LAVL's strong performance stems not only from its LAN-based value learning but also from its HIQL-style hierarchical subgoal-based policy learning. In contrast, NEFD directly predicts actions from the observed history and distant goal, avoiding potential subgoal prediction errors of LAVL. NEFD also achieves larger gains on other stitch tasks over LAVL, including Pointmaze-large-stitch (100% vs. 87%) and Humanoidmaze-large-stitch (82% vs. 72%).
We also report the near-future horizon truncation rates on the giant-stitch datasets in the table below. As described in Appendix A.4, when exceeds the sampled in-trajectory goal index , the Teacher's near-future state is truncated to ; for randomly sampled goals, it is truncated to the trajectory endpoint when .
| Dataset | Horizon | HIQL | LAVL | NEFD |
|---|---|---|---|---|
| Pointmaze-giant-stitch-v0 | 25 | 12.50% | 12.50% | 12.79% |
| Antmaze-giant-stitch-v0 | 25 | 12.50% | 12.50% | 12.79% |
| Humanoidmaze-giant-stitch-v0 | 100 | 25.00% | 25.00% | 25.28% |
Response to Reviewer RqgZ. (Part3)
Q2: How sensitive is NEFD to the near-future horizon , the temperature , and the distillation weight ? Could you provide a sensitivity analysis and default recommendations across domains?
We have already investigated the sensitivity of NEFD to the near-future horizon in Appendix B.3.3 (Table 10), where we vary to and . The results show that NEFD maintains strong performance under these variations, with the default horizon achieving the best or tied-best results across all three tasks. To further assess robustness over a wider range, we extend this analysis to and on the same three tasks. The additional results are presented below.
| Dataset | QRL | HIQL | NEFD | ||||
|---|---|---|---|---|---|---|---|
| Pointmaze-giant-navigate-v0 | 68±7 | 46±9 | 100±0 | 99±1 | 100±0 | 99±1 | 100±0 |
| Antmaze-giant-navigate-v0 | 14±3 | 65±5 | 81±1 | 81±1 | 84±1 | 83±2 | 88±2 |
| Scene-play-v0 | 5±1 | 38±3 | 93±2 | 95±2 | 93±3 | 94±1 | 95±2 |
We further investigate the sensitivity of NEFD to the goal progress weighting temperature , in addition to the default , on the same three tasks. The results show that NEFD maintains consistently strong performance across these settings, indicating that the method is relatively insensitive to variations in .
| Dataset | QRL | HIQL | NEFD() | |||
|---|---|---|---|---|---|---|
| Pointmaze-giant-navigate-v0 | 68±7 | 46±9 | 100±0 | 100±0 | 100±0 | 100±0 |
| Antmaze-giant-navigate-v0 | 14±3 | 65±5 | 82±3 | 84±1 | 86±2 | 88±2 |
| Scene-play-v0 | 5±1 | 38±3 | 86±5 | 95±1 | 88±2 | 95±2 |
We also examine the sensitivity to the representation distillation coefficient by evaluating and , alongside the default , on the same three tasks. NEFD maintains stable performance across these variations, suggesting that the method is not highly sensitive to the choice of .
| Dataset | QRL | HIQL | NEFD() | ||
|---|---|---|---|---|---|
| Pointmaze-giant-navigate-v0 | 68±7 | 46±9 | 99±1 | 99±1 | 100±0 |
| Antmaze-giant-navigate-v0 | 14±3 | 65±5 | 85±1 | 84±2 | 88±2 |
| Scene-play-v0 | 5±1 | 38±3 | 92±1 | 94±2 | 95±2 |
Q3: How robust is goal-progress weighting to value-function misestimation? For example, what happens when you intentionally perturb V or evaluate weighting with earlier/later snapshots?
We thank the reviewer for this suggestion. To evaluate the robustness of goal-progress weighting to value-function estimation errors, we conduct two additional experiments:
(1) Value Perturbation. We add Gaussian noise to the estimated goal progress, with noise levels relative to the standard deviation of the original progress estimates. (2) Value Snapshot Analysis. We use value-function checkpoints saved at , and of a standard NEFD training run. These checkpoints are then frozen and used only to compute distillation weights. In contrast, NEFD follows the original training procedure, where the value function is continuously updated alongside the policy and its current estimates are used for weighting.
| Dataset | QRL | HIQL | NEFD | |||
|---|---|---|---|---|---|---|
| Pointmaze-giant-navigate-v0 | 68±7 | 46±9 | 100±0 | 100±0 | 98±1 | 100±0 |
| Antmaze-giant-navigate-v0 | 14±3 | 65±5 | 83±1 | 84±1 | 82±0 | 88±2 |
| Scene-play-v0 | 5±1 | 38±3 | 95±1 | 91±4 | 92±1 | 95±2 |
| Dataset | QRL | HIQL | NEFD | |||
|---|---|---|---|---|---|---|
| Pointmaze-giant-navigate-v0 | 68±7 | 46±9 | 98±1 | 100±0 | 99±1 | 100±0 |
| Antmaze-giant-navigate-v0 | 14±3 | 65±5 | 84±2 | 82±4 | 85±2 | 88±2 |
| Scene-play-v0 | 5±1 | 38±3 | 91±2 | 94±1 | 93±1 | 95±2 |
Response to Reviewer RqgZ. (Part1)
Thank you for your thorough review and insightful comments, which have helped us improve the clarity and completeness of our work. Below, we address each of your concerns in detail.
Weaknesses
We thank the reviewer for highlighting these recent and highly relevant baselines. CoGHP [1], TRL [2], and QHyer [3] are strong methods for offline goal-conditioned reinforcement learning, offering valuable advances in hierarchical policy learning, divide-and-conquer value learning, and hybrid sequence modeling respectively. We have added citations to these works and discussed their in the Related Work section.
To ensure a fair comparison, all methods are evaluated on the same standard OGBench datasets. We report results from the original papers when available and otherwise reproduce them using official implementations, while QHyer results are taken from its original paper due to the unavailability of its code. As shown in the table below, these results demonstrate that NEFD outperforms these strong recent baselines.
[1] Choi J, Lee S H, Seo S W. Chain-of-goals hierarchical policy for long-horizon offline goal-conditioned rl. In ICML, 2026.
[2] Park S, Oberai A, Atreya P, et al. Transitive rl: Value learning via divide and conquer. In ICLR, 2026.
[3] Lei X, Wang J, Zhang X, et al. QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL. In ICML, 2026.
We evaluated the average computational cost of SAW, HIQL, and NEFD on the OGBench manipulation tasks using NVIDIA RTX 3090 GPUs under official defaurt setting.
NEFD incurs higher training costs due to its history-conditioned Transformer backbone, additional teacher-conditioned computation, and action and representation distillation. At inference, only the Student Transformer is used, without the privileged Teacher or distillation objectives. The higher inference latency also reflects the use of a Transformer-based policy rather than the lighter architectures of SAW and HIQL. Nevertheless, NEFD maintains a low inference latency of 1.129 ms, well within the time budget for 20 Hz real-world robotic control [1].
[1] Reed S, Zolna K, Parisotto E, et al. A generalist agent[J]. In TMLR, 2022.