acceptodds
Under review as a conference paper at ICLR 2027

Near to Far Self Distillation for Offline Goal-Conditioned Reinforcement Learning

Abstract

Offline goal-conditioned reinforcement learning (GCRL) aims to learn policies for reaching diverse goals from a fixed dataset. Existing methods use value estimates or subgoals to guide policy learning, but supervision for current decisions can remain weak when goals are distant. Inspired by the dense predictive supervision of on-policy distillation (OPD), we use teacher predictions as direct learning targets. This raises two challenges in offline GCRL: (1) the Guidance Selection Challenge, identifying privileged information that provides useful guidance toward the distant goal; and (2) the Information Gap Challenge, transferring such guidance to a student without access to the privileged information. To address these challenges, we propose NEFD, a Near-to-Far Self-Distillation method for offline GCRL. We observe that near-future states reveal local outcomes of recorded behavior and thus provide informative privileged references for current decisions. NEFD uses these states as teacher-only information and weights teacher guidance by estimated goal progress. To bridge the information gap, NEFD performs action and representation self-distillation within a shared actor: the former matches teacher action distributions, while the latter uses teacher decision representations as intermediate targets. Experiments on 29 OGBench datasets show that NEFD improves policy learning when distant goals provide weak supervision for current decisions.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, goal-conditioned reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RqgZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NEFD, a Near-to-Far Self-Distillation method for offline goal-conditioned reinforcement learning. The approach employs a teacher policy conditioned on privileged near-future states along with the distant goal, providing dense action-level and representation-level supervision to a student that observes only the distant goal. The distillation is modulated by a goal-progress signal estimated from a learned value function. The method achieves strong empirical gains across 29 OGBench datasets, particularly in long-horizon navigation and multi-step manipulation, where distant goals provide weak guidance for current decisions.

Critical strengths
  1. Clarity of presentation: The method is clearly described, with precise notation, a concise architecture overview, and an end-to-end algorithm. The roles of the teacher/student conditions, the loss terms, and the weighting scheme are easy to follow.

  2. Experimental rigor and validation: Extensive evaluation is conducted on 29 OGBench datasets spanning locomotion and manipulation, in both state and visual modalities, with results averaged over multiple seeds following benchmark protocols. The ablations are thorough, separately isolating (i) the role of privileged information (OPD-Large vs. OPD-Privilege), (ii) action versus representation distillation, and (iii) goal-progress weighting; additional analyses examine performance as a function of goal distance and the predictability of the near future from student features.

Critical weaknesses

1.Missing comparisons and discussions to several recent and strong baselines on OGBench: [1] Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL [2] Transitive RL: Value Learning via Divide and Conquer [3] QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL These are highly relevant to long-horizon GCRL and could affect SOTA claims.

  1. Limited reporting on computational cost (training time and inference time, memory). Distilling both actions and representations with dual condition runs presumably increases runtime; fairness of compute vs baselines is not discussed.
Questions for authors
  1. The near-future state does not cross trajectories, so its role in stitching is unclear. For an in-trajectory goal g=sjg=s_j, the teacher sees smin⁡(i+H,j)s_{\min(i+H,j)}; for a random goal, it sees smin⁡(i+H,T)s_{\min(i+H,T)} (Appendix A.4). Progress weighting can still score whether a local step moves toward an off-trajectory goal, but the teacher never sees a stitched path. Please report how often i+Hi+H is truncated on short stitch trajectories, and note that the gain over LAVL is small on the hardest stitch tasks (8383 vs.\ 8282 on Antmaze-giant-stitch; 8383 vs.\ 8080 on Humanoidmaze-giant-stitch).

  2. How sensitive is NEFD to the near-future horizon H, the temperature β, and the distillation weight λrep? Could you provide a sensitivity analysis and default recommendations across domains?

  3. How robust is goal-progress weighting to value-function misestimation? For example, what happens when you intentionally perturb V or evaluate weighting with earlier/later snapshots?

  4. Could you elaborate on potential degeneracies from sharing parameters between teacher and student? Did you observe failure cases where the model collapses to teacher ≈ student predictions regardless of s_{i+H}, and how did you mitigate them?

6 comments
AuthorsOpenReview ↗

Response to Reviewer RqgZ. (Part1)

Thank you for your thorough review and insightful comments, which have helped us improve the clarity and completeness of our work. Below, we address each of your concerns in detail.

Weaknesses

W1: Missing comparisons and discussions to several recent and strong baselines on OGBench: [1] Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL [2] Transitive RL: Value Learning via Divide and Conquer [3] QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL These are highly relevant to long-horizon GCRL and could affect SOTA claims.

We thank the reviewer for highlighting these recent and highly relevant baselines. CoGHP [1], TRL [2], and QHyer [3] are strong methods for offline goal-conditioned reinforcement learning, offering valuable advances in hierarchical policy learning, divide-and-conquer value learning, and hybrid sequence modeling respectively. We have added citations to these works and discussed their in the Related Work section.

To ensure a fair comparison, all methods are evaluated on the same standard OGBench datasets. We report results from the original papers when available and otherwise reproduce them using official implementations, while QHyer results are taken from its original paper due to the unavailability of its code. As shown in the table below, these results demonstrate that NEFD outperforms these strong recent baselines.

DatasetCoGHP[1]TRL[2]Qhyer[3]NEFD
Pointmaze-large-navigate-v091±833±5-100±0
Pointmaze-giant-navigate-v079±89±4-100±0
Antmaze-large-navigate-v090±346±5-94±1
Antmaze-giant-navigate-v078±82±0-88±2
Cube-single-play-v052±695±284±496±3
Cube-double-play-v025±130±556±275±2
Cube-triple-play-v012±317±110±562±9
Scene-play-v078±777±253±295±2

[1] Choi J, Lee S H, Seo S W. Chain-of-goals hierarchical policy for long-horizon offline goal-conditioned rl. In ICML, 2026.

[2] Park S, Oberai A, Atreya P, et al. Transitive rl: Value learning via divide and conquer. In ICLR, 2026.

[3] Lei X, Wang J, Zhang X, et al. QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL. In ICML, 2026.

W2: Limited reporting on computational cost (training time and inference time, memory). Distilling both actions and representations with dual condition runs presumably increases runtime; fairness of compute vs baselines is not discussed.

We evaluated the average computational cost of SAW, HIQL, and NEFD on the OGBench manipulation tasks using NVIDIA RTX 3090 GPUs under official defaurt setting.

MethodTraining StepsTraining Time (h)GPU Memory (MiB)Inference Latency (ms)
SAW1M1.1995350.556
HIQL1M1.0104940.516
NEFD1M3.3638461.129

NEFD incurs higher training costs due to its history-conditioned Transformer backbone, additional teacher-conditioned computation, and action and representation distillation. At inference, only the Student Transformer is used, without the privileged Teacher or distillation objectives. The higher inference latency also reflects the use of a Transformer-based policy rather than the lighter architectures of SAW and HIQL. Nevertheless, NEFD maintains a low inference latency of 1.129 ms, well within the time budget for 20 Hz real-world robotic control [1].

[1] Reed S, Zolna K, Parisotto E, et al. A generalist agent[J]. In TMLR, 2022.

AuthorsOpenReview ↗

Response to Reviewer RqgZ. (Part2)

Questions

Q1: The near-future state does not cross trajectories, so its role in stitching is unclear. For an in-trajectory goal g=sjg = s_j, the teacher sees smin⁡(i+H,j)s_{\min(i+H,j)}; for a random goal, it sees smin⁡(i+H,T)s_{\min(i+H,T)} (Appendix A.4). Progress weighting can still score whether a local step moves toward an off-trajectory goal, but the teacher never sees a stitched path. Please report how often i+Hi + H is truncated on short stitch trajectories, and note that the gain over LAVL is small on the hardest stitch tasks (83 vs.\ 82 on Antmaze-giant-stitch; 83 vs.\ 80 on Humanoidmaze-giant-stitch).

The Teacher's near-future states are indeed sampled within the same trajectory. However, the distant goal gg can be sampled from other trajectories, allowing the Teacher to receive cross-trajectory goal conditioning. Trajectory stitching does not require the Teacher to observe a complete stitched path. Similar to HIQL, SAW, and LAVL, NEFD uses a goal-conditioned value function trained through TD learning, which propagates goal-conditioned value information through transitions across the offline dataset, allowing behaviors from different trajectories to be stitch when reaching distant goals.

NEFD does not aim to introduce a new stitching mechanism. Instead, as discussed in Section 3, it addresses the weak supervision for current decisions under distant goals. The observed near-future state provides the Teacher with a more direct reference for current decision and is also used to estimate progress toward the distant goal, which determines the weight of the corresponding teacher guidance. These roles remain consistent across stitch, navigate, and manipulation tasks, rather than being specific to trajectory stitching.

To further examine the contribution of our proposed components to trajectory stitching, we extend the distillation and goal progress weighting ablations in Tables 3 and 4 to stitch envionments. The results are presented below.

DatasetQRLHIQLw/o Lact+Lrep\mathcal{L}_{\mathrm{act}}+\mathcal{L}_{\mathrm{rep}}w/o Lact\mathcal{L}_{\mathrm{act}}w/o Lrep\mathcal{L}_{\mathrm{rep}}w/o act weightw/o rep weightNEFD
Antmaze-gaint-stitch-v00±02±20±077±552±574±261±483±3
Humanoid-giant-stitch-v00±03±21±065±145±368±257±283±2

We acknowledge that the gains over LAVL are modest on the hardest stitch tasks, including Antmaze-giant-stitch (83% vs. 82%) and Humanoidmaze-giant-stitch (83% vs. 80%). LAVL's strong performance stems not only from its LAN-based value learning but also from its HIQL-style hierarchical subgoal-based policy learning. In contrast, NEFD directly predicts actions from the observed history and distant goal, avoiding potential subgoal prediction errors of LAVL. NEFD also achieves larger gains on other stitch tasks over LAVL, including Pointmaze-large-stitch (100% vs. 87%) and Humanoidmaze-large-stitch (82% vs. 72%).

We also report the near-future horizon truncation rates on the giant-stitch datasets in the table below. As described in Appendix A.4, when i+Hi+H exceeds the sampled in-trajectory goal index jj, the Teacher's near-future state is truncated to sjs_j; for randomly sampled goals, it is truncated to the trajectory endpoint sTs_T when i+H>Ti+H>T.

DatasetHorizon HHHIQLLAVLNEFD
Pointmaze-giant-stitch-v02512.50%12.50%12.79%
Antmaze-giant-stitch-v02512.50%12.50%12.79%
Humanoidmaze-giant-stitch-v010025.00%25.00%25.28%
AuthorsOpenReview ↗

Response to Reviewer RqgZ. (Part3)

Q2: How sensitive is NEFD to the near-future horizon HH, the temperature ββ, and the distillation weight λrepλ_{rep}? Could you provide a sensitivity analysis and default recommendations across domains?

We have already investigated the sensitivity of NEFD to the near-future horizon HH in Appendix B.3.3 (Table 10), where we vary HH to 0.8H0.8H and 1.2H1.2H. The results show that NEFD maintains strong performance under these variations, with the default horizon achieving the best or tied-best results across all three tasks. To further assess robustness over a wider range, we extend this analysis to 0.6H0.6H and 1.4H1.4H on the same three tasks. The additional results are presented below.

DatasetQRLHIQL0.6H0.6H0.8H0.8H1.2H1.2H1.4H1.4HNEFD
Pointmaze-giant-navigate-v068±746±9100±099±1100±099±1100±0
Antmaze-giant-navigate-v014±365±581±181±184±183±288±2
Scene-play-v05±138±393±295±293±394±195±2

We further investigate the sensitivity of NEFD to the goal progress weighting temperature β∈{2,4,5}\beta \in \{2, 4, 5\}, in addition to the default β=3\beta=3, on the same three tasks. The results show that NEFD maintains consistently strong performance across these settings, indicating that the method is relatively insensitive to variations in β\beta.

DatasetQRLHIQLβ=2\beta=2β=4\beta=4β=5\beta=5NEFD(β=3\beta=3)
Pointmaze-giant-navigate-v068±746±9100±0100±0100±0100±0
Antmaze-giant-navigate-v014±365±582±384±186±288±2
Scene-play-v05±138±386±595±188±295±2

We also examine the sensitivity to the representation distillation coefficient λrep\lambda_{\mathrm{rep}} by evaluating λrep=4\lambda_{\mathrm{rep}}=4 and λrep=12\lambda_{\mathrm{rep}}=12, alongside the default λrep=8\lambda_{\mathrm{rep}}=8, on the same three tasks. NEFD maintains stable performance across these variations, suggesting that the method is not highly sensitive to the choice of λrep\lambda_{\mathrm{rep}}.

DatasetQRLHIQLλrep=4\lambda_{rep}=4λrep=12\lambda_{rep}=12NEFD(λrep=8\lambda_{rep}=8)
Pointmaze-giant-navigate-v068±746±999±199±1100±0
Antmaze-giant-navigate-v014±365±585±184±288±2
Scene-play-v05±138±392±194±295±2

Q3: How robust is goal-progress weighting to value-function misestimation? For example, what happens when you intentionally perturb V or evaluate weighting with earlier/later snapshots?

We thank the reviewer for this suggestion. To evaluate the robustness of goal-progress weighting to value-function estimation errors, we conduct two additional experiments:

(1) Value Perturbation. We add Gaussian noise to the estimated goal progress, with noise levels σ∈{0.10,0.20,0.50}\sigma \in \{0.10, 0.20, 0.50\} relative to the standard deviation of the original progress estimates. (2) Value Snapshot Analysis. We use value-function checkpoints saved at 2525\\%, 5050\\% and 7575\\% of a standard NEFD training run. These checkpoints are then frozen and used only to compute distillation weights. In contrast, NEFD follows the original training procedure, where the value function is continuously updated alongside the policy and its current estimates are used for weighting.

DatasetQRLHIQLσ=0.10\sigma=0.10σ=0.20\sigma=0.20σ=0.50\sigma=0.50NEFD
Pointmaze-giant-navigate-v068±746±9100±0100±098±1100±0
Antmaze-giant-navigate-v014±365±583±184±182±088±2
Scene-play-v05±138±395±191±492±195±2
DatasetQRLHIQL2525\\%5050\\%7575\\%NEFD
Pointmaze-giant-navigate-v068±746±998±1100±099±1100±0
Antmaze-giant-navigate-v014±365±584±282±485±288±2
Scene-play-v05±138±391±294±193±195±2
Reviewer_EgE7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at a problem in offline goal-conditioned RL: when the goal is far away, it is hard to tell which action is useful right now. The idea is to let the teacher see a near-future state from the recorded trajectory, so it has more information to learn the current action. The student does not see this future state and learns from the teacher by matching its action distribution and internal features. Both use the same network. The method also gives more weight to segments that make more progress toward the goal, estimated by a value function. The authors evaluate it on 29 OGBench datasets and report gains on navigation and manipulation tasks, especially some of the harder state-based manipulation tasks.

Critical strengths
  1. I think the teacher construction makes sense, and the experiments support the motivation. When the final goal is far away, the near-future state gives the teacher a concrete reference for what the recorded actions lead to. This information is already available in the offline data, and the student does not need it at test time. The comparison with a larger teacher is useful here: increasing teacher size alone gives little improvement, while providing the near-future state helps substantially. The representation distillation also contributes beyond matching actions. For example, removing it reduces Antmaze-giant-navigate success from 88% to 73%. These results make the proposed way of transferring the teacher’s guidance more convincing to me.

  2. The improvement on the harder state-based manipulation tasks is substantial. NEFD reaches 75% success on Cube-double compared with 42% for the strongest baseline, and 62% on Cube-triple compared with 11%. The larger gain on Cube-triple is particularly interesting because the task requires composing more operations before reaching the final goal. The trajectory-stitching results are also strong, including on the giant layouts where several baselines struggle. Together, these results support the paper’s claim that the method helps when reaching a distant goal requires a sequence of useful local decisions.

Critical weaknesses
  1. I think the difference from SAW could be made clearer. Both methods use local information to help train a policy for a distant goal. I understand that NEFD makes different choices in the teacher inputs, representation distillation, and progress weighting, but the comparison with SAW also involves differences in the actor architecture and value learning. The ablations show that these components help within NEFD, but I would still like to see how much of the gap remains when SAW uses the same backbone and value-learning setup. This would directly affect my assessment of the methodological contribution.

  2. I would particularly like to see ablations on Cube-double and Cube-triple. The gains on these tasks are large, especially the improvement from 11% for the strongest baseline to 62% on Cube-triple. This is one of the results I find most interesting. However, the main ablations are on Pointmaze, Antmaze, and Scene, so I am still not sure what accounts for such a large improvement on Cube. Adding a matched No Distill baseline and separately removing action and representation distillation on these tasks would make the results more convincing. The full method clearly works well here, but the source of its largest gains is not sufficiently explained.

Questions for authors
  1. Could you clarify which parts of the actor architecture and value-learning setup differ between NEFD and SAW in the reported experiments? Have you compared them using the same backbone and value-learning setup? If so, how much of the gap remains? I mainly want to understand how much of the improvement comes from the proposed distillation design.

  2. Do you have No Distill results and separate action/representation distillation ablations on Cube-double and Cube-triple? These tasks show the largest gains, especially the improvement from 11% to 62% on Cube-triple, but the main ablations do not cover them. If these results are available, could you share them? This would help explain where the large improvement comes from.

  3. For the same state and goal, the dataset may contain several different but valid routes. The teacher sees the near-future state from one particular route, while the student does not. Could matching these teacher predictions cause the student to average different actions and produce an unsuitable action? Have you observed this in practice, and how does the current method handle it?

  4. In the representation analysis, a higher R² shows that the student’s features better predict the near-future state recorded in the dataset. But this does not directly show that the student makes better goal-conditioned decisions. Do you have evidence that distinguishes better prediction of the recorded behavior from better use of the goal to choose actions? This would help me understand why representation distillation works.

  5. Progress weighting depends on the value function’s estimate of progress. The appendix compares different value functions, but have you directly checked whether the highly weighted segments actually help reach the goal? In multistage tasks such as Cube, some necessary actions may not show clear short-term progress. Could these segments receive low weights, and have you observed this issue in the experiments?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EgE7. (Part1)

We sincerely appreciate the time and effort you devoted to reviewing our work. Your constructive feedback has helped us strengthen our experimental analysis and provide more rigorous evidence for our key claims. Below, we provide detailed responses to each of your concerns.

Weaknesses

W1: I think the difference from SAW could be made clearer. Both methods use local information to help train a policy for a distant goal. I understand that NEFD makes different choices in the teacher inputs, representation distillation, and progress weighting, but the comparison with SAW also involves differences in the actor architecture and value learning. The ablations show that these components help within NEFD, but I would still like to see how much of the gap remains when SAW uses the same backbone and value-learning setup. This would directly affect my assessment of the methodological contribution.

We thank the reviewer for this valuable suggestion. To isolate the effects of backbone architecture and value learning, we conduct additional controlled experiments with SAW. Specifically, SAW+Transformer replaces SAW's original MLP actor with the same history-conditioned Transformer used in NEFD, SAW+LAN incorporates LAN value learning while retaining the original MLP actor, and SAW+Transformer+LAN combines both modifications. All variants retain SAW's original policy learning mechanism. As shown below, NEFD consistently outperforms SAW even when both the backbone and value learning are matched, supporting the effectiveness of our proposed distillation mechanisms beyond these differences.

DatasetSAW(Original)SAW+TransformerSAW+LANSAW+Transformer+LANNEFD
Pointmaze-giant-navigate-v068±870±168±472±3100±0
Antmaze-giant-navigate-v073±429±072±348±288±2
Scene-play-v063±666±464±169±195±2

Regarding the methodological differences, SAW bootstraps a flat policy from a separately trained subgoal-conditioned policy through action supervision. In contrast, NEFD conditions both Teacher and Student on the same distant goal, while providing the Teacher with the observed near-future state as additional privileged information. NEFD further uses estimated goal progress to weight the Teacher guidance, which is transferred through both action and representation self-distillation.

W2: I would particularly like to see ablations on Cube-double and Cube-triple. The gains on these tasks are large, especially the improvement from 11% for the strongest baseline to 62% on Cube-triple. This is one of the results I find most interesting. However, the main ablations are on Pointmaze, Antmaze, and Scene, so I am still not sure what accounts for such a large improvement on Cube. Adding a matched No Distill baseline and separately removing action and representation distillation on these tasks would make the results more convincing. The full method clearly works well here, but the source of its largest gains is not sufficiently explained.

We thank the reviewer for highlighting the substantial improvements on Cube-double and Cube-triple. To further investigate the contributions of NEFD's individual components, we extend our ablation studies to both tasks. Specifically, we include a matched variant that removes both action and representation distillation (w/o Lact+Lrep\mathcal{L}_{\mathrm{act}}+\mathcal{L}_{\mathrm{rep}}), remove each distillation objective separately, and disable goal progress weighting for each objective. All other training and evaluation settings remain unchanged.

These multistage manipulation tasks require a sequence of intermediate actions, such as lifting and placing objects, for which the distant goal alone provides weak supervision. NEFD addresses this challenge by leveraging observed near-future states as privileged information to provide more informative supervision for these intermediate decisions. Action distillation transfers near-future-informed action guidance, while representation distillation helps the Student encode intermediate outcomes. Goal progress weighting further adjusts the strength of this supervision according to the estimated contribution of each near-future outcome toward the final goal.

Datasetw/o Lact+Lrep\mathcal{L}_{\mathrm{act}}+\mathcal{L}_{\mathrm{rep}}w/o Lact\mathcal{L}_{\mathrm{act}}w/o Lrep\mathcal{L}_{\mathrm{rep}}w/o act weightw/o rep weightNEFD
Cube-double-play-v06±562±342±167±250±475±2
Cube-triple-play-v02±154±136±252±441±262±9
AuthorsOpenReview ↗

Response to Reviewer EgE7. (Part2)

Questions

Q1: Could you clarify which parts of the actor architecture and value-learning setup differ between NEFD and SAW in the reported experiments? Have you compared them using the same backbone and value-learning setup? If so, how much of the gap remains? I mainly want to understand how much of the improvement comes from the proposed distillation design.

We thank the reviewer for highlighting this important concern. In the original comparison, SAW uses two separate MLP policies and its original GCIVL-based value learning setup, whereas NEFD employs a shared causal Transformer over state–action histories and adopts LAN value learning from LAVL.

We have conducted additional controlled comparisons using the same backbone and value learning setup, as reported in our response to Weakness 1.

Q2: Do you have No Distill results and separate action/representation distillation ablations on Cube-double and Cube-triple? These tasks show the largest gains, especially the improvement from 11% to 62% on Cube-triple, but the main ablations do not cover them. If these results are available, could you share them? This would help explain where the large improvement comes from.

We have already addressed this question in our response to Weakness 2. For clarity, the No Distill setting corresponds to w/o Lact+Lrep\mathcal{L}_{\mathrm{act}}+\mathcal{L}_{\mathrm{rep}} in our ablation table, where both action and representation distillation are removed while the remaining training objectives are retained.

Q3: For the same state and goal, the dataset may contain several different but valid routes. The teacher sees the near-future state from one particular route, while the student does not. Could matching these teacher predictions cause the student to average different actions and produce an unsuitable action? Have you observed this in practice, and how does the current method handle it?

We thank the reviewer for raising this concern. Theoretically, uniform action distillation may lead to unsuitable action averaging when different valid routes induce conflicting teacher predictions for the same history and distant goal. NEFD mitigates this issue through goal progress weighting, which emphasizes Teacher guidance associated with greater estimated progress toward the distant goal.

Controlled experiment. We construct 16 Antmaze-giant-navigate cases with two distinct valid routes sharing the same initial state and goal. A frozen Teacher produces different action targets conditioned on the corresponding near-future states. For example, if two Teacher actions are (1,0)(1,0) and (−1,0)(-1,0) in a two-dimensional illustration, their midpoint is (0,0)(0,0), representing the average of two distinct action targets. Using 1,600 training samples, we compare uniform and goal progress weighted action distillation under identical settings, varying only the supervision weights. We evaluate both Students on 100 held-out cases using two metrics: (1) Actions Near Teacher Midpoint, the percentage of Student actions within 25% of the distance between two sufficiently distinct Teacher actions from their midpoint; and (2) MSE to Teacher Midpoint, the mean squared error between Student actions and the midpoint of the corresponding Teacher actions. A lower percentage of actions near the midpoint indicates less action averaging, while a higher MSE indicates that Student actions deviate further from the midpoint.

MethodActions Near Teacher Midpoint (%) ↓\downarrowMSE to Teacher Midpoint ↑\uparrow
w/o Goal Progress Weighting28±40.052±0.08
Goal Progress Weighting6±10.137±0.07

As shown above, goal progress weighting substantially reduces the tendency to average conflicting Teacher predictions (28% to 6%). These results demonstrate that progress weighting mitigates action averaging under conflicting supervision.

AuthorsOpenReview ↗

Response to Reviewer EgE7. (Part3)

Q4: In the representation analysis, a higher R² shows that the student’s features better predict the near-future state recorded in the dataset. But this does not directly show that the student makes better goal-conditioned decisions. Do you have evidence that distinguishes better prediction of the recorded behavior from better use of the goal to choose actions? This would help me understand why representation distillation works.

We thank the reviewer for this insightful observation. To directly examine whether representation distillation improves goal-conditioned decisions, we compare NEFD with its variant without Lrep\mathcal{L}_{\mathrm{rep}} on Pointmaze-giant-navigate. we conduct an additional goal-conditioned action quality analysis on 2,000 shared decision points from Pointmaze-giant-navigate. At each point, different policies select actions from the same state and distant goal, and each action is independently executed for one step. We estimate goal-conditioned state values V^MC(s,g)\hat V_{\mathrm{MC}}(s,g) as the negative average remaining steps to the goal through Monte Carlo estimation, using 16 environment rollouts under a shared, fixed BFS-based navigation controller. We then measure the estimated goal-value improvement as δMC=V^MC(st+1,g)−V^MC(st,g)\delta_{\mathrm{MC}}=\hat V_{\mathrm{MC}}(s_{t+1},g)-\hat V_{\mathrm{MC}}(s_t,g). A higher δMC\delta_{\mathrm{MC}} indicates better current action selection toward the distant goal, allowing us to directly evaluate whether NEFD's dense Teacher supervision improves local decision quality. Following Figures 3 and 4, Goal Distance denotes the shortest-path distance to the goal in maze cells.

As shown in the table below, NEFD consistently selects actions with higher estimated goal-value improvement across all goal distances, including distant goals, demonstrating the benefit of representation distillation for current action selection.

Goal DistanceHIQLSAWw/o Lrep\mathcal{L}_{\mathrm{rep}}NEFD
60.82±0.090.85±0.031.11±0.031.18±0.01
120.83±0.120.86±0.061.22±0.021.28±0.01
180.81±0.080.80±0.051.04±0.021.14±0.02
240.60±0.060.68±0.100.96±0.021.04±0.04
310.48±0.120.35±0.120.77±0.080.89±0.02

To further understand this improvement, we extend Figure 4cc to evaluate the actual HH-step outcomes of frozen policy rollouts, rather than recorded dataset trajectories. As shown below, the higher rollout R2R^2 suggests that representation distillation helps the Student better encode where its own policy is likely to reach after HH steps, while the higher estimated goal-value improvement indicates that these future states are more favorable for reaching the distant goal.

HH-step Rollout Prediction (R2R^2)

Datasetw/o Lrep\mathcal{L}_{\mathrm{rep}}NEFD
Pointmaze-giant-navigate-v00.778 ± 0.0030.879 ± 0.004
Antmaze-giant-navigate-v00.586 ± 0.0080.659 ± 0.019
Scene-play-v00.599 ± 0.0120.800 ± 0.006

Estimated HH-step Goal-value Improvement

Datasetw/o Lrep\mathcal{L}_{\mathrm{rep}}NEFD
Pointmaze-giant-navigate-v011.699 ± 0.18011.954 ± 0.116
Antmaze-giant-navigate-v04.215 ± 0.1675.085 ± 0.193
Scene-play-v04.246 ± 0.0476.281 ± 0.057
AuthorsOpenReview ↗

Response to Reviewer EgE7. (Part4)

Q5: Progress weighting depends on the value function’s estimate of progress. The appendix compares different value functions, but have you directly checked whether the highly weighted segments actually help reach the goal? In multistage tasks such as Cube, some necessary actions may not show clear short-term progress. Could these segments receive low weights, and have you observed this issue in the experiments?

We thank the reviewer for these insightful questions. We conduct additional analyses to examine the contribution of highly weighted segments and the potential underweighting of necessary behaviors in multistage tasks.

(1) Contribution of highly weighted segments. To examine the contribution of highly weighted segments to goal reaching performance, we conduct an additional ablation, w/o Top-2020\\% Guidance. Specifically, we exclude the 20% of positions with the highest estimated goal progress in each minibatch from both action and representation distillation, while keeping all other training settings unchanged.

DatasetQRLHIQLw/o Top-2020\\% GuidanceNEFD
Pointmaze-giant-navigate-v068±746±996±1100±0
Antmaze-giant-navigate-v014±365±575±388±2
Cuble-double-play-v01±06±252±175±2

(2) Potential underweighting of necessary behaviors in Cube. We have also observed cases in Cube where necessary actions receive low progress weights despite being important for reaching the final goal. Ideally, an accurate goal-conditioned value function should recognize the long-term contribution of these actions, even when they show little immediate progress, and assign them appropriate weights. However, value overestimation and underestimation remain common challenges in RL, particularly in offline RL. Our progress estimates are also approximate, and accurately evaluating such necessary behaviors remains an open challenge in the field.

Reviewer_JYPCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Summary

The paper argues that, in offline goal-conditioned RL, distant goals provide weak supervision for the current action. Value-guided methods struggle to separate actions whose values differ only slightly, and subgoal-based methods depend on the quality of generated subgoals. Inspired by on-policy distillation (OPD) with privileged teachers in LLMs, the authors propose NEFD (Near-to-Far Self-Distillation).

  • One actor, two views. A single goal-conditioned causal Transformer actor is run in two views. In the teacher view, the condition token also encodes the state H steps ahead in the dataset trajectory, used as privileged information. In the student view, it encodes only the goal. Both views are trained with AWR, using the teacher-side advantage V(s_{i+1}, s_{i+H}) − V(s_i, s_{i+H}) and the student-side advantage V(s_{i+1}, g) − V(s_i, g).
  • Two distillation losses. The student is trained to match the teacher's action distribution (a KL term with stop-gradient) and its projected pre-action-head representation (an MSE term with VICReg-style variance/covariance regularisation).
  • Goal-progress weighting. Both losses are weighted by an exponentiated, batch-standardised goal-progress estimate, V(s_{i+H}, g) − V(s_i, g).
  • Value function. The value function is LAVL's Latent Alignment Network.

On 29 OGBench datasets (22 state-based, 7 visual), NEFD reports the best or tied-best mean on 28. Ablations cover the privileged teacher, the two distillation objectives, the progress weighting, and several hyperparameters.

Critical strengths

Pros

  1. The empirical evaluation is broad and the results are consistently strong. NEFD is evaluated on 29 OGBench datasets spanning navigation, stitching, manipulation, and visual observations, and achieves the best or tied-best mean success on nearly all of them. The gains are especially large on multi-stage manipulation, e.g., Cube-triple 62% vs. 11% for the strongest baseline. A distance-binned analysis (Fig. 3) shows that NEFD's performance degrades more slowly with goal distance than the baselines'.
Critical weaknesses

Cons

  1. The motivation for the method is unclear and is not supported experimentally.

    • The paper motivates NEFD by analogy to OPD in LLMs, but never establishes how OPD's success relates to recommending better local targets along an offline trajectory.
    • The analogy is also loose: OPD distils on student-generated sequences, whereas NEFD distils on fixed dataset sequences. Mechanically this is teacher-forced, privileged-information distillation rather than on-policy distillation.
    • The key design choice, using the near-future state s_{t+H} as the teacher's privileged input, is asserted rather than justified. A teacher conditioned on s_{t+H} is essentially a subgoal-conditioned policy with a hindsight subgoal, which is close to how SAW trains a flat policy from a subgoal-conditioned one. The paper does not explain why this is the right privileged information, or what it offers beyond existing subgoal-based supervision.
    • No alternative is compared, such as a generated subgoal, a different choice of future state, or a learned subgoal. H is varied only by ±20% (Table 10).
    • It is also unclear why the teacher's guidance should be reliable on suboptimal data, where s_{t+H} may lead away from g.
  2. It is unclear why the method works or which problem it actually solves.

    • The improvements over the strongest baseline are often marginal, e.g., Antmaze large-navigate 94 vs. 93, giant-navigate 88 vs. 85, and giant-stitch 83 vs. 82 against LAVL. On Visual-AntMaze, NEFD is close to CRL (95/86/51 vs. 94/84/47).
    • NEFD reuses LAVL's value function (LAN plus continuity regularisation), and its actor is a K = 20 history Transformer while the baselines use MLP actors. So it is hard to tell how much of the gain comes from the proposed distillation itself, as opposed to the inherited value function and the different architecture.
    • The paper says NEFD addresses weak supervision for current decisions under distant goals, but never measures this directly. For example, it does not show that the student's action ranking or action-value discrimination improves for distant goals.
    • In particular, there is no explanation of why NEFD performs well on very long-horizon tasks such as Humanoidmaze-giant.
    • The representation probe (Fig. 4c) shows only that student features predict the near-future displacement better; it does not show that this causes the performance gain.

Overall, the paper provides neither a clear motivation for why the method should be designed this way nor an analysis of why it works. Combined with the confounds above, this makes the reported improvements difficult to trust.

Questions for authors

Already mentioned in the weaknesses section.

12 comments
AuthorsOpenReview ↗

Response to Reviewer JYPC. (Part1)

We sincerely thank the reviewer for the thoughtful and constructive feedback. The reviewer’s suggestions have helped us strengthen our experimental analysis and improve the overall rigor of the paper. We address each of the reviewer’s comments in detail below.

Weaknesses

W1.1: The paper motivates NEFD by analogy to OPD in LLMs, but never establishes how OPD's success relates to recommending better local targets along an offline trajectory.
W1.2: The analogy is also loose: OPD distils on student-generated sequences, whereas NEFD distils on fixed dataset sequences. Mechanically this is teacher-forced, privileged-information distillation rather than on-policy distillation.

We thank the reviewer for highlighting this distinction. As stated in the Introduction, NEFD is inspired by the dense predictive supervision of OPD, rather than its on-policy sampling mechanism. We do not claim that NEFD is an OPD method in our paper. Instead, we adopt the idea of using dense Teacher predictions to provide local supervision for the Student's current decisions. We agree that our presentation may have caused some confusion and have clarified this distinction in the Discussion and Related Work sections.

NEFD is not simply teacher-forced regression to recorded dataset actions. The Teacher provides predictive distributions that directly supervise the Student's current generated action,such action is not fixed data, as defined in Eq. (5): $ \mathcal{L}_i^{\mathrm{act}} = D_{\mathrm{KL}}\left( \mathrm{sg}\left[ \pi_\theta(\cdot \mid \mathcal{H}_{t,\leq i}^{T}) \right] \parallel \pi_\theta(\cdot \mid \mathcal{H}_{t,\leq i}^{S}) \right). $

To further support this motivation, we conduct an additional goal-conditioned action quality analysis on 2,000 shared decision points from Pointmaze-giant-navigate. At each point, different policies select actions from the same state and distant goal, and each action is independently executed for one step. We estimate goal-conditioned state values V^MC(s,g)\hat V_{\mathrm{MC}}(s,g) as the negative average remaining steps to the goal through Monte Carlo estimation, using 16 environment rollouts under a shared, fixed BFS-based navigation controller. We then measure the estimated goal-value improvement as δMC=V^MC(st+1,g)−V^MC(st,g)\delta_{\mathrm{MC}}=\hat V_{\mathrm{MC}}(s_{t+1},g)-\hat V_{\mathrm{MC}}(s_t,g). A higher δMC\delta_{\mathrm{MC}} indicates better current action selection toward the distant goal, allowing us to directly evaluate whether NEFD's dense Teacher supervision improves local decision quality. Following Figures 3 and 4, Goal Distance denotes the shortest-path distance to the goal in maze cells.

As shown below, NEFD consistently achieves higher estimated goal-value improvement across all goal distances, including distant goals. Removing either action or representation distillation from the privileged Teacher reduces this improvement. These results provide direct empirical evidence that privileged Teacher improves current action selection under distant goals.

Goal DistanceHIQLSAWw/o Lact+Lrep\mathcal{L}_{\mathrm{act}}+\mathcal{L}_{\mathrm{rep}}w/o Lact\mathcal{L}_{\mathrm{act}}w/o Lrep\mathcal{L}_{\mathrm{rep}}NEFD
60.82±0.090.85±0.030.51±0.061.07±0.031.11±0.031.18±0.01
120.83±0.120.86±0.060.59±0.041.24±0.011.22±0.021.28±0.01
180.81±0.080.80±0.050.48±0.061.13±0.031.04±0.021.14±0.02
240.60±0.060.68±0.100.44±0.090.99±0.030.96±0.021.04±0.04
310.48±0.120.35±0.120.28±0.010.84±0.040.77±0.080.89±0.02
AuthorsOpenReview ↗

Response to Reviewer JYPC. (Part2)

W1.3: The key design choice, using the near-future state st+Hs_{t+H} as the teacher's privileged input, is asserted rather than justified. A teacher conditioned on s_{t+H} is essentially a subgoal-conditioned policy with a hindsight subgoal, which is close to how SAW trains a flat policy from a subgoal-conditioned one. The paper does not explain why this is the right privileged information, or what it offers beyond existing subgoal-based supervision.

We thank the reviewer for this valuable comment.

(1) Why Near-future States as Privileged Information? As discussed in the Introduction, distant goals can provide weak or ambiguous supervision for current decisions. We choose the observed near-future state as privileged information because it reveals the local outcome of recorded behavior, providing the Teacher with a more direct reference for current action prediction.

To further validate this design choice, we conduct additional experiments with three new NEFD variants, replacing the Teacher's observed near-future state with: (1) Random Future State, a randomly sampled future state from the same recorded trajectory; (2) Generated Subgoal, an intermediate goal state predicted by a high-level policy following HIQL; and (3) Generated Latent Subgoal, an intermediate goal representation predicted by a high-level policy following HIQL's latent subgoal formulation. As shown in the table below, all three variants using alternative privileged information consistently underperform NEFD across the evaluated tasks, validating the effectiveness of our design choice.

DatasetQRLHIQLRandom Future StateGenerated SubgoalGenerated Latent SubgoalNEFD
Pointmaze-giant-navigate-v068±746±927±264±480±2100±0
Antmaze-giant-navigate-v014±365±50±06±171±388±2
Scene-play-v05±138±30±063±542±795±2

(2) Beyond Existing Subgoal-based Supervision. While NEFD also leverages hindsight information, it differs from existing subgoal-based methods in three key aspects:

  • Observed hindsight without prediction errors. Unlike HIQL, NEFD directly uses observed near-future states, avoiding errors from learned subgoal prediction.
  • Hindsight as privileged information. SAW learns subgoal-conditioned and distant-goal-conditioned policies with different objectives, potentially introducing learning errors due to a mismatch between reaching intermediate subgoals and the final goal. In contrast, NEFD conditions both Teacher and Student on the same distant goal gg, using st+Hs_{t+H} only as additional privileged information for the Teacher.
  • Goal progress weighting. Rather than treating all hindsight outcomes equally, NEFD emphasizes teacher guidance associated with greater estimated progress toward gg, reducing the influence of less favorable local outcomes.

To further examine these design choices, we introduce an additional Teacher w/o gg variant, which removes distant goal conditioning from the Teacher while retaining it for the Student, and report its results alongside the existing w/o act weight and w/o rep weight ablations in the table below.

All three variants retain the same near-future states as privileged information. Removing any of these additional design components consistently leads to performance degradation across all three tasks, demonstrating the importance of NEFD's additional design choices beyond subgoal-based supervision alone.

DatasetQRLHIQLTeacher w/o ggw/o act weightw/o rep weightNEFD
Pointmaze-giant-navigate-v068±746±984±497±287±3100±0
Antmaze-giant-navigate-v014±365±578±483±176±288±2
Scene-play-v05±138±384±690±381±195±2

W1.4: No alternative is compared, such as a generated subgoal, a different choice of future state, or a learned subgoal. H is varied only by ±20% (Table 10).

As discussed in our response to W1.3, we have conducted additional experiments comparing the observed near-future state with alternative privileged inputs, including randomly sampled future states, generated subgoals, and generated latent subgoals. We further extend the horizon ablation in Table 10 with two additional variants, 0.6H0.6H and 1.4H1.4H, expanding the evaluated range from ±20% to ±40%. The results are presented in the table below.

DatasetQRLHIQL0.6H0.6H0.8H0.8H1.2H1.2H1.4H1.4HNEFD
Pointmaze-giant-navigate-v068±746±9100±099±1100±099±1100±0
Antmaze-giant-navigate-v014±365±581±181±184±183±288±2
Scene-play-v05±138±393±295±293±394±195±2
AuthorsOpenReview ↗

Response to Reviewer JYPC. (Part3)

W1.5: It is also unclear why the teacher's guidance should be reliable on suboptimal data, where st+Hs_{t+H} may lead away from gg.

We thank the reviewer for raising this concern. OGBench itself contains a large number of suboptimal trajectories in its offline datasets, and its task setting requires learning effective goal-conditioned policies from such data. NEFD does not assume that every observed near-future state provides reliable guidance. Specifically, we estimates the progress of each near-future state toward the distant goal using the goal-conditioned value difference V(st+H,g)−V(st,g)V(s_{t+H},g)-V(s_t,g). Teacher guidance associated with greater estimated progress receives higher weights in both action and representation distillation, while guidance from less goal-relevant segments is downweighted. As shown in Table 4 of our paper, removing goal-progress weighting from either distillation objective consistently reduces performance, demonstrating its effectiveness.

DatasetQRLHIQLw/o act weightw/o rep weightNEFD
Pointmaze-giant-navigate-v068±746±997±287±3100±0
Antmaze-giant-navigate-v014±365±583±176±288±2
Scene-play-v05±138±390±381±195±2

W2.1: The improvements over the strongest baseline are often marginal, e.g., Antmaze large-navigate 94 vs. 93, giant-navigate 88 vs. 85, and giant-stitch 83 vs. 82 against LAVL. On Visual-AntMaze, NEFD is close to CRL (95/86/51 vs. 94/84/47).

The strongest baselines vary across environments. LAVL is the strongest baseline on Antmaze, whereas CRL is the strongest baseline on Visual-AntMaze. Importantly, across all 29 OGBench datasets, NEFD achieves an average success rate of 84.4%, surpassing LAVL (72.6%) and CRL (28.2%) by 11.8 and 56.2 percentage points, respectively. Particularly substantial gains are observed on multi-stage manipulation tasks, where NEFD improves success rates over LAVL from 42% to 75% on Cube-double-play-v0 and from 11% to 62% on Cube-triple-play-v0.

W2.2: NEFD reuses LAVL's value function (LAN plus continuity regularisation), and its actor is a K = 20 history Transformer while the baselines use MLP actors. So it is hard to tell how much of the gain comes from the proposed distillation itself, as opposed to the inherited value function and the different architecture.

We thank the reviewer for raising this important concern. To further assess the impact of architectural differences, we additionally evaluate SAW and LAVL with the same history-conditioned Transformer backbone as NEFD. For SAW, we also adopt LAN value learning. Both variants use the same history lengths as NEFD, while retaining their respective policy-learning mechanisms.

DatasetSAWLAVLSAW+Transformer + LANLAVL+TransformerNEFD
Pointmaze-giant-navigate-v068±891±472±393±4100±0
Antmaze-giant-navigate-v073±485±248±274±288±2
Scene-play-v063±688±469±159±495±2

Furthermore, Table 3 of the paper directly isolates the contributions of action and representation self-distillation. All variants use the same Transformer backbone and LAN value function, allowing us to evaluate the effects of the proposed distillation objectives under identical architectural and value-learning settings. The results show that removing either distillation objective consistently reduces performance, demonstrating the contributions of both action and representation self-distillation.

DatasetQRLHIQLw/o Lact\mathcal{L}_{\mathrm{act}}w/o Lrep\mathcal{L}_{\mathrm{rep}}NEFD
Pointmaze-giant-navigate-v068±746±994±288±3100±0
Antmaze-giant-navigate-v014 ±365±582±273±288±2
Scene-play-v05±138±387±785±495±2

Comments on the paper

1 comment

Dear Reviewer JYPC,

The authors have provided a follow-up response to address your remaining concerns. Please acknowledge if you have read it and discuss with the authors if any clarifications are needed to help you complete your assessment. Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.