acceptodds
Under review as a conference paper at ICLR 2027

Intermediate-Goal-Informed Scorer for Long-Horizon Goal-Conditioned RL

Abstract

Long-horizon goal-conditioned reinforcement learning faces a central challenge in reliably distinguishing which primitive actions are useful for distant goals. As the goal becomes more distant, action-dependent differences can become weak relative to estimation noise, making primitive-action discrimination increasingly difficult. Prior work has addressed this challenge through hierarchical methods using intermediate goals and flat methods learning long-horizon values or goal-reaching relations. However, hierarchical methods introduce additional complexity, while accurate long-horizon estimation in flat methods does not necessarily ensure effective primitive-action discrimination. To address this challenge, we propose the Intermediate-Goal-Informed Scorer (IGIS), which learns a more discriminative representation for primitive-action scoring. IGIS constructs a goal-conditioned action scorer from state-action and state-goal representations and strengthens the distant-goal representation by distilling shorter-horizon information from intermediate-goal representations along offline trajectories. This enables more effective discrimination among candidate actions while requiring neither subgoal generation nor hierarchical execution at test time. Across challenging long-horizon locomotion and manipulation benchmarks, IGIS shows consistent gains over strong baselines.

Primary area:
reinforcement learning
Keywords:
Goal-Conditioned Reinforcement Learning, Offline Reinforcement Learning, Long-Horizon Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7wdXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes IGIS, which builds on transitive value learning and bilinear value networks to distill intermediate-goal information into a final-goal action scorer without hierarchical execution at test time.

Critical strengths
  • IGIS improves success rates over TRL from 75% to 96% on Humanoidmaze-giant and from 47% to 71% on Puzzle-4×6. These gains provide meaningful empirical support for the approach.

  • IGIS outperforms TRL with a bilinear critic alone. This helps establish that the proposed distillation contributes beyond the architectural substitution while preserving flat action selection at test time.

Critical weaknesses
  • SAW already uses intermediate-goal supervision to learn a flat final-goal policy, while BVN and TRL provide the bilinear architecture and transitive value learning used here. The main additional contribution is the relevance-weighted representation interpolation and distillation, whose advantages over policy-level bootstrapping are not sufficiently established under comparable value-learning conditions.

  • The ablations support adding distillation to TRL+BVN, but do not isolate the contribution of the relevance weight ρ\rho or compare representation-level distillation against simpler scalar-score distillation. Consequently, the necessity of the specific proposed design remains unclear.

  • IGIS uses environment-specific mixing coefficients that coincide with the best-performing values in the reported sweep. However, the paper does not clearly specify whether the baselines were given comparable tuning budgets, making it difficult to distinguish improvements due to the algorithm itself from advantages due to hyperparameter tuning.

  • Restricted evaluation of generality and robustness. All experiments use oracle goal representations and 100M-transition datasets from four OGBench environments. Performance without oracle goal information, with smaller datasets, or under degraded trajectory coverage is not evaluated, despite the method’s acknowledged dependence on informative intermediate states.

  • GAS (ICML 2025) and AQM (ICML 2026) perform graph-based planning using learned temporal-distance or quasimetric structure without a learned subgoal generator. Like IGIS, they do not require such a generator, but they select and use graph nodes as intermediate goals during execution. Comparing goal-reaching performance and training and inference costs in the same environments would help establish the practical benefits of IGIS’s elimination of intermediate-goal selection and use at test time.

  • The related-work taxonomy mixes execution structure with training-time use of intermediate goals: SAW and IGIS are discussed under hierarchical GCRL despite its flat execution, while TRL is classified as flat despite also using intermediate states during training. This obscures the shared principles and precise differences among SAW, TRL, and IGIS.

Questions for authors

Please refer to the weaknesses above for the issues that require clarification.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7wdX (1/2)

Thank you for the detailed review and constructive feedback. The reviewer’s comments on how IGIS differs from SAW helped us better clarify the distinction between the two approaches.


W1. Insufficient Differentiation from Prior Work and Validation of Contributions

Comparison with SAW. As the reviewer pointed out, both SAW and IGIS leverage intermediate-goal information during training while executing a flat policy at test time. However, SAW directly transfers intermediate-goal information to the final-goal policy through policy-level bootstrapping, whereas IGIS distills intermediate-goal representations into the final-goal representation to learn an action scorer that replaces the conventional scalar critic during policy extraction. Therefore, the distinction of IGIS lies not in the use of intermediate goals itself, but in the specific mechanism of incorporating this information into primitive-action scoring through representation-level distillation. This allows IGIS to leverage shorter-horizon, action-relevant information to better distinguish primitive actions for distant goals. We have revised the Related Work section to clarify these similarities and differences in their approaches.

Comparison under comparable value-learning conditions. We agree that comparing SAW's policy-level supervision and IGIS's representation-level distillation under comparable value-learning conditions would provide a more direct analysis of their respective effects. One way to conduct such a comparison would be to adapt SAW's policy-level bootstrapping to TRL's transitive value-learning framework. However, this would require modifying how SAW's policy-level bootstrapping interacts with TRL's value estimates, resulting in a new algorithmic variant rather than a direct comparison with the original SAW.


W2. Insufficient Ablations of Relevance Weighting and Distillation

Constant-relevance ablation. To isolate the contribution of relevance weighting, we added an ablation with ρ=1\rho=1 in Appendix F.3. The results show that IGIS with critic-based relevance weighting achieves higher average success rates than the constant-relevance setting in both environments, with a larger improvement in puzzle. These findings support that adjusting the contribution of intermediate-goal representations through critic-based relevance weighting can provide additional benefits over constant weighting.

EnvironmentConstant relevance (ρ=1\rho=1)IGIS (Ours)
humanoidmaze-giant92$\pm$296$\pm$1
puzzle-4x660$\pm$971$\pm$3

Representation vs. scalar-score distillation. For a direct comparison, we constructed a baseline that distills a mixture of final-goal and intermediate-goal scalar action scores, using the same intermediate-goal information as IGIS (Appendix F.3). The results on humanoidmaze-giant and puzzle-4x6 show that IGIS achieves higher average success rates than scalar-score distillation in both environments. These findings support the benefit of distilling intermediate-goal information directly into the state-goal representation for action scoring, rather than transferring it solely through scalar action scores.

EnvironmentScalar-score distillationIGIS (Ours)
humanoidmaze-giant75$\pm$1196$\pm$1
puzzle-4x663$\pm$671$\pm$3

W3. Hyperparameter selection

Hyperparameter selection and tuning protocol. We selected the mixing coefficient λ\lambda for IGIS by sweeping {0.25, 0.5, 0.75, 1.0} for each environment and choosing the value with the highest evaluation success rate. For IGIS, we retained TRL's existing settings and tuned only the newly introduced mixing coefficient λ\lambda. For the other baselines, we used the settings provided by TRL or their respective official implementations, without conducting additional matched hyperparameter searches. However, as shown in Table 3, IGIS achieved higher average success rates than TRL across multiple λ\lambda settings. These results support that IGIS's performance gains do not depend solely on the optimal λ\lambda selected for each environment.

AuthorsOpenReview ↗

Response to Reviewer 7wdX (2/2)

W4. Limited Evaluation of Generality and Robustness

Non-oracle evaluation. In the original experiments, we used oracle goal representations to maintain consistency with the experimental setting used in the TRL paper. However, oracle goal representations are not required for training IGIS. To verify this, we conducted additional experiments without oracle goal representations while retaining IGIS's intermediate-goal representation distillation. The results show that IGIS achieves higher average success rates than TRL under the same non-oracle setting in all three environments. These findings support that IGIS's performance improvements over TRL persist without oracle goal representations.

EnvironmentHIQL (oraclerep)TRL (oraclerep)TRL (w/o oraclerep)IGIS (oraclerep)IGIS (w/o oraclerep)
puzzle-4x58$\pm$499$\pm$188$\pm$399$\pm$191$\pm$3
puzzle-4x67$\pm$447$\pm$444$\pm$771$\pm$354$\pm$6
humanoidmaze-giant19$\pm$875$\pm$774$\pm$796$\pm$185$\pm$2

Standard-size OGBench evaluation. We conducted additional comparisons on standard-size OGBench datasets, covering 50 tasks across 10 environments (Section 5.3). Following TRL, we evaluated the average success rate across all tasks. IGIS achieved the highest average success rate among all compared methods. These results support that the performance improvements of IGIS are not limited to the 100M-transition datasets and four environments, but can also extend to more diverse environments and smaller datasets.

GCFQLGCIQLSAC+BCCRLQRLSAWHIQLTRLIGIS(Ours)
Average Success Rate55547381241495563

Trajectory coverage limitation. Limited trajectory coverage is an important challenge in offline GCRL. As acknowledged in the Limitations section, IGIS relies on intermediate states observed in the offline dataset, so insufficient trajectory coverage may limit access to informative intermediate-goal information and affect critic estimation. However, our work focuses on improving primitive-action discrimination for distant goals given a fixed offline dataset. Therefore, we leave robustness evaluation under limited trajectory coverage and the development of approaches to directly address this limitation to future work.


W5. Lack of Comparison with Graph-Based Planning

Comparison with graph-based planning. Unlike graph-based planning methods such as GAS [1] and AQM [2], IGIS solves long-horizon tasks through primitive-action scoring for distant goals, without intermediate-goal selection or graph planning at test time. As our evaluation focuses on assessing the contribution of this action-scoring mechanism, we primarily selected policy-based GCRL methods as baselines. We agree that comparing GAS and AQM would be useful for evaluating the overall performance and computational costs of graph-based planning and policy-based GCRL approaches. However, such a comparison serves a different purpose from isolating the contribution of IGIS's action-scoring mechanism, and was therefore not included in our evaluation.

[1] Baek, Seungho, et al. "Graph-assisted stitching for offline hierarchical reinforcement learning." arXiv preprint arXiv:2506.07744 (2025).

[2] Kobanda, Anthony, et al. "Adaptive Quasimetric Mapping: Principled Topological Abstraction for Robust Offline Goal-Conditioned Navigation." Forty-third International Conference on Machine Learning. 2026.


W6. Inconsistent Categorization in Related Work

Related Work categorization. In the Related Work section, we categorized prior studies based not only on their test-time execution structure but also on their primary learning mechanisms and key characteristics. Although TRL uses intermediate states, we discussed it under Flat Long-Horizon GCRL because it focuses on addressing the limitations of conventional TD learning through transitive value learning. In contrast, we discussed SAW under Hierarchical GCRL because it introduces policy-level bootstrapping to transfer the benefits of hierarchical policies to a flat policy. IGIS builds on TRL's transitive value learning, connecting it to Flat Long-Horizon GCRL. Its connection to Hierarchical GCRL lies in leveraging shorter-horizon guidance from intermediate goals to improve distant-goal action selection, while retaining flat execution through representation distillation. We understand the reviewer's concern that this categorization may obscure the similarities and differences among SAW, TRL, and IGIS. We have revised the Related Work section to clarify these distinctions.

Thank you for the detailed response and additional experiments. The new ablations and evaluations without oracle goal representations and on standard-size datasets address several of my concerns and strengthen the empirical evidence. However, concerns remain regarding the incremental contribution over closely related methods and the fairness of hyperparameter tuning across baselines. Accordingly, I am revising my recommendation from clear rejection to weak rejection.

We thank the reviewer for their detailed feedback and positive assessment of our additional experiments. We also appreciate the reviewer’s decision to revise their recommendation. The reviewer's constructive comments have helped us better clarify IGIS's contribution and strengthen its empirical evidence.

We would like to further highlight the distinction between SAW and IGIS. Both methods leverage intermediate-goal information during training while maintaining a flat policy at test time. However, SAW focuses on policy learning, whereas IGIS uses intermediate-goal information to learn a separate action scorer that complements the existing critic. Specifically, SAW transfers the benefits of intermediate-goal-conditioned policies to the final-goal policy through policy-level bootstrapping. In contrast, IGIS distills intermediate-goal representations to improve primitive-action discrimination for distant goals. Thus, the two methods differ in both how they leverage intermediate-goal information and the purpose for which it is used.

Regarding hyperparameter tuning, we agree that an equal-budget hyperparameter search was not conducted across all baselines. However, IGIS retains most hyperparameters shared with TRL, limiting additional tuning to the newly introduced mixing coefficient λ\lambda. We would also like to highlight that, as shown in the results below, IGIS achieves higher success rates than TRL across multiple λ\lambda settings, suggesting that its performance improvements are not solely attributable to selecting the best-performing λ\lambda for each environment.

TRLIGIS (λ=0.25\lambda=0.25)IGIS (λ=0.5\lambda=0.5)IGIS (λ=0.75\lambda=0.75)IGIS (λ=1.0\lambda=1.0)
humanoidmaze-giant75$\pm$783$\pm$592$\pm$495$\pm$396$\pm$1
puzzle-4x647$\pm$471$\pm$362$\pm$1057$\pm$847$\pm$10
cube-quadruple41$\pm$651$\pm$453$\pm$452$\pm$444$\pm$4
Reviewer_hbhXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the IGIS for long horizon offline goal conditioned RL. The key idea is to use intermediate states sampled from offline trajectories as training time supervision, while avoiding explicit subgoal generation and hierarchical execution at test time. IGIS combines a transitive bilinear critic with intermediate goal representation distillation. The method is evaluated on several tasks against flat offline GCRL, long horizon value learning, and hierarchical baselines. The proposed method achieves strong improvements, and ablations comparing TRL, TRL+BVN, and IGIS suggest that the gains are not explained by bilinear factorization alone.

Critical strengths
  1. The idea of using intermediate-goal information as training-time supervision while retaining flat action selection at test time is conceptually appealing and may offer a useful compromise between hierarchical and flat approaches.
  2. The empirical results are promising on several challenging tasks. In addition, the TRL versus TRL+BVN versus IGIS ablation also suggests that the gains are not explained by bilinear factorization alone, but are related to the proposed intermediate-goal representation distillation.
Critical weaknesses
  1. The paper argues that IGIS improves primitive action discrimination, but the experiments mainly report final task success rates. There is no direct evaluation of action ranking quality, score margins, or the relationship between the relevance score rho and the correctness of action selection. Moreover, rho is computed from the learned critic itself, which raises concerns about error propagation or self-reinforcing supervision.
  2. The mixing coefficient lambda, goal relabeling ratios, and policy extraction settings are task dependent, while the paper does not clearly state how these values were selected or whether the same tuning budget was used for all baselines. In addition, the oracle goal-representation setting and the lack of inference cost measurements limit the strength of the claims about generality and practical advantage.
Questions for authors
  1. How were the task specific values of lambda, the goal relabeling ratios, and the policy extraction hyperparameters selected? Were they chosen using an independent validation set, and did all baselines receive an equivalent tuning budget?
  2. Can the authors provide direct evidence for the claimed improvement in primitive action discrimination, such as pairwise action-ranking accuracy, action score margins, or correlation between rho and the quality of the selected action? It would also be useful to compare against random intermediate goals, uniformly weighted intermediate goals, and a version with rho detached or removed.
  3. What are the training and inference costs of IGIS relative to TRL and hierarchical baselines such as HIQL?
  4. In the oracle goal representation setting, what information is available to the teacher during training and to the policy at test time?

My current rating mainly due to the concerns above. I would be open to raising the rating if the authors can clarify the hyperparameter selection protocol and provide direct evidence for improved action discrimination.

5 comments
AuthorsOpenReview ↗

Response to Reviewer hbhX (1/2)

Thank you for the detailed review and constructive feedback. We appreciate the reviewer’s comments highlighting the need for more direct evaluation of action discrimination.


W1 & Q2: Insufficient Action Discrimination Evidence & Relevance Reliability

Action-discrimination diagnostic. We thank the reviewer for pointing out the need for direct action-level validation. To address this concern, we conducted an additional action-discrimination diagnostic on humanoidmaze-giant, comparing IGIS (λ=1\lambda=1), IGIS without intermediate-goal mixing (λ=0\lambda=0), and TRL. We have included this analysis in Section 5.6 and Appendix F.2. At each measurement point along successful evaluation trajectories, we evaluated the same flow-generated candidate actions using all three scorers and analyzed action-score dispersion by measuring the average pairwise absolute difference between scores after transforming them to the same probability scale. The results show that IGIS (λ=1\lambda=1) maintains relatively stable action-score dispersion across goal distances, including distant goals, whereas IGIS (λ=0\lambda=0) and TRL exhibit small score gaps for distant goals that tend to increase as the agent approaches the goal. These findings suggest that intermediate-goal representation distillation helps maintain score separation among candidate actions even for distant goals. This improved action-score separation, together with the higher goal-reaching success rates observed in our experiments, supports our claim that IGIS improves primitive-action discrimination for distant goals and contributes to learning effective goal-reaching behavior. Further details are provided in Section 5.6 and Appendix F.2.

Critic estimation error and self-reinforcing supervision. Since ρ\rho is computed from learned critic estimates, estimation errors may lead to inaccurate representation supervision. However, IGIS applies stop-gradient to ρ\rho and the teacher representations, preventing direct feedback from the distillation loss and thereby partially mitigating the risk of self-reinforcing supervision. To evaluate the practical benefit of critic-based relevance weighting, we conducted an ablation with ρ\rho fixed at 1 in Appendix F.3 of the revised paper. The results show that IGIS with critic-based relevance weighting achieves higher average success rates than the constant-relevance setting in both environments, with a larger improvement in puzzle-4x6. These findings support that adjusting the contribution of intermediate-goal representations through relevance weighting can provide additional benefits, even though ρ\rho does not guarantee accurate relevance estimates or correct action rankings.

EnvironmentConstant relevance (ρ=1\rho=1)IGIS (Ours)
humanoidmaze-giant92$\pm$296$\pm$1
puzzle-4x660$\pm$971$\pm$3
AuthorsOpenReview ↗

Response to Reviewer hbhX (2/2)

W2 & Q1 & Q3 : Hyperparameter Fairness, Generality & Computational Cost

Hyperparameter selection and tuning protocol. We selected the mixing coefficient λ\lambda for IGIS by sweeping {0.25, 0.5, 0.75, 1.0} for each environment and choosing the value with the highest evaluation success rate. This selection was based on evaluation results without an independent validation set, and the full sweep results are reported in Table 3. For IGIS, we followed TRL's existing goal relabeling ratios and policy-extraction settings and tuned only the newly introduced mixing coefficient λ\lambda. For the other baselines, we used the settings provided by TRL or their respective official implementations, without conducting additional matched hyperparameter searches. However, as shown in Table 3, IGIS achieved higher average success rates than TRL across multiple λ\lambda settings. These results support that IGIS's performance gains do not depend solely on the optimal λ\lambda selected for each environment.

Non-oracle evaluation. In the original experiments, we used oracle goal representations to maintain consistency with the experimental setting used in the TRL paper. However, oracle goal representations are not required for training IGIS. To verify this, we conducted additional experiments without oracle goal representations while retaining IGIS's intermediate-goal representation distillation. The results show that IGIS achieves higher average success rates than TRL under the same non-oracle setting in all three environments. These findings support that IGIS's performance improvements over TRL persist without oracle goal representations.

EnvironmentHIQL (oraclerep)TRL (oraclerep)TRL (w/o oraclerep)IGIS (oraclerep)IGIS (w/o oraclerep)
puzzle-4x58$\pm$499$\pm$188$\pm$399$\pm$191$\pm$3
puzzle-4x67$\pm$447$\pm$444$\pm$771$\pm$354$\pm$6
humanoidmaze-giant19$\pm$875$\pm$774$\pm$796$\pm$185$\pm$2

Computational cost. In response to the reviewer's request, we compared the training time, parameter count, and inference latency of IGIS, TRL, and HIQL on humanoidmaze-giant. The results show that IGIS achieves improved goal-reaching performance with computational costs comparable to those of TRL and HIQL.

AgentTraining Time#ParamsInference Latency
HIQL4h23.5M1.6ms
TRL4h23.0M1.4ms
IGIS5h19.6M1.8ms

Q4: Oracle Goal Representation

Oracle goal representations are task-specific goal information provided by OGBench. Specifically, humanoidmaze uses the target XY position, cube uses the positions of individual cubes, and puzzle uses the binary states of buttons. During training, the teacher critic performs transitive value learning using full goal observations and intermediate goals sampled from trajectories. The intermediate-goal information is then distilled into an oracle-goal-conditioned action-scoring representation. At test time, action selection uses only the current observation and the oracle goal representation.

Thank you for the clarifications and additional experiments, which address several of my concerns, particularly regarding computational cost and the oracle setting. Could you clarify the scope of the action discrimination claim, given that increased score dispersion does not directly establish more accurate action ranking? Could you also clarify whether the reported success rates reuse the evaluations used to select λ, and discuss how this selection procedure affects the interpretation of the comparisons with baselines?

Reviewer_GabDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the Intermediate-Goal-Informed Scorer (IGIS) for long-horizon offline goal-conditioned RL. It combines TRL’s transitive value learning with a bilinear critic, then distills a relevance-weighted mixture of intermediate-goal and final-goal representations into a separate action scorer. The resulting policy uses only the current state and final goal at inference. Experiments on four challenging OGBench datasets show substantial improvements over TRL, particularly on humanoid navigation and larger puzzles.

Critical strengths
  • Substantial improvements on difficult benchmarks. In Table 1, IGIS improves over TRL from 75 ± 7 to 96 ± 1 on humanoidmaze-giant and from 47 ± 4 to 71 ± 3 on puzzle-4x6. These are meaningful gains, reported across four seeds.

  • A useful architectural ablation. On humanoid maze, TRL+BVN achieves only 61 ± 10, compared with 96 ± 1 for IGIS (Table 2). This provides strong evidence that the humanoid improvement cannot be explained simply by replacing the critic with a bilinear architecture.

  • Simple inference and flexible policy extraction. Intermediate goals are required only during training, and the scorer supports both gradient-based actor learning and candidate-action ranking without a subgoal generator or hierarchical execution.

Critical weaknesses
  • The advantage over the closest alternative approaches remains untested. SAW, which is appropriately cited and evaluated, already uses training-time intermediate-goal supervision to learn a flat final-goal policy. IGIS’s distinct contribution is therefore the particular representation-distillation mechanism. Dual Advantage Fields is especially relevant because it also combines bilinear goal representations with local action scoring, but is only cited. Multistep Quasimetric Estimation is another relevant recent long-horizon baseline that is cited but absent from the experiments. Comparisons with these methods would better establish the contribution’s significance.
  • The central action-discrimination explanation lacks direct validation. The experiments measure goal-reaching success, but provide no action-ranking accuracy, signal-to-noise measurements, or analysis of how these change with goal distance. Appendix A establishes an interpolation identity, not that the resulting action preferences become more accurate. Moreover, the relevance weight in Eq. 8 measures transitive consistency for the dataset action; a waypoint appropriate for that action need not preserve the ranking of alternative actions that reach the final goal through a different route. This limitation deserves a controlled diagnostic.
  • The ablations do not fully isolate intermediate-goal representation distillation. A λ = 0 control with the identical student architecture and oracle-goal distillation procedure is missing; the description of TRL+BVN does not establish that it supplies this exact control. There is also no constant-relevance control or comparison with distilling the equivalent mixture of scalar action scores. These distinctions matter because the bilinear architecture already accounts for much of the manipulation improvement: 47 → 64 → 71 on puzzle-4x6 and 41 → 50 → 53 on cube-quadruple for TRL, TRL+BVN, and IGIS, respectively (Table 2).
  • Generality is insufficiently established. Evaluation covers four datasets from three environment families, all with 100M transitions and oracle goal representations. Results on standard-size datasets, broader OGBench tasks, and full-observation goals would help establish the method’s scope. The main results also use environment-specific mixing coefficients without explaining their selection protocol. Table 3 shows that this choice matters: increasing λ from 0.25 to 1 improves humanoid performance but reduces puzzle-4x6 success from 71 ± 3 to 47 ± 10.
Questions for authors
  • What benefit comes specifically from intermediate-goal representation distillation? Please compare with a λ = 0 variant that retains the identical student network, oracle-goal inputs, and policy-extraction procedure, and clarify whether TRL+BVN already provides this control. Would a scalar critic trained on the corresponding mixture of final-goal and intermediate-goal scores achieve similar gains?

  • Does IGIS improve the correctness of action rankings as goal distance increases? A controlled experiment with known action values or rollout-based estimates would help establish this. In particular, when a better route bypasses the sampled intermediate goal, does the relevance weight ρ prevent misleading supervision? How does it compare with ρ = 1?

  • How broadly do the reported improvements hold? Under matched training and tuning budgets, how does IGIS compare with Multistep Quasimetric Estimation (MQE)? Do the gains persist on a representative set of standard-size OGBench datasets and without oracle goal representations? Please clarify how the environment-specific λ values were selected.

3 comments
AuthorsOpenReview ↗

Response to Reviewer GabD (1/3)

Thank you for the detailed review and constructive feedback. We appreciate the reviewer’s suggestion to include more targeted ablations to clarify the contribution of intermediate-goal representation distillation.


W1 & Q3: Comparison with SAW, DAF & MQE

Comparison with SAW. As the reviewer pointed out, both SAW and IGIS leverage intermediate-goal information during training while executing a flat policy at test time. However, SAW directly transfers intermediate-goal information to the final-goal policy through policy-level bootstrapping, whereas IGIS distills intermediate-goal representations into the final-goal representation to learn an action scorer that replaces the conventional scalar critic during policy extraction. Therefore, the distinction of IGIS lies not in the use of intermediate goals itself, but in the specific mechanism of incorporating this information into primitive-action scoring through representation-level distillation. This allows IGIS to leverage shorter-horizon, action-relevant information to better distinguish primitive actions for distant goals. We have revised the Related Work section to clarify these similarities and differences in their approaches.

Comparison with DAF. We agree with the reviewer that a comparison with DAF would be valuable. While DAF combines global goal-reaching representations with local action-effect information for action scoring, IGIS instead distills intermediate-goal information into the final-goal representation. However, to the best of our knowledge, the official implementation of DAF is not publicly available, making a direct experimental comparison difficult.

Comparison with MQE. We used the official implementation of MQE and increased its network capacity to a level comparable to TRL and IGIS for the 100M-transition datasets, considering that the original MQE experiments were conducted on standard-size datasets. We compared all three methods on the same 100M-transition datasets without oracle goal representations. The results show that IGIS achieves higher average success rates than both MQE and TRL in all three environments, with particularly large improvements over MQE in the puzzle environments.

EnvironmentMQE (w/o oraclerep)TRL (w/o oraclerep)IGIS (w/o oraclerep)
humanoidmaze-giant70$\pm$274$\pm$785$\pm$2
puzzle-4x55$\pm$288$\pm$391$\pm$3
puzzle-4x68$\pm$344$\pm$754$\pm$6

W2 & Q2: Action-Discrimination Validation & Relevance Reliability

Action-discrimination diagnostic. We thank the reviewer for pointing out the need for direct action-level validation. To address this concern, we conducted an additional action-discrimination diagnostic on humanoidmaze-giant, comparing IGIS (λ=1\lambda=1), IGIS without intermediate-goal mixing (λ=0\lambda=0), and TRL. We have included this analysis in Section 5.6 and Appendix F.2. At each measurement point along successful evaluation trajectories, we evaluated the same flow-generated candidate actions using all three scorers and analyzed action-score dispersion by measuring the average pairwise absolute difference between scores after transforming them to the same probability scale. The results show that IGIS (λ=1\lambda=1) maintains relatively stable action-score dispersion across goal distances, including distant goals, whereas IGIS (λ=0\lambda=0) and TRL exhibit small score gaps for distant goals that tend to increase as the agent approaches the goal. These findings suggest that intermediate-goal representation distillation helps maintain score separation among candidate actions even for distant goals. This improved action-score separation, together with the higher goal-reaching success rates observed in our experiments, supports our claim that IGIS improves primitive-action discrimination for distant goals and contributes to learning effective goal-reaching behavior. Further details are provided in Section 5.6 and Appendix F.2.

Meaning and limitations of relevance weighting. As the reviewer correctly points out, ρ\rho is computed from critic estimates for the dataset action and does not account for the rankings of alternative actions. Therefore, relevance weighting may not fully reflect the relative quality of alternative actions when a better route exists. However, a lower ρ\rho reduces the contribution of the intermediate-goal representation while relatively preserving the final-goal representation, thereby mitigating excessive reliance on a particular intermediate goal.

AuthorsOpenReview ↗

Response to Reviewer GabD (2/3)

Constant-relevance ablation. To evaluate the effect of relevance weighting, we added an ablation with ρ=1\rho=1 in Appendix F.3. The results show that IGIS with critic-based relevance weighting achieves higher average success rates than the constant-relevance setting in both environments, with a larger improvement in puzzle-4x6. These findings support that adjusting the contribution of intermediate-goal information can provide additional benefits during training, even though ρ\rho does not guarantee accurate relevance estimates or correct action rankings.

EnvironmentConstant relevance (ρ=1\rho=1)IGIS (Ours)
humanoidmaze-giant92$\pm$296$\pm$1
puzzle-4x660$\pm$971$\pm$3

W3 & Q1: Insufficient Ablations of Representation & Scalar-Score Distillation

Intermediate-goal mixing ablation. TRL+BVN and IGIS (λ=0\lambda=0) both exclude intermediate-goal representation mixing, but differ in their distillation procedures. TRL+BVN directly distills scalar Q-values, whereas IGIS (λ=0\lambda=0) distills the final-goal representation. Therefore, TRL+BVN does not provide the exact control requested by the reviewer. To address this, we added a λ=0\lambda=0 ablation that removes only intermediate-goal mixing while retaining the identical IGIS architecture, oracle-goal inputs, and policy-extraction procedure (Table 3). The results show substantial improvements from intermediate-goal representation mixing, particularly on humanoidmaze-giant. These findings highlight the contribution of intermediate-goal representation mixing to IGIS's performance gains, beyond the effects of the bilinear architecture and final-goal representation distillation.

00.250.50.751.0
humanoidmaze-giant71$\pm$783$\pm$592$\pm$495$\pm$396$\pm$1
puzzle-4x667$\pm$371$\pm$362$\pm$1057$\pm$847$\pm$10
cube-quadruple51$\pm$251$\pm$453$\pm$452$\pm$444$\pm$4

Representation vs. scalar-score distillation. For a direct comparison, we constructed a baseline that distills a mixture of final-goal and intermediate-goal scalar action scores, using the same intermediate-goal information as IGIS in Appendix F.3 of the revised paper. The results show that IGIS achieves higher average success rates than scalar-score distillation. These findings support the benefit of distilling intermediate-goal information directly into the state-goal representation for action scoring, rather than transferring it solely through scalar action scores.

EnvironmentScalar-score distillationIGIS (Ours)
humanoidmaze-giant75$\pm$1196$\pm$1
puzzle-4x663$\pm$671$\pm$3

W4 & Q3: Generality & Environment-Specific Mixing Coefficients

Non-oracle evaluation. In the original experiments, we used oracle goal representations to maintain consistency with the experimental setting used in the TRL paper. However, oracle goal representations are not required for training IGIS. To verify this, we conducted additional experiments without oracle goal representations while retaining IGIS's intermediate-goal representation distillation. The results show that IGIS achieves higher average success rates than TRL under the same non-oracle setting in all three environments. These findings support that IGIS's performance improvements over TRL persist without oracle goal representations.

EnvironmentHIQL (oraclerep)TRL (oraclerep)TRL (w/o oraclerep)IGIS (oraclerep)IGIS (w/o oraclerep)
puzzle-4x58$\pm$499$\pm$188$\pm$399$\pm$191$\pm$3
puzzle-4x67$\pm$447$\pm$444$\pm$771$\pm$354$\pm$6
humanoidmaze-giant19$\pm$875$\pm$774$\pm$796$\pm$185$\pm$2

Standard-size OGBench evaluation. In response to the reviewer's request, we conducted additional experiments on standard-size OGBench datasets, covering 50 tasks across 10 environments (Section 5.3). Following TRL, we evaluated the average success rate across all tasks. The results show that IGIS achieves the highest average success rate among all compared methods. These results support that the performance improvements of IGIS are not limited to the 100M-transition datasets and four environments, but can also extend to more diverse environments and smaller datasets.

GCFQLGCIQLSAC+BCCRLQRLSAWHIQLTRLIGIS(Ours)
Average Success Rate55547381241495563
AuthorsOpenReview ↗

Response to Reviewer GabD (3/3)

Mixing coefficient selection and tuning protocol. We selected the mixing coefficient λ\lambda for IGIS by sweeping {0.25, 0.5, 0.75, 1.0} for each environment and choosing the value with the highest evaluation success rate. However, as shown in Table 3, IGIS achieved higher average success rates than TRL across multiple λ\lambda settings. These results support that IGIS's performance gains do not depend solely on the optimal λ\lambda selected for each environment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful evaluations and constructive feedback. Based on your suggestions, we have clarified several aspects of the paper and strengthened its empirical support through additional experiments and analyses. We summarize the main revisions below.

Paper Revision Summary

All revisions are highlighted in blue in the revised manuscript.

  • We extended our evaluation beyond the four challenging long-horizon environments by adding experiments on 50 tasks across 10 additional OGBench environments in Section 5.3, with detailed experimental settings and full results provided in Appendices E.2 and F.1. (GabD, 7wdX)
  • We added the λ=0\lambda=0 ablation results to Table 3 to more clearly isolate the effect of intermediate-goal representation mixing. (GabD)
  • We added constant-relevance (ρ=1\rho=1) and scalar-score distillation ablations in Appendix F.3 to examine the effects of relevance weighting and representation-level distillation, respectively. (GabD, hbhX, 7wdX)
  • We added an action-score dispersion analysis in Section 5.6 and Appendix F.2 to examine action-score separation across goal distances, particularly for distant goals. (GabD, hbhX)
  • We revised the Related Work section to clarify the similarities and differences between SAW and IGIS in their learning mechanisms and to refine the description of TRL's approach. (GabD, 7wdX)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.