- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes IGIS, which builds on transitive value learning and bilinear value networks to distill intermediate-goal information into a final-goal action scorer without hierarchical execution at test time.
Critical strengths
-
IGIS improves success rates over TRL from 75% to 96% on Humanoidmaze-giant and from 47% to 71% on Puzzle-4×6. These gains provide meaningful empirical support for the approach.
-
IGIS outperforms TRL with a bilinear critic alone. This helps establish that the proposed distillation contributes beyond the architectural substitution while preserving flat action selection at test time.
Critical weaknesses
-
SAW already uses intermediate-goal supervision to learn a flat final-goal policy, while BVN and TRL provide the bilinear architecture and transitive value learning used here. The main additional contribution is the relevance-weighted representation interpolation and distillation, whose advantages over policy-level bootstrapping are not sufficiently established under comparable value-learning conditions.
-
The ablations support adding distillation to TRL+BVN, but do not isolate the contribution of the relevance weight or compare representation-level distillation against simpler scalar-score distillation. Consequently, the necessity of the specific proposed design remains unclear.
-
IGIS uses environment-specific mixing coefficients that coincide with the best-performing values in the reported sweep. However, the paper does not clearly specify whether the baselines were given comparable tuning budgets, making it difficult to distinguish improvements due to the algorithm itself from advantages due to hyperparameter tuning.
-
Restricted evaluation of generality and robustness. All experiments use oracle goal representations and 100M-transition datasets from four OGBench environments. Performance without oracle goal information, with smaller datasets, or under degraded trajectory coverage is not evaluated, despite the method’s acknowledged dependence on informative intermediate states.
-
GAS (ICML 2025) and AQM (ICML 2026) perform graph-based planning using learned temporal-distance or quasimetric structure without a learned subgoal generator. Like IGIS, they do not require such a generator, but they select and use graph nodes as intermediate goals during execution. Comparing goal-reaching performance and training and inference costs in the same environments would help establish the practical benefits of IGIS’s elimination of intermediate-goal selection and use at test time.
-
The related-work taxonomy mixes execution structure with training-time use of intermediate goals: SAW and IGIS are discussed under hierarchical GCRL despite its flat execution, while TRL is classified as flat despite also using intermediate states during training. This obscures the shared principles and precise differences among SAW, TRL, and IGIS.
Questions for authors
Please refer to the weaknesses above for the issues that require clarification.
4 comments
Response to Reviewer 7wdX (2/2)
W4. Limited Evaluation of Generality and Robustness
Non-oracle evaluation. In the original experiments, we used oracle goal representations to maintain consistency with the experimental setting used in the TRL paper. However, oracle goal representations are not required for training IGIS. To verify this, we conducted additional experiments without oracle goal representations while retaining IGIS's intermediate-goal representation distillation. The results show that IGIS achieves higher average success rates than TRL under the same non-oracle setting in all three environments. These findings support that IGIS's performance improvements over TRL persist without oracle goal representations.
| Environment | HIQL (oraclerep) | TRL (oraclerep) | TRL (w/o oraclerep) | IGIS (oraclerep) | IGIS (w/o oraclerep) |
|---|---|---|---|---|---|
| puzzle-4x5 | 8$\pm$4 | 99$\pm$1 | 88$\pm$3 | 99$\pm$1 | 91$\pm$3 |
| puzzle-4x6 | 7$\pm$4 | 47$\pm$4 | 44$\pm$7 | 71$\pm$3 | 54$\pm$6 |
| humanoidmaze-giant | 19$\pm$8 | 75$\pm$7 | 74$\pm$7 | 96$\pm$1 | 85$\pm$2 |
Standard-size OGBench evaluation. We conducted additional comparisons on standard-size OGBench datasets, covering 50 tasks across 10 environments (Section 5.3). Following TRL, we evaluated the average success rate across all tasks. IGIS achieved the highest average success rate among all compared methods. These results support that the performance improvements of IGIS are not limited to the 100M-transition datasets and four environments, but can also extend to more diverse environments and smaller datasets.
| GCFQL | GCIQL | SAC+BC | CRL | QRL | SAW | HIQL | TRL | IGIS(Ours) | |
|---|---|---|---|---|---|---|---|---|---|
| Average Success Rate | 55 | 54 | 7 | 38 | 12 | 41 | 49 | 55 | 63 |
Trajectory coverage limitation. Limited trajectory coverage is an important challenge in offline GCRL. As acknowledged in the Limitations section, IGIS relies on intermediate states observed in the offline dataset, so insufficient trajectory coverage may limit access to informative intermediate-goal information and affect critic estimation. However, our work focuses on improving primitive-action discrimination for distant goals given a fixed offline dataset. Therefore, we leave robustness evaluation under limited trajectory coverage and the development of approaches to directly address this limitation to future work.
W5. Lack of Comparison with Graph-Based Planning
Comparison with graph-based planning. Unlike graph-based planning methods such as GAS [1] and AQM [2], IGIS solves long-horizon tasks through primitive-action scoring for distant goals, without intermediate-goal selection or graph planning at test time. As our evaluation focuses on assessing the contribution of this action-scoring mechanism, we primarily selected policy-based GCRL methods as baselines. We agree that comparing GAS and AQM would be useful for evaluating the overall performance and computational costs of graph-based planning and policy-based GCRL approaches. However, such a comparison serves a different purpose from isolating the contribution of IGIS's action-scoring mechanism, and was therefore not included in our evaluation.
[1] Baek, Seungho, et al. "Graph-assisted stitching for offline hierarchical reinforcement learning." arXiv preprint arXiv:2506.07744 (2025).
[2] Kobanda, Anthony, et al. "Adaptive Quasimetric Mapping: Principled Topological Abstraction for Robust Offline Goal-Conditioned Navigation." Forty-third International Conference on Machine Learning. 2026.
W6. Inconsistent Categorization in Related Work
Related Work categorization. In the Related Work section, we categorized prior studies based not only on their test-time execution structure but also on their primary learning mechanisms and key characteristics. Although TRL uses intermediate states, we discussed it under Flat Long-Horizon GCRL because it focuses on addressing the limitations of conventional TD learning through transitive value learning. In contrast, we discussed SAW under Hierarchical GCRL because it introduces policy-level bootstrapping to transfer the benefits of hierarchical policies to a flat policy. IGIS builds on TRL's transitive value learning, connecting it to Flat Long-Horizon GCRL. Its connection to Hierarchical GCRL lies in leveraging shorter-horizon guidance from intermediate goals to improve distant-goal action selection, while retaining flat execution through representation distillation. We understand the reviewer's concern that this categorization may obscure the similarities and differences among SAW, TRL, and IGIS. We have revised the Related Work section to clarify these distinctions.
Thank you for the detailed response and additional experiments. The new ablations and evaluations without oracle goal representations and on standard-size datasets address several of my concerns and strengthen the empirical evidence. However, concerns remain regarding the incremental contribution over closely related methods and the fairness of hyperparameter tuning across baselines. Accordingly, I am revising my recommendation from clear rejection to weak rejection.
We thank the reviewer for their detailed feedback and positive assessment of our additional experiments. We also appreciate the reviewer’s decision to revise their recommendation. The reviewer's constructive comments have helped us better clarify IGIS's contribution and strengthen its empirical evidence.
We would like to further highlight the distinction between SAW and IGIS. Both methods leverage intermediate-goal information during training while maintaining a flat policy at test time. However, SAW focuses on policy learning, whereas IGIS uses intermediate-goal information to learn a separate action scorer that complements the existing critic. Specifically, SAW transfers the benefits of intermediate-goal-conditioned policies to the final-goal policy through policy-level bootstrapping. In contrast, IGIS distills intermediate-goal representations to improve primitive-action discrimination for distant goals. Thus, the two methods differ in both how they leverage intermediate-goal information and the purpose for which it is used.
Regarding hyperparameter tuning, we agree that an equal-budget hyperparameter search was not conducted across all baselines. However, IGIS retains most hyperparameters shared with TRL, limiting additional tuning to the newly introduced mixing coefficient . We would also like to highlight that, as shown in the results below, IGIS achieves higher success rates than TRL across multiple settings, suggesting that its performance improvements are not solely attributable to selecting the best-performing for each environment.
| TRL | IGIS () | IGIS () | IGIS () | IGIS () | |
|---|---|---|---|---|---|
| humanoidmaze-giant | 75$\pm$7 | 83$\pm$5 | 92$\pm$4 | 95$\pm$3 | 96$\pm$1 |
| puzzle-4x6 | 47$\pm$4 | 71$\pm$3 | 62$\pm$10 | 57$\pm$8 | 47$\pm$10 |
| cube-quadruple | 41$\pm$6 | 51$\pm$4 | 53$\pm$4 | 52$\pm$4 | 44$\pm$4 |
Response to Reviewer 7wdX (1/2)
Thank you for the detailed review and constructive feedback. The reviewer’s comments on how IGIS differs from SAW helped us better clarify the distinction between the two approaches.
Comparison with SAW. As the reviewer pointed out, both SAW and IGIS leverage intermediate-goal information during training while executing a flat policy at test time. However, SAW directly transfers intermediate-goal information to the final-goal policy through policy-level bootstrapping, whereas IGIS distills intermediate-goal representations into the final-goal representation to learn an action scorer that replaces the conventional scalar critic during policy extraction. Therefore, the distinction of IGIS lies not in the use of intermediate goals itself, but in the specific mechanism of incorporating this information into primitive-action scoring through representation-level distillation. This allows IGIS to leverage shorter-horizon, action-relevant information to better distinguish primitive actions for distant goals. We have revised the
Related Work sectionto clarify these similarities and differences in their approaches.Comparison under comparable value-learning conditions. We agree that comparing SAW's policy-level supervision and IGIS's representation-level distillation under comparable value-learning conditions would provide a more direct analysis of their respective effects. One way to conduct such a comparison would be to adapt SAW's policy-level bootstrapping to TRL's transitive value-learning framework. However, this would require modifying how SAW's policy-level bootstrapping interacts with TRL's value estimates, resulting in a new algorithmic variant rather than a direct comparison with the original SAW.
Constant-relevance ablation. To isolate the contribution of relevance weighting, we added an ablation with ρ=1 in
Appendix F.3. The results show that IGIS with critic-based relevance weighting achieves higher average success rates than the constant-relevance setting in both environments, with a larger improvement in puzzle. These findings support that adjusting the contribution of intermediate-goal representations through critic-based relevance weighting can provide additional benefits over constant weighting.Representation vs. scalar-score distillation. For a direct comparison, we constructed a baseline that distills a mixture of final-goal and intermediate-goal scalar action scores, using the same intermediate-goal information as IGIS (
Appendix F.3). The results on humanoidmaze-giant and puzzle-4x6 show that IGIS achieves higher average success rates than scalar-score distillation in both environments. These findings support the benefit of distilling intermediate-goal information directly into the state-goal representation for action scoring, rather than transferring it solely through scalar action scores.Hyperparameter selection and tuning protocol. We selected the mixing coefficient λ for IGIS by sweeping {0.25, 0.5, 0.75, 1.0} for each environment and choosing the value with the highest evaluation success rate. For IGIS, we retained TRL's existing settings and tuned only the newly introduced mixing coefficient λ. For the other baselines, we used the settings provided by TRL or their respective official implementations, without conducting additional matched hyperparameter searches. However, as shown in
Table 3, IGIS achieved higher average success rates than TRL across multiple λ settings. These results support that IGIS's performance gains do not depend solely on the optimal λ selected for each environment.