- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This article addresses the long-horizon problem from the perspective of mutual information between actions and the target of re-labeling, and proposes the Sobolev bootstrapping method, which is simple and easy to follow. Moreover, the experimental results are quite comprehensive, and the ablation experiments are also quite thorough. It is a relatively solid article.
Critical strengths
I think the strength of this article lies in its novel perspective. It obtains observations through experiments and presents a new design concept for goal-conditioned strategies from the perspective of mutual information. Moreover, it is simple and easy to implement.
Critical weaknesses
-
The same method yields different numbers in different tables. For instance, in Table 6, S²AW on cube-double-play-v0, humanoidmaze-giant-navigate-vo, and scene-play-v0 are 50±6, 38±6, and 73±6 respectively, while in Table 1 they are 57±6, 39±5, and 74±6. Table 2 states "Other settings are the same as Table 1", but SAW on cube-single is 76±8, while in Table 1 it is 72±5. Additionally, the hyperparameter table in Table 3 omits the "puzzle-3x3" and "puzzle-4x4" datasets, while Table 1 reports the results for these two datasets.
-
The statement about the number of seeds is contradictory. Table 1 claims 8 seeds. Table 8 and Table 9 claim 4 seeds, yet they provide exactly the same S²AW numbers as those in Table 1. Table 4 claims 4 seeds, but the λ=0 column is exactly the same as the SAW in Table 1 . The author needs to clarify which numbers are reused and based on how many seeds each one is.
-
There is a lack of ablation that can attribute the gains to the "narrow-field sensitivity structure". For example: 1) Change the sub-goal step number k of the teacher and use a long-field teacher as a control group. 2) Perform zero-order distillation by adding noise to the input of the teacher. 3) Simply increase the KL weight of SAW.
-
The validity of the Jacobian indicator is questionable. In Section 3, the ℓ1 proportion of the target Jacobian is used. The decrease in the proportion could be due to a decline in target sensitivity, or it could simply be an increase in state sensitivity. The ℓ1 norm also depends on the scale of each input dimension: position, velocity, and angle are mixed together, and the target also includes mechanical arm body states unrelated to the success determination. The article does not explain whether normalization was performed, nor does it explain how the Jacobian of the flow matching strategy is calculated (whether it is fixed noise or the average of the noise).
-
Section 3 uses proportion, while Section 5.3 replaces it with the absolute target Jacobian norm, and only compares it with SAW. If the teacher's state Jacobian is large (similar to the feedback gain in tracking control), matching it will increase the state Jacobian of the student. Students may also increase the target Jacobian structurally to fit the action. This does not mean that the target sensitivity has truly recovered; it should also report the proportion.
-
Theoretical assumptions do not match the experimental data. Proposition 3.2 requires that the behavioral strategy be Markovian, but the original OGBench text (§7.2) clearly states that the "play" data is collected by "non-Markovian expert policies", and the cube and scene experiments in Section 3 use this type of data. At this point, A_t → S_{t+k} → S_{t+k'} does not necessarily form a conditional Markov chain, and the monotonic decrease of mutual information cannot be guaranteed. Even distant future states may reveal hidden intentions, providing more information for the current action. The authors should at least discuss this gap.
Questions for authors
-
The current experimental results show that the gains in the navigation task are smaller compared to those in the manipulation task. Could the author explain why this is the case? Additionally, I would like to know how well you performed in the visual tasks.
-
Why didn't the author report the results as SAW did in the stitching-type tasks? If the performance in the manipulation task is acceptable, then it is likely that the stitching ability of AW is also good. Please provide the results and conduct an analysis.
-
Given that AW is a relatively novel goal-conditioned strategy design algorithm, how does its performance compare with the latest ones, such as that described in [1, 2]?
[1] Lei, X., Wang, J., Zhang, X., & Wang, D. (2026). QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL. ICML, 2026.
[2] Choi J, Lee S H, Seo S W. Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL. ICML, 2026.
-
Could the author provide a table and visualizations of the AW and RIS/SAW comparisons in the actual ogbench maze environment to demonstrate their contributions? Furthermore, although the method in this paper has been improved, the computational load has also increased. Could the author provide an explanation regarding whether there is any room for improvement in future work?
-
All three manipulations in Section 3.1 (larger γ, larger H_train, and random-goal mixing) also reduce the share of training pairs whose goals lie near the test-time distance H_plan. For example, under Uniform [1, H_train], only a 32/H_train fraction of relabeled goals falls within 32 steps. Could the degradation in Figures 1 and 5–7 simply reflect fewer short-horizon training goals rather than lower action-goal MI? Does it persist if the number of short-horizon samples is held fixed and distant goals are only added on top?
6 comments
W4: Validity of the Jacobian indicator
- We agree that examining the absolute norms captures true sensitivity changes better. To address this, we have added Figures 12 & 13 (page 21 of the Appendix) in the updated draft to demonstrate that our observations hold for the absolute goal Jacobian norm as well.
- For flow-based policies, we average the resulting Jacobian norms across 4 initial noise samples and 1,024 state–goal pairs per evaluated offset. We differentiate through GCFBC’s full flow integration and GCFQL’s deployed one-step actor, before final action clipping.
- All derivatives are computed in OGBench’s native observation coordinates, including its predefined feature scaling, without additional per-dimension standardization.
W5: Goal Jacobian norm proportion in Section 5.3
We agree — to address this, we have added Figure 14 (page 22 of the Appendix) in the updated draft, which shows that the same trend is reflected in the Jacobian norm proportion as well as the absolute norms.
W6: Mismatch between theory and non-Markovian expert policies in OGBench
We agree that within-subtask plan memory prevents a direct application of Proposition 3.2 at arbitrary temporal offsets. However, once the current subtask ends, OGBench samples a fresh target and plan from the boundary state, so subsequent behavior is independent of the current plan. Then, although there may be a theory-application gap for immediate states, the implementation is consistent with our information-loss argument for any goals beyond the current subtask. We thank the reviewer for pointing out this detail and will revise the discussion to acknowledge this gap between the theoretical guarantee and its application to within-plan play data.
Q1: Why are gains in the navigation task are smaller compared to those in the manipulation task?
- As noted in our response to Reviewer yTCs (Q1), one of the benefits of matching the short-horizon teacher’s sensitivity structure is improved generalization to distractors in combinatorial environments. In navigation, the full observation space is typically relevant for whole-body control, whereas environments such as multi-
cubemanipulation andscenecontain extrinsic object-related features that are not relevant to the immediate choice of action. - Extension to pixel-based observations. We acknowledge that the current method has limited efficacy and is approximately 3.3 times more computationally expensive than SAW in visual environments, and agree that extending S$^2$AW to pixel observations is an important direction for future work. Please refer to our response to reviewer iaGc, Q4 for a proof of concept to bridge this gap.
Q2: Evaluation on trajectory stitching benchmarks
We trained S²AW agents on a representative sample of environments with stitch datasets. S$^2$AW improves performance over base SAW in all tested stitch datasets, and exhibits particularly large gains in antmaze-giant-stitch. Please refer to our response to reviewer iaGc, W2 & Q1 for the result table.
Q3: Comparison to QHyer and CoGHP
S²AW exceeds or matches the performance of QHyer and CoGHP on a representative subset of environments. We evaluated the official implementation of CoGHP on these environments across four seeds, adapting their cube-manipulation configuration from the noisy dataset to the play dataset. Because an official implementation for QHyer is unavailable, we report their published numbers where applicable. S²AW baseline results are drawn directly from our main results table.
| Environment | S$^2$AW | CoGHP | QHyer |
|---|---|---|---|
| antmaze-giant-navigate | – | ||
| cube-double-play | |||
| cube-triple-play | |||
| scene-play |
Q4: Room for improvement regarding computational cost
-
Inspired by the reviewer’s proposed zero-order distillation baseline, we can reframe our objective as the trace of the squared difference matrix, , where is the state Jacobian difference between student and teacher. Using Hutchinson's trace estimator, we project the Jacobian onto randomly sampled vectors to estimate our Sobolev bootstrapping loss in expectation [1]. Empirically, using a small largely recovers our performance but provides a limited 8% runtime speedup. Therefore, while we emphasize the theoretical insights of our analysis, we agree that developing more computationally efficient method variants is an important direction for future work.
Dataset (SAW) Full antmaze-giant-navigate cube-double-play cube-triple-play scene-play
Q5: Does results in Section 3.1 persist if the number of short-horizon samples is held fixed and distant goals are only added on top?
- The reviewer makes a great point that the performance degradation shown by training on longer-horizon goals could simply be due to less training samples in the test-time distribution of goal distances . To address this, we rerun the oracle planner experiments while fixing 1/8th of the sampled goals to the goal distance interval and then sampling the remainder from the interval . We show degradation based on the distribution of goal distances independent of the proportion of goals sampled from , confirming that performance degradation is due to exposure to longer-horizon goals. The full plots can be seen in Figure 11, page 21 of our updated submission.
References
- Czarnecki et al. (2017). Sobolev Training for Neural Networks. NeurIPS 2017.
We thank the reviewer for their thoughful and thorough critiques, and address them point-by-point below.
W1 & 2: Inconsistencies and missing details.
cube-singleis a typo (incorrectly adding 4%). We will correct it to match Table 1 and the original SAW paper.puzzleenvironments are identical to those used for thesceneenvironment. We have updated the manuscript to include them.W3: Additional ablations
In the following results, duplicated S$^2$AW and SAW results are copied from the main results table. The new ablations are evaluated across three seeds due to computational limitations.
Change the sub-goal step number k of the teacher. Increasing the SAW teacher goal horizon by k∈{10,20,50} underperforms S$^2$AW, supporting the role of horizon in better sensitivity targets.
Perform zero-order distillation by adding noise to the input of the teacher. While zero-order distillation with matched noisy inputs yields performance gains in
scene-playandantmaze-giant-navigate, S$^2$AW provides more consistent improvements. We compute a standard deviation vector σ across the training dataset, where σi represents the standard deviation of observation dimension i. We then perturb both the student and teacher state inputs with a shared Gaussian noise scaled by this σ.Simply increase the KL weight of SAW. While scaling the SAW KL weight yields notable gains in
cube-triple-play, S$^2$AW provides more consistent improvements across all environments.kl_alpha×2kl_alpha×10kl_alpha×20