acceptodds
Under review as a conference paper at ICLR 2027

An Informational Curse of Horizon in Goal-Conditioned Policy Learning

80 views

Abstract

The difficulty of learning goal-reaching policies is often attributed to a "curse of horizon" that manifests as bias accumulation in temporal-difference backups and noisy advantage estimates. In this work, we identify an additional *informational* curse of horizon in goal-conditioned policy learning, where increasing the goal relabeling horizon can significantly reduce policy generalization and performance. Through a series of controlled experiments with oracle planners, we decouple the goal horizons sampled during training from those that the policy is asked to reach at test time. Even when evaluated only on a sequence of nearby subgoals, goal-conditioned behavioral cloning (BC) policies suffer from severe, training horizon-dependent performance degradation that is mitigated by reinforcement learning (RL) objectives. We explain this phenomenon as a horizon-dependent decrease in the conditional mutual information between actions and hindsight-relabeled goals, and find empirically that both BC and RL policies trained on longer-horizon goals exhibit a shift in sensitivity from goal to state information, as measured by the policy's input Jacobians. Motivated by this observation, we find that distilling the input Jacobians of short-horizon policies into long-horizon policies yields significant performance gains, especially in combinatorial manipulation tasks. Taken together, our results highlight goal relabeling horizon as an important consideration when learning generalist policies from offline data.

Primary area:
applications to robotics, autonomy, planning
Keywords:
behavior cloning, reinforcement learning, goal-conditioned, offline, hierarchical, long horizon, sobolev, bootstrapping

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_vKbPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This article addresses the long-horizon problem from the perspective of mutual information between actions and the target of re-labeling, and proposes the Sobolev bootstrapping method, which is simple and easy to follow. Moreover, the experimental results are quite comprehensive, and the ablation experiments are also quite thorough. It is a relatively solid article.

Critical strengths

I think the strength of this article lies in its novel perspective. It obtains observations through experiments and presents a new design concept for goal-conditioned strategies from the perspective of mutual information. Moreover, it is simple and easy to implement.

Critical weaknesses
  1. The same method yields different numbers in different tables. For instance, in Table 6, S²AW on cube-double-play-v0, humanoidmaze-giant-navigate-vo, and scene-play-v0 are 50±6, 38±6, and 73±6 respectively, while in Table 1 they are 57±6, 39±5, and 74±6. Table 2 states "Other settings are the same as Table 1", but SAW on cube-single is 76±8, while in Table 1 it is 72±5. Additionally, the hyperparameter table in Table 3 omits the "puzzle-3x3" and "puzzle-4x4" datasets, while Table 1 reports the results for these two datasets.

  2. The statement about the number of seeds is contradictory. Table 1 claims 8 seeds. Table 8 and Table 9 claim 4 seeds, yet they provide exactly the same S²AW numbers as those in Table 1. Table 4 claims 4 seeds, but the λ=0 column is exactly the same as the SAW in Table 1 . The author needs to clarify which numbers are reused and based on how many seeds each one is.

  3. There is a lack of ablation that can attribute the gains to the "narrow-field sensitivity structure". For example: 1) Change the sub-goal step number k of the teacher and use a long-field teacher as a control group. 2) Perform zero-order distillation by adding noise to the input of the teacher. 3) Simply increase the KL weight of SAW.

  4. The validity of the Jacobian indicator is questionable. In Section 3, the ℓ1 proportion of the target Jacobian is used. The decrease in the proportion could be due to a decline in target sensitivity, or it could simply be an increase in state sensitivity. The ℓ1 norm also depends on the scale of each input dimension: position, velocity, and angle are mixed together, and the target also includes mechanical arm body states unrelated to the success determination. The article does not explain whether normalization was performed, nor does it explain how the Jacobian of the flow matching strategy is calculated (whether it is fixed noise or the average of the noise).

  5. Section 3 uses proportion, while Section 5.3 replaces it with the absolute target Jacobian norm, and only compares it with SAW. If the teacher's state Jacobian is large (similar to the feedback gain in tracking control), matching it will increase the state Jacobian of the student. Students may also increase the target Jacobian structurally to fit the action. This does not mean that the target sensitivity has truly recovered; it should also report the proportion.

  6. Theoretical assumptions do not match the experimental data. Proposition 3.2 requires that the behavioral strategy be Markovian, but the original OGBench text (§7.2) clearly states that the "play" data is collected by "non-Markovian expert policies", and the cube and scene experiments in Section 3 use this type of data. At this point, A_t → S_{t+k} → S_{t+k'} does not necessarily form a conditional Markov chain, and the monotonic decrease of mutual information cannot be guaranteed. Even distant future states may reveal hidden intentions, providing more information for the current action. The authors should at least discuss this gap.

Questions for authors
  1. The current experimental results show that the gains in the navigation task are smaller compared to those in the manipulation task. Could the author explain why this is the case? Additionally, I would like to know how well you performed in the visual tasks.

  2. Why didn't the author report the results as SAW did in the stitching-type tasks? If the performance in the manipulation task is acceptable, then it is likely that the stitching ability of S2S^2AW is also good. Please provide the results and conduct an analysis.

  3. Given that S2S^2AW is a relatively novel goal-conditioned strategy design algorithm, how does its performance compare with the latest ones, such as that described in [1, 2]?

[1] Lei, X., Wang, J., Zhang, X., & Wang, D. (2026). QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL. ICML, 2026.

[2] Choi J, Lee S H, Seo S W. Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL. ICML, 2026.

  1. Could the author provide a table and visualizations of the S2S^2AW and RIS/SAW comparisons in the actual ogbench maze environment to demonstrate their contributions? Furthermore, although the method in this paper has been improved, the computational load has also increased. Could the author provide an explanation regarding whether there is any room for improvement in future work?

  2. All three manipulations in Section 3.1 (larger γ, larger H_train, and random-goal mixing) also reduce the share of training pairs whose goals lie near the test-time distance H_plan. For example, under Uniform [1, H_train], only a 32/H_train fraction of relabeled goals falls within 32 steps. Could the degradation in Figures 1 and 5–7 simply reflect fewer short-horizon training goals rather than lower action-goal MI? Does it persist if the number of short-horizon samples is held fixed and distant goals are only added on top?

6 comments

We thank the reviewer for their thoughful and thorough critiques, and address them point-by-point below.

W1 & 2: Inconsistencies and missing details.

  • Different reported results in different tables. Thank you for catching both discrepancies! Table 6 reports S²AW results obtained with an earlier Sobolev weight, and will be updated with the final values from Table 1. In Table 2, the SAW score on cube-single is a typo (incorrectly adding 4%). We will correct it to match Table 1 and the original SAW paper.
  • Missing puzzle dataset hyperparameters. The hyperparameters for the puzzle environments are identical to those used for the scene environment. We have updated the manuscript to include them.
  • Inconsistency between results and claimed number of seeds. We apologize for the confusion. Wherever applicable, we reported the 8-seed results directly from our main results table, and some ablations used 4 seeds. We will add a clarifying note to the manuscript to explicitly state this.

W3: Additional ablations

  • In the following results, duplicated S$^2$AW and SAW results are copied from the main results table. The new ablations are evaluated across three seeds due to computational limitations.

  • Change the sub-goal step number k of the teacher. Increasing the SAW teacher goal horizon by k∈{10,20,50}k\in \{10,20,50\} underperforms S$^2$AW, supporting the role of horizon in better sensitivity targets.

    cube-double-playcube-triple-playscene-playantmaze-giant-navigate
    SAW40±740{\scriptstyle \pm 7}4±24{\scriptstyle \pm 2}63±663{\scriptstyle \pm 6}73±473{\scriptstyle \pm 4}
    S$^2$AW57±6\mathbf{57}{\scriptstyle \pm 6}30±5\mathbf{30}{\scriptstyle \pm 5}74±6\mathbf{74}{\scriptstyle \pm 6}87±3\mathbf{87}{\scriptstyle \pm 3}
    SAW, k+10k+1027±227{\scriptstyle \pm 2}8±18{\scriptstyle \pm 1}53±253{\scriptstyle \pm 2}75±375{\scriptstyle \pm 3}
    SAW, k+20k+2029±229{\scriptstyle \pm 2}12±212{\scriptstyle \pm 2}42±142{\scriptstyle \pm 1}70±170{\scriptstyle \pm 1}
    SAW, k+50k+5010±310{\scriptstyle \pm 3}13±413{\scriptstyle \pm 4}31±231{\scriptstyle \pm 2}43±343{\scriptstyle \pm 3}
  • Perform zero-order distillation by adding noise to the input of the teacher. While zero-order distillation with matched noisy inputs yields performance gains in scene-play and antmaze-giant-navigate, S$^2$AW provides more consistent improvements. We compute a standard deviation vector σ\sigma across the training dataset, where σi\sigma_i represents the standard deviation of observation dimension ii. We then perturb both the student and teacher state inputs with a shared Gaussian noise scaled by this σ\sigma.

    cube-double-playcube-triple-playscene-playantmaze-giant-navigate
    S$^2$AW57±6\bf57{\scriptstyle \pm 6}30±5\bf30{\scriptstyle \pm 5}74±6\bf74{\scriptstyle \pm 6}87±3\bf87{\scriptstyle \pm 3}
    SAW40±740{\scriptstyle \pm 7}4±24{\scriptstyle \pm 2}63±663{\scriptstyle \pm 6}73±473{\scriptstyle \pm 4}
    0.01σ0.01\sigma39±539{\scriptstyle \pm 5}5±15{\scriptstyle \pm 1}62±362{\scriptstyle \pm 3}77±377{\scriptstyle \pm 3}
    0.05σ0.05\sigma43±443{\scriptstyle \pm 4}9±29{\scriptstyle \pm 2}59±459{\scriptstyle \pm 4}85±1\bf85{\scriptstyle \pm 1}
    0.2σ0.2\sigma34±434{\scriptstyle \pm 4}17±317{\scriptstyle \pm 3}72±3\bf72{\scriptstyle \pm 3}57±457{\scriptstyle \pm 4}
    0.3σ0.3\sigma14±014{\scriptstyle \pm 0}8±38{\scriptstyle \pm 3}62±362{\scriptstyle \pm 3}25±425{\scriptstyle \pm 4}
  • Simply increase the KL weight of SAW. While scaling the SAW KL weight yields notable gains in cube-triple-play , S$^2$AW provides more consistent improvements across all environments.

    cube-double-playcube-triple-playscene-playantmaze-giant-navigate
    SAW40±740{\scriptstyle \pm 7}4±24{\scriptstyle \pm 2}63±663{\scriptstyle \pm 6}73±473{\scriptstyle \pm 4}
    S$^2$AW57±6\mathbf{57}{\scriptstyle \pm 6}30±5\mathbf{30}{\scriptstyle \pm 5}74±6\mathbf{74}{\scriptstyle \pm 6}87±3\mathbf{87}{\scriptstyle \pm 3}
    SAW, kl_alpha ×2\times 232±332{\scriptstyle \pm 3}15±315{\scriptstyle \pm 3}56±456{\scriptstyle \pm 4}70±270{\scriptstyle \pm 2}
    SAW, kl_alpha ×10\times 1019±219{\scriptstyle \pm 2}22±522{\scriptstyle \pm 5}48±348{\scriptstyle \pm 3}62±262{\scriptstyle \pm 2}
    SAW, kl_alpha ×20\times 2014±114{\scriptstyle \pm 1}31±6\mathbf{31}{\scriptstyle \pm 6}50±350{\scriptstyle \pm 3}65±165{\scriptstyle \pm 1}

W4: Validity of the Jacobian indicator

  • We agree that examining the absolute norms captures true sensitivity changes better. To address this, we have added Figures 12 & 13 (page 21 of the Appendix) in the updated draft to demonstrate that our observations hold for the absolute goal Jacobian norm as well.
  • For flow-based policies, we average the resulting Jacobian norms across 4 initial noise samples and 1,024 state–goal pairs per evaluated offset. We differentiate through GCFBC’s full flow integration and GCFQL’s deployed one-step actor, before final action clipping.
  • All derivatives are computed in OGBench’s native observation coordinates, including its predefined feature scaling, without additional per-dimension standardization.

W5: Goal Jacobian norm proportion in Section 5.3

We agree — to address this, we have added Figure 14 (page 22 of the Appendix) in the updated draft, which shows that the same trend is reflected in the Jacobian norm proportion as well as the absolute norms.

W6: Mismatch between theory and non-Markovian expert policies in OGBench

We agree that within-subtask plan memory prevents a direct application of Proposition 3.2 at arbitrary temporal offsets. However, once the current subtask ends, OGBench samples a fresh target and plan from the boundary state, so subsequent behavior is independent of the current plan. Then, although there may be a theory-application gap for immediate states, the implementation is consistent with our information-loss argument for any goals beyond the current subtask. We thank the reviewer for pointing out this detail and will revise the discussion to acknowledge this gap between the theoretical guarantee and its application to within-plan play data.

Q1: Why are gains in the navigation task are smaller compared to those in the manipulation task?

  • As noted in our response to Reviewer yTCs (Q1), one of the benefits of matching the short-horizon teacher’s sensitivity structure is improved generalization to distractors in combinatorial environments. In navigation, the full observation space is typically relevant for whole-body control, whereas environments such as multi-cube manipulation and scene contain extrinsic object-related features that are not relevant to the immediate choice of action.
  • Extension to pixel-based observations. We acknowledge that the current method has limited efficacy and is approximately 3.3 times more computationally expensive than SAW in visual environments, and agree that extending S$^2$AW to pixel observations is an important direction for future work. Please refer to our response to reviewer iaGc, Q4 for a proof of concept to bridge this gap.

Q2: Evaluation on trajectory stitching benchmarks

We trained S²AW agents on a representative sample of environments with stitch datasets. S$^2$AW improves performance over base SAW in all tested stitch datasets, and exhibits particularly large gains in antmaze-giant-stitch. Please refer to our response to reviewer iaGc, W2 & Q1 for the result table.

Q3: Comparison to QHyer and CoGHP

S²AW exceeds or matches the performance of QHyer and CoGHP on a representative subset of environments. We evaluated the official implementation of CoGHP on these environments across four seeds, adapting their cube-manipulation configuration from the noisy dataset to the play dataset. Because an official implementation for QHyer is unavailable, we report their published numbers where applicable. S²AW baseline results are drawn directly from our main results table.

EnvironmentS$^2$AWCoGHPQHyer
antmaze-giant-navigate87±3\mathbf{87}{\scriptstyle \pm 3}73±473{\scriptstyle \pm 4}–
cube-double-play57±6\mathbf{57}{\scriptstyle \pm 6}6±06{\scriptstyle \pm 0}56±2\mathbf{56}{\scriptstyle \pm 2}
cube-triple-play30±5\mathbf{30}{\scriptstyle \pm 5}7±07{\scriptstyle \pm 0}10±510{\scriptstyle \pm 5}
scene-play74±6\mathbf{74}{\scriptstyle \pm 6}75±2\mathbf{75}{\scriptstyle \pm 2}53±253{\scriptstyle \pm 2}

Q4: Room for improvement regarding computational cost

  • Inspired by the reviewer’s proposed zero-order distillation baseline, we can reframe our objective as the trace of the squared difference matrix, tr⁡(ΔJΔJT)\operatorname{tr}(\Delta J \Delta J^T), where ΔJ\Delta J is the state Jacobian difference between student and teacher. Using Hutchinson's trace estimator, we project the Jacobian onto nn randomly sampled vectors ϵ\epsilon to estimate our Sobolev bootstrapping loss in expectation [1]. Empirically, using a small n=3n=3 largely recovers our performance but provides a limited 8% runtime speedup. Therefore, while we emphasize the theoretical insights of our analysis, we agree that developing more computationally efficient method variants is an important direction for future work.

    Datasetn=0n=0 (SAW)n=3n=3Full
    antmaze-giant-navigate73±473{\scriptstyle \pm 4}85±3\mathbf{85}{\scriptstyle \pm 3}87±3\mathbf{87}{\scriptstyle \pm 3}
    cube-double-play40±740{\scriptstyle \pm 7}51±651{\scriptstyle \pm 6}57±6\mathbf{57}{\scriptstyle \pm 6}
    cube-triple-play4±24{\scriptstyle \pm 2}28±7\mathbf{28}{\scriptstyle \pm 7}30±5\mathbf{30}{\scriptstyle \pm 5}
    scene-play63±663{\scriptstyle \pm 6}71±9\mathbf{71}{\scriptstyle \pm 9}74±6\mathbf{74}{\scriptstyle \pm 6}

Q5: Does results in Section 3.1 persist if the number of short-horizon samples is held fixed and distant goals are only added on top?

  • The reviewer makes a great point that the performance degradation shown by training on longer-horizon goals could simply be due to less training samples in the test-time distribution of goal distances [1,Hplan][1, H_\mathrm{plan}]. To address this, we rerun the oracle planner experiments while fixing 1/8th of the sampled goals to the goal distance interval [1,32][1, 32] and then sampling the remainder from the interval [33,Htrain][33, H_\mathrm{train}]. We show degradation based on the distribution of goal distances independent of the proportion of goals sampled from [1,Hplan][1, H_\mathrm{plan}], confirming that performance degradation is due to exposure to longer-horizon goals. The full plots can be seen in Figure 11, page 21 of our updated submission.

References

  1. Czarnecki et al. (2017). Sobolev Training for Neural Networks. NeurIPS 2017.
Reviewer_iaGcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a critical yet overlooked phenomenon in offline goal-conditioned reinforcement learning (GCRL), termed the "informational curse of horizon", where increasing the goal relabeling horizon degrades a policy's ability to reach nearby goals due to the decay of action-goal mutual information. To mitigate this issue, the authors introduce Sobolev bootstrapping (S2AW\text{S}^2\text{AW}), a policy extraction objective that distills the input state Jacobians of a short-horizon teacher policy into a long-horizon student policy. Empirical evaluations on the OGBench suite validate the insights and demonstrate marked performance improvements, particularly on combinatorial manipulation tasks. Overall, this is a well-motivated, conceptually novel, and technically solid paper that addresses a fundamental challenge in goal-conditioned learning. I lean towards acceptance.

Critical strengths
  1. Compelling Problem Formulation & Conceptual Insights: The identification of the "informational curse of horizon" is both novel and insightful. Decoupling training horizon from evaluation horizon via oracle planners reveals an essential failure mode in goal-conditioned policies that cannot be explained purely by task complexity or value learning error.

  2. Solid Theoretical & Mechanistic Analysis: The paper connects the empirical degradation to a decay in action-goal conditional mutual information and formalizes it through total variation distance bounds and Fisher information geometry. The use of policy input Jacobians provides a concrete, interpretable diagnostic tool for monitoring goal sensitivity collapse.

  3. Rigorous Empirical Validation: The authors provide thorough experimental evidence across diverse locomotion and manipulation benchmarks. The inclusion of independent scripted-trajectory attention analyses and matched-architecture ablations demonstrates notable scientific rigor.

Critical weaknesses
  1. Vulnerability to Critic Quality in Subgoal Advantage Weighting: The proposed method relies on an exponential subgoal advantage weight, exp⁡(V(sg,g)−V(s,g))\exp(V(s_g, g) - V(s, g)), to filter out suboptimal or irrelevant in-trajectory states. However, over extended horizons, value functions are prone to bias accumulation and noisy advantage estimates. When the critic becomes unreliable on distant goals, this weighting mechanism fails, causing the student policy to distill noisy or misleading input Jacobians from the teacher. This vulnerability is empirically evident in the RIS ablation (Table 7), where matching unweighted or noisy teacher Jacobians results in severe performance drops.

  2. Absence of Evaluation on Trajectory Stitching Benchmarks: The evaluation is limited exclusively to the -navigate-v0 and -play-v0 datasets, leaving out the more demanding -stitch-v0 splits in OGBench. In fragmented stitching datasets, distant global goals cannot be sampled via in-trajectory hindsight relabeling, requiring cross-trajectory random goals. However, Section 3.1 explicitly demonstrates that random goals cause action-goal mutual information to collapse to zero, leading to severe performance degradation across both BC and RL algorithms. Furthermore, fixed large-offset subgoal sampling (e.g., k=100k=100) is susceptible to boundary truncation failures in short trajectory fragments.

  3. Computational Overhead from Higher-Order Differentiation: Jacobian matching requires double backpropagation to optimize input gradient alignment. As reported in Appendix G (Table 5), this increases wall-clock training time by 60% to 280% compared to baseline policy bootstrapping. This additional computational burden poses practical challenges for scaling the framework to pixel-level observations or large-scale foundation policy architectures.

Questions for authors
  1. Evaluation on Trajectory Stitching Benchmarks: Could the authors provide experimental results of S2AW\text{S}^2\text{AW} on the OGBench -stitch-v0 datasets (e.g., pointmaze/antmaze/humanoidmaze-*-stitch-v0)? In highly fragmented datasets where trajectories are short, how does S2AW\text{S}^2\text{AW} resolve the dilemma between "the collapse of action-goal mutual information caused by cross-trajectory random goals" and "the truncation failure of large-step in-trajectory subgoals"?

  2. Disentangling Critic Failure from Actor Degradation in Long Horizons: In tasks where the policy still struggles (e.g., cube-quadruple-play-v0 with a 3% success rate), is this failure primarily bottlenecked by the inaccurate critic value span ΔQ\Delta_Q and noisy advantage weighting A(s,sg,g)A(s, s_g, g) misleading the teacher Jacobians? If an oracle or more accurate value function is provided for computing the advantage weights, does S2AW\text{S}^2\text{AW} exhibit significant gains on such ultra-long-horizon tasks?

  3. Robustness of Jacobian Matching Against Subgoal Noise: As observed in the RIS ablation (Table 7), distilling noisy or unweighted teacher Jacobians causes performance degradation (e.g., from 81% to 59% in cube-single). Does first-order Jacobian distillation have an inherently lower tolerance to noisy or false-positive subgoals compared to zeroth-order action distillation (KL divergence)? How sensitive is S2AW\text{S}^2\text{AW} to stochastic environments where action-free GCIVL incurs optimistic bias? Furthermore, from an information-geometric perspective, have the authors considered the role of second-order information (e.g., input Hessians or local curvature of the policy)? Could constraining the curvature (second derivatives) further prevent sharp, erratic policy shifts and improve generalization in contact-rich manipulation, or does the computational cost of higher-order automatic differentiation make second-order distillation prohibitive?

  4. Extension to Pixel-based Observations: Given that computing input Jacobians requires higher-order differentiation and scales with input dimensionality, what is the expected computational feasibility and behavior of Sobolev bootstrapping when moving from state-based inputs to high-dimensional image observations?

4 comments

W1 & Q2: Vulnerability to critic quality in subgoal advantage weighting

  • While we agree that S$^2$AW relies on the quality of value learning, this is generally true of all advantage-weighted methods and we note that our method is agnostic to the choice of value learning algorithm. This allows it to benefit from advancements in value learning methods, which we demonstrate by replacing the MLP backbone for implicit value learning with a Latent Alignment Network (LAN) [1]. This produces a clear performance improvement in the hardest cube-quadruple-play environment. The baseline S$^2$AW results are drawn from our main results table, while these new LAN experiments are evaluated across 4 seeds.

    envLAN + SAWLAN + S$^2$AWS$^2$AW
    cube-triple-play16±616{\scriptstyle \pm 6}31±3\textbf{31}{\scriptstyle \pm 3}30±530{\scriptstyle \pm 5}
    cube-quadruple-play1±11{\scriptstyle \pm 1}16±6\textbf{16}{\scriptstyle \pm 6}3±23{\scriptstyle \pm 2}

W2 & Q1: Absence of evaluation on trajectory stitching benchmarks

  • This is a great question — in the OGBench stitch datasets, it is standard to sample 1/2 of the actor goals randomly rather than from uniformly distributed in-trajectory offsets, which may result in the dilemma noted by the reviewer and shown in Figure 10. In this case, the subgoal advantage term is the core mechanism that mitigates the informational issues by matching local sensitivity structure more heavily when the sampled in-trajectory subgoal is relevant to the randomly sampled goal.

  • To empirically validate this, we trained S²AW agents on the most challenging navigation environments with stitch datasets and report the results below. S$^2$AW improves performance over base SAW in all tested stitch datasets, and exhibits particularly large gains in antmaze-giant-stitch.

    HIQLSAWS$^2$AW
    antmaze-giant-stitch2±22 {\scriptstyle \pm 2}2±22 {\scriptstyle \pm 2}28±9\textbf{28} {\scriptstyle \pm 9}
    humanoidmaze-giant-stitch3±23 {\scriptstyle \pm 2}6±36 {\scriptstyle \pm 3}11±3\textbf{11} {\scriptstyle \pm 3}

Q3: Robustness of Jacobian matching against subgoal noise

  • Does first-order Jacobian distillation have an inherently lower tolerance to noisy or false-positive subgoals compared to zeroth-order action distillation (KL divergence)?

    First-order Jacobian distillation is more robust against noisy subgoals. We replace in-trajectory subgoals with random states in the dataset under probability p∈{0,0.25,0.5}p\in\{0, 0.25,0.5\}. The teacher, the value function and the one-step AWR term are still trained on clean data. Jacobian-only distillation degrades the least in both environments. All results are evaluated with 3 seeds.

    Methodcube-single-play, p = 0p = 0.25p = 0.5cube-double-play, p = 0p = 0.25p = 0.5
    AWR only83±283{\scriptstyle \pm 2}83±283{\scriptstyle \pm 2}83±283{\scriptstyle \pm 2}26±126{\scriptstyle \pm 1}26±126{\scriptstyle \pm 1}26±126{\scriptstyle \pm 1}
    KL (SAW)76±276{\scriptstyle \pm 2}15±415{\scriptstyle \pm 4}3±23{\scriptstyle \pm 2}39±339{\scriptstyle \pm 3}2±02{\scriptstyle \pm 0}1±01{\scriptstyle \pm 0}
    KL + Jacobian (S$^2$AW)88±388{\scriptstyle \pm 3}38±338{\scriptstyle \pm 3}15±315{\scriptstyle \pm 3}58±558{\scriptstyle \pm 5}7±17{\scriptstyle \pm 1}4±14{\scriptstyle \pm 1}
    Jacobian only90±090{\scriptstyle \pm 0}83±583{\scriptstyle \pm 5}72±272{\scriptstyle \pm 2}30±430{\scriptstyle \pm 4}25±225{\scriptstyle \pm 2}17±217{\scriptstyle \pm 2}
  • How sensitive is S$^2$AW to stochastic environments where action-free GCIVL incurs optimistic bias?

    We find that Sobolev bootstrapping is more robust against environment stochasticity. To demonstrate this, we created stochastic versions of OGBench manipulation tasks where the dataset records the oracle's commanded action, but the simulator executes it under (unobserved) Gaussian noise with σ∈{0.2,0.3}\sigma \in \{0.2, 0.3\}, resulting in stochastic transitions during data collection and rollout. Sobolev bootstrapping improves over KL alone at every noise level. All results are evaluated with 3 seeds.

    Methodcube-single-play, σ = 0σ = 0.2σ = 0.3cube-double-play, σ = 0σ = 0.2σ = 0.3
    KL (SAW)76±276{\scriptstyle \pm 2}50±250{\scriptstyle \pm 2}29±129{\scriptstyle \pm 1}39±339{\scriptstyle \pm 3}22±122{\scriptstyle \pm 1}10±210{\scriptstyle \pm 2}
    KL + Jacobian (S$^2$AW)88±388{\scriptstyle \pm 3}69±269{\scriptstyle \pm 2}46±246{\scriptstyle \pm 2}58±558{\scriptstyle \pm 5}46±546{\scriptstyle \pm 5}16±216{\scriptstyle \pm 2}
  • Role of second-order information (e.g., input Hessians or local curvature of the policy).

    Incorporating second-order curvature terms yields no performance benefits beyond first-order Jacobian matching. Because full Hessian computation is prohibitively expensive, we estimated curvature using random directional second derivatives. We tested two variants: matching the teacher's curvature, which hurts performance because the teacher's curvature grows excessively large, and a teacher-free curvature penalty, which also proves harmful. Thus, second-order matching is computationally expensive and lacks empirical utility in this setting, though task-specific curvature regularization remains an interesting avenue for future work. ww below denotes weight. All results are evaluated with 3 seeds.

    Curvature term added to S$^2$AWcube-single-playcube-double-play
    None88±388{\scriptstyle \pm 3}58±558{\scriptstyle \pm 5}
    Match teacher curvature (w=0.1)54±1854{\scriptstyle \pm 18}42±642{\scriptstyle \pm 6}
    Match teacher curvature (w=1.0)18±218{\scriptstyle \pm 2}20±420{\scriptstyle \pm 4}
    Curvature penalty (w=0.1)80±280{\scriptstyle \pm 2}56±356{\scriptstyle \pm 3}
    cube-single-play, bsz=1024ms per updateRelative to KL
    KL (SAW)2.032.031.0×1.0\times
    KL + Jacobian (S$^2$AW)2.782.781.4×1.4\times
    S$^2$AW + directional curvature, 1 direction3.763.761.9×1.9\times
    S$^2$AW + directional curvature, 4 directions6.276.273.1×3.1\times
    S$^2$AW + full Hessian matching25.4125.4112.5×12.5\times

Q4: Extension to Pixel-based Observations

  • We agree that extending S$^2$AW to pixel observations is an important direction for future work, as the current method has limited efficacy and is approximately 3.3 times more computationally expensive than SAW in visual environments.

  • As a proof of concept to bridge this gap, we experimented with latent space alignment. By having the long-horizon and short-horizon actors share the same state encoder (and thus the same latent space), we can perform sobolev bootstrapping in the lower-dimensional latent space. While sharing the state encoder (SSE) causes a performance drop relative to standard SAW, we observe benefits from performing sobolev bootstrapping in the latent space (SSE + SBL) without incurring any additional wall-clock time over the SAW baseline. In practice, we use the value function state encoder for both actors. Experiments are run across four seeds.

    DatasetSAW + SSESAW + SSE + SBL
    visual-cube-double-play-v05±125{\scriptstyle \pm 12}18±1518{\scriptstyle \pm 15}
    visual-cube-triple-play-v011±611{\scriptstyle \pm 6}12±812{\scriptstyle \pm 8}
    visual-scene-play-v07±197{\scriptstyle \pm 19}27±1027{\scriptstyle \pm 10}

References

  1. Kang et al. (2026). Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning. ICML 2026.

Thank you for the detailed feedback. I think my main concerns have been addressed basically. Therefore, I would like to maintain my original positive rating.

Reviewer_yTCsOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Training with more distant hindsight goals can degrade performance on nearby goals, particularly for BC policies.
  • Policies trained with more distant goals exhibit lower relative goal sensitivity at matched state–goal distances.
  • The paper proposes S2AW, which distills a short-horizon teacher’s state Jacobians into a long-horizon student.
Critical strengths
  • The paper shows that long-horizon learning requires not only critic improvements but also attention to how the actor uses goals under hindsight goal relabeling. This offers an interesting perspective on actor training beyond the critic.
  • Training with different goal distances changes the actor’s state and goal Jacobians. Using this finding, the authors add a distillation loss to SAW, improving performance on both maze and cube manipulation tasks.
Critical weaknesses

Finding–Method Connection

The paper shows that state Jacobians vary with the training horizon, but does not sufficiently explain why this motivates matching a short-horizon teacher’s state Jacobians. A teacher trained on nearby goals may provide better supervision, but why state Jacobians are particularly useful features to distill remains unclear. Similarly, a gap remains between the mutual-information analysis in Theorem 3.1 and state Jacobians.

Comparison with Hierarchical Policies

Although flat policies offer advantages such as reduced compounding error, critic improvements combined with hierarchical policies have already achieved strong performance on most maze tasks. However, methods such as OTA are absent from the baselines in Table 1. The additional value of achieving similar performance with a flat policy therefore remains unclear.

Questions for authors
  • Figure 2 reports the relative goal-Jacobian norm (ratio). Should the absolute norm also be examined? The ratio could increase even if the absolute norm decreases. Is the ratio alone sufficient to characterize goal and state sensitivity?
5 comments

We thank the reviewer for their insights and hope that the below rebuttal clears up our argument's flow and addresses their concerns about soundness.

W1: The motivation for matching short-horizon state Jacobians is insufficiently justified.

Theorem 3.1 shows that when hindsight goals carry little information about the immediate action and the critic weakly distinguishes actions, the target policy is close on average to the state-only behavior policy. Nearby goals can retain more action information (Prop. 3.2), motivating a short-horizon teacher. Figure 2 shows a corresponding empirical shift in relative sensitivity toward the state as the training horizon increases.

As observed in [1], the key bottleneck in offline RL is not fitting the offline dataset, but rather generalization to new states during online rollouts. Then, two policies can agree on the action at a recorded state yet respond differently to movements in distractor objects, with significant effects on downstream performance. The teacher and student see different goals but share the current state, so matching the state Jacobian transfers the teacher’s local response to that perturbation. On the four-cube distractor task, this yields 35% success versus 1–2% for uniform state-Jacobian penalties (Table 2).

W2: Missing hierarchical baselines (e.g., OTA) obscure the proposed flat policy's value.

We would like to clarify that we have already included additional hierarchical and horizon reduction baselines in Appendix H.3, including OTA-V [2], DQC [3], and SHARSA [4]. We show that S$^2$AW either matches or exceeds the performance of all hierarchical baselines.

Q1: In Figure 2, the goal-Jacobian norm ratio could increase even if the absolute norm decreases.

We agree that examining the absolute norms captures true sensitivity changes better. To address this, we have added Figures 12 & 13 (page 21 of the Appendix) in the updated draft to demonstrate that our observations hold for the absolute goal Jacobian norm as well.

References

  1. Park et al. (2024). Is Value Learning Really the Main Bottleneck in Offline RL? NeurIPS 2024.
  2. Ahn et al. (2025). Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning. NeurIPS 2025.
  3. Li, Park, & Levine (2026). Decoupled Q-Chunking. ICLR 2026.
  4. Park et al. (2025). Horizon Reduction Makes RL Scalable. NeurIPS 2025.

I thank the authors for their response.

W1.

I agree that Section 3 motivates distilling from a short-horizon teacher. However, it does not justify why the distillation target should be the state Jacobian.

Line 354 states that Section 3 suggests that the input Jacobian of a short-horizon teacher may also provide useful supervisory signals, but the metric Section 3 relies on is relative goal sensitivity, which does not show that the teacher’s state Jacobian is a good supervisory signal. If the justification is instead generalization to new states during online rollouts, as stated in the response, this is an out-of-distribution generalization argument that Section 3 does not address.

W2.

Appendix H.3 compares against OTA only on antmaze-giant-navigate. However, Table 6 of the original OTA paper reports results that outperform HIQL on humanoidmaze-navigate-{large,giant}. Since these results are absent from Table 1, S2AW appears to have a considerable advantage on the navigate tasks.

Could the authors clarify why hierarchical baselines with n-step value learning are excluded from the main table? I understand that hierarchical policies suffer from compounding errors, but is there a specific reason for excluding n-step-based hierarchical methods?

Q1.

I have checked Figures 12 & 13. Thank you for the additional analysis.

Thanks for the continued engagement and thoughtful critiques!

W1: Lack of justification for the state Jacobian as the distillation target

We agree that our previous response appealed to rollout generalization that Section 3 does not necessarily address. Instead, we refer to the state-only limit discussed in Theorem 3.1, which provides more direct motivation: a goal-agnostic policy cannot adapt its response to the same scene when the requested goal changes. As an example, in multi-object manipulation, the same scene requires different responses depending on which object must be manipulated. Preserving goal-dependent structure in the policy’s response to state information is critical in such cases, and we therefore investigate whether a short-horizon teacher conditioned on a suitable subgoal provides a useful prior for this sensitivity structure.

State-Jacobian distillation transfers this sensitivity structure by extending action supervision to the teacher’s local response to state information. On a more practical note, the teacher and student share the current state, despite receiving different goals, and so it would only make sense to use the state Jacobian. A state perturbation represents the same physical change for both policies, whereas perturbing a nearby subgoal and a final goal need not induce corresponding action responses. Our additional assumption is that the teacher’s local feedback remains useful when the subgoal remains appropriate under these changes. We will introduce this rationale explicitly in Section 4.2 and present the analyses in Section 5 as empirical support for this hypothesis.

W2: Missing nn-step hierarchical baselines

At its core, our analysis and method focuses on policy extraction, which is orthogonal to value learning methods for standard offline GCRL algorithms. Algorithms such as OTA focus on mitigating bias and compounding error using nn-step value backups, which is a useful and impactful direction for improving performance but not necessarily a relevant baseline for our work.

Indeed, we have verified that our method benefits from newer value learning methods in our response to Reviewer iaGc (W1 & Q2). Second, OTA did not report state-based manipulation results, and it performs poorly there without further hyperparameter tuning, as shown in Appendix H.3. We agree that a more comprehensive comparison in the navigation environments should include OTA’s navigation results and will update our main result table accordingly.

Reviewer_MW36OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the "informational curse of horizon" in goal-conditioned policy learning, demonstrating that increasing the goal relabeling horizon during training reduces the policy's sensitivity to goal information. To mitigate this issue, the authors propose "Sobolev bootstrapping," a method that alleviates the problem by distilling the input Jacobians of a short-horizon teacher policy. The proposed method is systematically validated through experiments constructed from three perspectives: decoupling training and testing horizons, applying multi-dimensional interventions to the training goal distribution, and quantifying and benchmarking local policy sensitivity.

Critical strengths

The authors extend the challenges introduced by long horizons from traditional optimization difficulties, such as bias accumulation in temporal-difference backups and noisy advantage estimates, to the information-theoretic level. Furthermore, they prove that due to trajectory merging, the conditional mutual information between actions and future states monotonically decreases as the horizon increases.

Sobolev bootstrapping not only improves empirical performance metrics but also prevents the network from exploiting state-only optimization shortcuts through first-order gradient distillation, providing a novel regularization perspective with physical and information-geometric significance for policy learning.

Critical weaknesses

Box 3.3 discussing Fisher information regularization establishes a rigorous connection between the input Jacobian and the Fisher information matrix under fixed-variance Gaussian policies. However, several experiments in the paper employ flow-matching policy classes or other complex distributions. Under these non-Gaussian policies, the validity of the Jacobian norm as a proxy for local policy sensitivity lacks further theoretical guarantees or empirical validation. (Additionally, Box 3.3 is not explicitly numbered in the manuscript.)

Sobolev bootstrapping requires maintaining an additional short-horizon teacher policy and computing input Jacobians, which introduces significant computational overhead (as shown in Appendix G, where training time is noticeably increased). The authors also acknowledge this limitation in the Discussion section.

Questions for authors

Under these non-Gaussian policies, the validity of the Jacobian norm as a proxy for local policy sensitivity lacks further theoretical guarantees or empirical validation.

How can we dynamically maintain the Fisher information between actions and goals during long-horizon training, thereby avoiding the reliance on explicit teacher policies and costly Jacobian computations?

1 comment

We thank the reviewer for their time and positive impression of our work, and address their concerns around flow policies and computational expense below.

W1 & Q1: Validity of Jacobian norm as a proxy for local policy sensitivity for non-Gaussian policies.

To illustrate the applicability of Fisher information to flow policies, consider the following case: suppose we fix the state and integrate the flow twice from the same initial noise sample zz, once at goal gg and once at g+ϵug+\epsilon u. To first order, the sampled action changes by ϵJg(z)u\epsilon J_g(z)u, where Jg(z)J_g(z) is the goal Jacobian of the final action. At the level of the full distribution, Wasserstein distance is the smallest average action displacement over all pairings, and shared noise gives one such pairing. Thus, even for non-Gaussian action distributions, Ez∣Jg(z)u∣1\mathbb E_z|J_g(z)u|_1 upper-bounds how fast the action distribution can move along the direction of goal change uu. Our plotted ℓ1\ell_1 goal Jacobian norm expresses this local sensitivity relative to state sensitivity. On cube-triple, longer training horizons reduce this measure of sensitivity across goal offsets and also decrease task success with fixed-distance oracle subgoals for both flow policies (Figs. 5 and 9). We will extend our argument in Box 3.3 to note its application to flow policies with the modified Wasserstein distance interpretation, which has been previously used in the context of conservative regularization [1].

W2: Sobolev bootstrapping creates high computational overhead.

We agree that double gradient descent is indeed costly and noted this in our conclusion. To move towards cheaper alternatives, we test a lightweight approximation with Hutchinson’s trace estimator in our response to Reviewer vKbP (Q4), which provides a small speedup while retaining significant gains. We also tested cheaper finite-difference approximations with matching teacher and student outputs at the same perturbed state, which avoids explicit input Jacobians in our response to Reviewer vKbP (W3). This came within two percentage points of S²AW on scene and antmaze-giant for the best noise scale, although it failed to recover the performance of full Jacobian distillation on cube-double and cube-triple.

Q2: How can we maintain Fisher information efficiently without costly teacher policies?

This is an interesting question! One of our key arguments in the paper is that the Fisher information degrades as the policy is trained on longer horizon goals, but simply getting rid of them makes the long-horizon goals presented at test time OOD — this is the core motivation for keeping a separate teacher policy and cannot be easily removed. We did try some teacher-free baselines in Appendix H.4 that simply perform uniform Jacobian norm regularization. While ℓ1\ell_1 regularization demonstrated some performance benefits in simpler environments such as cube-single and double, it showed substantial degradation as the number of cubes increased compared to an explicit teacher.

References

  1. Park et al. (2025). Flow Q-Learning. ICML 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.