acceptodds
Under review as a conference paper at ICLR 2027

Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

14 views

Abstract

Existing approaches to offline goal-conditioned reinforcement learning (GCRL) struggle with long-horizon tasks. Discounting shrinks value differences between distant states until they fall below the function approximation error, leaving the agent with no signal for ranking states. Temporal abstraction—treating environment steps as a single transition—restores this signal at long range, but no single fixed suits all state-goal distances: large preserves value differences across long temporal distances while collapsing distinctions between nearby states, and small does the reverse. We make this trade-off explicit and introduce Generalized Implicit Temporal Abstraction (GITA), which conditions a single value function on . GITA trains one policy by aggregating advantage-weighted supervision across multiple values, so scales assigning larger positive advantages to a state-goal pair contribute more strongly to its update. GITA does not need to choose between local resolution and long-range signal; it retains both without committing to a single . On OGBench, GITA outperforms a broad range of offline GCRL baselines, raising average success rate across all tasks by 25 percentage points (73% relative improvement) over HIQL. It also improves over the strongest fixed- method, OTA, by 7 percentage points (14% relative).

Primary area:
reinforcement learning
Keywords:
Goal-Conditioned Reinforcement Learning, Offline Reinforcement Learning, Temporal Abstraction, Multi-Scale Reinforcement Learning, Long-Horizon, Hierarchical Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zMMnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies offline goal-conditioned reinforcement learning. For distant goals, discounting shrinks the value differences between neighboring states until they fall below the network’s error, so the agent can’t rank subgoals. Temporal abstraction, which treats k steps as one transition, restores this signal, but a fixed k blurs nearby states when it is large and loses distant ones when it is small. The authors propose GITA, which conditions one value function on k and trains a single policy by weighting advantages across several k values, so the most informative scale dominates for each state-goal pair. On OGBench, GITA outperforms prior methods, including the fixed-k method OTA, especially on long-horizon tasks.

Critical strengths

The paper clearly shows why one fixed step size can’t work for both near and far goals, and its easy fix gives the biggest gains on the hardest long-distance tasks.

Critical weaknesses

the method doesn’t need you to pick k anymore, but it still tunes γ for every task, and the results swing a lot depending on that choice. In AntMaze, the wrong γ drops success to almost zero. So it feels like the tuning just moved from k to γ instead of really going away.

Questions for authors

If I understand the analysis correctly, in your idealized model jumping k steps gives the same values as just using a discount of γ to the power 1/k. So the theory doesn’t really separate GITA from conditioning on the discount instead. What do you think makes k-step jumps work better in practice? Since the policy is a single Gaussian trained with weights from several scales, I wonder what happens when small and large k prefer different subgoals. Could the policy end up averaging them into a subgoal in between that neither scale actually wants, for example one inside a wall?

Reviewer_nfe1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Generalized Implicit Temporal Abstraction (GITA), an offline goal-conditioned reinforcement learning method designed to address the trade-off between long-range value propagation and local state discrimination. GITA conditions a shared value function on a temporal abstraction factor and trains a single high-level policy by aggregating advantage-weighted supervision across multiple scales. The authors also provide an analysis of the reachability–resolution trade-off. Experiments evaluate GITA on OGBench, reporting improvements over several baselines.

Critical strengths
  1. This paper is well-motivated: a fixed temporal abstraction factor trades off long-range value propagation against local discrimination, and the preferred scale depends on the state-goal distance. GITA addresses this limitation by conditioning a shared value function on the abstraction factor and aggregating scale-specific exponentiated advantages to train a single high-level policy, which is novel and inspiring.

  2. Experiments show promising empirical evidence on multiple tasks, supported by targeted ablations. GITA improves substantially over HIQL and often over the directly relevant fixed-scale OTA baseline. The ablations also indicate that conditioning the value function on the abstraction factor and retaining multi-scale policy supervision are useful design choices. These results support the practical relevance of the proposed approach beyond its theoretical motivation.

Critical weaknesses
  1. The central claim of implicit, state-goal-dependent scale selection is not fully justified by the proposed mechanism or the evidence. GITA averages exponentiated advantages across a predefined set of scales, so a scale with a larger advantage receives greater relative influence. However, this does not prove that the dominant scale is the one that most reliably ranks candidate subgoals for each state-goal pair. This weakens the central methodological interpretation, even if the empirical performance gains are genuine.

  2. The proposed method involves several critical parameters whose sensitivities are not sufficiently discussed, e.g. K\mathcal{K}. It’s also unclear how is this choice is related to the discount factor. These are critical for understanding the role of different scales.

Questions for authors
  1. The paper states that the scale assigning the largest AWR weight is also the scale that most reliably ranks candidate subgoals. How is this relationship established? To my understanding, a large value for certain scale cannot represent value for other scales, therefore it may also not represent the general action value. Would this design lead to overestimation issue of value function?

  2. How sensitive is the method to the kk candidate set?

1 comment

Thanks the authors for the detailed rebuttal. Given the added experiment results and other reviews, I believe the current ratings reflect my evaluation of this paper.

Reviewer_SjPqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the loss of value signal in long-horizon offline goal-conditioned RL. Under discounting, value differences between nearby states shrink as the goal moves away and eventually fall below the function-approximation error. Temporal abstraction with a fixed stride k, as in OTA, shortens value propagation but blurs distinctions between nearby states. The authors analyse this trade-off with a relaxed shortest-path model. They show that the high-level advantage factors into a reachability term that grows with k and a resolution term that shrinks with k, so the advantage-maximising k grows with state–goal distance. They support this empirically with an Advantage–Progress Correlation analysis. Building on this, they propose GITA:

  • A single high-level value network is conditioned on k (via FiLM) and trained with k sampled from a geometric distribution.
  • The high-level policy is trained with AWR, using the average of exponentiated advantages over a fixed set of k values. Scales with larger positive advantages implicitly dominate each update.

On 23 OGBench tasks, GITA reports a 25 pp average improvement over HIQL and 7 pp over OTA, and ablations examine k-conditioning and the aggregation rule.

Critical strengths

Pros

  1. The paper clearly identifies a limitation of OTA: a single fixed abstraction factor cannot suit all state–goal distances. It supports this both analytically (the reachability/resolution factorisation and the growth of the optimal k with distance) and empirically (the APC analysis shows the best k increasing with distance). It then proposes a new temporal abstraction method that resolves this limitation by learning multiple timescales in one value function, without explicitly selecting k.
Critical weaknesses

Cons

  1. The experimental results are somewhat inconsistent.

    • On HumanoidMaze navigate-large, OTA outperforms GITA (82 vs. 73).
    • On the state-based Cube and Scene tasks, the non-hierarchical GCIQL wins by a wide margin over GITA, HIQL and OTA.
    • On Visual-AntMaze navigate, CRL is the best method.

    In addition, Cube single and double have short horizons per task, so they are hard to regard as long-horizon problems and are not well suited to evaluating a temporal abstraction method.

  2. Per-task tuning of the discount factor γ, and high sensitivity to it.

    • The paper contrasts GITA's single candidate set K with OTA's per-task k, but GITA re-tunes γ per task (from {0.98, 0.99, 0.995}, plus 0.97 for manipulation).
    • Because α = γ, the same γ also determines the distribution from which k is sampled during value learning. In effect, the per-task tuning has moved from k to γ rather than been removed.
    • Appendix Table 8 shows strong sensitivity. At γ = 0.995, AntMaze performance collapses (navigate-large 0.7%, stitch-giant 0.0%), likely because E[k] = 200 matches the trajectory length.
    • Under the best single global γ = 0.99, several AntMaze tasks fall to or below OTA's reported performance: stitch-giant 16.6 vs. OTA's 29, stitch-large 60.2 vs. 79, navigate-giant 65.8 vs. 70. These are final-checkpoint numbers, so the comparison is approximate.
    • The authors should report results with a shared γ, or decouple α from γ, so that the gains can be separated from the extra tuning.
  3. The analysis and ablations only partially support the actual design of GITA.

    • The theory (Prop. 1 and its corollaries) uses an idealised deterministic shortest-path value with a continuous relaxation of ⌈d/k⌉. The APC experiment that confirms it trains independent value functions V_k per k.
    • GITA instead uses a single shared network trained with OTA-style targets on suboptimal offline data. The paper doesn't characterise what V(s, g; k) converges to in that setting, or whether the per-k orderings survive parameter sharing.
    • There is no formal argument that "exponentiate then average" approximates selecting k*_opt; Figure 4 shows this only qualitatively.
    • Two of the four ablations are not significantly different from GITA overall: Mean Advantage Before Exponentiation (p_adj = 0.87) and GITA with Geometric k (p_adj = 0.37). So the evidence that the specific aggregation order and the fixed Fibonacci set K matter rests mainly on single-task collapses (AntMaze explore-large and stitch-giant).
Questions for authors

Questions

  1. Why is the candidate set K chosen as Fibonacci numbers ({3, 5, 8, 13, 21})? Appendix E justifies only the range of K: the smallest value lies below OTA's smallest task-specific k and the largest exceeds OTA's largest. It gives no reason for the Fibonacci spacing itself. Is there a principled motivation, for example roughly logarithmic coverage of scales, given that k enters the network as log k? How sensitive is GITA to this choice? Results with other spacings and sizes of K would clarify whether the specific set matters:

    • powers of two, e.g. {2, 4, 8, 16, 32};
    • uniform spacing, e.g. {4, 8, 12, 16, 20};
    • a smaller or larger |K|.

    The "GITA with Geometric k" ablation is not significantly different from GITA overall (p_adj = 0.37), which suggests the exact set may not be important. If so, the paper should say this explicitly rather than present the Fibonacci set as a design choice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.