- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Selective Shortcut Value Learning (SSVL) for long-horizon offline goal-conditioned reinforcement learning. The method introduces single-discount shortcut targets into primitive action-value learning, uses a frozen reference value function to selectively compress segment rewards, and adds a Shortcut Value Consistency (SVC) loss to regularize state-value learning. Experiments on state-based and visual OGBench tasks show competitive results against several existing methods. However, the methodological novelty is somewhat limited, and the theoretical analysis and experimental evidence do not fully establish the necessity and general effectiveness of the proposed components.
Critical strengths
- The paper addresses a relevant problem in long-horizon offline GCRL, where sparse rewards make value propagation and primitive-action selection difficult.
- SSVL retains a single primitive goal-conditioned policy at inference without requiring hierarchical planning or subgoal prediction.
- The method achieves promising results on several OGBench tasks, particularly state-based manipulation and visual AntMaze navigation. The paper also includes useful ablations of its main components.
Critical weaknesses
- The methodological novelty relative to existing temporal abstraction methods is limited. The main idea of replacing the conventional multi-step discount factor \gamma^kwith a single discount factor \gamma is closely related to the temporal abstraction mechanism used in OTA (Ahn et al., 2025). The main distinction appears to be applying shortcut supervision to primitive action-value learning rather than the high-level state-value learning used by OTA. Although this change introduces a different credit-assignment setting, the underlying shortcut propagation mechanism remains largely the same. The authors should clarify what new learning principle is introduced beyond transferring existing temporal abstraction to the primitive-action critic, and whether the performance gains can be attributed to this modification alone.
- The theoretical results rely on restrictive assumptions and provide limited guarantees for the proposed algorithm. Proposition 3.1 establishes improved normalized value separation on an idealized fixed-length chain, but this setting does not capture the stochastic transitions and heterogeneous trajectory continuations encountered in offline RL. More importantly, Theorem 3.4 relies on Assumption 3.3, which requires the logged continuation to preserve a positive preference margin between primitive actions. This is precisely the property that may fail in long-horizon offline datasets. The theorem therefore provides a conditional target-ordering guarantee rather than demonstrating that SSVL can generally recover correct primitive-action preferences. The authors should discuss how frequently these assumptions are satisfied in the evaluated datasets and provide empirical evidence connecting the theoretical conditions to actual policy performance.
- The proposed selective reward compression has limited independent contribution in the reported ablations. In Figure 2(a), replacing exact k-step propagation with single-discount continuation already improves success from 41.5 to 86.1. The matched Flat OTA-style variant achieves 88.3, while selective SSVL without SVC reaches 91.2. These results indicate that most of the improvement comes from the existing single-discount shortcut mechanism, whereas the additional benefit of selective reward compression is relatively modest in this experiment. Although Figure 2(b) shows benefits from a shortcut-aligned reference on another task, it does not fully establish the general necessity of selective compression. The authors should provide broader comparisons between uniform shortcut targets and selective targets to clarify the independent contribution of the proposed selection mechanism.
- The consistency objective is not fully consistent with the selective critic target. Eq. 30 imposes the unit-reward shortcut relation (V\left(s_t,g\right)=-1+\gamma V\left(s_t^+,g\right))on all sampled nonterminal shortcut edges. However, Eqs. 27–28 use different reward targets depending on whether a segment is accepted or rejected by the reference gate. For rejected segments, the critic retains the accumulated segment reward, while SVC still enforces the unit-reward relation. This creates a mismatch between the supervision applied to the critic and the state-value function. The authors acknowledge that SVC is auxiliary, but do not sufficiently analyze whether these potentially conflicting objectives introduce bias or instability. A comparison with a consistency loss constructed from the same selective targets would help justify the current design.
- The experimental results are inconsistent across tasks, and the limitations of shortcut learning are not sufficiently investigated. Table 1 shows substantial performance differences across environments. For example, SSVL achieves only 23 on AntMaze large-stitch compared with OTA's 84, and 24 on Visual Cube double-noisy compared with OTA's 65. Similarly, SSVL obtains 67 on HumanoidMaze giant-navigate, substantially below OTA's 92. These failures are particularly relevant because the method targets long-horizon goal-conditioned learning. Although the authors discuss possible limitations in primitive-action supervision, the explanation remains speculative. The paper should provide more direct analysis of whether these failures arise from incorrect shortcut action ordering, unreliable logged continuations, or the selective reference mechanism.
Questions for authors
The paper presents a practical extension of shortcut-based value learning to primitive-action policies and reports promising results on several challenging tasks. However, the central propagation mechanism is closely related to existing temporal abstraction methods, particularly OTA, and the additional contributions from selective reward compression and value consistency are not consistently established. The theoretical analysis provides conditional action-ordering results but does not directly guarantee improved policy learning under realistic offline data distributions. Given the limited methodological novelty and the substantial performance degradation on several long-horizon tasks, the current evidence is not sufficient to fully support the paper's main contribution claims.
Thank you for the clarifications and additional experiments, which substantially address my main concerns. The representation controls make the experimental interpretation more credible, but also narrow the evidence attributable to shortcut learning, since much of the improvement at K=1 comes from representation choices. The new cross-task ablations and action-ordering analyses strengthen the remaining evidence. Overall, I will maintain my score.