- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Short-to-Long (S2L), a framework that transfers features from short-horizon value networks to a long-horizon behavior learner through detached connections. The method uses multiple discount factors while keeping the final control objective unchanged. The authors provide theoretical and numerical analyses of bootstrap target sensitivity and gradient isolation. Experiments on Atari, online continuous control, and offline MuJoCo tasks demonstrate performance improvements across different reinforcement learning settings.
Critical strengths
-
The paper addresses an important challenge in reinforcement learning, where long-horizon value targets can introduce instability during representation learning. The motivation is clear and supported by a detailed analysis of discount factors and bootstrap target sensitivity.
-
The paper evaluates S2L across multiple reinforcement learning settings and includes useful ablation studies. The target-drift measurements and gradient isolation analysis provide valuable insights into how discount factors affect value learning.
Critical weaknesses
-
The proposed method mainly combines multiple discount factors, independent value networks, and stop-gradient feature transfer. Similar ideas have been explored in multi-horizon value learning and auxiliary representation learning, particularly in [1] and [2]. Although the paper discusses these studies, the fundamental advantage of the proposed architecture over existing approaches is not sufficiently clear. The contribution appears to be primarily empirical rather than a substantial methodological advance. Reference: [1] William Fedus, Carles Gelada, Yoshua Bengio, Marc G. Bellemare, and Hugo Larochelle. 2019. Hyperbolic Discounting and Learning over Multiple Horizons. arXiv preprint arXiv:1902.06865. https://arxiv.org/abs/1902.06865 [2] Joshua Romoff, Peter Henderson, Ahmed Touati, Emma Brunskill, Joelle Pineau, and Yann Ollivier. 2019. Separating Value Functions Across Time-Scales. In Proceedings of the 36th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 97, pp. 5468–5477. https://proceedings.mlr.press/v97/romoff19a.html
-
Although the experiments cover multiple environments, the paper provides limited insight into why different tasks benefit from different discount horizons. For example, the continuous-control improvements emerge mainly in later training stages, while some Atari and offline RL tasks show negative results. A more systematic interpretation of these differences would strengthen the paper and clarify when S2L is most useful.
Questions for authors
-
Beyond using separate networks and detached feature transfer, what is the main conceptual contribution of S2L compared with previous multi-discount value learning methods?
-
Can the authors use the existing experimental results to explain which types of environments benefit most from short-horizon features and which depend more strongly on long-horizon value learning?
-
Can the authors explain why S2L performs worse during early training in continuous-control tasks and why some Atari and offline RL environments show negative gains? Do these observations indicate limitations of the proposed feature transfer mechanism?
3 comments
Comment on rebuttal
I thank the authors for their rebuttal and agree the other reviewers’ concerns are valid. While the idea is interesting, lingering questions on novelty and presentation lead me to change my score to Weak Reject.
Thank you for considering our rebuttal. Could you specify which questions about novelty remain unresolved after our response? It would help to identify which aspect of S2L you consider already covered by prior work and why our explanation does not establish a meaningful distinction. We would appreciate the opportunity to address these specific concerns before the discussion closes.
The central idea of S2L is that representation learning and decision-making need not use the same temporal horizon. Long-horizon decisions require accounting for distant rewards, but learning useful features from long-horizon value targets can be difficult because those targets change during training. Shorter-horizon targets are less sensitive to distant bootstrap changes and can therefore provide more stable supervision for feature learning. S2L asks whether a long-horizon agent can benefit from features learned under this shorter-horizon supervision without shortening its decision horizon.
Fedus et al. use multiple horizons to jointly train a shared representation. That representation remains subject to long-horizon supervision, so their approach does not separate the horizon used to learn the source features from the horizon required for decision-making. This separation is the conceptual contribution of S2L; separate networks and detached transfer are how we implement it.
Romoff et al. instead decompose a long-horizon value function into components across time-scales. Their focus is how to estimate the long-horizon value through that decomposition. Our focus is which supervision should shape the features used to learn that value.
Could you clarify which aspect of this conceptual distinction you consider already covered by prior work?
Thank you for recognizing the motivation, experimental breadth, and analysis. We address the conceptual distinction and the task-dependent results below.
In Fedus et al. (https://arxiv.org/abs/1902.06865), multiple discount-specific heads train a shared representation. Their auxiliary-task variant also preserves the behavior discount; S2L’s distinction is that the transferred representations remain trained exclusively by their respective source losses. In Romoff et al. (https://proceedings.mlr.press/v97/romoff19a.html), a long-horizon value is decomposed into a short-horizon value and learned differences across discounts. S2L instead transfers hidden features, while the final learner directly learns its own long-horizon value without summing source predictions.
The analysis explains why this distinction matters: smaller discounts attenuate distant bootstrap perturbations, while the local shared-feature model shows how a long-horizon loss can restrict the stable step-size range of shared-feature updates. The experiments test the corresponding design choices. Removing transfer, setting all discounts to .99, or removing gradient isolation reduces full-training AUC by .0797, .0652, and .0731, respectively (Table 4). S2L also outperforms the shared-trunk MDH-4 baseline and its parameter-matched widened version (Section 5.4). These controls provide direct evidence for the proposed organization of representation learning beyond simply training with multiple discounts.
First, the five-game source-horizon study selects effective horizons of 8 for Qbert and Asterix, 10 for Phoenix, 32 for Seaquest, and 64 for Breakout (Appendix E). Thus, even within Atari, the most useful source horizon varies substantially. These are preferred source horizons in a limited development sweep; every behavior learner still uses γ=.99.
Second, offline gains depend on the available data. The mean AUC improvements are +0.45, +3.58, and +6.14 for simple, medium, and expert datasets, respectively (Table 6). HalfCheetah changes from −1.63 on simple data to +17.78 on expert data. This shows that the same environment can benefit very differently depending on its dataset. The results therefore support dependence on both supervision horizon and data conditions, rather than a universal division into environments that “need short” or “need long” horizons.
The negative Atari and offline results identify a limitation of fixed feature transfer: reducing target sensitivity does not guarantee that the resulting features are useful for every task. Moreover, gradient isolation protects the source objectives but prevents the behavior loss from adapting those sources to its particular needs. The final learner retains its own trainable representation and long-horizon objective, but the added features can still be unhelpful. We will make this trade-off and the observed dependence on training stage, source horizon, and dataset more explicit in the discussion.