acceptodds
Under review as a conference paper at ICLR 2027

SHORT-TO-LONG: Detached Features for Long-horizon Value Learning

Abstract

Learning useful representations from continually changing value targets remains a challenge in deep reinforcement learning. We study the temporal horizon of this supervision: smaller discounts make these targets move less by attenuating distant bootstrap changes, but using them for behavior changes the control objective. We introduce Short-to-Long (S2L), which supplies a long-horizon behavior learner with features trained by smaller-discount source networks. Detached connections prevent the behavior value loss from updating the sources, allowing each source to follow its own value objective while the behavior discount remains unchanged. Across Atari, online continuous control, and offline continuous control, the approach yields aggregate performance gains, improving the paired terminal median on 40 of 57 Atari games. Matched Atari ablations support the roles of feature transfer, smaller source discounts, and gradient isolation. Fixed-trajectory measurements further show reduced drift in source value targets with mixed discounts.

Primary area:
reinforcement learning
Keywords:
Reinfrocment Learning, Q Learning, Learning Stability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YvhNOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Short-to-Long (S2L), a framework that transfers features from short-horizon value networks to a long-horizon behavior learner through detached connections. The method uses multiple discount factors while keeping the final control objective unchanged. The authors provide theoretical and numerical analyses of bootstrap target sensitivity and gradient isolation. Experiments on Atari, online continuous control, and offline MuJoCo tasks demonstrate performance improvements across different reinforcement learning settings.

Critical strengths
  1. The paper addresses an important challenge in reinforcement learning, where long-horizon value targets can introduce instability during representation learning. The motivation is clear and supported by a detailed analysis of discount factors and bootstrap target sensitivity.

  2. The paper evaluates S2L across multiple reinforcement learning settings and includes useful ablation studies. The target-drift measurements and gradient isolation analysis provide valuable insights into how discount factors affect value learning.

Critical weaknesses
  1. The proposed method mainly combines multiple discount factors, independent value networks, and stop-gradient feature transfer. Similar ideas have been explored in multi-horizon value learning and auxiliary representation learning, particularly in [1] and [2]. Although the paper discusses these studies, the fundamental advantage of the proposed architecture over existing approaches is not sufficiently clear. The contribution appears to be primarily empirical rather than a substantial methodological advance. Reference: [1] William Fedus, Carles Gelada, Yoshua Bengio, Marc G. Bellemare, and Hugo Larochelle. 2019. Hyperbolic Discounting and Learning over Multiple Horizons. arXiv preprint arXiv:1902.06865. https://arxiv.org/abs/1902.06865 [2] Joshua Romoff, Peter Henderson, Ahmed Touati, Emma Brunskill, Joelle Pineau, and Yann Ollivier. 2019. Separating Value Functions Across Time-Scales. In Proceedings of the 36th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 97, pp. 5468–5477. https://proceedings.mlr.press/v97/romoff19a.html

  2. Although the experiments cover multiple environments, the paper provides limited insight into why different tasks benefit from different discount horizons. For example, the continuous-control improvements emerge mainly in later training stages, while some Atari and offline RL tasks show negative results. A more systematic interpretation of these differences would strengthen the paper and clarify when S2L is most useful.

Questions for authors
  1. Beyond using separate networks and detached feature transfer, what is the main conceptual contribution of S2L compared with previous multi-discount value learning methods?

  2. Can the authors use the existing experimental results to explain which types of environments benefit most from short-horizon features and which depend more strongly on long-horizon value learning?

  3. Can the authors explain why S2L performs worse during early training in continuous-control tasks and why some Atari and offline RL environments show negative gains? Do these observations indicate limitations of the proposed feature transfer mechanism?

3 comments

Thank you for recognizing the motivation, experimental breadth, and analysis. We address the conceptual distinction and the task-dependent results below.

  1. Conceptual contribution relative to multi-discount learning. S2L uses shorter-horizon value objectives to train persistent feature sources that the long-horizon learner can use without changing how those sources are trained. Gradient isolation makes this separation effective: the behavior loss cannot redirect the source representations toward its own longer-horizon target.

In Fedus et al. (https://arxiv.org/abs/1902.06865), multiple discount-specific heads train a shared representation. Their auxiliary-task variant also preserves the behavior discount; S2L’s distinction is that the transferred representations remain trained exclusively by their respective source losses. In Romoff et al. (https://proceedings.mlr.press/v97/romoff19a.html), a long-horizon value is decomposed into a short-horizon value and learned differences across discounts. S2L instead transfers hidden features, while the final learner directly learns its own long-horizon value without summing source predictions.

The analysis explains why this distinction matters: smaller discounts attenuate distant bootstrap perturbations, while the local shared-feature model shows how a long-horizon loss can restrict the stable step-size range of shared-feature updates. The experiments test the corresponding design choices. Removing transfer, setting all discounts to .99, or removing gradient isolation reduces full-training AUC by .0797, .0652, and .0731, respectively (Table 4). S2L also outperforms the shared-trunk MDH-4 baseline and its parameter-matched widened version (Section 5.4). These controls provide direct evidence for the proposed organization of representation learning beyond simply training with multiple discounts.

  1. Which environments benefit from shorter versus longer horizons? The existing results reveal two useful patterns.

First, the five-game source-horizon study selects effective horizons of 8 for Qbert and Asterix, 10 for Phoenix, 32 for Seaquest, and 64 for Breakout (Appendix E). Thus, even within Atari, the most useful source horizon varies substantially. These are preferred source horizons in a limited development sweep; every behavior learner still uses γ=.99.

Second, offline gains depend on the available data. The mean AUC improvements are +0.45, +3.58, and +6.14 for simple, medium, and expert datasets, respectively (Table 6). HalfCheetah changes from −1.63 on simple data to +17.78 on expert data. This shows that the same environment can benefit very differently depending on its dataset. The results therefore support dependence on both supervision horizon and data conditions, rather than a universal division into environments that “need short” or “need long” horizons.

  1. Early-training disadvantages and negative results. In continuous control, the mean difference changes from −12.71 in the first half to +27.03 in the second half, reaching +39.09 at the endpoint. A plausible explanation is that source features initially provide little useful information and become beneficial as their value predictors learn. The frozen-source ablation supports the importance of learned source features, although it does not by itself establish the cause of the early disadvantage.

The negative Atari and offline results identify a limitation of fixed feature transfer: reducing target sensitivity does not guarantee that the resulting features are useful for every task. Moreover, gradient isolation protects the source objectives but prevents the behavior loss from adapting those sources to its particular needs. The final learner retains its own trainable representation and long-horizon objective, but the added features can still be unhelpful. We will make this trade-off and the observed dependence on training stage, source horizon, and dataset more explicit in the discussion.

Reviewer_YvhNOpenReview ↗

Comment on rebuttal

I thank the authors for their rebuttal and agree the other reviewers’ concerns are valid. While the idea is interesting, lingering questions on novelty and presentation lead me to change my score to Weak Reject.

Thank you for considering our rebuttal. Could you specify which questions about novelty remain unresolved after our response? It would help to identify which aspect of S2L you consider already covered by prior work and why our explanation does not establish a meaningful distinction. We would appreciate the opportunity to address these specific concerns before the discussion closes.

The central idea of S2L is that representation learning and decision-making need not use the same temporal horizon. Long-horizon decisions require accounting for distant rewards, but learning useful features from long-horizon value targets can be difficult because those targets change during training. Shorter-horizon targets are less sensitive to distant bootstrap changes and can therefore provide more stable supervision for feature learning. S2L asks whether a long-horizon agent can benefit from features learned under this shorter-horizon supervision without shortening its decision horizon.

Fedus et al. use multiple horizons to jointly train a shared representation. That representation remains subject to long-horizon supervision, so their approach does not separate the horizon used to learn the source features from the horizon required for decision-making. This separation is the conceptual contribution of S2L; separate networks and detached transfer are how we implement it.

Romoff et al. instead decompose a long-horizon value function into components across time-scales. Their focus is how to estimate the long-horizon value through that decomposition. Our focus is which supervision should shape the features used to learn that value.

Could you clarify which aspect of this conceptual distinction you consider already covered by prior work?

Reviewer_brKAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors tried to motivate the need for transferring features from differently trained value networks based on different discount factors. Discount factors affect the optimization differently, and relying on only one value is claimed by the authors to be less effective. This work introduces S2L, where multiple source networks are trained separately with different discount factors while passing features from short horizon (low discount factor) to longer horizon (higher discount factor) while stopping any gradient flow. Doing so shows performance improvement on Atari games, continuous control, as well as offline RL.

Critical strengths
  • Overall, the empirical results show a promising trend across all experiments.
  • The idea of combining features from differently trained value networks with different discount factors is interesting, yet the motivation is not clear to me due to the writing of the paper.
Critical weaknesses
  • The paper is very hard to follow. The writing seems very AI-generated to me, without careful adjustment for the human eye.
  • Not background/ preliminaries section is included in this paper, which makes reading even harder. The paper jumps directly to the methodology and even some technicalities regarding the evaluated problem settings, for example, hyperparameters used.
  • Quantities are not defined in place; for instance, in Section 2.2, ll was not defined in the text, and I needed to understand the quantity by checking the diagram in Figure 1.
  • Results are shown in an unclear and less convincing way, for example, in Figure 2. Why show only the second half of the training?
  • The gain in performance for S2L could be due to the use of many networks in comparison, for instance, to default scaled PQN or Simba. Or do they match in parameter count?
  • In the case of scaled PQN, I believe there might be an issue in the formulation. If the different source networks were trained using the same data, technically the driven policies (from source networks and their Q-functions) are different, and hence the transitions should not be shared since it will no longer be on-policy. If the method does take care of that, then my argument is invalid.

Overall, I believe my assessment has been heavily affected by the writing of this paper, not by my knowledge of the literature. The paper is unclear in many parts, and the motivation is not crystal clear.

Questions for authors
  • What are the 720 matched pairs exactly in Table 1?
  • Why only 4 source networks, and what are the values of the discount factors?
  • There is inconsistency in the aggregation method used for reporting the results; some results use the mean, others use IQM. What is the reason for that?
  • Why not pass features at different block levels?
Reviewer_S4Z5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses value learning in deep reinforcement learning. The proposed method, Short-to-Long, trains separate value networks with different discount factors and transfers intermediate features from shorter-horizon to longer-horizon networks through stop-gradient connections. This design aims to stabilize representation learning while retaining the original long-horizon control objective. Experiments show aggregate performance improvements in online and offline reinforcement learning, and ablations examine the contributions of feature transfer, discount choices, and gradient isolation.

Critical strengths
  • Experimental breadth: The evaluation covers Atari, online continuous control, and offline RL, with ablations examining the main design choices.
  • Simple approach: The method offers a conceptually simple approach to representation learning under non-stationary value targets.
Critical weaknesses

Clarity and presentation: I found the manuscript exceptionally difficult to follow. The problem setting and intended objective are not clearly established before the method is introduced. Although equations are provided, important concepts and notation are insufficiently explained, and the method and baseline descriptions do not form a clear, self-contained account. The organization compounds these difficulties. Figure 1 also needs substantial improvement: visually inconsistent elements and apparent overlays make the presentation difficult to interpret. Taken together, these issues made understanding the method and assessing its implementation unnecessarily difficult. Addressing them would require substantial rewriting and restructuring, and I do not support acceptance in its current form.

Questions for authors

I have no additional questions. My concerns relate primarily to clarity, organization, and presentation and require substantial manuscript revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.