acceptodds
Withdrawn from ICLR 2027

Deep Q-Learning Without a Target Network

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A target network is the standard device for stabilizing deep Q-learning, at the price of storing a full copy of the online network and tuning its refresh period. We ask what a target actually needs to store and answer by lowering the storage level from a full copy, through partial copies, to none. A first-order analysis shows that a target decouples only the parameter directions it stores, and in deep networks negative feedback from a frozen copy is what keeps partial targets alive. Removing storage entirely exposes spurious solutions at which the loss vanishes while the values are wrong. A bounded multiplicative output scaling, which scales the network's own prediction by a factor slightly above one before regressing it onto the bootstrap, repairs this with a value-error bound. We call the resulting method Tempered Q-learning. In a stress test where the full copy diverges, it survives while storing nothing, matches or exceeds the copy's return once the copy's period is matched in environment time, and on ALE Breakout reaches the full copy's mean return at a fraction of the spread. The replacement is two lines of code, with no second network and no synchronization period to tune.

Primary area:
reinforcement learning
Keywords:
deep reinforcement learning, deep Q-learning, target networks, temporal difference learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Mor7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers an interesting case whether DQN needs to maintain a target network. Throughout the paper, the authors present a set of theoretical analysis with counterexamples. Based on the bounded multiplicative output scaling results, they propose Tempered Q-learning and validate the method on several RL benchmarks. However, I find the theoretical part does not match well with the proposed method. Moreover, the paper is heavily written with semicommas, colons, and highly compressed sentences, which makes it difficult to follow and raises concerns about whether the presentation has been sufficiently proofread.

Critical strengths

The paper presents a detailed experimental reasoning process, including several carefully designed counterexamples that help understanding the proposed stability arguments.

Critical weaknesses
  1. The theoretical part is unnecessarily presented and loosely connected to the proposed method. Prop. 1-3 could be reduced to the synchronization-map analysis only. Prop. 4 analyzes only the local between-synchronization dynamics, while omitting the synchronization map that the paper itself shows can be essential for trajectory-level divergence. This makes its connection to the claimed global stability and failure behavior incomplete. Furthermore, the final method can be derived from the bounded scaling design and its empirical behaviour, the Prop. 8 and 10 appear dispensable.
  2. The resulting metho relies on explicitly constraining the online Q-values used for bootstrapping. In this sense, its underlying role appears closely related to that of a target network: both aim to suppress instability in the bootstrap target, either by using a more stable delayed target or by directly constraining the online Q estimate. This makes the conceptual distinction somewhat less clear than the paper suggests. Moreover, despite the extensive theoretical analysis and diagnostic experiments, the empirical results do not show a clear or consistent performance advantage among baselines. The main practical takeaway may therefore be a different way to stabilize Q-updates. Finally, the experimental evaluation covers too few tasks to convincingly establish the effectiveness and generality of the proposed method.
Questions for authors

No further questions. See in the weaknesses.

Reviewer_B9GhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the role of target networks in Q-learning across full-target, partial-target, and target-free settings. It uses local linearization and dynamical-systems analysis to characterize their update dynamics and proposes a negative-feedback explanation for the stabilization provided by partial targets. It introduces Tempered Q-learning, which applies bounded multiplicative scaling to online predictions while retaining unscaled values for bootstrapping. Although this changes the Bellman equation and the original fixed point. The empirical evaluation includes synthetic examples, CartPole, few MinAtar games, and ALE Breakout. The experiments vary target storage, loss function, update-to-data ratio, and synchronization period, and compare several stabilizing components. The reported results demonstrate substantial sensitivity to these choices and show that scaling can improve stability and achieve competitive returns in the tested settings. For constant scaling, the authors additionally reproduce its behavior using target-free TD with a reduced discount and rescaled rewards.

Critical strengths

None.

Understanding the role of target networks in optimization problems is an important research question with potential relevance beyond Q-learning, including drifting models and LLM training. However, I do not find the theoretical and empirical evidence in the current submission sufficiently convincing to identify a substantive strength that supports acceptance.

Critical weaknesses

1. The theoretical analysis does not sufficiently support its explanation of nonlinear training dynamics.

Section 4 analyzes a linearized model at synchronization points that are generally not equilibria or around stationary point. For nonlinear Q-networks, the exact update Jacobian also contains residual terms, whose omission requires justification beyond linearizing the network outputs. Moreover, an expansive direction of a frozen Jacobian describes local perturbation behavior; it does not by itself establish sustained expansion or divergence of the nonlinear training trajectory. The projected reduction further relies on a fast-adapter assumption that is not adequately validated in the main deep-learning experiments. Without controlling the approximation error and the evolution of the dynamics across updates and synchronizations, the analysis provides conditional mechanistic intuition rather than an established explanation of training stability. This gap is consequential because explaining the stabilizing role of partial targets is a central claimed contribution.

***2. The incremental benefit of adaptive scaling beyond discount reduction is not established. ***

The proposed Tempered Q-learning reweights the reward and discount factor in its Bellman self-consistency equation. Reduced discounts have established regularization effects in TD learning (Amit et al., 2020), and analyses of inexact fixed-target updates explicitly characterize how the discount affects the overall contraction of Q learning with fixed targets (Liu et al., 2026). The paper’s own reduced-discount control reproduces the constant-scaling variant, leaving the additional contribution of learned adaptive scaling unclear. Related work on reward centering also shows that value magnitude can materially affect learning at large discounts (Naik et al., 2024). A systematic comparison against appropriately tuned discount and reward-scale baselines is therefore needed to distinguish gains from adaptivity, shorter horizons, and altered value scales. Without this separation, the experiments do not convincingly establish a distinct algorithmic advantage for the proposed learned scaling.

Ron Amit, Ron Meir, and Kamil Ciosek. “Discount Factor as a Regularizer in Reinforcement Learning.” ICML 2020.

Liu, Z., Okolo, N., D’Orazio, R., Vucetic, D., Mitliagkas, I., & Gidel, G. A Surrogate Perspective on Convergence of Fixed-Target DQN. NeurIPS 2026.

Naik, Wan, Tomar, and Sutton. Reward Centering. RLC 2024.

Questions for authors

Applicability of the dynamical analysis.

What notion of stability or expansion is established by the linearization at non-equilibrium points? Under what conditions do these local conclusions extend to the nonlinear training trajectory across successive target updates? In particular, how are the residual-weighted curvature terms and the fast-adapter assumption controlled or empirically validated in the deep-network experiments?

Benefit beyond discount reduction.

Given that the reduced-discount control reproduces the constant-scaling variant, what additional benefit can be attributed specifically to learned adaptive scaling? Does this benefit persist against appropriately tuned fixed-discount and constant-scaling baselines?

Correctness of the learned values.

For a learned nonlinear hh, the self-consistency condition becomes h(Q)=BγQh(Q)=\mathcal B_\gamma Q, rather than the original Bellman equation. Beyond the conditional bounds for zero-loss solutions, what can be established about the existence and attainability of these solutions? How should the resulting predictions be interpreted relative to the original γ\gamma-discounted value function, and how is their accuracy evaluated?

Interpretation of the experimental comparisons.

How can the effects of target storage be separated from synchronization timing and finite optimization budgets? Could the authors report sensitivity to the synchronization period and within-period fitting error under matched data and gradient-update budgets? Since the return and value-bias results use different loss protocols, do the conclusions about stability and calibration hold simultaneously under a common protocol?

Contribution relative to prior partial-target methods.

Which theoretical insight or experimentally validated prediction goes beyond Vincent et al.’s partial-target results and empirical mechanism analyses? How should the component comparisons in this paper be interpreted relative to the performance of the complete prior methods with their original training configurations?

Practical resource benefit.

In the evaluated settings, how much does removing the target copy reduce measured peak training memory or improve throughput, feasible batch size, or model size? What is the incremental benefit over a frozen-last-layer baseline, which already stores only a small fraction of the full network?

T. Vincent, Y. Tripathi, T. Faust, A. Akg¨ul, Y. Oren, M. Kandemir, J. Peters, C. D’Eramo. Bridging the performance-gap between target-free and target-based reinforcement learning. ICLR, 2026

Reviewer_XKXGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to remove the so called target network from DQN. When using the target network, the agent deliberately uses a stale copy of the online network so as to improve stability. The paper argues that, similar to previous works findings, it is an overkill to maintain a second set of weights just for this purpose alone, and that there exist easier ways to achieve this purpose, such as by using a simple scaling factor.

Critical strengths
  • The paper does show that target network is an overkill, and that there are easier ways of taming DQN-style algorithms.
  • The paper attempts to go to fundamentals and discover the role of stale update, by for instance focusing on the well-known counterexamples of TD learning. I appreciate this.
Critical weaknesses
  • To start with, I do not like the presentation style of the paper. As someone with moderate expertise in the domain, I had a hard time understanding the paper in the early sections. The abstract, for instance gets really weird from the sentence starting with "A first-order analysis shows ..." onwards. At this point, the user does not know what negative feedback, parameter direction, spurious solutions or partial targets are. Using these non-standard terminologies only makes sense once they are properly defined. Only say things that are well defined as a general rule. Similarly in the introduction, what does "survival" mean in the first paragraph. It's so easy to lose the authors reading the abstract and intro.

  • Section 3 is really confusing to me. It is actually not true that target networks prevent TD from divergence in the cited counter example. All that target network does is to delay the explosion of the weights, which it is unclear why this is considered a win to me. Also I could not reproduce convergence of scale and shift approach. Basically, none of these approaches actually result in convergence necessarily. If the scaling is less than a threshold, then I agree there will be convergence, but then a) this is nothing but using a smaller gamma, which of course is known to help with convergence, simply because even in the on policy case the Bellman operator is a gamma contraction, and so in other words this insight is not new.

  • As for the empirical results, again, tempered TD is just TD with a smaller discount factor (technically reward is also multiplied by the inverse of the factor, which is unclear if this is a good idea). The learned approach brings some novelty in some sense, but actually judging from the final return in the table 1, this approach actually does worse than the simple TD with lower gamma approach (or the so called tempered TD), so it is hard to give much credit to it.

Questions for authors
  • Can I see the calculations for the claim that on the counter example, TD with target net actually converges? Yes, within online network updates, the coefficient is less than 1, but then if we look at successive updates after target network sync, then weights are blowing into infinity.
  • How is tempered TD different than using a smaller gamma? Is it possible to show experiments where the effectively smaller gamma is only multiplied by the target term, not the reward? Basically, the only difference is multiplication by 1/factor in the reward, so would like to understand the effect of this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.