- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper considers an interesting case whether DQN needs to maintain a target network. Throughout the paper, the authors present a set of theoretical analysis with counterexamples. Based on the bounded multiplicative output scaling results, they propose Tempered Q-learning and validate the method on several RL benchmarks. However, I find the theoretical part does not match well with the proposed method. Moreover, the paper is heavily written with semicommas, colons, and highly compressed sentences, which makes it difficult to follow and raises concerns about whether the presentation has been sufficiently proofread.
Critical strengths
The paper presents a detailed experimental reasoning process, including several carefully designed counterexamples that help understanding the proposed stability arguments.
Critical weaknesses
- The theoretical part is unnecessarily presented and loosely connected to the proposed method. Prop. 1-3 could be reduced to the synchronization-map analysis only. Prop. 4 analyzes only the local between-synchronization dynamics, while omitting the synchronization map that the paper itself shows can be essential for trajectory-level divergence. This makes its connection to the claimed global stability and failure behavior incomplete. Furthermore, the final method can be derived from the bounded scaling design and its empirical behaviour, the Prop. 8 and 10 appear dispensable.
- The resulting metho relies on explicitly constraining the online Q-values used for bootstrapping. In this sense, its underlying role appears closely related to that of a target network: both aim to suppress instability in the bootstrap target, either by using a more stable delayed target or by directly constraining the online Q estimate. This makes the conceptual distinction somewhat less clear than the paper suggests. Moreover, despite the extensive theoretical analysis and diagnostic experiments, the empirical results do not show a clear or consistent performance advantage among baselines. The main practical takeaway may therefore be a different way to stabilize Q-updates. Finally, the experimental evaluation covers too few tasks to convincingly establish the effectiveness and generality of the proposed method.
Questions for authors
No further questions. See in the weaknesses.