- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a novel method to improve the exploration strategy of the deep RL algorithms. The method is to sample in actions that have relatively low estimated Q values. The paper proves that sampling lower estimate Q values will result in larger TD errors and provide interesting mathematical statements that local maximals correlates to lower Q-values. Experiment show that by changing the exploration strategy, it yeilds higher training efficiency and asympotatic results comparing to other results.
Critical strengths
- The method is interesting and simple, but the gap is substantial empirically.
- The focus of the problem is long but overlooked field in deep RL algorithms.
Critical weaknesses
The explroation issue has been explored in many prior works like RND back in 2017/18/19. Though some of the methods are more complicated than the proposed method, it is still worth comparision with all the existing methods. the method focus on locomotin tasks only, however, it is unclear whether this method are useful in other RL tasks. the mathematcial results is not directly related to the results. why exploring large TD error placed will have better results and trainin efficiency?
Questions for authors
- explain the correlation from theory findings with the empricial results.
- compare with prior works on RL exploration.
We are truly glad to hear that you found that our method provides substantial improvements in training efficiency and higher asymptotic performance in a long standing overlooked field of deep RL algorithms, and further that our approach is interesting and simple.
The rather well known fact regarding higher temporal difference and faster learning is a foundational principle in reinforcement learning and can also be found in quite early work starting from [1], which is also cited in our paper.
[1] Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 1993.
The theoretical result of Theorem A.4 shows that sampling counteractive actions i.e. actions with values lower than the current policy on average, will lead to larger temporal difference updates. As we also discussed above, the fact that larger temporal differences lead to faster learning is a foundational principle in reinforcement learning [1]. In Section 3 through Theorem 3.6 and 3.12, we prove that counteractive actions can be sampled from the tails of the current policy distribution. In Section 4 and Section 5, we report empirical analysis results, and these results verify the theoretical analysis in Section 3 and demonstrate that CoAct-PG obtains larger temporal difference updates and indeed leads to faster learning and substantially better asymptotic performance.
We would like to kindly point out that RND [1] doubles the number of parameters that need to be trained and learned. The focus of these methods includes introducing high computational cost and sample complexity to obtain high scores for a few selected games, while our objective is to increase sample efficiency across the entire benchmark without introducing computational complexity which is perfectly aligned with very recent work [2]. Our algorithm introduces zero additional computational cost and complexity while achieving effective and efficient learning across the entire benchmark. Further note that prior work (NeurIPS 2025 Spotlight) [2] already demonstrated that Counteractive RL performs substantially better than count-based methods (UCB), epsilon greedy, NNs and CFN.
[1] Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov. Exploration by Random Network Distillation, ICLR 2019.
[2] Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learning, NeurIPS 2025. [Spotlight Presentation]