- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors show that learning the Q function alongside QCS can increase the benchmark performance by a great extent.
Critical strengths
- The proposed algorithm demonstrates remarkable performance on multiple D4RL benchmarks.
- The algorithm is simple and easy to understand.
Critical weaknesses
- The proposed algorithm is quite incremental in the sense that it's basically QCS + critic learning, and learning the critic with an actor regulariser is a widely used technique in the field of Offline RL.
- Justification for the critic learning algorithm is not provided. What's the rationale behind equation (3)? There are multiple ways to implement TD-learning. For example, one may use an n-step TD loss with a fixed n(1, 2, 3, ). Instead of restricting the computation of the n-step return inside the sub-trajectory , one may also use samples that come after time-step . Ablation experiments comparing the performance of these methods is missing.
- Experiments on more realistic datasets such as OGBench(Park et al., 2025) would make the empirical results more impactful.
References
Park, S., Frans, K., Eysenbach, B., & Levine, S. (2025, May). Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (Vol. 2025, pp. 94937-94982).
Questions for authors
- Why is equation (1) so small?
- n-step returns are inherently biased. Unlike the online setting where the bias can be reduced through additional sampling, this becomes a huge problem in the offline setting. Theoretical analysis of the n-step bias and its effect on the overall algorithm would be very interesting.