- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduced a reward calibration method for offline reinforcement learning. The idea is to use compute a coverage-aware reward signal derived from offline episode returns which is adjusted over training helping algorithms to overcome detrimental Bellman updates initially.
Critical strengths
- From a mathematical perspective, the method seems to be transparently laid out and it is compatible, in principle, with any offline RL algorithm.
- First promising indicators on standard offline RL datasets when compared with other offline RL methods are given
Critical weaknesses
- I think the major weakness of the paper is the type of experiments conducted. To me, it sees that work here is half done. The paper is about a reward calibration method that should be agnostic to the base learner. Nevertheless, in the experiments, it is only run in combination with TD3-BC and only compared with vanilla offline RL algorithms that do not use any calibration. Why not combine it with CQL or IQL as well? I think there have to be experiments showing the effect of the method on more algorithms. Moreover, it must be benchmarked against other reward calibration methods. In its current form, the experiments in 5.1 are, at best, very early signals that something has some promising direction than a robust evaluation. Even with these promising signals, the experiments currently do not enough to prove that the proposed method has the right formulation carying the success. Structural ablation studies on the key design choices are missing For instance: How does data quality, data size, or episode length affect the method? How do hyperparameters of the offline RL algorithms affect performance? Can the proposed calibration compensate bad hyperparameters of the vanilla algorithm?
- The paper presents a very special construction of a reward calibration. Beside the fact I find central parts of the construction hard to understand, the construction comes largely without motivation, intuition, or theoretic gurantees why it is supposed to works. The authors do not enough to communicate the larger picture or give inuition why the design of their method is how it is. To name one example: Why are the weights computed using the very special form of given in Equation 5?
Questions for authors
See my questions in the weakness section.