- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses offline to online RL with a pre-trained value function and learning a reward function on the fly. The main proposal is the reward gauge which aligns the online reward with the offline reward basis which can be extracted from the pre-trained value function via the inverse Bellman operator. The paper further addresses active data selection by querying for the most informative and anwerable data pairs. They estimate information gain using Laplace approximation. Experiment are conducted in D4RL environments showing that the proposed method approaches WSRL with known online reward, significantly outperform no-gauge, and benefit from active selection.
Critical strengths
- The paper is well written. The proposed method is sound and easy to understand.
- The paper situated itself among prior work WSRL and PEBBLE very well, ran very reasonable ablations, and demonstrated expected effects.
Critical weaknesses
- While reward gauge and the offline to online setting are clear contributions from this paper, I find the active selection an unclear contribution. There already exist a handful of work applying active selection which are also based on approximate Bayesian inference, e.g., [1, 2].
[1]. Poddar, S., Wan, Y., Ivison, H., Gupta, A., & Jaques, N. (2024). Personalizing reinforcement learning from human feedback with variational preference learning. [2]. Biyik, E., & Sadigh, D. (2018, October). Batch active preference-based learning of reward functions.
Questions for authors
NA
2 comments
Thank the authors for the response. I have no other questions.
Rebuttal to Reviewer mPZf
The reviewer is right that active query selection with approximate Bayesian inference is already established in preference learning, and we do not claim it as new. Poddar et al. (2024) choose queries to identify a user's latent preference, and Biyik and Sadigh (2018) choose a diverse batch of queries at once for a reward that is linear in features. In both, every query receives a label. Our teacher may abstain, so we weight each candidate pair by a calibrated estimate of the probability that the teacher answers it, computed from the return gap implied by the frozen critic. Starting from a pretrained checkpoint is what makes this estimate available. We also score a pair by how much its label reduces the posterior variance of the reward over the whole pool. We will cite both papers and state these differences in Section 1.1.