- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a method for learning from offline demonstrations for which a labelled preference dataset is also available. The method combines a diffusion policy approach to behavior cloning with a q-value regularization, where the q-value objective if modified to include an estimate of the value of a state action pair according to the preference data. The preference score is distinct from the reward function learned from the labelled preference data. The D4RL dataset is used as the source of training data, and several baselines are compared against. The results suggest the proposed method PSAC improves upon prior work.
Critical strengths
None
Critical weaknesses
- The paper does not provide sufficient evidence to justify the proposed modification to policy learning is the source of the gains. The policy architecture and behavior cloning objective is based on a diffusion policy, whereas the baselines that are compared against would have relied on a MLP. The use of a diffusion policy alone could explain a large portion of the observed performance gains. Additionally, another contributing factor could be the increase in number of parameters that are used in the PSAC method. As the overall size of the models used in the different approaches (PSAC vs. baselines) is not mentioned, this is difficult to assess. Contributing to understanding the benefits of the proposed preference score, there are no experiments examining how the preference score mitigates negative effects on policy learning from a poor reward function. The need for the preference score is motivated to address errors in the reward function. However, the preference score appears to be learned from the same data as the reward function, so it is not clear how the preference score addresses errors in the learned reward function. There should be experiments explicitly looking at noise in the reward model to show that the introduced preference score is able to mitigate different amounts of noise in the reward function. Additionally, there should be experiments looking at the extent to which the preference score and the reward are different and provide distinct information to the learning process.
- The paper does not appropriately motivate the design decisions made and how these are distinct from what has been done in prior work (Note about prior work: there are only two sentences to cover all prior work on offline RL). For example, why exactly is the preference score expected to address errors in the reward function when it is trained on the same data? What is the motivation and reason for select the given form of the preference score? How is the preference score meaningfully distinct from the reward? Can the learned reward be used in place of the preference score? What is the motivation for incorporating the Q-value as an auxiliary to the BC loss? Is the addition of the Q-value as an auxiliary loss as contribution of this work, because there isn't a citation for it? What about the prior works motivates that modification to critic learning should have performance gains?
- Much of the space of the paper is taken up with unnecessary results. For example, the complete Table 1 with results per demonstration quality despite no analysis at that level, an image of the UniTree task that provides no additional information, and the weirdly massive Table 3 for an experiment that bring limited additional analysis and learnings as an experiment distinct from the D4RL experiments. This has led to a lack of space for related work, a conclusion, and appropriate ablations.
Questions for authors
- Why exactly is the preference score expected to address errors in the reward function when it is trained on the same data?
- What is the motivation and reason for select the given form of the preference score?
- How is the preference score meaningfully distinct from the reward?
- Can the learned reward be used in place of the preference score?
- What is the motivation for incorporating the Q-value as an auxiliary to the BC loss?
- Is the addition of the Q-value as an auxiliary loss to the BC loss a contribution of this work, because there isn't a citation for it?
- What about the prior works motivates that modification to critic learning should have performance gains?