- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper aims to learn a reward function for state-action pairs from trajectory-level scores. The authors propose two local reward constraints: a contrastive ordering constraint that directly compares actions at similar states based on their trajectory-level scores, and a smoothness constraint that assigns similar rewards to similar state-action pairs. The proposed method falls into the field of inverse reinforcement learning (IRL). Accordingly, in the experimental section, the authors compare their method with state-of-the-art IRL methods.
Critical strengths
-
This paper observes that a higher-scoring trajectory is more likely to contain a higher-quality action than a lower-scoring trajectory when the states are similar. The authors use this observation to learn reward functions in several IRL tasks and achieve improved performance. Although this observation does not always hold, the authors empirically demonstrate its effectiveness on the benchmarks reported in the paper.
-
I think the proposed method can be easily integrated into many RL algorithms, and this plug-and-play property offers some potential for extending the method to more benchmarks or real-world tasks where only trajectory-level scores are available.
Critical weaknesses
-
My major concern is the motivation of this paper, which is also summarized in Assumption 1. The authors assume that a higher-scoring trajectory is more likely to contain a higher-quality action when the states are similar. However, practical scenarios may severely violate this assumption, and I do not think it generally holds in real-world tasks. As we all know, the trajectory-level score is influenced by all state-action pairs along the trajectory. Thus, I doubt whether the proposed method can be successfully extended to more practical tasks beyond the benchmarks considered in this paper, even though it is straightforward to apply the algorithm to other tasks. If the authors can thoroughly address this major concern, I would be willing to raise my score.
-
In the experimental section, the authors only consider three baseline methods, among which only one (R4) is relatively recent. I think it would be better to include some additional baselines to further demonstrate the strengths of the proposed method.
Questions for authors
Could the authors provide some failure cases where the proposed method does not work well? I think such cases would be useful for exploring the limitations of the method and clarifying the applicability of Assumption 1.