- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a method, called Flow-KTO, for leveraging deployment feedback for policy improvement. To method actually applies KTO-style objective on the positive/negative rollouts, combined with a BC loss and a drift estimate for stabilizing. Moreover, the method consider the asynchronization between action generation and observation in the real-world. The method is evaluated in real-wolrd deployment, and demonstrates improvement by leveraging on manipulation tasks.
Critical strengths
- The method demonstrate performance gain compared to existing methods on real-world tasks.
Critical weaknesses
- This paper is hard to follow. There are lots of equations in the method section while the notations are not well explained.
- The evaluation is limited. And the details about experimental settings are missing.
Questions for authors
- Can the author provide more justification on some experimental designs, such as the fraction of initial policy failed rollouts in the training set?
- There is an assumption that the offset between the generated action chunk and the observation is fixed. Is this a strong assumtion in real-world tasks? If so, does it have a large influence on the results?