- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to study the alignment problem between imitation learning polices and human feedback and preference. The authors propose Feedback Manipulation Regularization, which as a corrective term to improve the alignment quality. Evaluations on Safety Gym demonstrates the superior performance of the proposed method, FMR
Critical strengths
-
This paper studies the alignment problem between imitation learning robot policies and human preference, which is an important and timely topic;
-
The authors provide theoretical analysis for their proposed method FMR.
Critical weaknesses
-
I think the authors need to compare with very simple data selection methods (keep "good" data, discard "bad" data, only imitate from "good" data)
-
The authors need to discuss why they select the evaluation tasks. It seems that the authors add massive adaptations from Safe Gym. The authors need to justify why they select these evaluation tasks and whether their massive adaptations create the settings that do not resemble real settings and favor the proposed method.
-
The authors need to illustrate whether the proposed method works with modern imitation learning methods such as diffusion based or flow matching based.
Questions for authors
See the weakness points I list.
2 comments
Thanks for the additional results. My concerns have been solved. As I already indicate positive scores, I will keep it.
We would like to thank the reviewer for their time and feedback. Below we address both your questions and weaknesses. Any changes made have been highlighted in the paper.
[W1 Data Filtering] We agree, and we explored this idea early on. We used feedback to filter the original DI data, while keeping all original DE data in DE. For ReCOIL, data receiving positive feedback is moved to DE, while data receiving negative or no feedback remains in DI. For BC, which cannot learn from DI, data receiving positive or no feedback is moved to DE and data receiving negative feedback is removed, since also removing data without feedback would greatly reduce the dataset size. In both cases, performance falls short of FMR (see tables below). BC with filtering generally improves success and misalignment over baseline BC, but fails to reduce misalignment in low-overlap settings with scarce aligned data (e.g., PathBB at the 10-50 ratio). We hypothesize that this could be because filtering removes useful information, including general state-space coverage and information about undesirable behaviors. ReCOIL with filtering performs worse than its baseline in all settings, likely because misaligned data that incorrectly receives positive feedback is assigned to DE, where its influence is amplified. In contrast, FMR learns not only what the agent should do but, equally importantly, what it should avoid doing, which is exactly the information that filtering discards or dilutes. We have added these results to Appendix C.
PathM with demonstration filtering
PathBB with demonstration filtering
[W2 Environment Selection] Our task selection and modifications are motivated in Section 5.1 and Appendix B: Safety Gymnasium’s cost signals provide a principled measure of misalignment, which we use only for evaluation. Since the original navigation tasks do not define a specific intended behavior beyond avoiding hazards, we modified them only to create clear alignment scenarios in which a single designated path to the goal is aligned. For the velocity tasks, we slightly adjusted the velocity thresholds for certain tasks to allow room for distinct gait patterns, as we found some of the preset thresholds too restrictive. Both modifications were made to provide clearer and more accurate alignment evaluation, and they do not favor FMR or any other method, as hazards are unobservable to the agent and all methods are trained on the same demonstrations and feedback.
[W3 Algorithm Selection] Our selected baselines span classical imitation learning methods (BC, IQL) to prominent methods designed specifically for learning from noisy demonstrations (DemoDICE, ReCOIL), which reflects the setting of this work. Adapting FMR to diffusion and flow-matching policies is likely less straightforward. We believe this is a interesting direction that warrants its own investigation in future work.