- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Learn to Verify, then Revise (LVR), a method to learn verification checks from the feedback which is then used to guide the revisions without training the robot policy control. The paper shows that execution stability is not enough to determine task success as nearly half of the tasks with stable robot execution still fail to satisfy the given instructions.
LVR learns verification checks based on trajectory features, thresholds, and applicability conditions using feedback from a language model. The method selects clarification questions based on how much their answers are expected to improve verification decisions. Once the checks are learned, they are kept fixed and used to identify which motions need revision. When a check fails, the planner receives feedback about the failed requirement and generates a revised motion.
The method is evaluated on simulated Microduck robot tasks. LVR achieves 87.2% task success with 59 selected revisions, compared to 76.6% success with 141 revisions when all tasks are revised. The updated verifier also improves task success from 74.4% to 89.7% compared to the verifier before feedback.
Critical strengths
-
The problem is well motivated. The paper identifies stable robot execution can fail to satisfy the given instructions.
-
LVR does not need any updates to the robot control policy, as verification checks are learned from language-model feedback and uses them to decide which motions need revision.
-
Experiments show that LVR achieves 41/47 successful tasks with 59 selected revisions, compared to 36/47 with 141 revisions for repair-all on recorded revision chains. The updated verifier also improves success from 29/39 to 35/39 tasks compared to the pre-feedback verifier.
Critical weaknesses
- In Table 2, direct labeling achieves lower classification loss than LVR, while LVR increases false rejections compared to the fixed verifier. Also in Experiment C, the learned checks also fail to improve classification on held-out tasks. It is unclear whether the proposed query selection method performs better than simpler baselines or that the learned checks generalize to new tasks.
- In Table 3, the updated verifier improves task success but also selects more revisions. As the pre-feedback and updates verifiers uses different number of revisions, it is not clear if the improvement comes from more revisions or learning better checks.
- The main revision results use recorded revision chains with stopping decisions applied offline. So, the reported reduction in revisions reflects the revisions selected by the policy rather than the actual computation used during the experiments.
- The evaluation is limited to one simulated robot, one model. It is unclear if the results would hold with other simulated robot, model or real robot settings.
Questions for authors
- Could you compare the pre-feedback and updated verifiers using the same number of revisions?
- Why do the learned checks fail to improve classification on held-out tasks in Experiment C?
- Could you compare the proposed query selection method with random queries and direct labeling in terms of final task success, using the same feedback and revision budget?
- Could you evaluate LVR with online stopping instead of applying stopping decisions to recorded revision chains? Since the reported reduction in revisions is based on offline stopping, how much computation does LVR actually save when feedback and verification costs are included?
- Could you test whether the improvements generalize to different robots or models?
Rebuttal Acknowledgement
I thank the authors for their clarification and new experiments. My concern about revision allocation is partly addressed. As random queries can also learn verification checks, I am not sure what impact that would have on the final task success. I think the benefit of their method is not established over the simpler ones. Therefore, I will maintain my original rating.