- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates unsupported strategy selection in generalist robot policies: a policy attempts an interaction that the test object does not afford, despite receiving an instruction specifying the feasible strategy. The proposed method recolors objects according to strategy-conditioned training-data statistics and deletes object nouns when doing so increases an attention-based score on the requested interaction part. Experiments cover three simulated tasks, with additional transfer evaluations on two other DROID-trained policies.
Critical strengths
The failure mode is practically relevant, and the proposed interventions are straightforward to interpret.
Critical weaknesses
- Method described in main paper is unclear. The method combines demonstration retrieval, color selection, noun deletion, and attention scoring, but provides neither equations nor pseudocode describing the complete algorithm. This makes decision sequence difficult to verify. For example, the attention score lacks a self-contained mathematical definition.
- The method’s central selection rules are insufficiently justified. The observation edit selects the most frequent color among demonstrations of the requested strategy and defines “typical” colors through a greater-than-50% coverage rule. Neither the particular threshold nor the superiority of this selection rule is convincingly validated. And what if the objects in the training data are not a single color?
- Noun deletion may be too limited in specific scenes. For example, changing “take the lid off the box by lifting the knob” to “take the lid off by lifting the knob” removes information that distinguishes the box from other containers. In a scene containing both a box and another container with a knobbed lid, the original instruction can uniquely identify the target, whereas the edited instruction may become ambiguous. Consequently, the experiments demonstrate the benefit of noun deletion under favorable referential conditions. This limits the practical scope of the instruction-editing component and calls for an explicit semantic-preservation condition.
- The experimental scope is too narrow to establish the generality of the proposed procedure. The main evaluation uses three tasks, one principal object per task, and ten object poses with three rollouts each. Held-out poses test variation around the same objects, rather than generalization to new instances or environments. Consequently, the paper does not establish whether its edit-selection rules work beyond the specific objects used to develop and analyze the method or solve problem mentioned in W3.
- The cross-policy results provide limited evidence of practical transfer. Table 6 transfers edits selected for π0.5-DROID and uses asset recoloring rather than the online pipeline. Some improvements are small, and no uncertainty estimates are reported. Since the thirty rollouts are grouped within ten poses, pose-level variability also matters.
Questions for authors
- Tables 9 and 10 appear to report the same pooled attention-score changes over the ten held-out poses (layers-11-to-13), but give different values. Why?
- On the existing tasks, how does the attention-based rule compare with simply deleting eligible object nouns, and how does the color-selection rule compare with a fixed recoloring baseline?