- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies why VLA policies may identify an object from an indirect description yet fail to pick it up. The authors introduce KnowAct, which is a benchmark with a contrastive knowledge utilisation (KU) score that measures how much knowledge-based instructions steer object selection (relative to literal ones). Further, the authors introduce referential self-distillation (RSD), which trains the action head on indirect instructions to match a teacher, given a translated object name. On GR00T and pi0.5 in RoboTwin, the authors show that explicit translation reachers a KU of 0.86 vs. 0.1 for direct execution. RSD beats a matched DAgger control, and the gain seems to hold for held-out objects.
Critical strengths
- The proposed KU metric makes intuitive sense and seems well-designed. The paper proposes a careful way to measure whether VLAs act on knowledge its backbone can report, which is an interesting and important problem. This seems like a good contribution since existing benchmarks mostly rely on aggregate success, which may not provide us with a full picture.
Critical weaknesses
-
The authors mention that GR00T action head reads layer 12, rather than the public layer 16, while the knowledge probe uses the language head above it. As such, it seems that the layers producing the explicit answer are 'invisible' to the action head. Furthermore, S0 is trained on literal demonstrations only. Could this explain the gap for GR00T? It would be great if the authors could report direct and translation KU at layer 16, and, perhaps more generally, explain why layer 12 was chosen?
-
Regarding RSD: I can see that it improves over literal continuation while falling short of translation. This makes sense, since the translator relies on the list of registered objects in the scene. But this is exactly where the advantage of RSD at test time would be (being able to improve without priveleged information). I think the authors should make this case explicit, and importantly compare against a free-form translator?
-
I don't particularly understand why the authors state "This is a KU-only replication under a source-VLM reference because component guardrails are unavailable." Why are they unavailable? I thought that CL and CK can be computed from the same rollouts as KU, and the paper also treats them as essential checks. Similarly, "These comparisons are descriptive because stratum-specific intervals are unavailable." Could the authors explain why they are unavailable?
Questions for authors
Please see "Weaknesses"