acceptodds
Withdrawn from ICLR 2027

Knowing Is Not Acting in VLA Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) policies can identify an object from an indirect description yet fail to select it when acting. We study this knowledge-action gap with KnowAct, which pairs literal and knowledge-based references to the target and an alternative object in balanced manipulation scenes. Its contrastive knowledge utilization (KU) measures knowledge-conditioned selection relative to literal control on a fixed set screened for reported knowledge and initial control. We also introduce Referential Self-Distillation (RSD), which trains direct execution against an EMA teacher conditioned on a translated object name. We evaluate GR00T N1.7 and in RoboTwin 2.0. On GR00T, RSD reaches KU 0.311 versus 0.210 for a collection-protocol-matched DAgger control; the paired item-bootstrap difference is 0.101 [0.067,0.135]. On , RSD reaches 0.345 versus 0.258 for DAgger, a paired difference of 0.087 [0.048,0.126]. Among 157 GR00T items, knowledge contrast improves on 110, worsens on 35, and is unchanged on 12. The effect is positive in each evaluated split; knowledge injection, readout probes, activation patching, and control experiments further delimit the finding. The item-bootstrap intervals condition on three trained runs and do not quantify retraining uncertainty.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, knowledge-action gap, robotic manipulation, knowledge utilization, referential self-distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qNZHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies why VLA policies may identify an object from an indirect description yet fail to pick it up. The authors introduce KnowAct, which is a benchmark with a contrastive knowledge utilisation (KU) score that measures how much knowledge-based instructions steer object selection (relative to literal ones). Further, the authors introduce referential self-distillation (RSD), which trains the action head on indirect instructions to match a teacher, given a translated object name. On GR00T and pi0.5 in RoboTwin, the authors show that explicit translation reachers a KU of 0.86 vs. 0.1 for direct execution. RSD beats a matched DAgger control, and the gain seems to hold for held-out objects.

Critical strengths
  1. The proposed KU metric makes intuitive sense and seems well-designed. The paper proposes a careful way to measure whether VLAs act on knowledge its backbone can report, which is an interesting and important problem. This seems like a good contribution since existing benchmarks mostly rely on aggregate success, which may not provide us with a full picture.
Critical weaknesses
  1. The authors mention that GR00T action head reads layer 12, rather than the public layer 16, while the knowledge probe uses the language head above it. As such, it seems that the layers producing the explicit answer are 'invisible' to the action head. Furthermore, S0 is trained on literal demonstrations only. Could this explain the gap for GR00T? It would be great if the authors could report direct and translation KU at layer 16, and, perhaps more generally, explain why layer 12 was chosen?

  2. Regarding RSD: I can see that it improves over literal continuation while falling short of translation. This makes sense, since the translator relies on the list of registered objects in the scene. But this is exactly where the advantage of RSD at test time would be (being able to improve without priveleged information). I think the authors should make this case explicit, and importantly compare against a free-form translator?

  3. I don't particularly understand why the authors state "This is a KU-only replication under a source-VLM reference because component guardrails are unavailable." Why are they unavailable? I thought that CL and CK can be computed from the same rollouts as KU, and the paper also treats them as essential checks. Similarly, "These comparisons are descriptive because stratum-specific intervals are unavailable." Could the authors explain why they are unavailable?

Questions for authors

Please see "Weaknesses"

Reviewer_YiJyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether VLA models can use factual knowledge when following instructions. The authors construct KnowAct, where the same object is referred to by its name or through a factual description. They measure knowledge utilization using target and alternative references, after checking knowledge and literal control. The proposed RSD trains the action modules with an EMA teacher that receives explicit object names. It uses velocity targets, a margin against the alternative object, and literal demonstrations. Experiments on GR00T and π0.5 compare RSD with DAgger and translation at inference. RSD improves over DAgger, including on held-out knowledge types and objects, but remains well below translation. Probing and activation patching are used to further examine the gap.

Critical strengths
  1. The evaluation checks whether the model knows the reference and can follow the literal instruction before measuring knowledge utilization. Comparing target and alternative references also accounts for object preferences that raw success rates would miss.

  2. RSD improves over matched DAgger training on held-out knowledge types and objects. The experiment with a fixed teacher and fixed data buffer also shows a benefit from velocity targets, so the gain is not explained only by collecting different training states.

Critical weaknesses
  1. The experiments mainly show that the policies handle an indirect object description worse than its literal name, even after the knowledge and control checks. To me, this is still a fairly narrow result about object selection. The held-out objects and six-object placement task do not change the type of knowledge being tested. The probes and patching also leave open whether the referent was fully computed or whether the action module failed to use it, as the authors acknowledge. I do not think these analyses explain much about knowledge use beyond this setting.
  2. RSD largely applies context distillation to a flow-matching policy, with learner-state collection following DAgger [1]. The main changes are the teacher-velocity targets and the alternative-name margin. The ablations show that both help, but the extension is still incremental. Proposition 2 uses a standard conditional-expectation argument to show lower variance under teacher self-consistency, and the measured variance reduction agrees with this. It is also unclear whether the training benefit is specific to factual knowledge or would occur when distilling between ordinary paraphrases.
  3. I do not see a clear reason to use RSD over explicit translation in the tested setting. GR00T target selection improves from 30.2% with DAgger to 38.0% with RSD, but its KU is 0.311 compared with 0.860 for translation. For π0.5, the KU values are 0.345 and 0.834. The translator runs only once per episode. Without measuring the time or compute saved, avoiding that call does not seem sufficient to justify the performance difference.
  4. Equation 1 uses both target and alternative references, but the screening description only explicitly checks knowledge of the target. Is knowledge of the alternative also verified? The knowledge probe uses one canonical layout, whereas action evaluation averages four layouts. If a reference is misunderstood in another layout, the lower contrast could come from that misunderstanding. For π0.5, the probe is run on the source VLM, and the separate knowledge and literal contrasts are not reported. These details affect whether the model actually knows what the action evaluation asks it to use.
  5. Only 157 of 480 items enter the main GR00T evaluation, and 129 enter the π0.5 evaluation. Screening makes sense for the diagnostic, but I would also like to know how RSD behaves on the excluded items. Section 3 says all methods are evaluated on all 480 items, so the full-set target-selection rates and component contrasts should be available. The current results do not show whether the improvement on the screened subset comes with a loss elsewhere.

[1] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011.

Questions for authors
  1. Does velocity distillation also help with ordinary paraphrases that require no factual knowledge? This comparison should keep the teacher/student instruction mismatch and the sampled-target control.

  2. How much time or compute is saved by removing the translation step?

  3. Does the knowledge screen check both references in all four layouts? What are the separate π0.5 knowledge and literal contrasts?

  4. How does RSD perform on all 480 items and on the items excluded by the screens?

Reviewer_TU6WOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on “knowledge-action gap” in VLA models. The paper define Knowledge Utilization(KU), which identifies literal contrast and knowledge contrast and measures how much knowledge changes the actions. They also introduce a Referential Self-Distillation (RSD) which tries to fill in the gap of the indirect instruction. They introduces KnowAct, a four-object selection benchmark and validate the method over GR00T N1.7 and π0.5 in RoboTwin 2.0.

Critical strengths
  • KU is measured only on the items that the model knows and controls, effectively scoping the experiment to only measuring the knowledge gap with minimal control failure. By reporting CL and CK together, it shows that performance gains are not because of degraded literal control.
  • There are experiments that measure different design choices to validate RSD over DAgger.
Critical weaknesses
  • The π0.5 arm uses a source PaliGemma VLM model that is not the actual π0.5 VLM model. A VLM backbone model trained jointly with an action expert in π0.5 can behave very differently from it’s original VLM model. “Reported knowledge” there may not reflect the policy’s own backbone.
  • The gain from RSD is moderate. RSD raises KU from 0.21 to 0.31, while simple test-time translation reaches 0.86. The paper does not have a clear explanation for just translating at inference rather than training RSD.
  • The experiment scope is too small. It only checks between 4 objects. There are also different dimensions of how the VLA model can act regarding to language, like slowly, carefully, location, direction, height, etc. None of this is measured and only object-relevant result is being graded.
  • There’s no real robot evaluation. The real world can have multiple different noises effecting the system such as failure in manipulation or hardware calibration, which can change the KU estimation.
Questions for authors
  • Would there be a way to measure the gap between a VLM backbone in a VLA and the source VLM?
  • What are RSD, DAgger, and translation scores on all 480 unscreened items? How does RSD behave on the 81 confidently-wrong items? Does it amplify wrong beliefs, as measured by BAA outside S?
  • What do the error-stage results from Appendix I.3 actually show? How much of the CK gain is referent choice rather than grasp or approach differences?
  • What advantage does RSD offer over explicit translation at deployment? Does combining RSD with translation exceed translation alone?
  • Does RSD still work well on real-world settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.