acceptodds
Under review as a conference paper at ICLR 2027

DOES THE POLICY KNOW WHERE THE BOX IS?

Abstract

Unified world-action models place a vision-language reasoner and a video-action diffusion generator in one network, on the assumption that the policy inherits the reasoner's ability to ground objects. We test this assumption without training on the released Cosmos3-Nano-Policy-DROID, a 16B Mixture-of-Transformers built on a Qwen3-VL reasoner. Its inference code shows why the assumption could fail: at act time the reasoner stream receives only the text prompt, while the camera canvas enters only the generator, as VAE latents. Paired pixel, text, and activation interventions on 500 post-training frames (200 for activations) drawn from 2,302, and 500 paired closed-loop episodes per condition, show three things. Served in VLM mode, the policy checkpoint's own reasoner localizes the target on 71–76% of single views but 46% of the policy's three-view canvas (its base VLM: 76–78% and 50%); the halved resolution costs 0.14 in hit rate and the three-view layout 0.12. The named target is exposed far more sharply to a linear probe in the reasoner's image tokens than in any of the generator's token groups (top-cell hit 0.55 against at most 0.39; position-only prior 0.24), although both towers carry it (token AUROC 0.85 against 0.81; equal probe mass on the mask), yet the generator's actions follow the object's pixels and re-target when it is moved. The action head reads the instruction's noun: deleting or swapping it costs 39–46 percentage points (pp) of closed-loop success. It does not act on location text: a location sentence shifts actions about as much as a location-free one (0.5 cm more, toward neither location), survives a matched causal knock-out, steers toward neither instance on 26 frames with a verified same-class distractor (75 candidates), and gives no closed-loop gain (−0.4 pp, 95% CI [−4.4, +3.4]). The tower that exposes the object is not the tower that acts, and location text does not bridge them.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.