acceptodds
Desk rejected at ICLR 2027

DOES THE POLICY KNOW WHERE THE BOX IS?

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified world-action models place a vision-language reasoner and a video-action diffusion generator in one network, on the assumption that the policy inherits the reasoner's ability to ground objects. We test this assumption without training on the released Cosmos3-Nano-Policy-DROID, a 16B Mixture-of-Transformers built on a Qwen3-VL reasoner. Its inference code shows why the assumption could fail: at act time the reasoner stream receives only the text prompt, while the camera canvas enters only the generator, as VAE latents. Paired pixel, text, and activation interventions on 500 post-training frames (200 for activations) drawn from 2,302, and 500 paired closed-loop episodes per condition, show three things. Served in VLM mode, the policy checkpoint's own reasoner localizes the target on 71–76% of single views but 46% of the policy's three-view canvas (its base VLM: 76–78% and 50%); the halved resolution costs 0.14 in hit rate and the three-view layout 0.12. The named target is exposed far more sharply to a linear probe in the reasoner's image tokens than in any of the generator's token groups (top-cell hit 0.55 against at most 0.39; position-only prior 0.24), although both towers carry it (token AUROC 0.85 against 0.81; equal probe mass on the mask), yet the generator's actions follow the object's pixels and re-target when it is moved. The action head reads the instruction's noun: deleting or swapping it costs 39–46 percentage points (pp) of closed-loop success. It does not act on location text: a location sentence shifts actions about as much as a location-free one (0.5 cm more, toward neither location), survives a matched causal knock-out, steers toward neither instance on 26 frames with a verified same-class distractor (75 candidates), and gives no closed-loop gain (−0.4 pp, 95% CI [−4.4, +3.4]). The tower that exposes the object is not the tower that acts, and location text does not bridge them.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RQK9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether Cosmos3-Nano-Policy-DROID transfers its Qwen3-VL reasoner's object grounding to action generation. It concludes that the generator follows object pixels and nouns, but location text does not bridge the reasoner's grounding to actions.

Critical strengths

The paper demonstrates substantial empirical effort, including a broad set of probes, paired interventions, controls, and closed-loop evaluations.

Critical weaknesses
  1. The central question is mismatched to the deployed architecture. The paper shows that there is no visual-reasoner-to-action pathway through which the reasoner's grounding of the current scene could be transferred. But this is the architecture and training setup, not an accidental missing pathway. Under this design, the generator is the component that is expected to ground visual observations and produce actions. Q2 provides evidence for exactly this explanation: the generator responds causally to target-pixel changes and re-targets after object movement. The policy may simply not need visual information from the reasoner. A model's not using a particular capability, or one pathway's not carrying information, does not imply that the system lacks that capability or information. Therefore, the premise that the reasoner's visual grounding should transfer to actions is absent for this model; even if this transfer question is meaningful for another architecture, Cosmos is not an appropriate model on which to test it. This is consequential because the paper treats an architectural property as a failed transfer.

  2. The Q3 location-text intervention is confounded by prompt distribution shift. The paper acknowledges that the appended sentences and relative-clause prompts are rare or absent in policy-training instructions. Its null result may therefore show that the policy does not interpret these prompt formats, rather than that it cannot use location language. The same-class-distractor test does not rule this out: it uses the same constructions, and its verified subset is small. A convincing test would use naturally occurring, in-distribution spatial referring expressions in scenes with genuinely ambiguous same-class instances.

  3. Q1--Q3 do not establish the paper's central claim. Q1 evaluates the reasoner in VLM mode, Q2 instead shows that the generator responds to target pixels and re-targets after movement, and Q3 constructs a text bridge. Thus these experiments can be explained by the architecture itself, but do not show that the reasoner's visual grounding was expected to transfer to actions and failed to do so. The presentation reinforces this overstatement. The abstract is overly dense with numerical results and the contribution list largely repeats result-level statistics. The paper does not clearly separate high-level contributions, empirical results, and the central claim. It should state its precise, supportable claim first and distinguish it from broader architectural implications.

Questions for authors
  1. Is there evidence from the Cosmos design or documentation that the reasoner is intended to process visual observations during policy inference, rather than functioning only as a text-conditioning component? If not, why is this architectural property framed as a failed transfer?

  2. How do Q1, which is off-policy, Q2, which shows direct generator grounding, and Q3, which adds an untrained text interface, jointly test the claimed failure of reasoner-to-action grounding transfer?

  3. Can the authors evaluate spatial referring expressions that occur naturally in the policy-training distribution, with genuinely ambiguous same-class instances where the expression is necessary to identify the target?

Reviewer_VsRjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the relationship between the reasoner's visual grounding ability and action generation in Cosmos 3 policies. Using representation probes, pixel and text interventions, activation ablations, and closed-loop simulation, it finds that the reasoner receives only text during policy execution and that additional location text does not yield a significant control benefit.

Critical strengths

The paper critically examines the assumption that a unified policy naturally inherits the capabilities of its reasoner. By inspecting the actual inference pathway and conducting intervention experiments, it highlights the distinction between a model possessing a capability and the policy actually using it. This perspective provides useful insight into existing design choices.

Critical weaknesses
  1. The general characterization of WAMs is factually inaccurate. The abstract assumes that a WAM consists of a vision-language reasoner and a video-action generation branch. This architecture is not a defining requirement of WAMs, whose central feature is joint or coordinated modeling of actions and future world states. DreamZero and Cosmos Policy, for example, use different architectures and do not fit this general characterization.

  2. The evaluation is restricted to Cosmos 3, limiting the generality of the conclusions. The main experiments focus on Cosmos3-Nano, while the supplementary Edge experiments remain within the same model family. Without evaluating other WAM architectures, it is unclear whether the observations reflect Cosmos 3-specific implementation and training choices or a more general phenomenon.

  3. The direct connection between the core experimental metrics and reasoning ability is not established. Object localization is not equivalent to reasoning, and whether location text changes actions does not directly establish whether the policy inherits reasoning ability. The authors need to define reasoning and provide a direct argument or experimental evidence explaining why localization is a valid test of that ability and its transfer. Otherwise, the conclusions should be restricted to visual grounding and the use of location text.

  4. Some key experimental design choices are insufficiently justified. The main per-token probes examine generator layers 8, 20, and 32, whereas the reasoner is sampled across layers 4-35. The rationale for these choices and the asymmetric layer coverage is not sufficiently explained. The authors should establish whether the selected layers are representative and whether the comparison remains valid under matched layer coverage.

Questions for authors
  1. Please clarify the definition of a WAM and the scope of this study. Which conclusions apply specifically to Cosmos 3?

  2. What direct evidence supports the connection between the localization metrics and reasoning ability? How do these metrics establish whether that ability transfers to action generation?

  3. What principles guided the selection of layers for probing, and how sensitive are the comparisons to this selection?

  4. Does the lack of benefit from location text arise from an architectural limitation or from training that did not establish the relevant input-behavior association? How does the available evidence distinguish these explanations?

Reviewer_CDzROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper interrogates a core assumption of unified vision–language–action (VLA) models: that a policy's action head inherits the visual grounding capability of a pretrained reasoner. Using Cosmos3-Nano-Policy-DROID as a case study, the authors analyze its inference and training code paths and conduct paired causal interventions on pixels, text, and activations across 2,302 in-distribution frames and 500 closed-loop episodes. The results show that at act time, the reasoner stream receives only text, while the camera canvas is fed solely into the diffusion-based action generator; although the reasoner can localize the named object in single images, the action head neither consumes location text nor uses its grounding representations, yet remains causally driven by the target's pixels and the noun in the instruction. The authors conclude that "the tower that exposes the object is not the tower that acts," challenging a common design premise in unified VLA architectures.

Critical strengths

Directly challenges the widely held belief that sharing a backbone suffices for grounding transfer, with implications for many recent unified VLA designs.

Critical weaknesses

Ground truth for localization relies on Grounding DINO + SAM with LLM-agent verification (70% correctness), which, while partially validated, remains imperfect and could bias absolute hit rates.

Questions for authors
  1. Could you report results with explicit visual prompts (e.g., overlayed boxes/masks) appended to the canvas, or direct structured spatial tokens to the action head, to test whether a non-text channel immediately yields gains without retraining?

  2. How sensitive are the Q1 results to alternative multi-view arrangements (e.g., different tilings, higher-resolution exteriors, or separate per-view queries) and does any such arrangement improve closed-loop outcomes when piped through the reasoner as text?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.