acceptodds
Withdrawn from ICLR 2027

From Pixels to Actions – Towards Native Vision-Language-Action Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action (VLA), encoder-free vision-language models, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z1dsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NEO-VLA embeds image patches directly into a shared vision-language backbone and gives a flow-matching action expert access to selected backbone layers. The paper studies how readout depth affects control, using policy comparisons, representation interventions, and spatial probes.

Critical strengths
  1. Section 3.3 makes readout depth an explicit policy choice through the layer map. This is a useful abstraction: a backbone may contain information that its action expert cannot readily access. Figures 1 and 2 explain the interface clearly and connect the architectural design to the question investigated in the experiments.

  2. Table 3 provides informative evidence within the native backbone. Frozen skip-28 achieves 90.9 average success versus 86.7 for end-28, with the same retained backbone depth and expert depth. The disjoint early/late maps reinforce the relevance of readout position. Section 5.2 also appropriately distinguishes the early-only configurations, which truncate the backbone, from the other comparisons.

  3. Figure 3 and Appendix B contain thoughtful controls. Scene replacement is compared with matched-magnitude noise, and the span analysis includes matched-random replacements. The wrist-frame policy is a particularly useful positive control: camera pose affects prediction when it defines the output coordinate frame. This makes the interpretation of the robot-frame policy's weak pose dependence more convincing.

  4. Appendix E connects accessibility to a concrete property of the representations. Table 12 reports similar peak depth and correspondence scores across backbones, with larger differences at the layers available to the action expert. Together with the encoder-access Qwen result in Table 2, this suggests a useful design principle that extends beyond the proposed native backbone.

Critical weaknesses
  1. The architectural conclusion in the abstract places more weight on encoder removal than Table 2 supports. Giving Qwen's expert intermediate encoder access raises its score from 79.5 to 90.3, close to frozen NEO-VLA's 90.9. Section 5.1 recognizes that the independently pretrained backbones differ in several respects. The clearest contribution is therefore the importance of accessible intermediate features; the scientific case for encoder removal itself is less developed. This distinction affects how I assess the novelty and scope of the work.

  2. Sections 3.2 and 5.2 explain the benefit through early fine-grained geometry and later semantic processing. Appendix E supports the accessibility of spatial information, but does not directly connect the decoded quantities to the information responsible for the action improvements. Moreover, Figure 3's instruction–image–instruction pattern suggests intertwined visual and linguistic processing. The proposed hierarchy is a plausible interpretation, although the results more securely establish useful joint representations at particular depths than a specific progression from geometry to semantics.

  3. The attachment-point analysis in Figure 3 identifies how a trained expert uses its inputs; its broader architectural implication is less direct. Weak sensitivity to a late attachment need not mean that the corresponding representation is intrinsically less useful for another readout. Table 3 also shows that adaptation changes the preferred configuration: early-only first-28 leads when frozen, while distributed skip-28 leads after fine-tuning. The paper acknowledges this distinction, but its general statement that action prediction relies primarily on early representations should retain that qualification.

Questions for authors
  1. In light of Table 2, do the authors regard encoder removal or intermediate representation access as the principal contribution?
  2. Should Figure 3 be read primarily as an account of this expert's learned dependence, or as evidence for a general hierarchy of representations useful for control?
Reviewer_gdPhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces NEO-VLA. Contrary to prevailing VLA paradigms relying on a separate pre-trained visual encoder, NEO-VLA adopts the encoder-free architecture of the NEO model, wherein an action expert hierarchically accesses KV caches at diverse layers within the backbone. A suite of controlled experiments is devised to systematically examine the influence of multi-depth visual representations on action prediction, and the effectiveness of the proposed architecture is validated on multiple robotic manipulation benchmarks.

Critical strengths
  1. This paper revisits the interaction between visual representations and action prediction in existing VLA models, investigating how visual features at different representation depths contribute to robotic control. This is a meaningful research question that provides valuable empirical insights into how visual representations support embodied control.

  2. This paper conducts evaluations on multiple robotic manipulation benchmarks and out-of-distribution scenarios. Combined with hierarchical feature analysis and intervention experiments, it reveals the importance of intermediate visual representations for action prediction. These results provide valuable empirical evidence for understanding the internal working mechanism of the VLA model.

Critical weaknesses
  1. The proposed framework largely builds upon the existing NEO architecture, directly adopting its encoder-free design, while employing a conventional flow-matching action expert for action prediction. The primary contribution is limited to exploring different configurations for accessing intermediate visual representations, without introducing a novel technical mechanism or a dedicated methodological design to address the identified research problem. Consequently, the paper reads more like an empirical research report rather than a work that offers substantial methodological innovation.

  2. Although the paper compares encoder-free and encoder-based VLA models, the experimental variables are not sufficiently disentangled. The compared models differ in several aspects beyond the presence or absence of a visual encoder, making it difficult to attribute the observed performance gains specifically to the encoder-free design. Furthermore, allowing the encoder-based baseline to access intermediate visual features also leads to competitive performance, suggesting that feature accessibility may be an important contributing factor. Therefore, the current experiments do not convincingly establish the independent benefits of removing the visual encoder.

Questions for authors
  1. Please refer to Weakness.

  2. The experimental evaluation is primarily conducted on simulation benchmarks, without demonstrating closed-loop manipulation on physical robots. Although the paper includes a small-scale offline analysis using real-robot observations, it does not evaluate actual task execution in real-world environments. This limits the evidence for the practical effectiveness and robustness of the proposed approach in physical robotic manipulation.

Reviewer_UN43OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the visual interface between a pretrained vision-language backbone and a continuous action policy. NEO-VLA adopts an existing native multimodal architecture: a lightweight convolutional embedder maps image patches directly into a shared vision-language backbone, and a separate flow-matching action expert attends to cached keys and values from selected backbone layers. The main empirical contribution is a study of readout depth, combining layer-map comparisons, activation replacement, input interventions, and linear spatial probes. The results indicate that early-layer access is particularly useful with a frozen backbone, while readout spanning early and late layers performs best after full fine-tuning. The paper also reports competitive results on LIBERO, LIBERO-plus, and RoboTwin 2.0.

Critical strengths
  1. The readout-depth study provides a useful diagnostic result. With a frozen backbone, disjoint first-14 and last-14 maps score 93.6% and 89.6% (Table 3). Matched-capacity expert comparisons, activation replacement, and the wrist-frame control strengthen the evidence for early-layer access.

  2. Competitive performance across benchmarks and observation shifts. Results include 98.6% on LIBERO, 88.7% on LIBERO-plus (94.1% under camera perturbations), and 68.05% RoboTwin Clean2Rand with 2.5K training episodes. These establish a viable policy, although cross-paper comparisons cannot isolate the source of its gains.

Critical weaknesses
  1. Limited methodological novelty and unclear architectural benefit. The native interface and Native-RoPE follow NEO (Section 3.1), while the action expert follows Black et al. (Section 3.4). The main addition is their integration for control with configurable layer attachments. In Table 2, encoder-access Qwen3-VL reaches 90.3%, close to the frozen native model's 90.9%, suggesting that improved feature access can recover much of the gap while retaining an encoder. Section 5.1 also acknowledges pretraining, resolution, and capacity differences. The distinctive benefit of removing the encoder therefore remains insufficiently established.

  2. The analysis does not yet establish a general mechanism. Early access is best with a frozen backbone, but full fine-tuning favors skip-28 (95.3% versus 94.6% for first-28; Table 3). Figure 3 couples backbone depth with attachment position within the expert: stronger early sensitivity may reflect conditioning routes rather than less accessible geometry in later representations. Appendix E leaves probe data, training details, and cross-backbone layer indexing unclear; peak scores are also similar across backbones. No matched readout-map comparison links early access to the robustness gains. The evidence thus supports a configuration-dependent preference more clearly than a general explanation, limiting the empirical contribution's significance.

Questions for authors
  1. Baseline controls. For Table 2's encoder-access variant, specify the exposed layers, trainable components, expert capacity, and token budget. Please reconcile the “same recipe and evaluation” caption with the encoder-unfrozen Qwen variant's different rollout settings and Gemma's larger expert (Appendix A). Which comparisons isolate the native architecture's benefit? A matched rollout check with existing checkpoints would help.

  2. Spatial probes. Please provide the dataset/split, probe-training and evaluation settings, and layer definitions for Table 12. Does encoder-based depth include the vision tower, language decoder, or both? Layer-wise curves would help distinguish representation quality from attachment selection and averaging effects.

  3. Mechanism and robustness. Can existing controls separate backbone readout depth from attachment position within the expert? Please identify the readout maps and adaptation settings used in Tables 4-6, and clarify whether matched comparisons under distribution shift link early access to robustness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.