acceptodds
Withdrawn from ICLR 2027

Seeing What Matters for Action: Bridging Visual Priors and Robotic Control in Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained visual foundation models provide rich semantic and spatial priors for Vision-Language-Action (VLA) policies, and recent methods often transfer these priors by directly aligning external visual features with VLA representations. However, direct alignment leaves two gaps between generic perception and robotic control: it does not explicitly prioritize visual information relevant to action prediction, and frame-wise supervision leaves cross-frame relational structure only indirectly constrained. We propose ActSight, a lightweight framework for action-sensitive, temporally structured visual transfer in VLA policies. ActSight consists of two complementary components. Control-Oriented Visual Refinement weights external visual supervision by each token’s influence on the action objective, guiding lightweight residual refinement that preserves the VLA's learned representation while incorporating control-relevant visual knowledge. Temporal-Aware Representation Alignment complements frame-wise supervision by explicitly preserving relational structure across neighboring observations, encouraging the adapted representation to retain how visual states change during interaction. Across multiple manipulation tasks, ActSight improves control performance over direct feature alignment and visual-enhancement baselines. These results demonstrate the effectiveness of organizing visual transfer around action relevance and temporal relational structure.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action, Action-Relevant Visual Representation, Temporal Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nTaYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActSight, a framework for transferring pretrained visual knowledge into vision-language-action models. It combines lightweight residual adaptation, action-loss-gradient-based weighting of visual supervision, and temporal alignment of token correspondences across neighboring observations.

Critical strengths
  1. The method organizes visual knowledge transfer around the requirements of robotic control, resulting in an informative design.

  2. The method introduces little deployment overhead and is evaluated across diverse manipulation settings. At inference, only the lightweight residual module is retained alongside the original VLA.

Critical weaknesses
  1. Some images in Figure 1 appear to originate from BridgeData V2, but they may have been modified using AI tools. If so, the authors should disclose this modification in the AI use statement. In particular, the details of the microwave differ across the images in ways that do not appear physically plausible.

2.Simulation evaluation omits episode counts, training seeds, and uncertainty, limiting assessment of robustness. Moreover, Figure 3 does not explicitly identify the benchmark or model scale; Figures 4 and 6 lack data-split and sample-selection details.

  1. The reported OpenPI results on SimplerEnv-WidowX differ from those reported in the original QDepth VLA paper, which is cited as the baseline source. In contrast, the reported results for QDepth VLA itself are consistent with those in the original paper.

  2. The choice of Qwen VL as the base model is less convincing than using an established VLA model such as OpenPI or OpenVLA.

Questions for authors
  1. Regarding Weakness 1, could the authors provide clarification?

  2. Could the authors explain why they selected Qwen VL as the base model? ActSight appears to be readily applicable to established VLA models such as OpenPI and OpenVLA OFT.

Reviewer_w6mzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ActSight, a framework for improving visual representations in VLA models through action-sensitive visual supervision and temporal relation alignment. The method uses gradients of the action loss with respect to visual tokens to weight feature alignment, performs adaptation through a residual pathway with gradient isolation, and aligns cross-frame correspondence distributions using confidence weighting. Experiments on LIBERO, SimplerEnv, and real-world robotic manipulation tasks show improved performance with limited deployment overhead.

Critical strengths

Useful improvements supported by controlled experiments. The paper includes same-backbone comparisons with FeatAlign and RelAlign, alongside ablations of the residual architecture, action-sensitive weighting, and temporal supervision. On SimplerEnv with the 0.8B backbone, ActSight achieves an average success rate of 72.9%, compared with 63.5% for FeatAlign and 64.6% for RelAlign (Table 1). These results support the framework’s practical value for robotic control.

Limited deployment overhead. External visual teachers are used only during training, while inference retains only a lightweight residual pathway. The small increase in inference latency reported in Table 3 makes the approach attractive from a deployment perspective.

Critical weaknesses

Limited technical novelty and additional insight. Relative to existing work on visual feature alignment for VLAs and spatiotemporal relational distillation, the main additions are action-gradient-based reweighting, residual adaptation with gradient isolation, and confidence-weighted alignment of adjacent-frame correspondences. These choices are reasonable, but the overall method remains an incremental extension of existing representation-transfer approaches. The results demonstrate the effectiveness of the combined framework, while the broader insights gained from this particular combination remain insufficiently established.

The independent value of the specific temporal-alignment formulation is not sufficiently established. RelAlign and ActSight differ simultaneously in their residual architecture, gradient pathways, action-sensitive weighting, and relational objectives. Their comparison therefore does not isolate the contribution of the temporal objective itself. Figure 3(c) demonstrates that adding the complete temporal module is beneficial, but does not establish whether a simpler adjacent-frame relation constraint would suffice under the same residual and action-weighting setup, or how much confidence weighting contributes beyond unweighted correspondence-distribution alignment. This limits the assessment of the technical contribution of this component.

Questions for authors

With the residual pathway, gradient isolation, action-sensitive weighting, and teacher features held fixed, how does performance change if the proposed temporal objective is replaced with simple adjacent-frame cosine-relation matching?

What happens when all correspondence confidence weights are set to one, while retaining the bidirectional soft correspondences and KL alignment? This comparison would help establish the independent value of cycle-consistency-based weighting.

How many training updates are used for joint training and for each stage of two-stage training in Figure 3(d)? Are the total action-supervised training budget and the number of passes over the data matched? How does the action-only baseline perform when continued for the same additional training budget?

Reviewer_cvYJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActSight, a two-stage training framework that distills external visual priors (DINOv3 and SigLIP 2) into a Qwen-VL-based Vision-Language-Action (VLA) policy through a lightweight residual MLP adapter. In the second stage, it weights token-level feature alignment by backpropagated action-loss gradient norms and aligns cross-frame token correspondence distributions weighted by cycle-consistency confidence. After training, the external vision teachers are discarded, and experiments on SimplerEnv-WidowX, LIBERO, and six Franka real-robot tasks report improvements over a vanilla Qwen-VL baseline and alignment variants.

Critical strengths
  1. The problem studied is practical, and discarding the external vision teachers after training keeps deployment lightweight (142.2 ms vs. 141.1 ms in Table 3) without adding onboard foundation-model overhead.
  2. The empirical evaluation covers both simulation benchmarks (SimplerEnv-WidowX and LIBERO) and six real-world Franka manipulation tasks with scene-level OOD perturbations.
Critical weaknesses
  1. The core premise behind the two proposed modules is not fully convincing. First, external vision teachers are typically introduced to provide visual cues that the policy's action loss fails to capture, so it is unclear why gating the teacher by the policy's own action-loss gradient norm is desirable: if the policy overlooks a task-relevant region, its action gradient there may also be small (which may explain why Joint Training from scratch struggles in Figure 3(d)). Moreover, Table 4 sets α=0.10\alpha = 0.10 on LIBERO (α=0.25\alpha = 0.25 on Bridge), which keeps the token weights wt,i=1+α(qt,i−qˉt)w _ {t,i} = 1 + \alpha(q _ {t,i} - \bar{q} _ t) within [0.90,1.10][0.90, 1.10] (or [0.75,1.25][0.75, 1.25]), raising questions about how much this weighting actually differs from uniform alignment. Second, for temporal alignment, the teacher in Section 3.2 is still computed from frame-wise DINOv3 features rather than a video model, and in Figure 7 the native VLA baseline (100.0%) already has higher Region Lift than both w/o Temporal-Aware (34.9%) and full ActSight (79.8%), suggesting that the second module mainly recovers cross-frame consistency lost during the first module's adaptation.
  2. The overall pipeline feels somewhat ad-hoc and over-complicated relative to existing representation alignment methods (REPA [1], Spatial Forcing [2], and VideoREPA [3]), combining a residual adapter, stop-gradient isolation, gradient-norm weighting, bidirectional softmax correspondence matrices, cycle-consistency filtering, and two-stage training. Despite this complexity, the gains over simpler alignment baselines are modest on LIBERO (e.g., 97.8% vs. 97.1% for RelAlign [3] in Table 2, with no error bars or trial counts reported), and on Spoon in Figure 3(b,c) full ActSight (79.2%) slightly trails Uniform weighting (83.3%) and w/o Temporal-Aware (83.3%).
  3. From a presentation standpoint, Sections 1–3 are quite repetitive in restating high-level terms ("action-relevance mismatch" and "temporal-structure mismatch"), yet leave several concrete technical details unclear. In particular, Equation (1) is written for a single target feature and a single projector, whereas Section 3.1.2 mentions using both DINOv3 and SigLIP 2 without explaining how two teachers with different feature dimensions are combined, and Section 3 does not write out the final combined training objective.
  4. It is unclear how well ActSight generalizes to mainstream VLA architectures, and some experimental controls appear incomplete. All experiments are conducted on a custom Qwen-VL model with a simple L1/L2 regression head (which gets 0.0% on Stack Cube in Table 1), rather than established VLA models such as OpenVLA-OFT or flow-based policies like π0\pi _ {0}, where stochastic denoising steps might affect per-step action gradients. In addition, Appendix A.2 notes that FeatAlign and RelAlign do not include the residual adapter (+3.1% in Figure 3(a)), and Table 3 omits training time and GPU memory overhead.

[1] Yu et al. 2025. Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. ICLR 2025.

[2] Li et al. 2026. Spatial Forcing: Implicit Spatial Representation Alignment for Vision-Language-Action Model. ICLR 2026.

[3] Zhang et al. 2026. VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models. NeurIPS 2025.

Questions for authors
  1. Given that α=0.10\alpha = 0.10 on LIBERO (α=0.25\alpha = 0.25 on Bridge) keeps wt,iw _ {t,i} within [0.90,1.10][0.90, 1.10] ([0.75,1.25][0.75, 1.25] on Bridge), to what extent does this weighting suppress background regions in practice, and how does ActSight compare to uniform weighting (α=0\alpha = 0) on LIBERO?
  2. In Figure 7, why does Region Lift under full ActSight (79.8%) remain below the native VLA baseline (100.0%), and how does Equation (3) compare against the simpler cross-frame cosine MAE of RelAlign [3] when both use the same residual adapter?
  3. Can ActSight be applied to established VLA models such as OpenVLA-OFT or flow-based policies like π0\pi _ {0} (where action gradients depend on random denoising steps)? Also, could the authors clarify how DINOv3 and SigLIP 2 are combined in Equation (1), and report the training time and GPU memory overhead in Table 3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.