acceptodds
Under review as a conference paper at ICLR 2027

DFA-VLA: Dynamic Fine-Grained Alignment for Vision-Language-Action Models in Robotic Manipulation

Abstract

With the rapid advancement of robotic hardware and software technologies, Vision-Language-Action (VLA) models have become pivotal for mapping mul- timodal inputs to continuous robot actions. However, mainstream VLA models often suffer from poor modeling of fine-grained visual elements (e.g., occluded regions and small objects) and an over-reliance on static cross-modal attention, restricting adaptability in complex, open environments. A plausible but underex- amined failure mode is attention dispersion under visual clutter: when distractor objects are present, a dense visual prefix may spread cross-modal attention across irrelevant tokens and degrade manipulation precision even when the target object is clearly visible. To address these limitations, we propose DFA-VLA (Dynamic Fine-Grained Alignment for Vision-Language-Action Models), integrating two key components: (1) the Multi-scale Visual-Semantic Modeling (MVSM) mod- ule, which preserves global scene context while extracting segment-guided local object features; and (2) the Dynamic Fine-grained Alignment and Fusion (DFAF) module, which dynamically isolates task-critical object tokens via text-guided to- ken retrieval and learnable gating before cross-modal fusion into a frozen language backbone. In real-world evaluations on 22 manipulation tasks (220 trials, scored 1, 0.5, or 0 for success, partial success, or failure), DFA-VLA trained for 10,000 steps (5,000 on LIBERO followed by 5,000 on BridgeData V2) obtains a mean score of 78.6% (95% Wilson interval [72.8%, 83.5%]) against 71.1% for the pub- lished OpenVLA policy, scoring higher on 16 of the 22 tasks, tying on 5, and lower on 1 (paired sign test p < 10 −3). In a paired 1,000-step clutter study on LIBERO- Spatial (32 scenes with 0/3/5/8 distractors), however, five-token prefixes, with or without segmentation, do not fit the demonstrations and reach only 3/32 and 1/32, while a LoRA-adapted dense control reaches 26/32 versus 22/32 for the published OpenVLA (exact McNemar p = 0.29). The two evaluations differ in budget, data, and scoring, and the physical evaluation has no matched dense or sparse control, so they are not directly comparable.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robotic Manipulation, Embodied AI, Multimodal Alignment, Cross-Modal Attention, Clutter Resilience, Sim-to-Real Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_iZvKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work presents DFA-VLA, an approach that replaces the 256-token dense visual prefix in OpenVLA with 5 sparse tokens. These tokens are selected via instruction-driven cosine retrieval and a learnable gate from a pool of original global tokens and MobileSAM-segmented local tokens. The paper investigates whether this 5-token prefix can serve as a viable substitute for the dense prefix in visually cluttered scenes. Simulation results (1,000 steps in LIBERO-Spatial, 32 paired scenes) indicate that it cannot. While real-world evaluations (22 tasks, 220 trials on two industrial robots, 10,000 training steps) show DFA-VLA achieving 78.6% versus public OpenVLA's 71.1%, the lack of proper matched controls renders these numbers incomparable.

Critical strengths
  1. The paper has a clear motivation and a straightforward approach. It reports its results honestly and offers some valuable insights.
Critical weaknesses
  1. The paper lacks comparisons with SOTA models. The baselines used here are somewhat outdated, leaving it unclear whether this method actually works on newer, more advanced architectures.
  2. The paper is poorly organized. Several key results and experiments are buried in the appendix; moving them to the main text would make the paper much more substantial and better support its claims.
  3. The simulation experiments are severely underpowered. They use 32 scenes and a single random seed, and the training loss clearly shows the model is underfitting, leading to poor performance. Because of this, the experiment ends up testing whether the sparse prefix underfits on a short training budget, rather than whether it can actually handle visual clutter.
Questions for authors

1, Although the approach saves on visual tokens, the added latency from MobileSAM is quite significant. How was this implemented in real-world physical evaluations, and are there more efficient alternatives? 2. What accounts for DFA-VLA's poor simulation performance contrasting with its superiority over OpenVLA in real-world tasks? Is there any supporting evidence from the training period? Can you provide a simulation baseline matched to the 10,000-step training budget?

1 comment
AuthorsOpenReview ↗

Response to Reviewer iZvK

Modern models and organization. The new comparison controls the intervention within OpenVLA. It does not test compatibility with newer VLA architectures or establish state-of-the-art performance. The revised manuscript states this scope and presents the controlled success counts, full-policy timing and mechanism diagnostic in the main results. A comparison to an unmatched published score would not repair this evidence gap.

Budget and sample size. We completed the requested matched 10,000-step simulation baselines. All three trainings start from the same checkpoint, use the same seed and demonstrate identical input-order and per-step data-chain hashes. Their final outcomes remain 26/32 dense, 10/32 sparse-global and 11/32 DFA on the reconstructed paired plan. This addresses the longer-budget control, while retaining the single-seed/eight-block limitation. It does not address statistical power across independent training seeds or unseen tasks. We report small-subset fit diagnostics separately; they cannot substitute for closed-loop evaluation.

Physical/simulation discrepancy and MobileSAM. The physical and simulation evaluations differ in scoring. The two physical methods used the same CR5 demonstrations according to the author, but a fully matched comparison of their training schedules and controller settings has not been established. We have recovered UMI gripper-range and SLAM/tag calibration records, and the author confirms that the CR5 demonstrations were collected with a mounted UMI gripper. The author confirms that the CR5 demonstrations were used to train the DFA-VLA physical policy associated with the reported 78.6% score. The author also confirms that the OpenVLA physical comparison was fine-tuned using the same CR5 demonstrations. Shared demonstration data do not alone establish that training schedules and controller settings were matched. The exact checkpoint lineage, training schedule and complete robot-base mapping remain unverified. We therefore cannot attribute the physical/simulation discrepancy to the method. The new simulation evidence does not show an advantage for DFA over the dense baseline. The complete-policy timings reported here are simulation measurements. The available physical documentation does not establish the MobileSAM schedule or controller-loop latency. The same-A40 comparison directly measures the added cost of the simulation implementation. Alternative segmenters or scheduling strategies would require new measurements before any efficiency claim.

Supplementary diagnostic.

To test whether five tokens simply cannot fit the training examples, we ran three matched small-subset conditions: top-5, top-32 and five tokens with one forced local slot. All three fit the same 64 training frames by step 50 (100% teacher-forced action-token accuracy). On 64 frames from different demonstrations of the same four tasks, token accuracies were 23.0%, 37.1% and 21.7%; the unadapted dense reference achieved 88.6%. Increasing the prefix or forcing a local slot therefore did not establish generalization in this diagnostic. The forced slot produced nonzero local-position gradients, whereas top-5 and top-32 selected no local tokens in these training evaluations. These observations support reporting routing and generalization limitations; they do not establish a universal causal explanation or a closed-loop benefit from either change. We have not launched another 10,000-step training on this basis.

Initialization and inherited pretraining. All three matched simulation trainings start from the same public openvla/openvla-7b-finetuned-libero-spatial checkpoint (revision 962318cec55ac10993ff0f5f43eda9a270b4c873) and use libero_spatial_no_noops for the additional 10,000 steps. We directly load the released checkpoint; its upstream foundation pretraining and official LIBERO fine-tuning were not performed in this work. This initialization inherits OpenVLA real-robot pretraining, including BridgeData V2, and subsequent public LIBERO adaptation. The reported simulation comparison therefore concerns the prefix intervention and subsequent adaptation within this shared initialization, rather than training a VLA from simulation alone. Inherited BridgeData V2 pretraining is distinct from the previously stated, still-unverified additional 5,000-step BridgeData V2 stage for the physical policy.

The author further confirms the physical training sequence: LIBERO adaptation followed by fine-tuning on the CR5 demonstrations collected with the mounted UMI gripper. Exact checkpoint identifiers and training step counts have not been independently recovered.

Reviewer_QQ6vOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Building upon OpenVLA, this paper integrates global image features with local features derived from segmentation. Through instruction-guided retrieval and a learnable gating mechanism, the visual prefix of the language model is compressed into five tokens, with adaptation handled via LoRA. Experiments include controlled comparisons in simulation as well as multi-task evaluations on physical robots.

Critical strengths
  1. The real-world experiments cover a broad range of tasks.
  2. Reducing the visual prefix of a VLA through instruction-guided visual token selection is a promising research direction.
Critical weaknesses
  1. Simulation results under controlled conditions do not support the current design. As shown in Table 1, DFA-VLA succeeded in only 1 out of 32 trials (3.1%), compared to 26/32 (81.3%) for the dense LoRA baseline and 22/32 (68.8%) for the published OpenVLA model. Even in the absence of additional distractors, DFA-VLA succeeded in only 1 out of 8 trials.

  2. The full pipeline is computationally expensive, and it has not been demonstrated that compressing the visual prefix yields actual efficiency gains. Table 3 indicates that the median inference latency for DFA- VLA on an A100 GPU is 3,305.1 ms per step (approximately 0.30 steps per second). The paper fails to report the full-pipeline latency for a dense baseline on the same hardware.

Questions for authors
  1. Following the second stage of training on BridgeData V2, how does the model generalize to the Zhichang CR5 and Lebai LM3 platforms without platform-specific training data? Please clarify the mapping of the 7- dimensional end-effector actions to these two robots, including coordinate systems, action scaling, de-normalization, and gripper control, and specify whether platform-specific calibration is required
1 comment
AuthorsOpenReview ↗

Response to Reviewer QQ6v

Clean and cluttered simulation. We agree that the clean-scene weakness prevents a claim of clutter robustness. At 10,000 steps, the dense/sparse-global/DFA counts are respectively 8/3/2 out of eight with zero added objects, 7/2/2 with three, 6/3/4 with five, and 5/2/3 with eight. DFA has three exclusive successes against sparse-global and two exclusive failures, a one-scene net difference. We treat these as exploratory paired observations from eight task/state blocks and one training seed. Added objects are anchored and collidable, so these conditions can change both image content and physical contacts. A visual-only control is required to attribute differences specifically to visual distraction.

Complete policy latency. The new same-A40 comparison includes preprocessing, segmentation and crop encoding when enabled, synchronized policy inference, decoding and action validation. Dense, sparse-global and DFA have median latencies of 277.8, 277.5 and 1,270.3 ms, and p95 values of 331.4, 849.5 and 2,675.3 ms. Thus the present DFA implementation compresses decoder input but is slower overall. We distinguish these rollout measurements from the submitted A100 microbenchmark and make no physical-loop timing claim from them.

Physical action mapping and calibration. Thank you for requesting clarification of the physical interface. The CR5 setup used a UMI gripper mounted on the robot arm; demonstrations were collected with the arm and gripper connected, and the retained physical trajectories were used primarily for CR5. CR5 and Lebai LM3 use different SDKs, and their respective SDKs were used during experimental data collection. The policy outputs end-effector position increments. Setup included camera–robot coordinate alignment, homing and gripper-range configuration, so we do not claim calibration-free transfer. OpenVLA standardizes the gripper channel to 0=closed and 1=open, but this convention must be distinguished from the platform-specific SDK mapping. We recovered UMI gripper-range and SLAM/tag calibration files, together with a monocular–inertial processing log. These establish retained preprocessing records, rather than the complete robot-base mapping. The author confirms that these CR5 demonstrations were used to train the DFA-VLA physical policy associated with the reported 78.6% score. The author also confirms that the OpenVLA physical comparison was fine-tuned using the same CR5 demonstrations; this does not independently establish identical training schedules or controller settings. Exact controller parameters, checkpoint lineage and training schedule still require verification; the retained demonstrations do not independently reconstruct the original scored trials.

Pretraining versus subsequent adaptation. OpenVLA inherits pretraining on real-robot data including BridgeData V2. This does not by itself establish an additional BridgeData V2 fine-tuning stage in our experiments. We are separating inherited pretraining from platform-specific demonstration use and verifying the physical checkpoint training history before retaining the previously stated 5,000-step BridgeData V2 recipe.

Initialization and inherited pretraining. All three matched simulation trainings start from the same public openvla/openvla-7b-finetuned-libero-spatial checkpoint (revision 962318cec55ac10993ff0f5f43eda9a270b4c873) and use libero_spatial_no_noops for the additional 10,000 steps. We directly load the released checkpoint; its upstream foundation pretraining and official LIBERO fine-tuning were not performed in this work. This initialization inherits OpenVLA real-robot pretraining, including BridgeData V2, and subsequent public LIBERO adaptation. The reported simulation comparison therefore concerns the prefix intervention and subsequent adaptation within this shared initialization, rather than training a VLA from simulation alone. Inherited BridgeData V2 pretraining is distinct from the previously stated, still-unverified additional 5,000-step BridgeData V2 stage for the physical policy.

The author further confirms the physical training sequence: LIBERO adaptation followed by fine-tuning on the CR5 demonstrations collected with the mounted UMI gripper. Exact checkpoint identifiers and training step counts have not been independently recovered.

Reviewer_fKzaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses attention dispersion in VLA policies under visual clutter. The hypothesis is that when distractor objects are present, a dense visual prefix spreads cross-modal attention over irrelevant tokens, degrading manipulation precision even when the target is clearly visible. The proposed DFA-VLA builds on OpenVLA and adds two modules. Multi-scale Visual-Semantic Modeling (MVSM) builds candidate tokens from the global visual tokens and from object crops obtained with MobileSAM masks. Dynamic Fine-grained Alignment and Fusion (DFAF) selects a small set of these candidates by instruction-guided cosine retrieval and applies a learnable scalar gate before the tokens enter the language backbone. On 22 real-robot tasks, DFA-VLA trained for 10,000 steps scores 78.6% against 71.1% for the published OpenVLA policy.

Critical strengths

None

Critical weaknesses

Clarity of contribution. The abstract and introduction mainly report numerical results, but they do not clearly state the paper’s conceptual contribution or the insight behind it. I recommend stating the problem, the key idea and why it should work before presenting results. The framing is also inconsistent: the title, abstract and method present DFA-VLA as an effective approach, while the conclusion describes the main controlled result as a “negative pilot finding”.

DFA-VLA's success rate is (1/32), while a dense LoRA control performs best (26/32). I would suggest the authors double-check with Section 5.1.

Figure 3 is the only evidence for reduced attention dispersion, but it shows a single example, and its caption states that it is “for illustration only” and not used in any result.

Questions for authors

I don't think there is any question that would substantially affect my assessment.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fKza

Contribution and framing. The revised manuscript explicitly describes the intervention in OpenVLA: replacing its dense visual prefix with five instruction-retrieved and gated tokens drawn from global and segmentation-derived candidates. The evidence does not establish improved robustness or efficiency. The revised manuscript states this boundary consistently in the motivation, abstract and conclusion.

Table 1. The submitted manuscript reports 1/32 for DFA-VLA and 26/32 for dense after the original 1,000-step pilot. We additionally completed matched 10,000-step training for dense, sparse-global and DFA-VLA. On a newly frozen shared 32-scene plan, they achieve 26/32, 10/32 and 11/32. Because the original scene manifest was not recovered, we identify this as a reconstructed follow-up, and do not pool these outcomes with the original pilot. Longer training does not establish superiority over dense.

Figure 3. Figure 3 in the submitted version does not quantify attention dispersion or establish that attention dispersion causes failure. The revised manuscript removes claims that rely on that inference. A separate diagnostic now records actual selected tokens, with the stored initial image and first action verified against the rollout. Across 32 initial observations, all have 16 valid local candidates and none selects a local token in the five-token prefix. Best local ranks are 186–231. This identifies a concrete routing limitation at those observations, without making a claim about every subsequent timestep.

Initialization and inherited pretraining. All three matched simulation trainings start from the same public openvla/openvla-7b-finetuned-libero-spatial checkpoint (revision 962318cec55ac10993ff0f5f43eda9a270b4c873) and use libero_spatial_no_noops for the additional 10,000 steps. We directly load the released checkpoint; its upstream foundation pretraining and official LIBERO fine-tuning were not performed in this work. This initialization inherits OpenVLA real-robot pretraining, including BridgeData V2, and subsequent public LIBERO adaptation. The reported simulation comparison therefore concerns the prefix intervention and subsequent adaptation within this shared initialization, rather than training a VLA from simulation alone. Inherited BridgeData V2 pretraining is distinct from the previously stated, still-unverified additional 5,000-step BridgeData V2 stage for the physical policy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.