acceptodds
Withdrawn from ICLR 2027

SMA-VLA: Proprioception-Conditioned Semantic-Motion Alignment for Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language-Action (VLA) models leverage pretrained Vision-Language Models (VLMs) to map visual observations and language instructions to robot actions, providing a promising paradigm for language-conditioned manipulation. However, strong in-distribution performance does not necessarily translate into robust behavior under distribution shifts, particularly when the robot starts from a different configuration, when an instruction is rephrased while preserving the same task semantics, or when visual layouts and distractors change. A key reason is that existing VLA models predominantly learn direct mappings from multimodal observations to expert actions, while how task semantics and current execution conditions influence the required motion remains insufficiently modeled. To address this problem, we propose SMA-VLA, a Vision-Language-Action framework centered on Proprioception-Conditioned Semantic–Motion Alignment (PC-SMA). The PC-SMA module incorporates the robot's current proprioceptive state into task conditioning and aligns the resulting task semantics with the coarse translational direction implied by expert demonstrations, thereby strengthening the otherwise implicit correspondence between task semantics and state-dependent motion. Multi-Level Context Enhancement integrates complementary information across VLM depths, providing richer semantic and spatial context for this alignment and reducing reliance on isolated layer representations. Patch-Focused Bridge uses the evolving action representation to emphasize action-relevant visual evidence, reducing interference from irrelevant scene factors and supporting more reliable motion prediction under visual shifts. Experiments on LIBERO, together with zero-shot transfer evaluations on LIBERO-Plus and LIBERO-Pro, show that SMA-VLA improves manipulation performance and enhances generalization under representative robot-state, instruction, and visual shifts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language-Action Models, Robotic Manipulation, Semantic–Motion Alignment, Out-of-Distribution Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vxzeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Main contribution: SMA-VLA extends VLA-Adapter (a 0.5B Qwen2.5 backbone with a Policy Bridge) using three add-on modules. The central claim is that conditioning task semantics on proprioception, and aligning them with the coarse direction of demonstrated motion, improves generalization when the robot’s initial state, the instruction wording, or the visual scene shifts.

Key innovations:

  1. MCE (Multi-Level Context Enhancement): a softmax-weighted mixture over VLM layers. Its difference from the final layer is injected into every layer as a gated, zero-initialized residual.
  2. PC-SMA (Proprioception-Conditioned Semantic–Motion Alignment):
  • A task query built from pooled language features plus gated proprioception.
  • The query applies a bounded multiplicative modulation to visual tokens.
  • A training-only auxiliary loss pushes a direction head toward the net translational direction of the expert chunk. The loss is a hinge outside a cone whose width adapts to how coherent the motion is.
  1. PFB (Patch-Focused Bridge): action-to-patch attention inside Bridge blocks. It adds a gated residual of focused context minus the uniform patch average.
Critical strengths
  1. Well-motivated problem. The paper argues that VLA policies under-use proprioception when mapping semantics to motion. This is a reasonable hypothesis, and the robot-state column of LIBERO-Plus is a natural testbed for it.
  2. Careful, conservative module design. All three modules use bounded, zero-initialized gates. The policy therefore starts as the original VLA-Adapter computation, which makes the additions low-risk and easy to adopt. The direction loss is confined to coarse translation, and the cone width scales with motion coherence. These are thoughtful details that avoid over-constraining multimodal or curved demonstrations.
  3. No inference-time cost from the auxiliary branch. The direction head and loss are training-only.
  4. Targeted out-of-distribution gains. The +12.3 pp on robot-state shifts and +6.7 pp on language shifts over the direct baseline are sizable. In the ablation (Table 4), they line up with the addition of PC-SMA, which supports the paper’s narrative.
  5. Honest reporting of failures. The paper does not hide that it underperforms on camera and noise shifts or on LIBERO-Pro positional swaps.
  6. Clear writing. The method section is concise and the equations are easy to follow.
Critical weaknesses

W1. Statistical significance of the main results.

  • No seeds, standard deviations, or confidence intervals are reported anywhere.
  • With 500 rollouts per suite at around 96% success, the binomial standard error per suite is about 0.9 pp. Most per-suite differences in -Table 1 are therefore within about one standard error, before even counting seed-to-seed variance.
  • SMA-VLA is slightly worse than VLA-Adapter on LIBERO-Object.
  • On LIBERO-Plus, the 0.46 pp overall margin over Anchor-Align is not meaningful without variance estimates.
  • The claim of “the highest overall success rate” needs to be softened or supported with multiple seeds.

W2. Provenance and comparability of baseline numbers.

  • Please state which baseline numbers were reproduced and which were copied from prior papers, and under what settings.
  • The VLA-Adapter and OpenVLA-OFT LIBERO numbers in Table 1 appear lower than the figures I recall from the original papers. (Reviewer note: verify this against the original papers before including it.) If the authors re-trained these baselines, the training budget and configuration should be specified.
  • In Table 2, baselines are reported to one decimal place, while SMA-VLA is reported to two. This suggests the baseline numbers were taken from other sources. It is then unclear whether they follow the same “train on LIBERO-Spatial only” protocol stated in Section 4.2.

W3. Compute fairness.

  • SMA-VLA is trained for 60K steps and then a further 20K steps of PFB refinement.
  • The ablation states that all variants share the same total step count. The main comparison against VLA-Adapter does not say the same.
  • Please report VLA-Adapter trained for the same 80K steps, as well as parameter counts and inference latency for each added module.

W4. The ablations do not isolate the claimed mechanism.

The paper’s central claim is that semantic–motion alignment drives generalization. However, PC-SMA bundles three separate things: proprioceptive conditioning of the query, token modulation, and the auxiliary direction loss. The following ablations are needed:

  • PC-SMA without the alignment loss (λ_SMA = 0).
  • PC-SMA without proprioception in the query.
  • A fixed cone in place of the coherence-adaptive cone, and no motion-coherence weighting.
  • Sensitivity to λ_SMA.
  • PC-SMA and PFB each without MCE.

Without the first two, the robot-state gain could come simply from feeding proprioception into visual conditioning rather than from alignment. A related question: in Table 4, MCE+PC-SMA and the full model obtain identical robot-state scores (64.86%). Is this a coincidence, or a copy error?

W5. The mechanism is under-supported.

  • The gradients of L_SMA are stopped at the VLM, so the loss trains only the query and proprioception projections and the direction head. - Its effect on actions therefore runs entirely through the bounded token modulation in Eq. 5.
  • The paper gives no evidence that this pathway does what is claimed. Useful evidence would include:
  • Direction-prediction accuracy in- and out-of-distribution.
  • Visualizations of the relevance distribution r.
  • Evidence that the query changes appropriately when the initial state is perturbed.
  • It is also not obvious why aligning with demonstrated directions, which come from in-distribution initial states, should help when starting states are perturbed. A short analysis would strengthen the paper.
  • The ablation shows mixed component effects:
  • PC-SMA reduces camera robustness, from 94.68% to 89.36%.
  • PFB reduces language robustness, from 84.10% to 81.54%.
  • MCE alone slightly reduces language robustness.

These patterns deserve discussion rather than the blanket statement that the modules are “complementary.”

W6. Novelty relative to Anchor-Align.

  • The related work notes that Anchor-Align already “promotes language-action consistency using motion-direction supervision.” This is close to the core of PC-SMA.
  • Anchor-Align appears to be contemporaneous. Even so, since the authors compare against it, they should state precisely what is new: proprioceptive conditioning, the adaptive cone, MCE/PFB, or some combination.
  • More broadly, MCE and PFB resemble existing ideas: layer mixing (M2-VLA, LayerRoute) and focused patch attention (FocusVLA, AVA-VLA). The overall contribution reads as a careful but incremental combination on top of VLA-Adapter.

W7. Limited evaluation scope.

  • All results are in simulation on the LIBERO family.
  • All out-of-distribution evaluation uses a single training suite (Spatial).
  • Generality would be much better supported by any of the following:
  • Out-of-distribution evaluation from other training suites.
  • A second benchmark, such as SimplerEnv, CALVIN, or RoboTwin.
  • A small real-robot study, especially for robot-state shifts, where real proprioceptive noise matters.
  • The 2.6% result on positional swaps suggests the policy still largely memorizes target locations. This sits uneasily with the claim of state-dependent motion reasoning and should be discussed in more depth.

W8. Reproducibility details. The reproducibility statement says Section 3 fully specifies the method, but several values are missing:

  • τ, ρ_s, g_M, g_T, and the bounds on g_P. λ_SMA.
  • How cone angles θ_t and weights w_t are computed from coherence.
  • The static-chunk threshold.
  • Which Bridge blocks are “eligible” for PFB.
  • The learning rate for the second stage.

W9. Qualitative results. Figure 3 shows single successful trials of SMA-VLA alongside failures of VLA-Adapter. Without per-condition statistics or representative failure cases for SMA-VLA, it adds little evidence.

Questions for authors
  1. Were the baselines in Tables 1–3 re-run under your protocol and compute budget? If not, what were their sources and training setups?
  2. Can you report mean ± std over at least three seeds for Tables 1, 2, and 4?
  3. What happens with λ_SMA = 0, and with proprioception removed from the query?
  4. How accurate is the direction head out-of-distribution, and does it correlate with rollout success?
  5. Why does PC-SMA hurt camera-shift performance?
  6. What is the inference latency and parameter overhead relative to VLA-Adapter?
  7. How exactly does PC-SMA differ from Anchor-Align’s motion-direction supervision?
Reviewer_e98WOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents SMA-VLA, an extension of VLA-Adapter for improving robustness under robot-state, instruction, and visual changes. The method combines cross-layer feature aggregation, proprioception-conditioned semantic–motion alignment, and action-conditioned visual attention. Experiments on LIBERO, LIBERO-Plus, and LIBERO-Pro report improvements over VLA-Adapter, particularly under representative robot-state, instruction, and visual shifts.

Critical strengths
  1. The motivation is clear and relevant.
  2. The method reports improvements over VLA-Adapter across several simulation settings, especially under robot-state and instruction changes.
Critical weaknesses
  1. Limited novelty and an unclear core contribution. The method mainly combines incremental modifications to feature aggregation, state conditioning, and visual attention. Motion-direction supervision is also used by Anchor-Align, as acknowledged in the paper. However, the authors do not clearly establish what meaningful insight comes from combining these components. Moreover, PC-SMA introduces state fusion, visual modulation, and direction supervision together, without separating their contributions. It remains unclear whether the improvements come from the proposed alignment mechanism or simply from additional conditioning.
  2. Insufficient experimental evidence. All experiments are conducted within the LIBERO benchmark family, with no real-robot experiments. The overall advantage over Anchor-Align on LIBERO-Plus is only 0.46 percentage points, without repeated-run results or uncertainty estimates. On LIBERO-Pro positional swaps, SMA-VLA achieves only 2.6%, compared with 22.6% for Anchor-Align. These results do not provide sufficiently strong evidence that the proposed combination offers a reliable and broadly applicable improvement in robustness.
Questions for authors
  1. The authors should separate the contributions of state conditioning, visual modulation, and direction supervision, and clarify what the proposed alignment adds beyond existing direction-supervision methods.
  2. Were the baselines trained and evaluated under comparable settings? Please clarify the evaluation protocol and report the mean and standard deviation of success rates across multiple independent runs.
  3. Real-robot comparisons under changes in initial robot state, instruction wording, and visual conditions are needed.
  4. What explains the low positional-swap success rate and the decrease in robustness to camera and noise perturbations when PC-SMA is added to MCE + PFB?
1 comment
Reviewer_e98WOpenReview ↗

Response to Authors

Thank the authors for their response and efforts. While some of my concerns have been addressed, I remain some concerned about the paper’s novelty and experimental evaluation. I will raise my rating to 2, but I believe the paper needs further improvement to meet the bar for acceptance at ICLR.

Reviewer_t5wjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SMA-VLA, an extension of VLA-Adapter intended to improve robustness under changes in robot state, language instructions, and visual conditions. It introduces three components: Multi-Level Context Enhancement (MCE), which combines representations across VLM layers; Proprioception-Conditioned Semantic–Motion Alignment (PC-SMA), which uses a state-conditioned task query for visual modulation and auxiliary motion-direction prediction; and a Patch-Focused Bridge (PFB), which adds action-conditioned visual attention.

The method is evaluated on standard LIBERO and through zero-shot transfer from LIBERO-Spatial to LIBERO-Plus and LIBERO-Pro. The results show improvements over VLA-Adapter, particularly under robot-state and instruction shifts. Component ablations provide evidence that the proposed modules improve performance in the evaluated configuration. However, they do not isolate the mechanisms within PC-SMA or establish an advantage over simpler alternatives using the same information.

Critical strengths
  1. The paper includes meaningful zero-shot robustness evaluation. Policies trained on the original LIBERO-Spatial split are evaluated directly under distribution shifts. Compared with VLA-Adapter, robot-state robustness on LIBERO-Plus improves from 52.60% to 64.86%, and instruction robustness improves from 85.10% to 91.79%. These are substantial improvements over the direct architectural baseline. The corresponding advantages over Anchor-Align—5.76 and 4.59 percentage points—also indicate that the method offers useful benefits on these particular axes.
  2. The component study provides evidence beyond the main benchmark comparison. With MCE fixed, Table 4 evaluates all four combinations of PC-SMA and PFB, and the paper states that the variants use the same total number of optimization steps. This supports the conclusion that the component packages contribute to performance, rather than leaving the entire improvement attributable to an uncontrolled training-budget difference.
Critical weaknesses
  1. The conceptual contribution and necessity of the specific architectural choices remain insufficiently established.

    MCE performs learned cross-layer aggregation with a shared residual; PC-SMA combines state-conditioned feature modulation with auxiliary direction prediction; and PFB introduces an additional action-conditioned visual attention residual. These are plausible inductive biases, but the paper does not establish what distinctive capability or learning advantage they provide over simpler mechanisms.

    The inputs are already available to the baseline, and the auxiliary direction target is derived from the existing expert actions. Consequently, the central question is whether the proposed organization of this information offers a demonstrable advantage over straightforward fusion and attention. No matched comparison is provided against alternatives such as direct state fusion, uniform layer aggregation, or an additional standard cross-attention block. The relationship to Anchor-Align, which already uses motion-direction supervision, also makes it important to isolate the benefit of the proposed state conditioning and continuous direction objective.

    Table 4 does not resolve this issue because PC-SMA changes several things simultaneously: proprioceptive fusion, visual relevance modulation, direction supervision, and the coherence-adaptive cone. Its gains therefore cannot be specifically attributed to “semantic–motion alignment.” The existing study establishes that the package helps, but provides limited evidence explaining why these particular designs are needed.

  2. The scope and consistency of the empirical gains do not yet convincingly justify the full design.

    On standard LIBERO, the average improvement over VLA-Adapter is 95.2% to 96.2%, with several suites already approaching saturation. The significance of further architectural specialization in this setting depends on whether it produces reliable improvements on more discriminating evaluations.

    The OOD results provide some such evidence, but reveal substantial trade-offs. Relative to Anchor-Align, the overall LIBERO-Plus improvement is only 0.46 percentage points, despite the stronger robot-state and language results. Camera and noise robustness decrease by 5.08 and 3.17 percentage points, respectively. On LIBERO-Pro, positional-swap success remains at 2.6%, compared with 22.6% for Anchor-Align. These results support selective robustness improvements more strongly than a general improvement in state-dependent visual grounding.

    The necessity of the complete architecture is also unclear: MCE+PC-SMA achieves 90.17% overall on LIBERO-Plus, compared with 90.76% for the full model. Without repeated-run uncertainty estimates, the stability of this additional gain is difficult to assess. All evaluations further remain within the LIBERO family and use a single 0.5B backbone. The evidence therefore leaves open whether these modifications offer a broadly transferable advantage or mainly benefit this particular model and benchmark configuration.

Questions for authors
  1. Clarification: Were the VLA-Adapter and Anchor-Align results obtained through retraining with the same initialization, data, and optimization budget as SMA-VLA, or were they taken from previous publications? Which comparisons are directly controlled?
  2. Controlled evidence: With state fusion and visual modulation held fixed, how much improvement remains when the direction loss is removed? How does the coherence-adaptive cone compare with a fixed cone or an ordinary cosine-direction loss? These comparisons would directly affect my assessment of the central alignment contribution.
  3. Controlled evidence: How does the proposed design compare with simpler fusion and attention alternatives under comparable capacity and training budgets? Are the improvements over Anchor-Align and the MCE+PC-SMA variant stable across independent training runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.