acceptodds
Under review as a conference paper at ICLR 2027

Used Does Not Mean Needed: Self-Attention in Interleaved Flow-Matching Action Experts

Abstract

Does a performance drop after deleting self-attention establish that an action expert needs it? We study flow-matching vision-language-action models whose action experts interleave separate cross-attention (CA) and self-attention (SA) modules. With CA retained, we distinguish deleting SA from a fixed dense-adapted policy from adapting a policy with SA disabled throughout. In GR00T N1.7 on LIBERO-10, dense inference, post-hoc deletion, and No-SA adaptation achieve 93.27%, 85.33%, and 93.60% success. In SmolVLA at two integration steps, their four-suite means are 76.40%, 23.84%, and 88.38% under matched adaptation configurations that differ only in the SA switch. Interventions in the fixed dense policies further show that dependence is conditional: early SA changes from a negative to a positive observed contribution when late SA is retained. This characterizes learned reliance without identifying the mechanism of successful adaptation. No-SA deployment reduces measured latency by 16.97% and 23.40% under the two models' respective timing protocols. Together, these results show that post-hoc dependence does not establish downstream architectural necessity in the studied CA–SA action experts.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Flow Matching, Action Experts, Self-Attention Ablation, Cross-Attention, Downstream Adaptation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CRQWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether self-attention is needed in VLA action expert by ablating the attention layers during training and inference time, and find that self-attention may not be necessary on some tasks.

Critical strengths

The paper studies the necessity of self-attention in action experts

Critical weaknesses
  1. The core research question whether post-hoc ablation loss implies a module must be retained during training has been well studied in model pruning, which the paper also cites. Re-confirming the conclusion for SA in two VLA action experts is useful but delivers very limited new information and novelty.
  2. The success rates on a few LIBERO tasks where some saturated at 100% show these tasks are solvable even without SA. It does not show SA is unnecessary for broader tasks, larger datasets, or settings that need a more expressive expert. Stronger benchmarks, and a well-tuned baseline are needed before the general conclusion is drawn. Additionally, the dense SmolVLA baseline (76.4% mean) seems to be below publicly reported results around 80%-90%. Without a faithful reproduction, the conclusion can be inconvincing.
Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer CRQW

Thank you. We revised the paper around a concrete architectural finding: matched downstream adaptation produces effective action experts with designated SA branches bypassed and cross-attention retained.

1. Contribution beyond the pruning principle. Pruning provides the methodological foundation (Section 7). Our VLA-specific contribution combines reproducible performance gains with evidence about retained computation. At ten integration steps and ten executed actions (S10/H10), three independent SmolVLA training seeds yield a 13.58-point No-SA gain, while deleting SA from the corresponding Dense checkpoints yields 1.15% success (Table 1). Freezing the entire VLM preserves a 6.62 ± 0.38-point advantage with the expert and external projections trainable (Table 4). Fixed-policy two-step interventions further show that adding early SA reverses its observed contribution when late SA is retained (Section 5.1). These findings characterize both effective simplified experts and conditional SA use in closed-loop control.

2. Stronger Dense reference. We reevaluated lerobot/smolvla_libero alongside our matched models. All use S10/H10 within each evaluation protocol:

EvaluationMatched DenseMatched No-SAPublic Dense
Clean LIBERO78.61 ± 2.0292.19 ± 0.6684.10
LIBERO-Plus12.61 ± 2.5018.93 ± 1.0527.70

Values are success percentages; ± denotes sample SD across three independent training seeds. Public Dense is one published checkpoint. Our matched pair uses batch 16 and a 30k decay horizon; the public configuration records batch 32 and 25k decay, with different input/preprocessing settings. Both use 25k updates. Matched retraining under the public recipe remains untested. Tables 2, 8, and 10 distinguish the controlled architectural comparison from this external reference.

3. Saturation and perturbations. On clean LIBERO-Long, matched No-SA improves success from 56.67% to 83.82% (Appendix A, Table 8). The Plus evaluation covers all 10,030 variants across four suites and seven perturbation categories, extending the matched-recipe gain to these perturbations.

GR00T's 5,037 Long/Object variants give 73.95/63.15/72.23% for Dense/deletion/No-SA, respectively: adaptation recovers 9.07 points over deletion (Table 3). Each suite has one adapted checkpoint pair. We corrected the clean-GR00T deletion attribution: 85.33% came from the public checkpoint (Appendix B). The revised comparison uses matched Dense/deletion weights.

The claim concerns downstream adaptation in these two interleaved architectures. Plus tests broader perturbations within LIBERO's simulator and base tasks. The revised introduction organizes the contribution around effective simplified experts, conditional computation, and measured deployment value.

Reviewer_CRQWOpenReview ↗

Reviewer Responses

We thank the authors for the new experiments.

  1. The reviewer's concern regarding the contribution and novelty has not been resolved. The authors suggest this work is a "VLA-specific contribution that combines reproducible performance gains with evidence about retained computation", which seems to be only empirical, while the methodology is stemmed from existing model pruncing studies.
  2. Furthermore, the applicability to SmolVLA does not suggest conclusions to broad VLA models. The reviewer also expects more insights regarding why no-SA imrpoves performance particularly on LIBERO-Plus, and if the improvements hold for other VLA models in general.

I keep the rating for now.

Reviewer_Nu4aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether the performance drop from removing self-attention in a trained VLA policy really means self-attention is necessary. The authors argue that it does not. They distinguish between deleting self-attention from a fixed dense policy and training a new policy without self-attention during downstream adaptation. The study focuses on flow-matching VLA action experts with separate cross-attention and self-attention blocks, while always keeping cross-attention. The main result is that dense policies may rely on self-attention at inference time, but separately adapted no-self-attention policies can match or exceed dense performance in the tested settings. On GR00T N1.7 for LIBERO-10, removing self-attention from the dense model hurts performance, but a No-SA adapted model recovers it. On SmolVLA, deletion is much more damaging, yet the No-SA adapted version performs even better than the dense baseline. The paper also analyzes which self-attention layers matter and reports inference-latency improvements for No-SA models.

Critical strengths

The paper makes a clear and useful point. Many ablation studies rely on post-hoc deletion, and this paper shows why that can be misleading when the real question is what architecture is needed after adaptation. The deletion-versus-adaptation comparison is the strongest part of the paper. It is simple, easy to interpret, and well supported by the experiments. The paper is also reasonably careful about scope. It does not claim that self-attention is never useful, only that dependence in a trained dense model is not the same as architectural necessity in this setting. The conditional layer analysis is interesting as well. It shows that the effect of keeping or removing self-attention depends on where it is placed, not just how much is kept. Finally, the latency results add practical value beyond the conceptual point.

Critical weaknesses

The biggest weakness is limited evidence across training runs. Most results appear to rely on repeated evaluation seeds rather than multiple independent training runs. The paper is transparent about this, but it still weakens the strength of the main claim. The scope is also narrower than the framing may initially suggest. The conclusions are really about two specific flow-matching VLA architectures with separate cross-attention and self-attention modules, so I would be careful about generalizing beyond that setting. Some key results also depend on a limited set of evaluation choices. For example, the SmolVLA story is centered on two integration steps, and it is not clear how robust the conclusion is across other decoding settings. The paper shows that deletion loss and adaptation success can differ, but the mechanism behind why No-SA adaptation works so well is still not very clear.

Questions for authors

Can you provide multi-seed training results for No-SA adaptation, especially for SmolVLA? How sensitive is the main conclusion to the number of integration steps? Does No-SA also improve training efficiency, not just inference latency? Do you have a stronger explanation for why No-SA adaptation works so well? How much do the conclusions depend on architecture-specific details of the self-attention modules? Did you compare against other reduced architectures besides simply removing self-attention? In the GR00T budget sweeps, how should we interpret the apparent instability across different placement choices?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Nu4a

Thank you. The revised evidence establishes effective simplified action experts through matched downstream adaptation and clarifies their retained computation.

1–2. Independent training and decoding settings. We adapted SmolVLA Dense and No-SA with training seeds 0/1/2, sharing the pretrained initialization and recipe. Table 1 reports success (%) across all four LIBERO suites:

Integration steps / executed actionsDenseSame-Dense deletionNo-SA adaptation
S10/H176.29 ± 2.310.81 ± 0.8187.44 ± 0.41
S10/H1078.61 ± 2.021.15 ± 0.5692.19 ± 0.66

Here ± is sample SD across independent training seeds, after pooling three evaluation repeats per checkpoint. Both horizons use the same checkpoints and a 50-action predicted chunk. No-SA improves every training-seed × suite comparison, with overall gains of 11.16 and 13.58 percentage points, respectively. These ten-step results extend the original S2 evidence; integration-step sensitivity on common checkpoints remains unmeasured (Section 3.1).

3. Training efficiency. Training time and memory savings remain unmeasured; both conditions use 25,000 updates. The measured SmolVLA inference benefit is 23.40% lower action-chunk latency at S2, batch one (Section 6, Table 6).

4. Explanation. Retained CA, FFNs, residual paths, positions, and flow-time conditioning allow action predictions to use shared multimodal context. Adaptation optimizes this computation for control. With the entire VLM frozen, three-seed No-SA retains a 6.62 ± 0.38 pp advantage over matched Dense at S10/H10 (Table 4). The fixed VLM continues supplying features; the expert and external projections, including the state projection, remain trainable. This supports the capability of retained pathways with fixed VLM weights. Optimization and regularization remain hypotheses for the gain over Dense.

5. Architecture dependence. SmolVLA SA accesses both prefix and action tokens; CA preserves a separate prefix route. GR00T Action-DiT SA also connects state tokens to action tokens, so bypassing it removes that route; separate VL-feature SA remains active (Section 2.1). On GR00T Plus Long/Object, Dense/deletion/No-SA achieve 73.95/63.15/72.23%; No-SA recovers 9.07 pp over deletion (Table 3). These results use one adapted pair per suite and characterize downstream adaptation in the two studied architectures.

6. Other reduced configurations. GR00T Uniform-8 retains eight of 64 SA executions. Adapted Uniform-8 achieves 92.27%, alongside Dense 93.27% and adapted No-SA 93.60%, over three evaluation repeats of fixed checkpoints (Appendix C, Table 13). Width and FFN reductions remain untested.

7. Placement variability. Equal budgets produce different closed-loop behavior: Uniform-8 achieves 91.93%, versus 83.40% for one fixed Random-8 mask. Additional random masks reach 88.20–94.13% in a separate batch. This supports conditional dependence on retained positions and identifies competitive alternatives to Uniform-8 (Section 5.2; Appendix D).

Correction: the historical GR00T deletion score, 85.33%, belongs to the public checkpoint. Appendix B corrects its earlier Dense-18k attribution.

Reviewer_M7QJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This is primarily an empirical architecture study paper. The key question is whether a performance drop of flow-matching VLA's action heads after deleting a module indicates that the module is necessary for downstream adaptation. The main results are: For self attention module (SA), deletion can fail while adaptation without this module succeeds. The study focuses on two architectures: GR00T and SmolVLA on LIBERO.

Critical strengths

This paper is quite easy to follow and the empirical observations are quite interesting. In particular, the SmolVLA result shows that post-hoc deletion of the SA (self attention) module in the action head reduces mean success from 76.40% to 23.84%, while No-SA adaptation (adaptation without the self attention module) achieves 88.38% with cross-attention retained. This implies for post-training adaptation there is indeed some redundancies in the model parameters, which suggests potential parameter-efficient adaptation methods.

Critical weaknesses
  1. The experiment setting is too limited: Only simulation results on LIBERO, which is already very saturated. Only considered two VLA architectures, in particular, no \pi-style VLA involved.

  2. All SmolVLA comparisons use only two integration steps and execute only a single action before replanning, which don't match common practices.

  3. The reported seeds repeat evaluation of fixed checkpoints rather than independent runs, which makes the reproducibility and consistency of the reported conclusion uncertain.

  4. No pre-training, or at least mid-training study on the role of self attention. So the implication around "used does not mean needed" is not fully justified.

Questions for authors

See the weakness part.

1 comment
AuthorsOpenReview ↗

Response to Reviewer M7QJ

The added matched adaptations establish effective simplified action experts: SmolVLA No-SA improves LIBERO success by 13.58 percentage points at ten integration steps and ten executed actions. We address your four concerns below.

1. Integration steps and action execution. We evaluated S=10 integration steps with H=1 or H=10 actions executed before replanning, retaining a predicted chunk of 50 actions (Section 3.1, Table 1):

SettingDenseSame-Dense deletionNo-SA adaptation
S10/H176.29 ± 2.310.81 ± 0.8187.44 ± 0.41
S10/H1078.61 ± 2.021.15 ± 0.5692.19 ± 0.66

Values are success percentages, mean ± sample SD across three independent training seeds. The advantage holds with ten consecutive actions between observations.

2. Independent training runs. Dense and No-SA were each adapted with seeds 0, 1, and 2, sharing the pretrained base and adaptation recipe, with the SA switch defining the intervention. Each condition and horizon pools three evaluation repeats into 6,000 episodes per training seed before computing training-seed statistics. No-SA improves every seed-by-suite comparison at both horizons (Appendix A, Table 7).

3. Saturation and broader evaluation. On clean LIBERO-Long, No-SA improves success from 56.67% to 83.82% at S10/H10. We additionally evaluated all 10,030 LIBERO-Plus variants across four suites and seven perturbation categories. Matched Dense/No-SA achieve 12.61 ± 2.50% / 18.93 ± 1.05%; public Dense achieves 27.70% under its different training recipe. GR00T's 5,037 Long/Object variants yield 73.95/63.15/72.23% for Dense/deletion/No-SA, recovering 9.07 points through adaptation relative to deletion (Tables 2–3). Plus extends perturbation coverage within the LIBERO simulator.

4. Architectures and training stage. The revised abstract and Sections 2.1 and 8 focus on downstream adaptation of these two architectures with cross-attention retained. GR00T has one adapted pair per suite. π-style architectures, pretraining, and mid-training with SA disabled remain untested. The supported design result is that adaptation under the intended deployment structure yields capable simplified experts.

Appendix B also corrects the original GR00T attribution: 85.33% deletion used the public checkpoint; 93.27% used Dense-18k. The revised Plus comparison uses matched Dense/deletion weights.

Comments on the paper

1 comment

Dear Reviewer M7QJ and Nu4a,

As the rebuttal discussion period is drawing to a close, we kindly encourage you to review the authors' responses to your previous concerns and provide any further feedback or clarification as appropriate.

Thank you for your time and cooperation.

Best regards,

Your Area Chairs

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.