acceptodds
Under review as a conference paper at ICLR 2027

Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?

Abstract

Vision-Language-Action (VLA) models enable instruction-driven robotic manipulation, but they inherit oversized language backbones from pretrained VLMs whose capacity far exceeds what is needed for short robotic instructions. This raises a basic question: how much of a VLA model is actually necessary for closed-loop control? In this work, we study architectural redundancy in VLA models by using transformer block removal as a controlled intervention. We introduce Drop-Then-Recovery (DTR), an analysis protocol that removes selected blocks from a pretrained VLA model and then fine-tunes the resulting model to measure whether the removed capacity was necessary for downstream control. To make this intervention reliable, we propose GateProbe, a one-shot virtual-gate sensitivity metric that ranks blocks by their contribution to the downstream action loss. Across multiple VLA architectures, manipulation benchmarks and even real-robot industrial scenarios, we find a strong asymmetry in post-removal recoverability: language backbones are highly redundant for standard robotic manipulation tasks, whereas vision and action pathways are substantially less tolerant to removal. On LIBERO, removing half of the LLM blocks even improves OpenVLA-OFT from 95.0% to 98.3% under the same downstream fine-tuning budget, and retaining only two language blocks still recovers baseline-level performance. These results suggest that current VLA benchmarks may exert limited pressure on deep language grounding and compositional instruction understanding, and that future VLA architectures should allocate capacity more deliberately across language, vision, and action components. The code will be released upon acceptance of the paper.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Model Compression, Architectural Redundancy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UhLqOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies architectural redundancy in VLA models via Drop-Then-Recovery (DTR), a method that removes redundant transformer blocks and fine-tunes the smaller model, paired with GateProbe, a one-shot virtual-gate metric that ranks blocks by their contribution to the downstream action loss. It finds that language backbones are highly redundant while vision and action pathways tolerate removal poorly. The paper concludes that current benchmarks under-test language grounding and calls for more capacity-balanced VLA designs.

Critical strengths
  1. Block skipping yields substantial speedups without significant accuracy loss: DTR-30 achieves 2.94× action speedup at 90% success on LIBERO-Goal, and DTR-16 even improves both accuracyand task speedup.

  2. The evaluation is comprehensive and convincing: four architectures, three simulation benchmarks, and real-robot experiments, including OOD robustness analysis.

  3. The paper is clearly written and easy to follow.

Critical weaknesses

When determining the importance of different LLM layers, representative datasets are used, but the source and proportion of representative datasets are not very clear. For example, if the simulator or real-world environment is changed, do we need to use anthor representative dataset for calibration again? How much data is needed as a representative dataset?

Questions for authors
  1. This article mainly argues that the understanding of instructions in embodied tasks is simple, and there is a lot of redundancy in the language model when processing these tasks. However, some datasets such as BEHAVIOR-1K have longer instructions. Do you think it is still possible to use such an aggressive block skipping strategy?

  2. Some representative works are not cited, such as DySL-VLA [1] and Deer-VLA [2]. Especially DySL-VLA uses the input-output similarity of each block to determine the importance of each block. How do you compare it with your method?

  3. Just for discussion, I noticed the robustness degradation under perturbations, while in VLA, the language model not only needs to understand the instructions, but also needs to handle the output of the vision encoder. Is it possible that aggressive layer skipping has caused a decrease in this ability of VLA, resulting in a decrease in accuracy?

[1] Yang Z, Qi Y, Xie T, et al. DySL-VLA: Efficient vision-language-action model inference via dynamic-static layer-skipping for robot manipulation[J]. arXiv preprint arXiv:2602.22896, 2026.

[2] Yue Y, Wang Y, Kang B, et al. Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution[J]. Advances in Neural Information Processing Systems, 2024, 37: 56619-56643.

6 comments
AuthorsOpenReview ↗

Response - 1

W: How are calibration data selected, and is recalibration necessary when the environment changes?

We thank the reviewer for the thoughtful questions regarding calibration data selection and cross-task applicability. We are happy to clarify our experimental setup and provide additional analyses.

1. Calibration data selection and computational overhead.

Under the DTR's setting, target-task training data are already available for recovery fine-tuning. Therefore, GateProbe can directly sample a small calibration subset from the existing training data, without requiring additional data collection or annotation.

For LIBERO, we combine the training data from all four suites and uniformly sample 64 batches × 8 = 512 samples (We use 512 samples for all benchmarks), with approximately the same suite proportions as the combined training set. Each batch requires only one forward and one backward pass, without parameter updates. For π0.5, calibration takes 24.9 seconds on a single H200 GPU (Table 8), representing a small overhead compared with recovery fine-tuning. We have not determined the minimum amount needed, but since the cost grows linearly with the number of batches, a larger calibration set remains inexpensive.

2. Calibration results show substantial consistency across tasks.

To investigate whether recalibration is necessary when the environment changes, we compare GateProbe rankings for π0.5 across nine calibration datasets, including LIBERO, LIBERO-Plus, and seven RoboTwin tasks. RoboTwin differs considerably from LIBERO in its environments, robot configurations, and manipulation tasks.

Calibration DatasetSpearman Correlation with LIBERODrop-9 Kept-Block OverlapDrop-16 Kept Blocks
LIBERO—9/9[0,5]
LIBERO-Plus0.928/9[0,5]
RoboTwin: Beat Block Hammer0.858/9[0,1]
RoboTwin: Click Bell0.787/9[0,2]
RoboTwin: Lift Pot0.847/9[0,5]
RoboTwin: Move Playingcard Away0.918/9[0,2]
RoboTwin: Open Microwave0.797/9[0,1]
RoboTwin: Place Dual Shoes0.767/9[0,1]
RoboTwin: Turn Switch0.828/9[0,2]

Across these datasets, Spearman correlations with LIBERO range from 0.76 to 0.92, and the Drop-9 configurations share 7–8 of the 9 retained blocks with the LIBERO selection. These results indicate that GateProbe importance rankings are largely consistent across different tasks.

Overall, GateProbe has low calibration overhead and exhibits substantial consistency in block selection across the evaluated environments. For new target tasks, especially under aggressive compression, we still recommend recalibration. We will include these experimental details and cross-task analyses in the revised paper.


Q1: Can aggressive layer dropping still work for tasks with longer instructions?

We thank the reviewer for this valuable question. We agree that evaluating tasks with more diverse linguistic expressions is important for understanding language-layer redundancy.

We believe that structural redundancy may still exist in more language-demanding tasks, although the extent of recoverable compression may decrease as language requirements increase. To further investigate this, we analyze how different degrees of instruction reformulation affect recovery performance on LIBERO-Plus.

In this analysis, only the instruction wording is changed, while the scene, initial state, target task, and evaluation seed remain fixed. Within each suite, we divide instructions into three groups based on their degree of reformulation, characterized by instruction length ratio (L), lexical overlap with the original instruction (J), and the proportion of subordinate clauses or indirect expressions (SUB).

π0.5 on LIBERO-Plus language perturbations:

Reformulation MagnitudeLJSUBFull ModelDrop-9Drop-16
Low1.200.520.05798268
Medium1.360.310.15696756
High1.680.080.55554647

As the degree of instruction reformulation increases, the performance difference between Drop-9 and the full model changes from +3 to −2 and −9 percentage points. Drop-16 performs below the full model across all three groups (−11, −13, and −8 points), although the degradation is not monotonic.

These results suggest that the high recoverability of language layers observed on standard LIBERO does not necessarily translate into equally strong performance under greater linguistic variation.

For benchmarks such as BEHAVIOR-1K, we believe the achievable compression level should be evaluated on the target tasks, and DTR provides a framework for conducting such an analysis. We appreciate the reviewer for highlighting this direction, which also helps clarify the scope of our findings.

AuthorsOpenReview ↗

Response - 2

Q2: Comparison with DySL-VLA and DeeR-VLA

We thank the reviewer for pointing out these two relevant works. They are closely related to our study, and we acknowledge that we overlooked discussing them in the original manuscript.

1. DySL-VLA: Comparison of block importance metrics.

DySL-VLA uses the similarity between a block's input and output representations to estimate its importance. This corresponds to the CosSim (Block Influence) metric already evaluated in our Table 3.

Under the same number of removed blocks and recovery fine-tuning settings:

ModelSettingCosSimGateProbe
π0.5Drop-990.294.0
π0.5Drop-1289.492.1
π0.5Drop-1681.992.2
OpenVLA-OFTDrop-1691.897.1
OpenVLA-OFTDrop-3093.595.9

GateProbe achieves higher post-recovery success rates in all five settings, particularly under aggressive compression.

We emphasize that this comparison concerns the similarity-based layer selection criterion used by DySL-VLA, rather than a full reproduction of DySL-VLA. The latter also incorporates dynamic layer skipping, prior-post guidance, and knowledge distillation.

2. DeeR-VLA: Dynamic inference versus structural compression.

DeeR-VLA dynamically determines the inference depth based on the input to avoid unnecessary computation. DySL-VLA similarly improves efficiency by dynamically adjusting computation during inference. In contrast, DTR physically removes transformer blocks before recovery fine-tuning, producing a structurally smaller model and evaluating whether its performance can be recovered.

Therefore, DySL-VLA and DeeR-VLA primarily investigate inference-time computational redundancy, whereas DTR focuses on structural redundancy. These approaches are complementary.

We will discuss both works in Related Work and clarify the connection between DySL-VLA's similarity criterion and our CosSim baseline, together with the experimental comparisons, in the appendix. We appreciate the reviewer for helping us improve the discussion of related work.


Q3: Could robustness degradation be related to the language backbone's role in processing visual features?

We agree that the language backbone in VLAs not only processes instructions but also participates in visual feature processing and cross-modal integration. Therefore, removing language layers may affect more than language understanding.

Our LIBERO-Plus results are consistent with this explanation. For π0.5 Drop-16, the performance decreases relative to the full model under different perturbations are:

PerturbationPerformance Change
Robot−24.2 points
Noise−15.0 points
Camera−12.7 points
Language−9.9 points
Light−4.7 points

Robot perturbations change the robot's initial configuration, while Noise and Camera perturbations primarily affect visual observations. Thus, the performance degradation is not limited to language perturbations.

Our real-robot experiments show a similar pattern. Under the green-light perturbation, the full model achieves 50% success, compared with 35% for Drop-16. When the container is removed, the corresponding success rates are 60% and 40%, respectively. The language instructions remain unchanged in both cases.

These results are consistent with the reviewer's explanation that the removed language layers may also contribute to visual integration and OOD robustness. However, our current experiments do not isolate the precise mechanisms responsible for the degradation.

Dear Reviewer UhLq,

Thank you very much for your encouraging feedback and valuable suggestions regarding our work.

As we approach the end of the discussion period, we wanted to briefly follow up and ask whether our responses have sufficiently clarified the questions you raised.

If there are any outstanding concerns or additional points you would like us to discuss, we would be more than happy to address them.

We truly appreciate your time and effort in reviewing our paper.

Best regards,

The Authors of Submission 3168

Reviewer_9APoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a one-shot metric, GateProbe, to rank transformer blocks. Using their Drop-Then-Recovery (DTR) protocol, the authors physically remove low-ranked transformer blocks from Vision-Language-Action (VLA) models, retrain the remaining layers, and evaluate the resulting task performance. They find that removing up to half of the language layer blocks does not reduce VLA performance. Notably, in the case of OpenVLA-OFT, model performance actually improves after block removal and retraining.

Critical strengths

S1. The proposed GateProbe operates without introducing additional variables into the model. It can be easily computed using the inner product between the loss gradient with respect to the layer's output and the residual difference (hi−hi−1h_i - h_{i-1}), effectively evaluating how the gradient direction aligns with the block's residual contribution. That is, GateProbe measures whether the activation difference aligns with the loss gradient on that layer. If the block's residual contribution aligns orthogonally or near zero with the gradient, it likely indicates that the block is not actively contributing to the model's loss reduction for that particular example. An interesting and simple idea.

Critical weaknesses

Major:

W1. The paper lacks a clear comparison with existing layer-dropping metrics, such as immediate performance degradation. Although the authors mention this briefly in line 056, it remains theoretically unclear how GateProbe differs from prior methods.

W2. The authors do not clarify whether GateProbe is the primary contributor to model compression, or if existing compression methods (e.g., Men et al., 2025; Gromov et al., 2025; Song et al., 2024; He et al., 2024) would achieve similar performance if applied to VLA models prior to policy training.

W3. The paper lacks a comparison with related work, even though several closely related studies exist—including [1], which is particularly relevant to the proposed method.

W4. Regarding the selection of redundant layers, the proposed method seems to simply select later layers. This appears to be a trivial solution, equivalent to extracting activations from an intermediate layer of the VLM for VLA policy training. IN LLM and VLM research the reducney prunign is already established.

[1] https://openreview.net/pdf?id=bFgykQRuHv

Minor:

W1. The authors do not provide the score distribution of GateProbe across different transformer blocks. Without visualizing or reporting how these importance scores are distributed across layers, it is difficult to determine whether the metric produces well-separated, meaningful rankings or if the scores are tightly clustered.

Questions for authors

Q1. Did the authors track the number of training steps? It is important to know this. Did the training steps help to reach the optimal accuracy, or was it the layer removal? All other variables were kept constant, so layer removal was the only variable.

Q2. Did the authors train the language layer, or only the policy head of the VLA? This is important because, if there is a 'drip' [leak] of information from a layer between two language layers, creating a short circuit will probably destroy the distribution of activations coming from the prior layer. This, in turn, gives another message: that the VLA model is probably an overfitted, task-dependent model that doesn't care about language at all.

Q3. What is the random drop baselien, without a random drop baseline, it is hard to evlaue the GateProve baselien. Table 1 shows that removign half of language layers produce better perfomance than the baselien result of the model.

Q4. Does static removal and GetedProbe removal are compare with the same amount of layer removal. It seesm , the GatedProve just perform better than static removal just becase gated removal only removes less than half.

I would suggest that the authors construct cleaner baselines for comparisons. Be precise about confounding effects and randomness. This evaluation needs a causal framework with controlled experiments rather than simply showing slight improvements.

8 comments
AuthorsOpenReview ↗

Response - 1

W1 & W2: How does GateProbe differ from existing methods, and how well do these methods work on VLAs?

1. The methods mentioned by the reviewer have corresponding comparisons in Table 3, with implementation details provided in Appendix E.

The main text mentions these methods in the Introduction (line 056) and Related Work, but does not explicitly connect them to the metrics in Table 3. We will clarify these connections in the revision:

Related WorkCorresponding Method in Our PaperLocation
Men et al., 2025 (Block Influence)CosSimAppendix E, Table 3
Gromov et al., 2025 (angular distance, contiguous removal)CosSim (contig.)Appendix E, Table 3
Song et al., 2024 (performance degradation after removal)PPL (loss change after removing each block)Appendix E, Table 3
He et al., 2024 (attention or MLP removal)CosSim; removal granularity: MHA / MLP / BlockAppendix E, Tables 2, 3, 5, Appendix K

Our main distinction is that we study structural redundancy in VLAs after recovery fine-tuning, rather than immediate performance after layer removal. VLAs typically require downstream fine-tuning to perform robotic tasks effectively. Directly removing complete transformer blocks without recovery can cause severe performance degradation. For example, OpenVLA-OFT drops from 98.0% to 18.0% on LIBERO-Goal after direct block removal (Table 5). Therefore, DTR measures structural redundancy through the success rate after recovery, while GateProbe provides a layer selection metric based on downstream action loss.


2. Differences between GateProbe and existing metrics.

GateProbe applies a virtual scalar α to the residual branch of each block (α = 1 in the original model) and uses the gradient of downstream action loss with respect to α as the block importance score.

  1. Compared with similarity based methods (Men et al., Gromov et al., CKA): These methods measure changes in hidden representations without directly considering whether those changes are useful for action prediction. GateProbe instead measures the sensitivity of downstream action loss to each block.
  2. Compared with parameter based metrics (Taylor, Fisher, etc.): These methods compute importance scores from model parameters and their gradients. GateProbe evaluates the entire residual branch in activation space. Setting α = 0 corresponds to skipping the block's residual computation, so the scoring granularity matches block removal.
  3. Compared with methods based on immediate loss changes after removal (Song et al.): Loss changes immediately after removal may not predict post-recovery performance. GateProbe estimates block sensitivity without individually removing blocks, and we evaluate the selected structures after recovery.

3. Existing metrics can also be applied to VLAs, but layer selection becomes more important under aggressive compression.

We compare different metrics using the same models, number of removed blocks, and recovery training settings.

π0.5 (Tables 3 and 9; batch size 32, 30K steps):

MethodDrop-9 Kept BlocksSRDrop-16 Kept BlocksSR
Full ModelAll 18 blocks91.7--
GateProbe[0–6,8,9]94.0[0,5]92.2
Taylor[0,1,3–9]94.2[5,6]85.2
PPL (Song et al.)[0,1,4,7,8,10,11,13,14]89.6[0,13]87.2
CosSim (Men et al., He et al.)[0–2,12–17]90.2[16,17]81.9
CosSim (contig.) (Gromov et al.)[0,1,11–17]88.0[16,17]81.9

OpenVLA-OFT (batch size 16, 50K steps):

MethodDrop-16 Kept BlocksSRDrop-30 Kept BlocksSR
Full ModelAll 32 blocks95.0--
GateProbe[0–5,7–14,17,18]97.1[1,5]95.9
Taylor[1,6–19,22]96.2[11,14]92.8
PPL (Song et al.)[6,11,12,14,15,19–23,25–28,30,31]93.0[23,31]89.1
CosSim (Men et al., He et al.)[0–2,7–18,31]91.8[0,1]93.5
CosSim (contig.) (Gromov et al.)[0–10,27–31]95.9[0,1]93.5

GateProbe is competitive with existing methods at moderate compression levels and generally performs better in the evaluated aggressive compression settings. For example, on π0.5 Drop-16, GateProbe achieves 92.2%, compared with 85.2% for Taylor and 81.9% for CosSim. On OpenVLA-OFT Drop-30, GateProbe is the only method in the table that exceeds the full model's performance. We note that Fisher selects the same blocks as GateProbe for π0.5 Drop-16 and achieves the same 92.2%.

Importantly, our main findings on language layer redundancy come from Tables 1 and 7, which use fixed removal rules and do not depend on GateProbe. We also acknowledge that the statement in Section 4.4 claiming that gradient based metrics better predict recoverability than static metrics is too broad. We will revise this statement and add the OpenVLA-OFT results.

AuthorsOpenReview ↗

Response - 2

W3: Comparison with related work [1] using adjacent layer CKA

We thank the reviewer for pointing out this related work. We added comparisons using the adjacent layer linear CKA criterion from [1], with the same number of removed blocks and recovery training settings as GateProbe.

ModelSettingMethodKept / Removed BlocksSpatialObjectGoalLongAvg.
OFTDrop-16CKA [1]Remove [5,6,12–14,20–30]98.097.295.693.896.2
OFTDrop-16GateProbeRemove [6,15,16,19–31]98.899.894.495.497.1
OFTDrop-30CKA [1]Keep [0,1]93.893.492.094.893.5
OFTDrop-30GateProbeKeep [1,5]96.899.492.494.895.9
π0.5Drop-9CKA [1]Keep [0,1,11–17]94.694.886.476.488.0
π0.5Drop-9GateProbeKeep [0–6,8,9]97.098.292.088.894.0
π0.5Drop-16CKA [1]Keep [0,1]94.696.289.672.888.3
π0.5Drop-16GateProbeKeep [0,5]96.696.890.684.692.2

CKA performs comparably to GateProbe on OpenVLA-OFT Drop-16 (96.2% vs. 97.1%), but GateProbe outperforms it by 2.4, 6.0, and 3.9 points in the other three settings. We will include these comparisons in the revision.


W4: Does GateProbe simply remove later layers?

We agree that GateProbe tends to remove later language layers, consistent with previous findings on redundancy in LLMs and VLMs.

Our main goal is not to introduce an entirely new pruning framework, but to study how much structural capacity different VLA components need. GateProbe helps identify blocks that can be removed and recovered, but its selections are not equivalent to simply removing the last few layers.

GateProbe does not always select the same layers as a simple depth based rule.

π0.5:

SettingSelection MethodKept BlocksSpatialObjectGoalLongAvg.
Drop-9Remove last 9[0–8]97.294.491.084.491.8
Drop-9GateProbe[0–6,8,9]97.098.292.088.894.0
Drop-16Remove last 16[0,1]94.696.289.672.888.3
Drop-16GateProbe[0,5]96.696.890.684.692.2

Layer depth alone does not always determine which blocks should be retained. For π0.5 Drop-16, replacing the retained block 1 with block 5 improves the average success rate by 3.9 points and LIBERO-Long by 11.8 points. GateProbe also outperforms keeping the first two blocks on OpenVLA-OFT Drop-30 (95.9% vs. 93.5%), although the two approaches perform similarly on Drop-16 (97.1% vs. 97.0%).

GateProbe rankings are consistent with recovery performance.

We remove 9 blocks from π0.5 based on low, medium, and high importance rankings, using the same recovery training settings.

Removed GroupRemoved BlocksSpatialObjectGoalLongAvg.
9 least important blocks[7,10–17]97.098.292.088.894.0
9 blocks with medium importance[2,3,4,7,9,10,11,12,13]93.895.290.684.090.9
9 most important blocks[0–6,8,9]94.695.088.475.288.3

Removing more important blocks leads to larger performance degradation after recovery (94.0 → 90.9 → 88.3), consistent with the GateProbe ranking.


Minor W1: Distribution of GateProbe scores

We compute the GateProbe scores using 512 LIBERO samples and will add visualizations in the revised paper.

π0.5:

Layer012345678
Score0.0290910.0040380.0037940.0036230.0035930.0044090.0044000.0035740.004218
Layer91011121314151617
Score0.0036300.0029490.0035310.0029200.0023510.0015460.0005390.0004040.000000

For π0.5, block 0 has a much higher score, while the middle blocks have relatively similar scores and later blocks have lower scores. Block 17 has a score of zero because the final PaliGemma output is not used for subsequent action prediction.

OpenVLA-OFT:

Layer012345678910
Score0.04620.07690.04900.03460.04520.08470.02870.06490.05370.04170.0440
Layer1112131415161718192021
Score0.03230.03260.03370.03050.02740.02130.02910.03670.02540.02040.0128
Layer22232425262728293031
Score0.02550.01300.01930.01500.01240.01400.00790.01840.00890.0123

OpenVLA-OFT shows a wider score range (0.0079–0.0847), with a general but not strictly monotonic decrease across layers.

AuthorsOpenReview ↗

Response - 3

Q1: Did you track training steps? Are the improvements due to training or layer removal?

Yes. We control the training budget using two settings:

  1. Fixed training steps (Tables 2, 3, 7, etc.): We use the same batch size and number of training steps within each model comparison (π0.5: batch size 32, 30K steps; OpenVLA-OFT: batch size 16, 50K steps). The pruned models require fewer FLOPs per step and therefore use fewer total training FLOPs.
  2. Matched training FLOPs (Table 4): We adjust the batch size and number of training steps to match the baseline's total training FLOPs. This allows the compressed models to process more training examples under the same compute budget, although the number of optimization steps varies across configurations.

Thus, performance recovery in the fixed step comparisons does not rely on additional training steps. The FLOPs matched experiments further evaluate performance under the same total compute budget. We will clarify the training settings for each table in the revision.


Q2: Are the language layers trained, or only the policy head?

The language layers are also trained during recovery, not just the policy head.

  • OpenVLA-OFT: We apply LoRA (rank 32) to all linear layers, including language layers, and fully train the action head.
  • π0.5: We use full fine-tuning of all trainable parameters (Table 14).

We agree that direct layer removal can change the input representations of subsequent layers. In Table 5, directly removing 8 complete blocks from an already fine-tuned OpenVLA-OFT model without recovery reduces LIBERO-Goal success from 98.0% to 18.0%, highlighting the importance of recovery fine-tuning.

Regarding language understanding, we agree that current VLA benchmarks place relatively limited demands on complex language understanding. For example, after removing half of the language blocks and fine-tuning, OpenVLA-OFT and π0.5 still achieve 97.8% and 93.4% on LIBERO-Goal, respectively (Table 1). This is consistent with our observation that standard manipulation tasks may not require the full language backbone capacity inherited from pretrained VLMs. It also motivates further study of VLA structural redundancy and more challenging language evaluations.


Q3: Random layer removal baseline

We add a random removal baseline to test whether recovery depends only on the number of removed blocks or also on their positions.

For OpenVLA-OFT, we randomly sample three different sets of 16 blocks to remove and use the same recovery training settings as GateProbe (batch size 16, 50K steps).

MethodRemoved BlocksSpatialObjectGoalLongAvg.
Full ModelNone97.298.495.688.695.0
Random Run 1[2,3,4,5,7,11,12,13,14,15,17,19,21,24,26,27]87.898.279.493.289.7
Random Run 2[1,2,4,7,8,13,15,16,18,19,20,26,27,28,29,30]97.299.095.891.495.9
Random Run 3[1,2,3,5,6,11,12,13,17,18,20,24,27,28,29,30]92.499.893.689.093.7
Random Mean ± Std16 / 3292.5 ± 4.799.0 ± 0.889.6 ± 8.991.2 ± 2.193.1 ± 3.1
GateProbe[6,15,16,19–31]98.899.894.495.497.1

Random removal can also recover high performance, but the choice of which blocks to remove still matters. GateProbe achieves 97.1%, compared with the random average of 93.1 ± 3.1% and the best random run at 95.9%.


Q4: Do fixed removal rules and GateProbe remove the same number of blocks?

Yes. For π0.5, both fixed Drop-Half and GateProbe Drop-9 remove 9 of 18 blocks. For OpenVLA-OFT, both remove 16 of 32 blocks. The more aggressive comparisons also use the same number of removed blocks (π0.5: 16/18; OpenVLA-OFT: 30/32). The methods differ in which blocks are selected, not in the number of blocks removed or the recovery training settings.


Experimental controls and randomness

We thank the reviewer for this suggestion. We will clarify the controlled variables in each comparison, distinguish fixed step and FLOPs matched settings, and report the variation across random layer removal configurations in the revision.

Reviewer_CRwbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies component-wise redundancy in VLA models through Drop-Then-Recovery (DTR): Transformer blocks are removed from a pretrained model and the remaining model is fine-tuned to measure recoverability. It also proposes GateProbe, a one-shot residual-gate sensitivity score for block selection. The central finding is that language blocks are far more recoverable than vision or action blocks on current manipulation benchmarks.

Critical strengths
  • The goal of identifying redundant parameters in VLA is interesting.

  • The evaluation is relatively thorough, covering multiple VLA architectures, multiple simulation benchmarks, and a real-robot deployment.

Critical weaknesses
  • The paper’s most comprehensive evidence is concentrated on LIBERO, a nearly saturated benchmark with short and largely templated instructions. The observation that many language blocks can be removed and recovered may therefore reflect the limited language demands of this benchmark rather than general redundancy in VLA language backbones. Indeed, the same intervention causes clear degradation on LIBERO-Plus, RoboTwin-Hard, and real-world OOD settings, suggesting that the removed capacity may serve as a robustness and generalization reserve.

  • Recoverability after downstream fine-tuning does not necessarily imply that the removed blocks were redundant in the pretrained model. The remaining parameters may relearn the evaluated task distribution during recovery,

Questions for authors
  • The paper lacks sufficiently strong comparisons with smaller models trained from scratch.
7 comments
AuthorsOpenReview ↗

Response - 1

W1: Is the observed redundancy primarily due to the limitations of LIBERO?

We thank the reviewer for raising this important concern. We agree that LIBERO's relatively simple instructions and high success rates may amplify the observed redundancy. However, our key finding is not merely that many language layers can be removed, but rather the pronounced asymmetry in recoverability across VLA components and how this recoverability changes with task distributions.

1. Recoverability differs substantially across VLA components, and the high recoverability of language layers is consistent across architectures.

First, under the same task and recovery fine-tuning settings, π0.5 retains 91.0% success with only two language blocks (vs. 91.7% for the baseline), whereas retaining only two vision or action blocks reduces success to 62.4% and 26.2%, respectively (Table 1). OpenVLA-OFT shows a similar asymmetry: removing 44.5% of language parameters achieves 98.3%, while removing only 3.6% of vision parameters reduces success from 95.0% to 83.9%.

These results suggest that LIBERO's simplicity alone cannot fully explain the substantial differences in recoverability across components. To further examine whether high language-layer recoverability depends on a particular architecture, we apply the same Drop-Half Language setting to four different VLAs. All four models recover to their baseline performance (Table 7):

ModelDropped LayersBaseline → After Dropping
OpenVLA-OFT16 / 3295.0 → 98.3
π0.59 / 1891.7 → 93.3
GigaBrain-013 / 2688.0 → 88.0
Lingbot-VLA18 / 3682.8 → 83.7

Thus, our findings go beyond compression results on a single benchmark: they reveal pronounced component-wise recoverability asymmetry on standard manipulation tasks, with consistent language-layer recoverability across different VLA architectures.

2. More challenging evaluations reveal the limits of recoverability.

We further evaluate on LIBERO-Plus, RoboTwin 2.0, and real-robot tasks (Section 6.2):

π0.5BaselineDrop-9Drop-16
LIBERO91.794.092.2
LIBERO-Plus81.477.668.8
RoboTwin 2.0 Easy41.741.136.6
RoboTwin 2.0 Hard28.021.413.4
Real-robot Env 163.365.055.0
Real-robot Env 275.071.766.7

Even on RoboTwin benchmark and real-robot setting, Drop-9 maintains performance close to the baseline. However, performance degradation becomes more pronounced under stronger distribution shifts: Drop-9 decreases by 6.6% on RoboTwin Hard, while Drop-16 decreases by 12.6% on LIBERO-Plus, including a 24.2% drop under Robot perturbations.

These results indicate that language-layer redundancy is not absolute: some capacity can be removed and recovered on standard tasks, but may remain important for robustness under OOD conditions.

3. Implications for VLA architectures and benchmarks.

Our experiments provide two key insights:

  • Architecture insight: Language, vision, and action components exhibit substantially different recoverability, suggesting that current VLA capacity allocation may not be well matched to the demands of standard manipulation tasks.
  • Benchmark insight: Language capacity that appears recoverable on standard tasks may still contribute to OOD robustness, indicating that success rates on standard benchmarks alone may underestimate the importance of this capacity for generalization.

We will further clarify that DTR measures structural redundancy under a given task distribution and recovery fine-tuning budget, rather than claiming that language layers are unnecessary across all tasks. These findings motivate both more balanced VLA architectures and more challenging benchmarks with stronger OOD evaluations.

AuthorsOpenReview ↗

Response - 2

W2: Does the pruned model simply relearn the downstream task through fine-tuning?

We thank the reviewer for raising this concern. We would like to emphasize that, for the VLAs evaluated in this work, even the full models require downstream fine-tuning to perform the target tasks. Therefore, both the full and pruned models must learn the downstream tasks through fine-tuning. The key difference is whether fine-tuning is performed on the complete pretrained model or on a model with some layers removed, under the same training settings.

Even if recovery involves relearning the task, this does not invalidate our central finding. Our goal is to investigate whether a small number of critical layers are sufficient to recover the full model's performance under the same downstream task and training budget. If so, the removed capacity is not essential for recovering performance on the evaluated task distribution. This is also how DTR is defined in the Introduction; we do not claim that the removed layers have no value in the original pretrained model.

A further question is whether such recovery simply reflects the ease of the downstream tasks: could a randomly initialized model of the same size achieve similar performance with the same training? If so, the results might primarily reflect the learnability of the downstream tasks rather than the value of reusing pretrained weights.

To investigate this, we conduct an initialization ablation, keeping the reduced architecture and downstream training settings fixed (batch size 32, 30K steps) while varying only the initialization:

Initialization (π0.5 / LIBERO)SpatialObjectGoalLongAvg
Pruned from pretrained model (DTR), Drop-997.098.292.088.894.0
Randomly initialized remaining LLM backbone, Drop-994.294.083.066.684.5
Fully random initialization, Drop-934.060.639.418.438.1
Pruned from pretrained model (DTR), Drop-1696.696.890.684.692.2
Randomly initialized remaining LLM backbone, Drop-1691.292.481.459.681.2

The results show that inherited pretrained weights remain highly valuable even under the same downstream training budget. With only two language blocks retained (Drop-16), DTR outperforms random initialization of the remaining language backbone by 11.0 points. For Drop-9, the fully randomly initialized model achieves only 38.1%, far below DTR's 94.0%.

Therefore, our findings have two implications: even if recovery involves relearning, a small number of critical layers remain sufficient to recover full-model performance on the evaluated tasks. Moreover, the initialization ablation shows that this recovery cannot be explained solely by learning from downstream data from scratch, but substantially benefits from inherited pretrained weights.


Q: How does DTR compare with training a smaller model from scratch?

We thank the reviewer for this suggestion. The initialization ablation study presented in W2 directly addresses this question. Under the same reduced architecture, downstream data, and training budget, DTR substantially outperforms randomly initialized smaller models. For example, when dropping 9 out of 18 language layers, DTR achieves 94.0%, compared with 84.5% when only the remaining LLM backbone is randomly reinitialized and 38.1% when the entire model is randomly initialized. The same trend holds under more aggressive layer removal.

Furthermore, not only do pretrained weights matter, but the choice of which layers to retain is also important. Even when retaining only two language layers, GateProbe selects layers [0, 5] and achieves 92.2%, whereas retaining the last two layers [16, 17] yields only 81.9%. These results further demonstrate that carefully selecting and reusing critical layers from a pretrained model can substantially improve the recovery performance of the compressed model.

Dear Reviewer CRwb,

Thank you again for your thoughtful comments and for taking the time to consider our responses.

As the discussion period approaches its conclusion, we wanted to kindly follow up and ask whether our additional analyses and clarifications have addressed your main concerns.

If there are any remaining questions or points you would like us to clarify further, please feel free to let us know. We would greatly appreciate any additional feedback while there is still time for discussion.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 3168

Comments on the paper

1 comment

Dear Reviewer,

Thank you again for your thoughtful feedback.

With less than 20 hours remaining in the discussion period, we wanted to gently ask whether our responses and revised manuscript have addressed your main concerns.

We would greatly appreciate any remaining feedback and look forward to hearing from you.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.