acceptodds
Withdrawn from ICLR 2027

Dense to MoE Adaptation for Compact Vision Language Action Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision language action (VLA) policies continue to grow in parameter count, making deployment on resource-constrained robot platforms difficult. The central goal is to reduce the number of LLM-side parameters retained in the deployed policy while preserving downstream task performance. Our approach, AdaDE, adapts selected dense feed forward blocks into mixture of experts (MoE) layers and derives expert retention masks from router statistics during fine tuning. The Dense2MoE conversion preserves the original dense FFN function at initialization, so expert deactivation can start without a separate recovery stage. Instead of using a fixed shutdown rule, expert masks are updated dynamically from router usage statistics, with staged training and expert protection to avoid early collapse. With 40% of the LLM parameters deactivated, AdaDE retains 95.7% average success in LIBERO and 42.0% average success across all 50 RobotWin2.0 tasks. These results suggest that dense to MoE adaptation with dynamic expert deactivation is a practical direction for reducing active VLA model size without severe performance loss.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action policies, robot learning, model compression, mixture of experts, structured sparsity, efficient robotics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rXksOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper applies a dense-to-MoE adaptation approach to vision-language-action (VLA) models. The experimental results demonstrate that this approach can reduce FLOPs and memory usage during inference while largely preserving task performance.

Critical strengths

The paper introduces dense-to-MoE adaptation to VLA models, demonstrating that the number of active parameters can be reduced with only a minor degradation in task performance.

Critical weaknesses
  1. Limited novelty. The techniques employed in this paper have already been widely explored in the LLM literature. The paper primarily applies existing methods to VLA models without introducing substantial methodological innovations.

  2. Lack of robotics-specific contributions. Neither the proposed method nor the experimental analysis provides significant insights specific to robotics. It remains unclear what new understanding or practical benefits this work offers beyond applying existing LLM compression techniques to VLA models.

  3. Increased inference latency. Although the proposed method reduces FLOPs and memory usage, it actually increases inference latency.

Questions for authors

No.

Reviewer_nQ1YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces AdaDE, which is a framework that distills experts from FFN layers from the LLM part of a pretrained VLA to make it possible to deactivate specific paths during training/evaluation. This should reduce parameter and memory requirements when running the model. In the experiments Pi 0.5 with 40 or 50 percent reduced LLM parameters is able to achieve almost the same performance on LIBERO as the VLA without reduction, but looses 4-8% on the RoboTwin 2.0 benchmark. The overall method is also evaluated on the real robot where performance only drops from 63% to 61% and uses 1.6GB of VRAM less (9.16 to 7.48).

Critical strengths
  • Method is evaluated on real robot tasks
  • Ablations show the benefit of the different introduced parts of the method
  • Experiments of other compact VLA variants show a clear comparison of AdaDE and these methods
Critical weaknesses
  • There is no compression baseline for Pi 0.5 and for OpenVLA it is only evaluated for LIBERO spatial
  • It is not clear why only the 40% and 50% setting was chosen to evaluate AdaDE and not also more fine-grained steps.
  • The only model which is adapted is Pi 0.5 which has a big LLM part in its VLA of over 50%. OpenVLA is only used for the compression experiments and only evaluated on LIBERO spatial. Therefore it is not clear what would happen with models that have smaller or bigger LLM parts and it is not clear how the overall parameter size influences the model performance when comparing from 1B to 4-5B parameters
  • Table 2 shows different compression schemes on an OpenVLA model, which displays that using AdaDE at 30% drops performance by 2 percent, whereas using 8-bit quantization keeps performance the same, but uses 3.2 GB of memory less. Does that mean using quantization is more efficient to reduce memory usage and keep performance compared to AdaDE?
  • The gains from AdaDE seem minimal. The real robot experiments for example show a reduction of 2 percent success rate for the 40% AdaDE model which is fine, but it only buys 1.6GB memory space (from 9.16 to 7.48) and the latency of the model gets worse.
  • The experiments do not include different seeds
  • The method does not read like an MoE, because there is no conditional computation happening. Could you clarify this? Also how would the novelty then hold against structured pruning and the MLPMoE (Novikov, 2025)?

** Minor Weaknesses **

Evaluation of a method is not a contribution but required to establish that the proposed method is actually working None of the references are working (for figures, tables, equations and citations)
Sentence in Line 45 introduces 3 terms which are not defined beforehand and therefore makes the sentence difficult to understand for someone who is not familiar with Line 48-49 is missing a reference to support this claim It is RoboTwin 2.0 not RobotWin 2.0 Empty Section 3 which directly starts with 3.1 Figure 2 is too small and the text is almost not readable, also the caption is overlapping with the main text Line 215 does not provide a appendix reference The paper layout is not optimized and shows many white spaces, which makes an overlap of figure captions and main text not necessary if optimized correctly

Questions for authors
  • Could you provide at least 3 different seeded models with success rates and standard deviation?
  • The claim is that the FFN blocks are on the LLM side, but it always then says that is covers the language-model and action-expert, what does this mean exactly? Figure 2 shows only LLM related FFN learned experts. How are the parameters counted?
  • Libero eval does not include Long/10. Why?
  • Please consider every weakness also a question that needs an answer
Reviewer_xhCUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to reduce the number of LLM-side parameters in a deployed vision-language-action (VLA) policy while keeping task performance. The method has two parts. (i) Dense2MoE: each selected SwiGLU FFN is split along its intermediate width into E = 10 contiguous channel slices ("experts"); a linear router is initialised to zero, so every expert gets weight 1/E, and with the factor N_r in Eq. (5) the converted block exactly reproduces the dense FFN at initialisation. (ii) AdaDE: during fine-tuning, an EMA of each expert's average router probability is tracked; after a 1k-step warmup, every 100 steps experts whose EMA score falls below τ_deact = 0.08 are switched off permanently, subject to a per-layer minimum of 5 of 10 active experts, a per-update removal budget, and protection of the top 25% experts by a Wanda-style score. An auxiliary loss keeps the weakest router probability above a floor. Masks are frozen at 15k of 40k steps, and inactive slices are physically removed for deployment. All retained experts are evaluated for every token (soft weighted sum, no top-k dispatch).

On π0.5 (PaliGemma Gemma-2B + Gemma-300M action expert), 36 LLM-side FFN modules are pruned. With "40%" / "50%" LLM deactivation, LIBERO Goal/Spatial/Object goes from 93.6/98.6/98.2 (base) to 92.2/97.4/97.6 and 91.6/96.8/97.4; the RobotWin2.0 50-task average goes from 46.7 to 42.0 and 38.1; total parameters go from 3.618B to 2.841B / 2.619B; and peak VRAM at batch size 1 goes from 7.12 GB to 5.69 / 5.25 GB. On OpenVLA, AdaDE is compared with LLM-Pruner, FLAP, quantization and unstructured pruning on LIBERO Spatial. A 7-task real-robot study reports 63.2% (base) vs 61.4% (40%) average success. Ablations compare against fixed-schedule removal, removal without Wanda protection, and different update intervals.

Critical strengths
  • A simple, exactly function-preserving entry point into an expert space, with broad evaluation. The width-partition construction with a zero-initialised router is easy to implement, and the 0% setting confirms it does not change behaviour (LIBERO 94.0/98.4/98.2 vs 93.6/98.6/98.2; RobotWin 46.4 vs 46.7). The evaluation covers all 50 RobotWin2.0 tasks with per-task results (Table 7), a second backbone (OpenVLA), memory measurements, and a real-robot study. This makes the paper's trade-offs visible rather than hidden.
  • A useful empirical observation about where VLA redundancy sits. The paper reports that the LLM-side FFNs tolerate structured removal much better than the vision encoder or the action module, and Figure 3(b) shows that, at a similar total ratio, the split between language-heavy and action-heavy modules changes the outcome. The ablation in Table 3 (fixed removal 88.0 vs AdaDE 95.0 vs no Wanda protection 90.0 at 50%) suggests that which experts are removed matters. This is relevant to anyone compressing VLA policies.
Critical weaknesses
  • I could not find clear evidence that the method reduces model size "without severe performance loss" (abstract), especially on the harder and longer-horizon settings. Observations from the paper:

    • LIBERO-Long (LIBERO-10), the longest-horizon LIBERO suite and the one where errors compound across sub-tasks, is not reported anywhere. The abstract's "95.7% average success in LIBERO" is the mean of Goal, Spatial and Object only ((92.2 + 97.4 + 97.6) / 3); the text does not state that one of the four suites is left out.
    • The three reported LIBERO suites are near saturation for the base policy (93.6–98.6), so they leave little room to show degradation. All ablations (Table 2(a), Table 3, Figure 3) use Spatial only, where the base policy scores 98.6.
    • On the harder benchmark, RobotWin2.0, the 50-task average drops from 46.7 to 42.0 (−4.7 points, about −10% relative) at 40% and to 38.1 (−8.6 points, about −18% relative) at 50%. The main-text table shows ten tasks selected for high dense-policy success, whose average drops by only 3.3 points (81.3 → 78.0). Several individual tasks drop much more, e.g. press_stapler 44 → 24 and rotate_qrcode 56 → 44 at 40%, and place_can_basket 50 → 32 and turn_switch 24 → 10 at 50% (Table 7).
    • All results appear to come from a single run, without seeds or confidence intervals. So it is not possible to tell which differences are within evaluation noise (e.g. open_laptop changes from 80 to 88 at 0%, where the function is unchanged).

    Given these observations, is it the case that 40–50% LLM-side deactivation preserves performance, or mainly that it preserves performance on near-saturated suites? How does the method behave on LIBERO-Long?

  • I could not find clear evidence of a practical deployment advantage, either in speed or in size compared with simpler alternatives. The introduction motivates the work with "limited compute, memory, and power budgets", and the conclusion states that AdaDE substantially reduces "active model size and computation". Observations from the paper:

    • Measured latency increases: 212 ms per forward pass for the dense policy vs 236 ms for AdaDE at 50% (Section 7). The FLOPs reduction (4.38 → 3.18 / 2.90 TFLOPs) is therefore not converted into wall-clock gains. On the real robot, the average execution time per task also increases from 46.0 s to 54.3 s (Table 6).
    • "40% of the LLM parameters" appears to refer to the LLM-side FFN weights only. In Table 1(b), the LLM column (2.043B) drops to 1.266B (−38%) and 1.044B (−49%), while the total model drops from 3.618B to 2.841B (−21%) and 2.619B (−28%). Peak VRAM drops by 1.25× and 1.36×.
    • In Table 2(a), 8-bit quantization of OpenVLA uses 7.9 GB at 84.6% Spatial success. AdaDE (20%) uses 12.4 GB at 84.3%, and AdaDE (30%) uses 11.1 GB at 82.0%. LLM-Pruner and FLAP, after SFT, reach 84.0% and 82.6% at similar parameter counts. AdaDE itself includes 40k steps of fine-tuning, so the "no separate recovery stage" comparison is not at equal training budget. Combining AdaDE with quantization is not evaluated.
    • Table 1(a) lists Efficient VLA at 1.48 TFLOPs with 91.6/95.8/97.8, compared with 2.921 TFLOPs and 91.6/96.8/97.4 for AdaDE (50%).
Questions for authors
  • Could the authors report LIBERO-Long for the dense π0.5, AdaDE (0%), AdaDE (40%) and AdaDE (50%), and give mean ± std over several seeds for the main LIBERO and RobotWin2.0 numbers? Please also state the source of the baseline numbers in Table 1(a). The π0.5 Goal value of 93.6 seems lower than what I recall from public reports, and I would like to confirm the setting.
  • With M_min = 5 of 10 experts per module (Table 5), no module can lose more than about half of its FFN width. The 50% setting removes 0.999B of the 2.043B LLM-side parameters (48.9%, Table 1(b)), which suggests that nearly every module ends at the same minimum. In that case, the layer-wise adaptivity would not differ from a uniform allocation. Is this correct? How does AdaDE compare with group-wise structured pruning of the same 10 channel slices (e.g. a Wanda or activation score per slice, no router), trained with the same 40k-step schedule? This would show whether the router and the MoE parameterisation are needed.
  • Is there any batch size or kernel implementation at which AdaDE is faster end to end than the dense policy? Since retained experts are all evaluated and their weights are fixed after training, could the compact checkpoint be merged back into a narrower dense FFN with a per-token gate, and what latency would that give?

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer xhCU

We thank Reviewer xhCU for the careful review. The additional results and corresponding changes are marked in blue in the revised PDF.

1. Long-horizon evaluation and variance

We evaluated the dense π0.5\pi_{0.5} baseline and AdaDE using three independent seeds under the same protocol.

The dense baseline obtains 91.8±0.791.8 \pm 0.7 on LIBERO-Long and 95.5±0.195.5 \pm 0.1 averaged over the four LIBERO suites. AdaDE obtains 91.7±0.591.7 \pm 0.5, 90.9±0.690.9 \pm 0.6, and 90.3±0.890.3 \pm 0.8 on LIBERO-Long at 0%, 40%, and 50% deactivation, with four-suite averages of 95.5±0.395.5 \pm 0.3, 94.6±0.194.6 \pm 0.1, and 94.1±0.294.1 \pm 0.2, respectively.

On all 50 RobotWin2.0 tasks, the corresponding averages are 46.59±0.6946.59 \pm 0.69, 42.37±0.4442.37 \pm 0.44, and 38.51±0.4838.51 \pm 0.48. Thus, degradation is larger on RobotWin2.0 than on the near-saturated LIBERO suites. We now describe the result as competitive performance rather than uniform preservation.

The π0.5\pi_{0.5} baseline uses the public pretrained checkpoint and official codebase, evaluated under our fixed protocol.

2. Deactivation ratio and adaptive allocation

The 40% and 50% settings denote approximate deactivation ratios within selected LLM-side FFN channel groups. Table 1(b) reports the resulting counts: LLM-side parameters decrease from 2.043B to 1.266B / 1.044B, and total parameters from 3.618B to 2.841B / 2.619B.

Mmin⁡=5M_{\min}=5 is a per-layer lower bound, not a uniform mask. Layers can remove different experts at different times according to router statistics, Wanda protection, and the update schedule.

We also evaluated a no-router baseline with the same ten channel slices and 40k-step budget. At 40%, Static Slice Pruning + Wanda obtains 92.5±1.792.5 \pm 1.7 on Spatial and 82.7±1.682.7 \pm 1.6 on Long, versus 97.5±0.297.5 \pm 0.2 and 90.9±0.690.9 \pm 0.6 for AdaDE. At 50%, it obtains 90.9±0.490.9 \pm 0.4 and 81.1±2.081.1 \pm 2.0, versus 96.9±1.096.9 \pm 1.0 and 90.3±0.890.3 \pm 0.8. This supports the benefit of adaptive routing beyond fixed channel partitioning.

3. Compact implementation and latency

The original multi-expert implementation introduces routing, aggregation, and small-matrix overhead. We therefore pack retained slices before deployment into a narrower gated FFN while preserving token-dependent router weighting.

Under a unified batch-size-one LIBERO protocol, the dense baseline takes 293.20 ms. The packed implementations at 20%, 30%, 40%, and 50% deactivation take 273.92, 262.39, 251.45, and 244.27 ms, respectively. We do not claim that the same speedup holds across all hardware or kernel implementations.

4. Other baselines

We agree that 8-bit quantization has a stronger memory-performance trade-off in the reported OpenVLA setting; we do not claim universal dominance.

LLM-Pruner and FLAP use additional SFT, so these are not equal-training-budget comparisons and are not used to claim optimization-budget superiority.

Efficient VLA targets a different efficiency axis; our claim is adaptive structured reduction of the deployed LLM-side FFN footprint while retaining competitive manipulation performance.

We thank the reviewer again for helping us clarify these points.

AuthorsOpenReview ↗

Response to Reviewer nQ1Y

We thank Reviewer nQ1Y for the detailed review. The additional experiments and corresponding clarifications are marked in blue in the revised PDF.

1. Additional evaluation and seeds

We agree that the original submission lacked multi-seed results and LIBERO-Long evaluation. We therefore evaluated the dense π0.5\pi_{0.5} baseline and AdaDE using three independent seeds.

The dense baseline obtains 91.8±0.791.8 \pm 0.7 on LIBERO-Long and 95.5±0.195.5 \pm 0.1 averaged over the four LIBERO suites. AdaDE obtains 91.7±0.591.7 \pm 0.5, 90.9±0.690.9 \pm 0.6, and 90.3±0.890.3 \pm 0.8 on LIBERO-Long at 0%, 40%, and 50% deactivation, respectively. The corresponding four-suite averages are 95.5±0.395.5 \pm 0.3, 94.6±0.194.6 \pm 0.1, and 94.1±0.294.1 \pm 0.2. These results are reported in Table 2.

We also added 20% and 30% compression settings. On OpenVLA, AdaDE obtains 84.3% and 82.0% LIBERO Spatial success at 20% and 30% compression, with peak VRAM of 12.4 GB and 11.1 GB, respectively.

We agree that the present study focuses primarily on π0.5\pi_{0.5}. Evaluating additional VLA architectures with different language/action parameter ratios is an important direction for future work.

2. Parameter accounting

In the main π0.5\pi_{0.5} experiments, AdaDE is applied to both language-model and action-expert FFN blocks. The LLM column in Table 1(b) includes these FFN parameters. The Flow column contains the remaining non-FFN flow/action-generation parameters and is unchanged.

The 40% and 50% labels refer to the approximate deactivation ratio within the selected LLM-side FFN channel groups; Table 1(b) reports the resulting actual parameter counts.

3. Quantization and compression baselines

We agree that 8-bit quantization provides a stronger memory-performance trade-off than AdaDE in the reported OpenVLA setting. We do not claim that AdaDE universally dominates quantization. The two methods operate on different compression axes and may be complementary.

We also added a no-router structured-pruning baseline using the same ten channel slices and the same 40k-step training budget. At 40%, Static Slice Pruning + Wanda obtains 92.5±1.792.5 \pm 1.7 on Spatial and 82.7±1.682.7 \pm 1.6 on Long, compared with 97.5±0.297.5 \pm 0.2 and 90.9±0.690.9 \pm 0.6 for AdaDE. At 50%, it obtains 90.9±0.490.9 \pm 0.4 and 81.1±2.081.1 \pm 2.0, compared with 96.9±1.096.9 \pm 1.0 and 90.3±0.890.3 \pm 0.8. This indicates that adaptive routing provides benefits beyond fixed structured pruning.

4. Clarification of the MoE formulation and novelty

We agree that AdaDE is not a conventional top-kk MoE with conditional expert dispatch. Dense2MoE partitions each FFN intermediate dimension into structured expert slices, and the router produces a soft weighted sum over the retained slices. During training, AdaDE dynamically removes slices according to router statistics and Wanda protection; after deployment, inactive slices are physically removed.

Thus, our contribution is not sparse top-kk execution. Instead, it is a function-preserving dense-to-expert reparameterization together with adaptive expert deactivation for VLA compression. The no-router comparison further shows that learned routing and adaptive removal outperform fixed slice pruning under the same training budget.

In contrast, MLPMoE primarily uses expert decomposition for model scaling or specialization, whereas AdaDE focuses on reducing the deployed footprint of pretrained VLA policies.

5. Presentation and scope

We corrected the cross-references, section numbering, appendix references, figure readability/layout, and the RoboTwin 2.0 naming issue in the revised PDF. We also added the previously omitted LIBERO-Long results and clarified that the method primarily targets LLM-side FFN compression.

We appreciate the reviewer’s comments, which helped us clarify the scope and limitations of the method.

AuthorsOpenReview ↗

Response to Reviewer rXks

We thank Reviewer rXks for the clear and important concerns. We agree that the main technical components are related to existing dense-to-MoE conversion, structured pruning, and expert-selection methods. Our intended contribution is not a new generic pruning primitive, but a VLA-oriented adaptation framework that combines function-preserving Dense2MoE conversion with adaptive deactivation and deployment analysis. The corresponding clarifications are marked in blue in the revised PDF.

1. Novelty and robotics-specific contribution

AdaDE targets a setting that differs from standard LLM compression: the compressed model must preserve closed-loop robotic manipulation behavior rather than only language-model loss or static prediction accuracy.

Our robotics-specific findings are:

  • language-model and action-expert FFNs tolerate deactivation differently from the vision and flow components;
  • action-heavy modules are more sensitive to aggressive removal;
  • fixed global removal is less reliable than layer-wise removal based on router statistics;
  • performance degradation is more pronounced on RobotWin2.0 than on the near-saturated LIBERO suites;
  • the learned compact structure transfers to physical robot tasks.

The no-router baseline further shows that the result is not explained solely by partitioning FFN channels into fixed groups. Under the same 40k-step budget, Static Slice Pruning + Wanda obtains 92.5±1.792.5 \pm 1.7 on Spatial and 82.7±1.682.7 \pm 1.6 on Long, while AdaDE obtains 97.5±0.297.5 \pm 0.2 and 90.9±0.690.9 \pm 0.6, respectively. At 50%, the corresponding results are 90.9±0.490.9 \pm 0.4 and 81.1±2.081.1 \pm 2.0 for Static Slice Pruning + Wanda, versus 96.9±1.096.9 \pm 1.0 and 90.3±0.890.3 \pm 0.8 for AdaDE.

2. Inference latency

We agree that the original multi-expert implementation can increase latency because of routing, aggregation, and multiple small expert projections. The original latency measurement reflected this implementation.

We additionally evaluated a packed retained-channel implementation. Retained slices are packed before deployment into a narrower gated FFN, while token-dependent router weights are broadcast over their corresponding channels. Under a unified batch-size-one LIBERO protocol, the dense baseline takes 293.20 ms, while the 20%, 30%, 40%, and 50% compact settings take 273.92, 262.39, 251.45, and 244.27 ms, respectively.

Thus, AdaDE does not guarantee latency reduction with the original multi-expert execution path, but the compact deployment implementation can reduce measured end-to-end latency while also reducing peak memory and deployed parameter footprint. We have revised the manuscript to state this trade-off more precisely.

We appreciate the reviewer’s feedback, which helped us better distinguish the general compression components from the VLA-specific findings and deployment implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.