acceptodds
Under review as a conference paper at ICLR 2027

Exploring What to Preserve for Reasoning and Action in Structured Pruning of Driving VLAs

Abstract

Reasoning vision–language–action (VLA) models for autonomous driving use chain-of-thought (CoT) reasoning to guide action generation, but incur substantial memory and computational costs. We study structured pruning for these models, focusing on preserving both scene understanding and trajectory prediction. Our analysis of Taylor importance in Alpamayo-1.5-10B shows that reasoning and trajectory objectives assign different importance to the same vision–language model (VLM) structures. In the action expert, MLP channels exhibit more stable importance rankings across denoising steps than query heads, and a small fraction of channels accounts for most of the measured importance. Based on these findings, we propose , which combines importance ranks from both objectives for VLM pruning and aggregates stepwise importance to select a fixed subset of expert MLP channels. At 24.0% total parameter reduction, VLM-only pruning with retains 94.0% of the dense model's LingoQA score and achieves the highest closed-loop scene scores among methods at the same budget. Removing 93.75% of expert MLP channels increases total parameter reduction to 39.4%, with nearly unchanged open-loop errors relative to VLM-only pruning and competitive closed-loop performance. The resulting model achieves a end-to-end inference speedup and 36.6% lower peak memory than the dense model, without post-pruning fine-tuning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLAs, Model Compression, Model Pruning, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_phZvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training-free structured pruning of the Alpamayo-1.5-10B driving VLA, which combines an autoregressive VLM producing a Chain-of-Causation with a flow-based action expert. The authors observe that cross-entropy and flow-matching objectives rank VLM heads and MLP channels differently, while action-expert MLP rankings are relatively stable across denoising steps. MoOP therefore retains VLM units that are important to either objective and prunes action-expert MLP channels using importance averaged across denoising steps, without post-pruning fine-tuning. At 24% total parameter reduction, it reports Test500/OOD minADE of 0.950/1.119, LingoQA of 68.8, and Origin150/Hard100 scene scores of 0.828/0.595.

Critical strengths
  1. The dual-objective ablation strongly supports the main VLM design. At the same 24% budget, MoOP substantially outperforms CE-only and FM-only pruning on LingoQA, open-loop errors, and Hard100 scene score (Tables 1–2, pp. 7–8).

  2. The paper demonstrates measured, deployment-relevant savings. A 39.4% parameter reduction yields a 1.372× measured end-to-end speedup and 36.6% lower peak memory without recovery fine-tuning, while aggressive expert MLP pruning has little effect on open-loop errors (Tables 3 and 5, pp. 9–10).

Critical weaknesses
  1. The closed-loop margins over the strongest pruning baseline are small. Improvements of 0.017 on Origin150 and 0.009 on Hard100 are insufficient by themselves to support a strong claim of closed-loop superiority, even though the overall performance is competitive (Table 2, p. 8).

  2. The principal scene score is biased toward the recorded route. Appendix B.4 shows safe alternative trajectories receiving much lower scores than MoOP solely because they deviate from the reference route. Since the reported closed-loop margin is already small, this metric behavior materially weakens the ranking claim (Appendix B.4, pp. 32–34).

Questions for authors
  1. Equation 12 describes off-road rate using binary rollout outcomes. With 200 Hard100 rollouts, a direct mean should change in increments of 0.005; how is the reported value 0.208 obtained? Please state the exact denominator and aggregation procedure (Table 2, p. 8; Equation 12, Appendix B).

  2. At the 93.75% expert-MLP pruning setting, how do random-channel and magnitude-channel selection perform under the same budget? These two controls would clarify whether the gain comes from MoOP's score or primarily from general expert overparameterization (Table 3, p. 9).

  3. Which decisions, if any, were made using the earlier 50-clip calibration set that contained 12 test clips—including the architecture, scoring formulation, unit types, pruning ratios, or hyperparameters? If any final design choice used those clips, can the corresponding selection and final evaluation be repeated without test-set exposure (Appendix A.3.1, p. 14)?

6 comments
AuthorsOpenReview ↗

Response to Reviewer phZv (1/5)

W1: Closed-loop margins and statistical significance

Thank you for the careful examination of our closed-loop results. We added confidence intervals and paired comparisons for all main results to Appendix C.1 (Table 7-13) and results at a higher pruning ratio to Tables 1, 2 and 4 and Appendix C.2 (Table 14-17). We clarified results and our contribution in Abstract (L23-27), Section 1 (L96-99, L111-112) and Section 6.2 (L460-470, L478-479, L487-505) in purple. Table 9,10,14 and 16 are provided below.

To quantify the uncertainty, we now report the 95% CI of every closed-loop metric, all computed with a scene-level cluster bootstrap. The results and the details of the procedure are provided in Table 7-9 in Appendix C.1.

As you noted, MoOP substantially outperforms CE-only and FM-only at the same budget, and this holds when the intervals are taken into account, as MoOP remains significantly better than both over the combined 250 scenes (Table 9 in Appendix C.1), confirming the effectiveness of the dual objective. We agree, however, that the margins over the strongest external baselines are small at the 40% budget on VLM. Over the combined scenes, the differences from Týr-the-Pruner (+0.029 [−0.005, +0.064]) and LLM-Pruner (+0.019 [−0.024, +0.061]) are not significant, so MoOP is comparable to them in closed-loop driving at this budget.

However, the methods separate more clearly when the budget is tightened. When 50% of the units are removed from each VLM layer as shown in Table 14-15, MoOP keeps scene scores of 0.866 on Origin150, while LLM-Pruner falls to 0.634 and Týr-the-Pruner to 0.742. In open-loop prediction, we additionally evaluated all methods on OOD-train, a larger set of 1,271 OOD clips disjoint from our calibration set (Table 10-11 in Appendix C.1). On this set, MoOP achieves a significantly lower minADE@6 than every compressed baseline, by 0.139 m relative to Týr-the-Pruner and 0.146 m relative to FM-only. MoOP therefore retains the dense model's performance on long-tail scenarios better than the other pruning methods.

We attribute these gains to MoOP's preservation of reasoning, as shown by the LingoQA results (Table 4 in Section 6.2). MoOP achieves 68.8% accuracy, close to the 73.2% of the dense model, whereas Týr-the-Pruner and LLM-Pruner reach only 34.2% and 44.8%, indicating that their reasoning has largely collapsed. In summary, MoOP is comparable to the strongest baselines at a 40% VLM pruning ratio, but it degrades less under more aggressive pruning and on long-tail scenarios while retaining the reasoning ability of the dense model.

Thank you for your careful examination again, which helped us present these results more precisely and clarify our contribution. We have revised Tables 1, 2 and 4, Abstract (L23-27), Section 1 (L96-99, L111-112) and Section 6.2 (L460-470, L478-479, L487-505) in purple accordingly.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer phZv (2/5)

W1: Closed-loop margins and statistical significance (continued)

Table 9: Paired differences from MoOP over the combined 250 scenes (95% CI). Scenes are resampled within each set. Differences are signed so that a positive value favors MoOP, that is, MoOP minus the method for the scene score and the method minus MoOP for the off-road and at-fault collision rates.

MethodExpert PruningScene ScoreOffroadAt-fault Coll.
Baseline✗+0.081 [+0.046, +0.116]+0.032 [+0.002, +0.062]+0.006 [−0.014, +0.026]
Wanda✗+0.200 [+0.147, +0.253]−0.004 [−0.042, +0.034]+0.082 [+0.044, +0.122]
LLM-Pruner✗+0.019 [−0.024, +0.061]−0.030 [−0.064, +0.004]+0.018 [−0.010, +0.046]
Týr-the-Pruner✗+0.029 [−0.005, +0.064]+0.022 [−0.010, +0.056]−0.004 [−0.024, +0.016]
FM-only✗+0.047 [+0.012, +0.081]+0.040 [+0.010, +0.070]+0.000 [−0.022, +0.024]
CE-only✗+0.173 [+0.127, +0.218]+0.018 [−0.018, +0.054]+0.030 [+0.004, +0.058]
MoOP (Ours)✓+0.020 [−0.005, +0.046]+0.014 [−0.008, +0.038]−0.004 [−0.022, +0.014]

Table 10: Open-loop prediction at 6.4 s with 95% confidence intervals (m).

MethodExpert PruningPruning RatioTest500 minADE@6 (m) ↓Test500 minFDE@6 (m) ↓OOD-Val262 minADE@6 (m) ↓OOD-Val262 minFDE@6 (m) ↓OOD-train minADE@6 (m) ↓OOD-train minFDE@6 (m) ↓
Baseline✗0%0.842 [0.765, 0.922]2.267 [2.009, 2.540]1.000 [0.847, 1.177]2.496 [2.041, 3.020]0.928 [0.856, 1.007]2.354 [2.144, 2.582]
Wanda✗24%2.975 [2.729, 3.233]8.269 [7.532, 9.038]2.790 [2.461, 3.141]6.884 [5.983, 7.842]2.832 [2.659, 3.013]6.873 [6.418, 7.348]
LLM-Pruner✗24%1.451 [1.312, 1.600]3.889 [3.485, 4.314]1.662 [1.432, 1.916]4.153 [3.525, 4.844]1.597 [1.485, 1.716]3.738 [3.450, 4.046]
Týr-the-Pruner✗24%0.950 [0.862, 1.044]2.570 [2.289, 2.867]1.177 [1.012, 1.366]2.990 [2.504, 3.543]1.178 [1.090, 1.272]2.768 [2.536, 3.014]
FM-only✗24%1.008 [0.917, 1.105]2.720 [2.423, 3.036]1.178 [1.011, 1.371]2.960 [2.475, 3.515]1.184 [1.098, 1.278]2.897 [2.661, 3.150]
CE-only✗24%1.584 [1.433, 1.746]4.500 [4.022, 5.007]1.546 [1.328, 1.792]4.010 [3.364, 4.733]1.505 [1.402, 1.614]3.737 [3.459, 4.031]
MoOP (Ours)✗24%0.950 [0.869, 1.034]2.546 [2.276, 2.832]1.119 [0.958, 1.307]2.851 [2.362, 3.410]1.038 [0.959, 1.124]2.541 [2.320, 2.778]
MoOP (Ours)✓39.4%0.951 [0.871, 1.035]2.539 [2.272, 2.822]1.118 [0.957, 1.304]2.844 [2.357, 3.400]1.039 [0.960, 1.124]2.537 [2.318, 2.772]

Table 14: Closed-loop driving on Origin150 at r = 0.5, with 95% confidence intervals (total pruning ratio is 30%).

MethodExpert PruningPruning RatioScene Score ↑Offroad ↓At-fault Coll. ↓
Baseline✗0%0.750 [0.698, 0.800]0.070 [0.037, 0.110]0.040 [0.017, 0.067]
Wanda✗30%0.350 [0.291, 0.410]0.280 [0.213, 0.350]0.180 [0.123, 0.240]
LLM-Pruner✗30%0.634 [0.570, 0.697]0.073 [0.037, 0.117]0.143 [0.093, 0.200]
Týr-the-Pruner✗30%0.742 [0.682, 0.800]0.073 [0.037, 0.117]0.087 [0.047, 0.130]
FM-only✗30%0.551 [0.494, 0.607]0.103 [0.060, 0.153]0.097 [0.057, 0.143]
CE-only✗30%0.279 [0.239, 0.322]0.170 [0.117, 0.230]0.140 [0.090, 0.197]
MoOP (Ours)✗30%0.866 [0.821, 0.908]0.047 [0.020, 0.080]0.037 [0.013, 0.063]

Table 16: Open-loop prediction at 6.4 s at r = 0.5, with 95% confidence intervals (m). The Baseline is the same as in Table 10.

MethodExpert PruningPruning RatioTest500 minADE@6 (m) ↓Test500 minFDE@6 (m) ↓OOD-Val262 minADE@6 (m) ↓OOD-Val262 minFDE@6 (m) ↓
Baseline✗0%0.842 [0.765, 0.922]2.267 [2.009, 2.540]1.000 [0.847, 1.177]2.496 [2.041, 3.020]
Wanda✗30%2.901 [2.658, 3.157]7.869 [7.158, 8.612]2.761 [2.439, 3.103]6.917 [6.028, 7.861]
LLM-Pruner✗30%2.533 [2.312, 2.766]6.662 [6.040, 7.319]2.628 [2.288, 2.993]6.613 [5.692, 7.600]
Týr-the-Pruner✗30%2.050 [1.854, 2.257]5.339 [4.788, 5.921]2.165 [1.905, 2.443]5.202 [4.525, 5.917]
FM-only✗30%2.822 [2.568, 3.094]8.278 [7.505, 9.102]2.436 [2.111, 2.788]6.753 [5.789, 7.799]
CE-only✗30%5.151 [4.761, 5.560]15.192 [14.014, 16.432]3.899 [3.468, 4.358]11.230 [9.930, 12.603]
MoOP (Ours)✗30%1.237 [1.126, 1.356]3.338 [2.980, 3.720]1.416 [1.217, 1.640]3.616 [3.027, 4.266]
AuthorsOpenReview ↗

Response to Reviewer phZv (3/5)

W2: Effect of the route-dependent scene score

Thank you for examining the appendix closely and for feedback that helps make our claims more rigorous. We added the analysis of alternative routes and the score lost to the 4 m rule to Appendix B.4 (L1800-1830, Table 6) in orange.

Appendix B.4 reports a limitation of the current AlpaSim closed-loop evaluation that we found during our experiments: some scenes admit more than one valid route, but only the recorded route is scored. We included these cases to document a part of the evaluation protocol that should be improved. Following your comment, we examined whether such cases affect the comparison between methods (Table 6 in Appendix B.4).

AlpaSim stops scoring once the ego deviates more than 4 m from the reference route. We count a rollout as taking an alternative route when scoring stopped under this 4 m rule and the ego then continued for at least 10 m without any off-road event or collision. We also estimate the score lost to the rule for every rollout stopped by it, not only for those on alternative routes. The lost score is the additional progress score that the ego accrued along the reference route after scoring stopped, counted until the end of the rollout.

The alternative-route fraction is similar across methods, at 5.0–8.0% on Origin150 (except 12.3% for CE-only) and 9.0–14.0% on Hard100, and its differences between MoOP and Týr-the-Pruner or LLM-Pruner are not significant. The score lost to the 4 m rule does not favor MoOP either. On Origin150, MoOP loses slightly more score to this rule (0.039 and 0.040 with expert pruning) than the Baseline, LLM-Pruner and Týr-the-Pruner (0.022–0.037), and on Hard100 the losses are similar (0.064–0.067 vs. 0.063–0.071). CE-only and FM-only lose more on Origin150 (0.083 and 0.045), but MoOP's margins over them are much larger than these differences. Even if this progress were credited back to every method, MoOP would still achieve the highest scene score among the methods at the same budget on both sets (0.867 on Origin150 and 0.663 on Hard100). The route dependence therefore lowers absolute scene scores to a similar degree across methods and does not account for the differences between them. Regarding the small closed-loop margins themselves, please see our response to W1.

Table 6: Fraction of alternative routes and score lost to the 4 m rule.

MethodOrigin150 Alt-route (%)Origin150 Loss from 4 m ruleHard100 Alt-route (%)Hard100 Loss from 4 m rule
Baseline6.30.03710.50.063
Wanda5.30.0409.00.094
LLM-Pruner5.00.02214.00.067
Týr-the-Pruner7.70.03011.00.071
FM-only6.30.04513.50.064
CE-only12.30.08311.50.139
MoOP5.70.03912.50.067
MoOP w/ Expert Pruning8.00.04013.50.064
Reviewer_6ExtOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies structured pruning of reasoning driving VLAs (Alpamayo-1.5-10B): Taylor importance under CoC CE vs. action flow-matching disagrees in late VLM layers, while expert MLP-channel ranks are stable across denoising steps and highly concentrated. MoOP keeps a unit if it ranks high under either objective, additionally drops up to 93.75% of expert MLP channels, and reports 24-39.4% parameter cuts with no recovery fine-tuning, retaining most LingoQA, competitive open-/closed-loop scores, 1.37× speedup, and 36.6% lower peak memory.

Critical strengths

The dual-role analysis is the main contribution: CE and FM need different late-layer VLM units, and a small expert-MLP subset carries almost all measured importance. Evaluation is appropriately multi-faceted (Open-loop, AlpaSim closed-loop, LingoQA, latency/memory) and the no-finetune protocol makes the 24% VLM comparison informative.

Critical weaknesses

Already stated in the Questions.

Questions for authors
  1. Could the authors clarify why the main experiments, including the methodology, are conducted based on pruning the ALPAMAYO model? Furthermore, is the MoOP method generalizable to other base models?
  2. The structured pruning section (lines 166-167) describes selecting attention query heads and intermediate MLP channels as pruning layers. Is there any evidence that these are estimated to be of low importance? Or is this a common practice in VLA? Are there supporting references?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6Ext (1/2)

Q1: Choice of Alpamayo and generalization to other models

Thank you for this question, which helped us examine whether our analyses and method generalize beyond Alpamayo-1.5. We have repeated both analyses and applied MoOP on another base model, Alpamayo 2 Edge, and added the results to Appendix D.3.

We chose Alpamayo for two reasons. First, driving in complex situations requires reasoning about the causes of a decision. Alpamayo is a representative, publicly available reasoning VLA for autonomous driving that generates explicit Chain-of-Causation (CoC) reasoning together with trajectories. Second, it is trained and evaluated on real-world data. The public PhysicalAI-AV dataset provides real-world multi-camera driving clips, and AlpaSim runs closed-loop evaluation in scenes reconstructed from real driving logs, whereas many driving VLAs are evaluated in synthetic simulators such as CARLA[1]. This allows us to evaluate reasoning, open-loop driving and closed-loop driving in a single real-world setting.

To examine generalization beyond Alpamayo-1.5, we repeated both analyses of Section 4 on Alpamayo 2 Edge [2], a compact 2.65B-parameter model of the next Alpamayo generation with a different VLM backbone (a SigLIP2 vision encoder and a 28-layer Cosmos 3 Edge language model) and a 28-layer flow-matching action expert. Unlike in Alpamayo-1.5, the CE–FM rank correlation of query heads is already low in the early layers, where CE importance is concentrated on CoC tokens and FM importance on vision tokens. As in Alpamayo-1.5, however, the correlation computed on CoC tokens remains low across the backbone, so the two objectives favor different units even for the same CoC tokens.

Expert MLP channels again show more stable importance rankings across denoising steps than query heads (mean Spearman ρ of 0.885 vs. 0.614), and their importance is again concentrated, although less strongly than in Alpamayo-1.5 (the lowest-scoring 90% of channels carry 12.5% of the importance, vs. 1.8%).

We also pruned Alpamayo 2 Edge with all methods at the same budget, removing 24.4% of all parameters from the VLM (Table 22 in Appendix D.3). MoOP gives the lowest open-loop errors among the pruned models on all three evaluation sets. On OOD-train, its minADE@6 is 1.322 m, compared with 1.457 and 1.456 m for FM-only and CE-only and 2.039, 2.343 and 2.666 m for Týr-the-Pruner, LLM-Pruner and Wanda (dense: 0.975 m). Additionally removing 75% of the expert MLP channels (27.5% in total) does not increase these errors.

[1] Dosovitskiy et al., 2017. CARLA: An Open Urban Driving Simulator. CoRL 2017.

[2] NVIDIA, 2026. Alpamayo 2 Edge. https://huggingface.co/nvidia/Alpamayo2-Edge.

Table 22: Open-loop prediction of pruned Alpamayo 2 Edge at 6.4 s (m). The row with expert pruning additionally removes 75% of the expert MLP channels.

MethodExpert PruningPruning RatioTest500 minADE@6 (m) ↓Test500 minFDE@6 (m) ↓OOD-Val262 minADE@6 (m) ↓OOD-Val262 minFDE@6 (m) ↓OOD-train minADE@6 (m) ↓OOD-train minFDE@6 (m) ↓
Baseline✗0%0.9162.7220.9642.4440.9752.480
Wanda✗24.4%2.4076.7412.5236.5942.6666.555
LLM-Pruner✗24.4%2.2366.1952.3916.0572.3435.836
Týr-the-Pruner✗24.4%1.6164.4661.9124.7752.0394.946
FM-only✗24.4%1.4413.8811.4223.6521.4573.601
CE-only✗24.4%1.4893.9421.6784.0941.4563.457
MoOP (Ours)✗24.4%1.1963.4061.3743.4491.3223.307
MoOP (Ours)✓27.5%1.1903.3961.3633.4151.3183.295
AuthorsOpenReview ↗

Response to Reviewer 6Ext (2/2)

Q2: Choice of query heads and MLP channels as pruning units

Thank you for this question, which helps clarify how we apply structured pruning. We have added the following explanation to Appendix A.4 and referred to it in Section 3.2 (L179–180) in green.

Query heads and intermediate MLP channels define the granularity of pruning, that is, the type of unit that can be removed. We did not choose these unit types because they were estimated to be less important than other types, and we did not compare importance across unit types in advance. Importance is measured afterwards, only to decide which individual heads and channels to remove within each type. Likewise, other units, such as layers, hidden dimensions and KV heads, are kept not because they are more important, but because of the architecture.

In Alpamayo, each expert layer ll attends to the KV cache of VLM layer ll, which is how the VLM conditions action generation. Removing layers or hidden dimensions would break this layer-to-layer conditioning, and removing KV heads would remove part of the KV cache itself, which is shared by several query heads under grouped-query attention[1]. Query heads and intermediate MLP channels, in contrast, can be removed without changing this interface.

They are also the standard units in structured pruning of transformers[2-5] like LLM-Pruner. Removing them deletes entire rows and columns of the weight matrices, so the matrices simply become smaller. For example, removing an MLP channel deletes one row of the up and gate projections and one column of the down projection. Unlike removing individual weights, which leaves scattered zeros that standard hardware cannot skip efficiently, this yields real speedups without specialized sparse kernels.

[1] Ainslie et al., 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023.

[2] Michel et al., 2019. Are Sixteen Heads Really Better than One? NeurIPS 2019.

[3] Voita et al., 2019. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL 2019.

[4] Kwon et al., 2022. A Fast Post-Training Pruning Framework for Transformers. NeurIPS 2022.

[5] Muralidharan et al., 2024. Compact Language Models via Pruning and Knowledge Distillation. NeurIPS 2024.

Thank you for the detailed rebuttal, and you have addressed my concern. I will raise my rating to 4. Solid rebuttal, good paper. Good Luck.

We sincerely thank the reviewer for the positive reassessment and for raising the score. Your questions on generalization beyond Alpamayo-1.5 and on the choice of pruning units helped us strengthen the paper. We will keep the Alpamayo 2 Edge results (Appendix D.3) and the rationale for the pruning units (Appendix A.4) in the final version.

Reviewer_ZDG5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies structured pruning for reasoning-based driving VLAs. The authors find that reasoning and action prediction rely on different VLM components, while MLP channel importance in the action expert remains relatively stable across denoising steps. Based on these findings, they propose MoOP, which combines CE and FM importance for VLM pruning and averages importance across denoising steps for expert pruning. Experiments on Alpamayo-1.5-10B show a 39.4% parameter reduction and a 1.37× speedup, while largely maintaining driving performance without fine-tuning.

Critical strengths
  1. The compression results are quite good. The method removes a large number of parameters without fine-tuning, and the authors also report actual latency and memory usage, which makes the results more convincing.
  2. The importance analysis is interesting, especially for the action expert. Removing 93.75% of its MLP channels with almost no additional loss in open-loop performance is a notable finding.
Critical weaknesses
  1. The paper shows that CE and FM importance differ significantly in later VLM layers, but the proposed method applies the same pruning ratio across all layers. I am not sure this fully takes advantage of the differences observed in the analysis. Also, the maximum-rank rule treats both objectives equally, without considering whether their relative importance changes across layers.
  2. The action expert results are interesting, but I am not fully convinced by the explanation. If MLP importance rankings are already stable across denoising steps, it is unclear how much averaging across steps actually helps. Also, removing 93.75% of the expert MLP channels only reduces end-to-end latency from 1032.4 ms to 977.8 ms. Most of the speedup seems to come from VLM pruning rather than expert pruning.
Questions for authors
  1. Given the different importance distributions across VLM layers, why is the same pruning ratio used for every layer?
  2. Since MLP channel importance is already stable across denoising steps, what is the benefit of averaging importance over all steps?
  3. Could the authors report variance or confidence intervals for the closed-loop results? Some of the differences between methods are quite small.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZDG5 (1/4)

W1 / Q1: Non-uniform layer-wise allocation and the max-rank rule

Thank you for the careful reading of our analysis and for this thoughtful question on how it connects to the method. We have added non-uniform layer-wise allocation experiment to Appendix D.1 in blue and discussions for max-rank rule to Appendix D.2 and refer to it in Section 5.1 L352-353 in blue.

Non-uniform layer-wise allocation. As described in Appendix A.2, we applied the same pruning ratio to every layer so that the comparison isolates the effect of the pruning criterion. CE-only, FM-only and MoOP remove exactly the same number of query heads and MLP channels in each layer, so the differences among them in Tables 1, 2 and 4 come only from which units each criterion removes.

Following your suggestion, we also evaluated a non-uniform layer-wise allocation guided by our analysis. We base it on the rank correlation computed on generated CoC tokens rather than on all tokens. Although the all-token correlation is high in the early layers, the CoC-token correlation remains low across much of the backbone (Figure 3a), so units in the early and middle layers can also serve CoC generation and action conditioning differently. We therefore assign a larger budget to layers with a lower CoC-token correlation, so that these layers are pruned less.

Two adjustments make this allocation stable. First, the CoC-token correlation fluctuates considerably between adjacent layers, which leads to abrupt changes in the pruning ratio across layers. As demonstrated in [1], it degrades performance so we scale the deviation of each layer's pruning ratio from the target pruning ratio by a=0.25a = 0.25. Second, the early layers are particularly sensitive to pruning as demonstrated in LLM-Pruner, so we keep the uniform pruning ratio in the first four layers. The total parameter reduction is the same as for MoOP (39.4%).

Under the same budget, this rank correlation based allocation lowers minADE@6 from 0.951 m to 0.939 m on Test500 and from 1.118 m to 1.086 m on OOD-Val262 (Table 18 in Appendix D.1). A more careful search over aa and the number of fixed layers may improve the performance further. As the reviewer suggested, our analysis can therefore also guide the allocation of the budget across layers, in addition to the selection of units within each layer. We have added this experiment to Appendix D.1 in blue.

Table 18. Non-uniform layer-wise allocation with 95% confidence intervals.

Budget allocationTest500 minADE@6 (m)↓Test500 minFDE@6 (m)↓OOD-Val262 minADE@6 (m)↓OOD-Val262 minFDE@6 (m)↓
Uniform0.951 [0.870, 1.035]2.539 [2.275, 2.819]1.118 [0.955, 1.303]2.844 [2.341, 3.397]
Non-uniform0.939 [0.856, 1.027]2.498 [2.233, 2.786]1.086 [0.928, 1.268]2.759 [2.268, 3.310]
Difference (non-uniform − uniform)−0.012 [−0.034, +0.011]−0.041 [−0.116, +0.038]−0.031 [−0.076, +0.011]−0.085 [−0.204, +0.031]

Max-rank rule. Regarding the max-rank rule, as you note, it does not weight the two objectives explicitly. This is a deliberate choice, because the CE and FM importance scores have different scales. FM scores are on average about 1.6× larger than CE scores. As the max-rank rule operates on within-layer ranks, it is unaffected by the different scales of the CE and FM scores, and it retains a unit if the unit ranks high under either objective. It also adapts to the rank correlation of each layer without a layer-specific weight: where the two rankings agree, the combined score closely follows the shared ranking, and where they diverge, the top-ranked units under each objective are retained.

To test whether the raw score scales can set this priority, we compared max-rank with max-raw, which takes the maximum of the raw CE and FM importance scores and thus implicitly favors FM in proportion to its larger scale (Table 21 in Appendix D.2). Max-raw gives 0.974 m and 1.173 m minADE@6 on Test500 and OOD-Val262 and 59.2 on LingoQA, compared with 0.950 m, 1.119 m and 68.8 for max-rank. This indicates that letting the scale of one objective determine its priority is harmful, and supports combining the objectives through their within-layer ranks. We have added this discussion to Appendix D.2 and refer to it in Section 5.1 L352-353 in blue.

Table 21. Combination rules.

Combination ruleLingoQA (%)↑Test500 minADE@6 (m)↓Test500 minFDE@6 (m)↓OOD-Val262 minADE@6 (m)↓OOD-Val262 minFDE@6 (m)↓
Mean-raw14.01.2163.2481.3733.533
Mean-rank67.0‾\underline{67.0}0.960‾\underline{0.960}2.607‾\underline{2.607}1.160‾\underline{1.160}2.967
Max-raw59.20.9742.6291.1732.961‾\underline{2.961}
Max-rank (MoOP)68.80.9502.5461.1192.851

[1] Yin et al., 2024. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. ICML 2024.

AuthorsOpenReview ↗

Response to Reviewer ZDG5 (2/4)

W2 / Q2: Step-wise expert scoring and the gain from expert pruning

Thank you for this thoughtful question on expert MLP pruning. We have added step-wise expert scoring experiment to Appendix D.1 and refer to it in Section 5.2 L368-370 in blue, and clarified cost gain from expert pruning in Section 6.3 L512-515 in blue.

Step-wise expert scoring. As you point out, because the MLP importance rankings of the action expert are stable across denoising steps, using the score of a single step instead of the step average makes little difference. To verify this, we pruned the expert at the 93.75% budget using the score of each single step (s = 0, …, 9) instead of the step-averaged score (Table 19 in Appendix D.1). All ten single-step selections give minADE@6 within 0.005 m of the step-averaged selection on both Test500 and OOD-Val262, far smaller than the 95% CI of each value. The choice of aggregation therefore has little effect, and this result directly supports Finding 2, that the MLP importance rankings of the expert are stable across denoising steps. We use the step average because it requires no choice of a particular step, and Finding 2 implies that any single step would select nearly the same channels. We have added this experiment to Appendix D.1 and refer to it in Section 5.2 L368-370 in blue.

We would like to clarify that our contribution here is not the averaging itself but Finding 2, that the importance ranking of the expert's MLP channels remains nearly the same across denoising steps. Because of this stability, the channels can be selected once and physically removed from the expert's MLP weights, and the resulting smaller expert is used at every denoising step.

Table 19. Single-step and step-averaged expert scoring at 93.75% expert MLP removal, with 95% confidence intervals. t is the flow-matching time of step s.

Step stTest500 minADE@6 (m)↓Test500 minFDE@6 (m)↓OOD-Val262 minADE@6 (m)↓OOD-Val262 minFDE@6 (m)↓
step 00.050.950 [0.870, 1.035]2.537 [2.273, 2.817]1.117 [0.955, 1.303]2.844 [2.342, 3.397]
step 10.150.950 [0.869, 1.035]2.537 [2.273, 2.816]1.117 [0.955, 1.303]2.844 [2.342, 3.397]
step 20.250.950 [0.870, 1.035]2.538 [2.274, 2.818]1.118 [0.955, 1.303]2.845 [2.342, 3.397]
step 30.350.951 [0.870, 1.036]2.539 [2.275, 2.819]1.118 [0.956, 1.304]2.846 [2.342, 3.399]
step 40.450.951 [0.870, 1.036]2.539 [2.274, 2.820]1.118 [0.955, 1.304]2.845 [2.342, 3.398]
step 50.550.951 [0.870, 1.036]2.540 [2.276, 2.820]1.118 [0.955, 1.303]2.844 [2.341, 3.397]
step 60.650.950 [0.870, 1.035]2.536 [2.273, 2.817]1.118 [0.955, 1.303]2.845 [2.343, 3.397]
step 70.750.950 [0.870, 1.035]2.539 [2.275, 2.819]1.118 [0.955, 1.303]2.845 [2.342, 3.397]
step 80.850.950 [0.870, 1.035]2.538 [2.274, 2.819]1.118 [0.955, 1.303]2.844 [2.342, 3.396]
step 90.950.946 [0.867, 1.030]2.514 [2.253, 2.793]1.117 [0.954, 1.303]2.832 [2.332, 3.386]
Step-averaged (MoOP)–0.951 [0.870, 1.035]2.539 [2.275, 2.819]1.118 [0.955, 1.303]2.844 [2.341, 3.397]

Gain from expert pruning. We agree that most of the end-to-end speedup comes from VLM pruning. Expert pruning reduces the latency of the denoising stage itself by 26.1% (204.3 to 150.9 ms, a 1.35× speedup), but because the action expert processes only a short sequence of action tokens, its effect on end-to-end latency is small (1032.4 to 977.8 ms, 5.3% lower). Its main benefit is therefore reduced memory. If the importance rankings were unstable across steps, each step would need its own set of channels, and the expert would have to keep every channel selected at any step, which would offset the memory savings. Because a single pruned expert serves all steps, expert pruning removes an additional 1.71B parameters (8.42B to 6.71B) and lowers peak memory by 3.19 GiB (17.45 to 14.26 GiB). We have clarified this discussion in Section 6.3 L512-515 in blue.

AuthorsOpenReview ↗

Response to Reviewer ZDG5 (3/4)

Q3: Closed-loop margins and statistical significance

We appreciate your question which helped us present results more precisely. We have added confidence intervals and paired comparisons for all main results to Appendix C.1 (Table 7-13) and results at a higher pruning ratio to Tables 1, 2 and 4 and Appendix C.2 (Table 14-17). We have clarified results and our contribution in Abstract (L23-27), Section 1 (L96-99, L111-112) and Section 6.2 (L460-470, L478-479, L487-505) in purple. Table 9,10,14 and 16 are provided below.

To quantify the uncertainty, we now report the 95% CI of every closed-loop metric, all computed with a scene-level cluster bootstrap. The results and the details of the procedure are provided in Table 7-9 in Appendix C.1.

As Reviewer phZv also noted, MoOP substantially outperforms CE-only and FM-only at the same budget, and this holds when the intervals are taken into account, as MoOP remains significantly better than both over the combined 250 scenes (Table 9 in Appendix C.1), confirming the effectiveness of the dual objective. We agree, however, that the margins over the strongest external baselines are small at the 40% budget on VLM. Over the combined scenes, the differences from Týr-the-Pruner (+0.029 [−0.005, +0.064]) and LLM-Pruner (+0.019 [−0.024, +0.061]) are not significant, so MoOP is comparable to them in closed-loop driving at this budget.

However, the methods separate more clearly when the budget is tightened. When 50% of the units are removed from each VLM layer as shown in Table 14-15 in Appendix C.2, MoOP keeps scene scores of 0.866 on Origin150, while LLM-Pruner falls to 0.634 and Týr-the-Pruner to 0.742. In open-loop prediction, we additionally evaluated all methods on OOD-train, a larger set of 1,271 OOD clips disjoint from our calibration set (Table 10-11 in Appendix C.1). On this set, MoOP achieves a significantly lower minADE@6 than every compressed baseline, by 0.139 m relative to Týr-the-Pruner and 0.146 m relative to FM-only. MoOP therefore retains the dense model's performance on long-tail scenarios better than the other pruning methods.

We attribute these gains to MoOP's preservation of reasoning, as shown by the LingoQA results (Table 4 in Section 6.2). MoOP achieves 68.8% accuracy, close to the 73.2% of the dense model, whereas Týr-the-Pruner and LLM-Pruner reach only 34.2% and 44.8%, indicating that their reasoning has largely collapsed. In summary, MoOP is comparable to the strongest baselines at a 40% VLM pruning ratio, but it degrades less under more aggressive pruning and on long-tail scenarios while retaining the reasoning ability of the dense model.

Thank you for your question again, which helped us present these results more precisely and clarify our contribution. We have revised Tables 1, 2 and 4, Abstract (L23-27), Section 1 (L96-99, L111-112) and Section 6.2 (L460-470, L478-479, L487-505) in purple accordingly.

(continued in the next comment)

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful reading and constructive feedback. We appreciate that the reviewers recognized our analysis of the dual roles of the VLM and of the concentrated importance in the action expert as the main contribution (ZDG5, 6Ext), and found that the dual-objective ablation strongly supports our VLM pruning design (phZv). We are also glad that the reviewers appreciated our strong compression results with measured speedup and memory savings without post-pruning fine-tuning (ZDG5, phZv), and our multi-faceted evaluation across open-loop prediction, closed-loop driving, driving QA and cost (6Ext).

To address the reviewers' concerns, we conducted additional experiments and analyses, summarized below:

  1. (ZDG5, phZv) Verified the statistical reliability of our results with confidence intervals and paired comparisons, showing that MoOP is comparable to the strongest baselines at r = 0.4 and significantly better than all other compressed methods at higher pruning ratio r = 0.5.
  2. (ZDG5, phZv) Further supported and extended our analysis through single-step expert scoring and random channel selection, which confirm Finding 2, and through non-uniform layer-wise allocation guided by the CE–FM rank correlation.
  3. (ZDG5) Validated the max-rank rule against alternative combination rules, showing that combining the raw CE and FM scores degrades reasoning because of their different scales.
  4. (6Ext) Demonstrated generalization by repeating our analyses and applying MoOP on Alpamayo 2 Edge.
  5. (ZDG5, 6Ext, phZv) Clarified the choice of pruning units and the cost benefits of expert pruning, and corrected two entries in Table 2.

We thank the reviewers again for their constructive feedback, which has substantially strengthened the paper. We are happy to address any remaining questions.

Changes in the revised manuscript are color-coded by reviewer: blue for ZDG5, green for 6Ext, orange for phZv, and purple for changes that address both ZDG5 and phZv.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.