- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies training-free structured pruning of the Alpamayo-1.5-10B driving VLA, which combines an autoregressive VLM producing a Chain-of-Causation with a flow-based action expert. The authors observe that cross-entropy and flow-matching objectives rank VLM heads and MLP channels differently, while action-expert MLP rankings are relatively stable across denoising steps. MoOP therefore retains VLM units that are important to either objective and prunes action-expert MLP channels using importance averaged across denoising steps, without post-pruning fine-tuning. At 24% total parameter reduction, it reports Test500/OOD minADE of 0.950/1.119, LingoQA of 68.8, and Origin150/Hard100 scene scores of 0.828/0.595.
Critical strengths
-
The dual-objective ablation strongly supports the main VLM design. At the same 24% budget, MoOP substantially outperforms CE-only and FM-only pruning on LingoQA, open-loop errors, and Hard100 scene score (Tables 1–2, pp. 7–8).
-
The paper demonstrates measured, deployment-relevant savings. A 39.4% parameter reduction yields a 1.372× measured end-to-end speedup and 36.6% lower peak memory without recovery fine-tuning, while aggressive expert MLP pruning has little effect on open-loop errors (Tables 3 and 5, pp. 9–10).
Critical weaknesses
-
The closed-loop margins over the strongest pruning baseline are small. Improvements of 0.017 on Origin150 and 0.009 on Hard100 are insufficient by themselves to support a strong claim of closed-loop superiority, even though the overall performance is competitive (Table 2, p. 8).
-
The principal scene score is biased toward the recorded route. Appendix B.4 shows safe alternative trajectories receiving much lower scores than MoOP solely because they deviate from the reference route. Since the reported closed-loop margin is already small, this metric behavior materially weakens the ranking claim (Appendix B.4, pp. 32–34).
Questions for authors
-
Equation 12 describes off-road rate using binary rollout outcomes. With 200 Hard100 rollouts, a direct mean should change in increments of 0.005; how is the reported value 0.208 obtained? Please state the exact denominator and aggregation procedure (Table 2, p. 8; Equation 12, Appendix B).
-
At the 93.75% expert-MLP pruning setting, how do random-channel and magnitude-channel selection perform under the same budget? These two controls would clarify whether the gain comes from MoOP's score or primarily from general expert overparameterization (Table 3, p. 9).
-
Which decisions, if any, were made using the earlier 50-clip calibration set that contained 12 test clips—including the architecture, scoring formulation, unit types, pruning ratios, or hyperparameters? If any final design choice used those clips, can the corresponding selection and final evaluation be repeated without test-set exposure (Appendix A.3.1, p. 14)?
6 comments
Response to Reviewer phZv (2/5)
W1: Closed-loop margins and statistical significance (continued)
Table 9: Paired differences from MoOP over the combined 250 scenes (95% CI). Scenes are resampled within each set. Differences are signed so that a positive value favors MoOP, that is, MoOP minus the method for the scene score and the method minus MoOP for the off-road and at-fault collision rates.
| Method | Expert Pruning | Scene Score | Offroad | At-fault Coll. |
|---|---|---|---|---|
| Baseline | ✗ | +0.081 [+0.046, +0.116] | +0.032 [+0.002, +0.062] | +0.006 [−0.014, +0.026] |
| Wanda | ✗ | +0.200 [+0.147, +0.253] | −0.004 [−0.042, +0.034] | +0.082 [+0.044, +0.122] |
| LLM-Pruner | ✗ | +0.019 [−0.024, +0.061] | −0.030 [−0.064, +0.004] | +0.018 [−0.010, +0.046] |
| Týr-the-Pruner | ✗ | +0.029 [−0.005, +0.064] | +0.022 [−0.010, +0.056] | −0.004 [−0.024, +0.016] |
| FM-only | ✗ | +0.047 [+0.012, +0.081] | +0.040 [+0.010, +0.070] | +0.000 [−0.022, +0.024] |
| CE-only | ✗ | +0.173 [+0.127, +0.218] | +0.018 [−0.018, +0.054] | +0.030 [+0.004, +0.058] |
| MoOP (Ours) | ✓ | +0.020 [−0.005, +0.046] | +0.014 [−0.008, +0.038] | −0.004 [−0.022, +0.014] |
Table 10: Open-loop prediction at 6.4 s with 95% confidence intervals (m).
| Method | Expert Pruning | Pruning Ratio | Test500 minADE@6 (m) ↓ | Test500 minFDE@6 (m) ↓ | OOD-Val262 minADE@6 (m) ↓ | OOD-Val262 minFDE@6 (m) ↓ | OOD-train minADE@6 (m) ↓ | OOD-train minFDE@6 (m) ↓ |
|---|---|---|---|---|---|---|---|---|
| Baseline | ✗ | 0% | 0.842 [0.765, 0.922] | 2.267 [2.009, 2.540] | 1.000 [0.847, 1.177] | 2.496 [2.041, 3.020] | 0.928 [0.856, 1.007] | 2.354 [2.144, 2.582] |
| Wanda | ✗ | 24% | 2.975 [2.729, 3.233] | 8.269 [7.532, 9.038] | 2.790 [2.461, 3.141] | 6.884 [5.983, 7.842] | 2.832 [2.659, 3.013] | 6.873 [6.418, 7.348] |
| LLM-Pruner | ✗ | 24% | 1.451 [1.312, 1.600] | 3.889 [3.485, 4.314] | 1.662 [1.432, 1.916] | 4.153 [3.525, 4.844] | 1.597 [1.485, 1.716] | 3.738 [3.450, 4.046] |
| Týr-the-Pruner | ✗ | 24% | 0.950 [0.862, 1.044] | 2.570 [2.289, 2.867] | 1.177 [1.012, 1.366] | 2.990 [2.504, 3.543] | 1.178 [1.090, 1.272] | 2.768 [2.536, 3.014] |
| FM-only | ✗ | 24% | 1.008 [0.917, 1.105] | 2.720 [2.423, 3.036] | 1.178 [1.011, 1.371] | 2.960 [2.475, 3.515] | 1.184 [1.098, 1.278] | 2.897 [2.661, 3.150] |
| CE-only | ✗ | 24% | 1.584 [1.433, 1.746] | 4.500 [4.022, 5.007] | 1.546 [1.328, 1.792] | 4.010 [3.364, 4.733] | 1.505 [1.402, 1.614] | 3.737 [3.459, 4.031] |
| MoOP (Ours) | ✗ | 24% | 0.950 [0.869, 1.034] | 2.546 [2.276, 2.832] | 1.119 [0.958, 1.307] | 2.851 [2.362, 3.410] | 1.038 [0.959, 1.124] | 2.541 [2.320, 2.778] |
| MoOP (Ours) | ✓ | 39.4% | 0.951 [0.871, 1.035] | 2.539 [2.272, 2.822] | 1.118 [0.957, 1.304] | 2.844 [2.357, 3.400] | 1.039 [0.960, 1.124] | 2.537 [2.318, 2.772] |
Table 14: Closed-loop driving on Origin150 at r = 0.5, with 95% confidence intervals (total pruning ratio is 30%).
| Method | Expert Pruning | Pruning Ratio | Scene Score ↑ | Offroad ↓ | At-fault Coll. ↓ |
|---|---|---|---|---|---|
| Baseline | ✗ | 0% | 0.750 [0.698, 0.800] | 0.070 [0.037, 0.110] | 0.040 [0.017, 0.067] |
| Wanda | ✗ | 30% | 0.350 [0.291, 0.410] | 0.280 [0.213, 0.350] | 0.180 [0.123, 0.240] |
| LLM-Pruner | ✗ | 30% | 0.634 [0.570, 0.697] | 0.073 [0.037, 0.117] | 0.143 [0.093, 0.200] |
| Týr-the-Pruner | ✗ | 30% | 0.742 [0.682, 0.800] | 0.073 [0.037, 0.117] | 0.087 [0.047, 0.130] |
| FM-only | ✗ | 30% | 0.551 [0.494, 0.607] | 0.103 [0.060, 0.153] | 0.097 [0.057, 0.143] |
| CE-only | ✗ | 30% | 0.279 [0.239, 0.322] | 0.170 [0.117, 0.230] | 0.140 [0.090, 0.197] |
| MoOP (Ours) | ✗ | 30% | 0.866 [0.821, 0.908] | 0.047 [0.020, 0.080] | 0.037 [0.013, 0.063] |
Table 16: Open-loop prediction at 6.4 s at r = 0.5, with 95% confidence intervals (m). The Baseline is the same as in Table 10.
| Method | Expert Pruning | Pruning Ratio | Test500 minADE@6 (m) ↓ | Test500 minFDE@6 (m) ↓ | OOD-Val262 minADE@6 (m) ↓ | OOD-Val262 minFDE@6 (m) ↓ |
|---|---|---|---|---|---|---|
| Baseline | ✗ | 0% | 0.842 [0.765, 0.922] | 2.267 [2.009, 2.540] | 1.000 [0.847, 1.177] | 2.496 [2.041, 3.020] |
| Wanda | ✗ | 30% | 2.901 [2.658, 3.157] | 7.869 [7.158, 8.612] | 2.761 [2.439, 3.103] | 6.917 [6.028, 7.861] |
| LLM-Pruner | ✗ | 30% | 2.533 [2.312, 2.766] | 6.662 [6.040, 7.319] | 2.628 [2.288, 2.993] | 6.613 [5.692, 7.600] |
| Týr-the-Pruner | ✗ | 30% | 2.050 [1.854, 2.257] | 5.339 [4.788, 5.921] | 2.165 [1.905, 2.443] | 5.202 [4.525, 5.917] |
| FM-only | ✗ | 30% | 2.822 [2.568, 3.094] | 8.278 [7.505, 9.102] | 2.436 [2.111, 2.788] | 6.753 [5.789, 7.799] |
| CE-only | ✗ | 30% | 5.151 [4.761, 5.560] | 15.192 [14.014, 16.432] | 3.899 [3.468, 4.358] | 11.230 [9.930, 12.603] |
| MoOP (Ours) | ✗ | 30% | 1.237 [1.126, 1.356] | 3.338 [2.980, 3.720] | 1.416 [1.217, 1.640] | 3.616 [3.027, 4.266] |
Response to Reviewer phZv (3/5)
W2: Effect of the route-dependent scene score
Thank you for examining the appendix closely and for feedback that helps make our claims more rigorous. We added the analysis of alternative routes and the score lost to the 4 m rule to Appendix B.4 (L1800-1830, Table 6) in orange.
Appendix B.4 reports a limitation of the current AlpaSim closed-loop evaluation that we found during our experiments: some scenes admit more than one valid route, but only the recorded route is scored. We included these cases to document a part of the evaluation protocol that should be improved. Following your comment, we examined whether such cases affect the comparison between methods (Table 6 in Appendix B.4).
AlpaSim stops scoring once the ego deviates more than 4 m from the reference route. We count a rollout as taking an alternative route when scoring stopped under this 4 m rule and the ego then continued for at least 10 m without any off-road event or collision. We also estimate the score lost to the rule for every rollout stopped by it, not only for those on alternative routes. The lost score is the additional progress score that the ego accrued along the reference route after scoring stopped, counted until the end of the rollout.
The alternative-route fraction is similar across methods, at 5.0–8.0% on Origin150 (except 12.3% for CE-only) and 9.0–14.0% on Hard100, and its differences between MoOP and Týr-the-Pruner or LLM-Pruner are not significant. The score lost to the 4 m rule does not favor MoOP either. On Origin150, MoOP loses slightly more score to this rule (0.039 and 0.040 with expert pruning) than the Baseline, LLM-Pruner and Týr-the-Pruner (0.022–0.037), and on Hard100 the losses are similar (0.064–0.067 vs. 0.063–0.071). CE-only and FM-only lose more on Origin150 (0.083 and 0.045), but MoOP's margins over them are much larger than these differences. Even if this progress were credited back to every method, MoOP would still achieve the highest scene score among the methods at the same budget on both sets (0.867 on Origin150 and 0.663 on Hard100). The route dependence therefore lowers absolute scene scores to a similar degree across methods and does not account for the differences between them. Regarding the small closed-loop margins themselves, please see our response to W1.
Table 6: Fraction of alternative routes and score lost to the 4 m rule.
| Method | Origin150 Alt-route (%) | Origin150 Loss from 4 m rule | Hard100 Alt-route (%) | Hard100 Loss from 4 m rule |
|---|---|---|---|---|
| Baseline | 6.3 | 0.037 | 10.5 | 0.063 |
| Wanda | 5.3 | 0.040 | 9.0 | 0.094 |
| LLM-Pruner | 5.0 | 0.022 | 14.0 | 0.067 |
| Týr-the-Pruner | 7.7 | 0.030 | 11.0 | 0.071 |
| FM-only | 6.3 | 0.045 | 13.5 | 0.064 |
| CE-only | 12.3 | 0.083 | 11.5 | 0.139 |
| MoOP | 5.7 | 0.039 | 12.5 | 0.067 |
| MoOP w/ Expert Pruning | 8.0 | 0.040 | 13.5 | 0.064 |
Response to Reviewer phZv (1/5)
W1: Closed-loop margins and statistical significance
Thank you for the careful examination of our closed-loop results. We added confidence intervals and paired comparisons for all main results to Appendix C.1 (Table 7-13) and results at a higher pruning ratio to Tables 1, 2 and 4 and Appendix C.2 (Table 14-17). We clarified results and our contribution in Abstract (L23-27), Section 1 (L96-99, L111-112) and Section 6.2 (L460-470, L478-479, L487-505) in purple. Table 9,10,14 and 16 are provided below.
To quantify the uncertainty, we now report the 95% CI of every closed-loop metric, all computed with a scene-level cluster bootstrap. The results and the details of the procedure are provided in Table 7-9 in Appendix C.1.
As you noted, MoOP substantially outperforms CE-only and FM-only at the same budget, and this holds when the intervals are taken into account, as MoOP remains significantly better than both over the combined 250 scenes (Table 9 in Appendix C.1), confirming the effectiveness of the dual objective. We agree, however, that the margins over the strongest external baselines are small at the 40% budget on VLM. Over the combined scenes, the differences from Týr-the-Pruner (+0.029 [−0.005, +0.064]) and LLM-Pruner (+0.019 [−0.024, +0.061]) are not significant, so MoOP is comparable to them in closed-loop driving at this budget.
However, the methods separate more clearly when the budget is tightened. When 50% of the units are removed from each VLM layer as shown in Table 14-15, MoOP keeps scene scores of 0.866 on Origin150, while LLM-Pruner falls to 0.634 and Týr-the-Pruner to 0.742. In open-loop prediction, we additionally evaluated all methods on OOD-train, a larger set of 1,271 OOD clips disjoint from our calibration set (Table 10-11 in Appendix C.1). On this set, MoOP achieves a significantly lower minADE@6 than every compressed baseline, by 0.139 m relative to Týr-the-Pruner and 0.146 m relative to FM-only. MoOP therefore retains the dense model's performance on long-tail scenarios better than the other pruning methods.
We attribute these gains to MoOP's preservation of reasoning, as shown by the LingoQA results (Table 4 in Section 6.2). MoOP achieves 68.8% accuracy, close to the 73.2% of the dense model, whereas Týr-the-Pruner and LLM-Pruner reach only 34.2% and 44.8%, indicating that their reasoning has largely collapsed. In summary, MoOP is comparable to the strongest baselines at a 40% VLM pruning ratio, but it degrades less under more aggressive pruning and on long-tail scenarios while retaining the reasoning ability of the dense model.
Thank you for your careful examination again, which helped us present these results more precisely and clarify our contribution. We have revised Tables 1, 2 and 4, Abstract (L23-27), Section 1 (L96-99, L111-112) and Section 6.2 (L460-470, L478-479, L487-505) in purple accordingly.
(continued in the next comment)