- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FastB2D addresses the high cost of closed-loop autonomous driving evaluation arising from redundant routes and uneven simulation costs. It uses model-response representations and cost-aware selection to retain 77 of 220 Bench2Drive routes, then estimates full-benchmark performance through predictive and calibrated estimators. The authors report over 70% less evaluation time and a success-rate MAE of 1.20 percentage points on held-out models. Further validation covers four metrics, new model families, and EmBench.
Critical strengths
-
Learning route representations from model evaluation results and incorporating simulation cost into anchor selection is well motivated for closed-loop evaluation. The method considers both how routes distinguish model performance and the computation they require, offering a useful approach to constructing compact evaluation sets. The EmBench experiments further support its applicability to another closed-loop benchmark.
-
The work addresses a problem of substantial practical value. FastB2D estimates four metrics from 77 routes, reports a test SR MAE of 1.20 percentage points, and reduces average evaluation time from 38.7 to 11.4 hours. These savings could enable more runs with different seeds and broader ablation studies for groups with limited compute. Reusing Bench2Drive routes should also facilitate adoption within existing evaluation workflows.
Critical weaknesses
-
The gains from route selection and its reproducibility remain unclear. In Table 4(a), calibration reduces random-selection Efficiency error from 3.38 to 1.37, close to the full method’s 1.13, leaving selection and estimation gains insufficiently separated. Probability-preserving transformations of Eq. (1) can also change clustering distances, yet normalization or parameter constraints are not clearly specified. Unclear CE normalization and metric aggregation further limit reproducibility of the 77-route budget choice.
-
The reliability of skill diagnosis and SR gating is not sufficiently established. Table 3 reports an 11.2-point Emergency Brake error for DriveTransformer and a 9.8-point Give Way error for PDM-Lite, which could change the identified model weaknesses. The displayed values yield corresponding MAEs of approximately 4.00 and 2.29, rather than 3.59 and 1.94, leaving the evaluation population and aggregation unclear. Aggregate OOD errors also do not establish when the SR gate is reliable: disagreement may reflect estimation error, and both estimators may agree incorrectly.
Questions for authors
-
How much does FastB2D reduce estimation error relative to random and other route selectors when all use the same calibrated estimator, SR gate, and simulation-frame budget? Please provide this comparison to separate selection gains from estimator gains.
-
What normalization or constraints are applied to Eq. (1)’s parameters before clustering to address transformations that preserve response probabilities but alter Euclidean distances? Please also specify the CE normalization range and metric aggregation formula used to choose the 77-route budget.
-
Which models, seeds, and aggregation formula underlie the Emergency Brake and Give Way MAEs in Table 3? Please reconcile the recalculated values of approximately 4.00 and 2.29 with the reported 3.59 and 1.94, and show whether full evaluation and proxy predictions identify the same weak skills for DriveTransformer and PDM-Lite.
-
For test models outside the construction performance range, could you report both SR estimators’ errors alongside the gate decisions? Do these results show that switching generally selects the lower-error estimator, rather than merely indicating disagreement?
3 comments
Response to Reviewer DCVK Part 2
[DCVK Q4] Could you clarify which model records are included in the skill MAEs in Table 3?
[DCVK A4] Thank you for checking the aggregation. The reported MAEs include two additional models with skill results. The displayed model list followed the four-metric comparison in Table 2. These two models lack Efficiency and Smoothness results and were therefore excluded from that comparison, while their available skill results remained included in the skill MAEs. This accounts for the difference between the reported MAEs and those recalculated from the displayed rows.
Their skill results are provided below. Each entry reports the full-benchmark score followed by the FastB2D prediction.
| Additional record | Overtaking | Merging | Emergency Brake | Give Way | Traffic Sign | Mean Ability |
|---|---|---|---|---|---|---|
| 1 | 73.3 / 67.4 | 48.8 / 47.7 | 84.2 / 82.1 | 50.0 / 49.4 | 60.0 / 62.6 | 63.3 / 61.8 |
| 2 | 62.2 / 61.2 | 69.4 / 65.2 | 88.3 / 86.7 | 50.0 / 50.2 | 81.6 / 77.5 | 70.3 / 68.2 |
We will correct the table presentation to clarify the aggregation scope. The existing skill scores and aggregate MAEs remain unchanged.
.
.
[DCVK Q5] How do individual skill-score errors affect model diagnosis, particularly for DriveTransformer and PDM-Lite?
[DCVK A5] We appreciate this careful and accurate observation. FastB2D prioritizes overall Success Rate, the most widely adopted closed-loop metric, without a separate skill-specific objective, as stated in lines 159–161. Skill estimates can therefore vary in accuracy. Overall-SR reliability is consistently validated across families and performance ranges in Tables 2 and 6 and Appendix F.5.
Skill scores provide supplementary references for identifying weaknesses. Across the eleven models in Table 3:
| Diagnostic measure | Result |
|---|---|
| Mean within-model Spearman correlation across five skills | 0.955 |
| Models with the same weakest skill | 10/11 |
The two weakest skills are preserved for both DriveTransformer and PDM-Lite. For DriveTransformer, these are Give Way and Traffic Sign. For PDM-Lite, Give Way remains the weakest skill, followed by Merging.
.
.
[DCVK Q6] How do the two SR estimators perform outside the construction range, and does the gate select the more accurate one?
[DCVK A6] Thank you for requesting this analysis. The gate selects the lower-error estimator in these OOD examples. Following Section 2.4, we use the threshold of 5 percentage points and switch to the calibrated estimator when the prediction gap exceeds this threshold.
| Model / record | SR range | Predictive error | Calibrated error | Prediction gap | Gate choice | Final error |
|---|---|---|---|---|---|---|
| DriveTransformer run2 | <50% | 9.92 | 2.14 | 12.06 | Calibrated | 2.14 |
| UniAD-Tiny | <50% | 16.77 | 1.18 | 15.59 | Calibrated | 1.18 |
| VAD | <50% | 15.88 | 2.57 | 18.45 | Calibrated | 2.57 |
| BridgeDrive-LEAD | >75% | 7.52 | 1.14 | 6.38 | Calibrated | 1.14 |
| UniAD-Base | <50% | 14.86 | 2.78 | 17.64 | Calibrated | 2.78 |
These comparisons complement Figs. 4 and 5 with direct evidence that switching can improve prediction accuracy beyond the construction range.
Thank you to the authors for the further clarification. My concern has been addressed. As I have given WA to this paper ready, I will maintain the score.
Response to Reviewer DCVK Part 1
We thank the reviewer for recognizing our well-motivated method and its practical value. We hope the clarifications and additional results below address your concerns.
.
.
[DCVK Q1]How much does route selection contribute beyond calibration when selectors use the same calibrated estimator, SR gate, and simulation-frame budget?[DCVK A1]Thank you for this careful observation. The learned estimator is part of FastB2D. The unweighted variant in Table 4(a) is an ablation that measures the contribution of metric-specific estimation, as discussed in Section 3, Analysis 5.Our calibrated estimator enables multi-metric prediction is closed loop AD. The LLM benchmark compression baselines considered here primarily target a single score. We learn separate anchor weights for each metric, allowing the same selected routes to estimate SR, DS, Efficiency, and Smoothness.
Our selected routes retain an advantage under the same calibrated estimators. We add a comparison that combines each selector with our calibrated weighted estimators. All methods select 77 routes, and we report their actual simulation-frame costs.
FastB2D achieves the lowest mean error on all four metrics while requiring the fewest simulation frames. Compared with random selection, it reduces SR MAE by 22.2% and frame cost by 24.1%. These results demonstrate that route selection contributes beyond calibration at the same route count.
.
.
[DCVK Q2]What normalization or constraints are used before clustering to address transformations that preserve success probabilities but change Euclidean distances?[DCVK A2]Thank you for raising this point. Hierarchical priors provide soft constraints on parameter location and scale. We use Normal priors on θi, aj, and bj, with Normal hyperpriors on their means and Gamma hyperpriors controlling their scales. These priors regularize estimation by discouraging extreme parameter values during fitting. Clustering uses the posterior means of [aj;bj] without additional normalization.We follow TinyBenchmarks and MiniLongBench and will release our code. Specifically, we adopt their hierarchical-prior configuration for parameter estimation within our response-based, cost-aware compression framework. We will supplement the manuscript with these details and release the training configurations and fitted parameters alongside the code to support reproducibility.
.
.
[DCVK Q3]How is CE normalized and aggregated, and how does it support the 77-route budget?[DCVK A3]Thank you for requesting this clarification. CE balances prediction accuracy and evaluation cost. Following the accuracy-cost aggregation principle used in RouterArena [1] and VL-RouterBench [2], we take the harmonic mean CE=2,Acc,Eff/(Acc+Eff). Here, Acc=1−(m−mmin)/(mmax−mmin), where m is SR MAE in the budget analysis and the bounds are its minimum and maximum across candidate budget points. This normalizes error to [0,1]. We define Eff=1−c, where c is the retained fraction of full-benchmark simulation frames.77 routes provide a practical default, and users can choose other budgets. They achieve low SR estimation error at a small fraction of full-benchmark frame cost, with a CE score near the peak in Fig. 3. We will also release configurations with different route counts, allowing users to choose the accuracy-cost balance that best suits their needs.
.
.
[1] RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers. ICLR 2026.
[2] VL-RouterBench: A Benchmark for Vision-Language Model Routing. CVPR 2026