acceptodds
Under review as a conference paper at ICLR 2027

FastB2D: Toward Cost-Efficient Closed-Loop Benchmark for Autonomous Driving

Abstract

We present FastB2D, a cost-efficient closed-loop benchmark proxy for autonomous driving (AD) development, built on the widely adopted yet costly Bench2Drive. FastB2D reduces the route count by 65% and total evaluation time by over 70%, while estimating Success Rate with a mean absolute error of  1.2% and a Spearman correlation of 0.9+ on held-out advanced AD model variants, and accurately recovering Driving Score, Driving Efficiency, and Driving Smoothness. FastB2D is constructed by addressing two structural inefficiencies we identify in closed-loop AD benchmarks: statistical route redundancy and highly imbalanced per-route evaluation cost, where a small number of long routes can dominate total testing time. Our cost-aware benchmark compression method learns compact route representations directly from model evaluation records and selects representative low-cost anchor routes through cost-aware clustering. Compared with existing general-purpose benchmark compression methods and compact AD benchmarks, FastB2D further reduces evaluation time by 27.2% on average, lowers performance-estimation error by 64% on average, and demonstrates stronger out-of-distribution generalization. All code and data will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Efficient, Closed-Loop test, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DCVKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FastB2D addresses the high cost of closed-loop autonomous driving evaluation arising from redundant routes and uneven simulation costs. It uses model-response representations and cost-aware selection to retain 77 of 220 Bench2Drive routes, then estimates full-benchmark performance through predictive and calibrated estimators. The authors report over 70% less evaluation time and a success-rate MAE of 1.20 percentage points on held-out models. Further validation covers four metrics, new model families, and EmBench.

Critical strengths
  1. Learning route representations from model evaluation results and incorporating simulation cost into anchor selection is well motivated for closed-loop evaluation. The method considers both how routes distinguish model performance and the computation they require, offering a useful approach to constructing compact evaluation sets. The EmBench experiments further support its applicability to another closed-loop benchmark.

  2. The work addresses a problem of substantial practical value. FastB2D estimates four metrics from 77 routes, reports a test SR MAE of 1.20 percentage points, and reduces average evaluation time from 38.7 to 11.4 hours. These savings could enable more runs with different seeds and broader ablation studies for groups with limited compute. Reusing Bench2Drive routes should also facilitate adoption within existing evaluation workflows.

Critical weaknesses
  1. The gains from route selection and its reproducibility remain unclear. In Table 4(a), calibration reduces random-selection Efficiency error from 3.38 to 1.37, close to the full method’s 1.13, leaving selection and estimation gains insufficiently separated. Probability-preserving transformations of Eq. (1) can also change clustering distances, yet normalization or parameter constraints are not clearly specified. Unclear CE normalization and metric aggregation further limit reproducibility of the 77-route budget choice.

  2. The reliability of skill diagnosis and SR gating is not sufficiently established. Table 3 reports an 11.2-point Emergency Brake error for DriveTransformer and a 9.8-point Give Way error for PDM-Lite, which could change the identified model weaknesses. The displayed values yield corresponding MAEs of approximately 4.00 and 2.29, rather than 3.59 and 1.94, leaving the evaluation population and aggregation unclear. Aggregate OOD errors also do not establish when the SR gate is reliable: disagreement may reflect estimation error, and both estimators may agree incorrectly.

Questions for authors
  1. How much does FastB2D reduce estimation error relative to random and other route selectors when all use the same calibrated estimator, SR gate, and simulation-frame budget? Please provide this comparison to separate selection gains from estimator gains.

  2. What normalization or constraints are applied to Eq. (1)’s parameters before clustering to address transformations that preserve response probabilities but alter Euclidean distances? Please also specify the CE normalization range and metric aggregation formula used to choose the 77-route budget.

  3. Which models, seeds, and aggregation formula underlie the Emergency Brake and Give Way MAEs in Table 3? Please reconcile the recalculated values of approximately 4.00 and 2.29 with the reported 3.59 and 1.94, and show whether full evaluation and proxy predictions identify the same weak skills for DriveTransformer and PDM-Lite.

  4. For test models outside the construction performance range, could you report both SR estimators’ errors alongside the gate decisions? Do these results show that switching generally selects the lower-error estimator, rather than merely indicating disagreement?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DCVK Part 1

We thank the reviewer for recognizing our well-motivated method and its practical value. We hope the clarifications and additional results below address your concerns.

.

.

[DCVK Q1] How much does route selection contribute beyond calibration when selectors use the same calibrated estimator, SR gate, and simulation-frame budget?

[DCVK A1] Thank you for this careful observation. The learned estimator is part of FastB2D. The unweighted variant in Table 4(a) is an ablation that measures the contribution of metric-specific estimation, as discussed in Section 3, Analysis 5.

Our calibrated estimator enables multi-metric prediction is closed loop AD. The LLM benchmark compression baselines considered here primarily target a single score. We learn separate anchor weights for each metric, allowing the same selected routes to estimate SR, DS, Efficiency, and Smoothness.

Our selected routes retain an advantage under the same calibrated estimators. We add a comparison that combines each selector with our calibrated weighted estimators. All methods select 77 routes, and we report their actual simulation-frame costs.

Route selectorFrame cost ↓SR MAE ↓DS MAE ↓EF MRE ↓SM MAE ↓
FastB2D27.09%1.766 ± 0.2090.980 ± 0.1491.136 ± 0.1051.428 ± 0.202
Random35.69%2.270 ± 0.2131.247 ± 0.1511.338 ± 0.1761.669 ± 0.175
TinyBenchmark37.32%1.997 ± 0.2061.045 ± 0.1001.149 ± 0.0671.530 ± 0.134
AnchorPoint42.05%5.508 ± 3.0471.011 ± 0.0791.190 ± 0.0671.861 ± 0.721
Scenario-label coverage34.68%2.473 ± 0.1521.210 ± 0.0941.221 ± 0.1801.612 ± 0.215

FastB2D achieves the lowest mean error on all four metrics while requiring the fewest simulation frames. Compared with random selection, it reduces SR MAE by 22.2% and frame cost by 24.1%. These results demonstrate that route selection contributes beyond calibration at the same route count.

.

.

[DCVK Q2] What normalization or constraints are used before clustering to address transformations that preserve success probabilities but change Euclidean distances?

[DCVK A2] Thank you for raising this point. Hierarchical priors provide soft constraints on parameter location and scale. We use Normal priors on θi\theta_i, aja_j, and bjb_j, with Normal hyperpriors on their means and Gamma hyperpriors controlling their scales. These priors regularize estimation by discouraging extreme parameter values during fitting. Clustering uses the posterior means of [aj;bj][a_j;b_j] without additional normalization.

We follow TinyBenchmarks and MiniLongBench and will release our code. Specifically, we adopt their hierarchical-prior configuration for parameter estimation within our response-based, cost-aware compression framework. We will supplement the manuscript with these details and release the training configurations and fitted parameters alongside the code to support reproducibility.

.

.

[DCVK Q3] How is CE normalized and aggregated, and how does it support the 77-route budget?

[DCVK A3] Thank you for requesting this clarification. CE balances prediction accuracy and evaluation cost. Following the accuracy-cost aggregation principle used in RouterArena [1] and VL-RouterBench [2], we take the harmonic mean CE=2,Acc,Eff/(Acc+Eff)\mathrm{CE}=2\\,\mathrm{Acc}\\,\mathrm{Eff}/(\mathrm{Acc}+\mathrm{Eff}). Here, Acc=1−(m−mmin⁡)/(mmax⁡−mmin⁡)\mathrm{Acc}=1-(m-m_{\min})/(m_{\max}-m_{\min}), where mm is SR MAE in the budget analysis and the bounds are its minimum and maximum across candidate budget points. This normalizes error to [0,1][0,1]. We define Eff=1−c\mathrm{Eff}=1-c, where cc is the retained fraction of full-benchmark simulation frames.

77 routes provide a practical default, and users can choose other budgets. They achieve low SR estimation error at a small fraction of full-benchmark frame cost, with a CE score near the peak in Fig. 3. We will also release configurations with different route counts, allowing users to choose the accuracy-cost balance that best suits their needs.

.

.

[1] RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers. ICLR 2026.

[2] VL-RouterBench: A Benchmark for Vision-Language Model Routing. CVPR 2026

AuthorsOpenReview ↗

Response to Reviewer DCVK Part 2

[DCVK Q4] Could you clarify which model records are included in the skill MAEs in Table 3?

[DCVK A4] Thank you for checking the aggregation. The reported MAEs include two additional models with skill results. The displayed model list followed the four-metric comparison in Table 2. These two models lack Efficiency and Smoothness results and were therefore excluded from that comparison, while their available skill results remained included in the skill MAEs. This accounts for the difference between the reported MAEs and those recalculated from the displayed rows.

Their skill results are provided below. Each entry reports the full-benchmark score followed by the FastB2D prediction.

Additional recordOvertakingMergingEmergency BrakeGive WayTraffic SignMean Ability
173.3 / 67.448.8 / 47.784.2 / 82.150.0 / 49.460.0 / 62.663.3 / 61.8
262.2 / 61.269.4 / 65.288.3 / 86.750.0 / 50.281.6 / 77.570.3 / 68.2

We will correct the table presentation to clarify the aggregation scope. The existing skill scores and aggregate MAEs remain unchanged.

.

.

[DCVK Q5] How do individual skill-score errors affect model diagnosis, particularly for DriveTransformer and PDM-Lite?

[DCVK A5] We appreciate this careful and accurate observation. FastB2D prioritizes overall Success Rate, the most widely adopted closed-loop metric, without a separate skill-specific objective, as stated in lines 159–161. Skill estimates can therefore vary in accuracy. Overall-SR reliability is consistently validated across families and performance ranges in Tables 2 and 6 and Appendix F.5.

Skill scores provide supplementary references for identifying weaknesses. Across the eleven models in Table 3:

Diagnostic measureResult
Mean within-model Spearman correlation across five skills0.955
Models with the same weakest skill10/11

The two weakest skills are preserved for both DriveTransformer and PDM-Lite. For DriveTransformer, these are Give Way and Traffic Sign. For PDM-Lite, Give Way remains the weakest skill, followed by Merging.

.

.

[DCVK Q6] How do the two SR estimators perform outside the construction range, and does the gate select the more accurate one?

[DCVK A6] Thank you for requesting this analysis. The gate selects the lower-error estimator in these OOD examples. Following Section 2.4, we use the threshold of 5 percentage points and switch to the calibrated estimator when the prediction gap exceeds this threshold.

Model / recordSR rangePredictive errorCalibrated errorPrediction gapGate choiceFinal error
DriveTransformer run2<50%9.922.1412.06Calibrated2.14
UniAD-Tiny<50%16.771.1815.59Calibrated1.18
VAD<50%15.882.5718.45Calibrated2.57
BridgeDrive-LEAD>75%7.521.146.38Calibrated1.14
UniAD-Base<50%14.862.7817.64Calibrated2.78

These comparisons complement Figs. 4 and 5 with direct evidence that switching can improve prediction accuracy beyond the construction range.

Thank you to the authors for the further clarification. My concern has been addressed. As I have given WA to this paper ready, I will maintain the score.

Reviewer_HpJXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FastB2D compresses Bench2Drive evaluation by selecting a cost-aware set of representative routes and estimating full-benchmark scores from anchor-route outcomes. It combines a multidimensional item-response model, route clustering and selection, and calibrated score estimation. The resulting benchmark uses 77 of 220 routes and is evaluated across planners, selection baselines, and several generalization settings.

Critical strengths

(+) The paper addresses an important practical bottleneck with a clear outcome: evaluation time decreases from 38.7 to 11.4 hours, while reported test-set mean absolute errors are 1.20 percentage points for success rate and 0.86 for driving score. This is a useful accuracy-cost trade-off for iterative development.

(+) The empirical analysis is extensive. It includes random and learned selection baselines, budget and cost trade-offs, low- and high-performing planners, additional model families, and transfer to EmBench. The appendix also examines the effect of excluding the original base models, which directly addresses a potential dependence concern.

Critical weaknesses

(-) FastB2D primarily addresses the trade-off between evaluation cost and fidelity to the full benchmark. While the reported savings and low average estimation errors are encouraging, their implications for practical model selection remain unclear. In particular, a low aggregate MAE and high rank correlation do not establish whether the proxy can reliably distinguish closely performing models. Reporting pairwise ranking accuracy as a function of the full-benchmark score gap, together with uncertainty estimates across evaluation budgets, would clarify when the efficiency gains justify the loss in evaluation fidelity.

(-) The agreement between FastB2D and the full Bench2Drive benchmark is demonstrated on a finite set of evaluated models. Although the experiments on unseen model families are encouraging, it remains unclear how representative these models are of the broader range of behaviors and failure patterns encountered in practice. Accurate score recovery on the tested models does not necessarily imply reliable evaluation of a new model whose failures are concentrated on omitted routes. A broader and more behaviorally diverse evaluation set, together with an analysis of the conditions under which the approximation breaks down, would strengthen the evidence for general applicability.

Questions for authors

Q1: FastB2D offers substantial evaluation savings at the cost of approximation error. Could the authors clarify whether this trade-off preserves reliable model selection, particularly when competing models have similar full-benchmark scores?

1 comment
AuthorsOpenReview ↗

Response to Reviewer HpJX

We thank the reviewer for recognizing our useful accuracy-cost trade-off and extensive empirical analysis. We appreciate your suggestions on model selection and generalization, which we address below.

.

.

HpJX q1 Can FastB2D reliably distinguish closely performing models, and how does reliability vary with score gap and evaluation budget?

HpJX a1 Thank you for this practical question. We directly measure pairwise ranking accuracy. The table groups held-out Bench2Drive model pairs by their full-benchmark SR gap and compares two route budgets.

Budget0.00 < gap ≤ 0.010.01 < gap ≤ 0.020.02 < gap ≤ 0.05Gap > 0.05
77 routes, 28.06% cost65.0 %79.0 %87.6 %100.0 %
110 routes, 43.18% cost70.8 %91.6 %99.7 %100.0 %

(1) The default budget provides useful ranking accuracy at low cost. With only 28.06% of full-benchmark cost, FastB2D achieves 79.0% accuracy for gaps of 0.01–0.02, 87.6% for gaps of 0.02–0.05, and 100% above 0.05, supporting efficient initial model screening.

(2) More evaluation improves close-model ranking while preserving substantial savings. Increasing the budget to 110 routes raises accuracy to 91.6% for gaps of 0.01–0.02 and 99.7% for gaps of 0.02–0.05, while using only 43.18% of full-benchmark simulation frames. We will release both budget configurations so users can choose between faster screening and more accurate comparisons.

.

.

HpJX q2 Does FastB2D generalize to diverse behaviors and failures concentrated on omitted routes, and when can the approximation fail?

HpJX a2 Thank you for raising this concern.

(1) FastB2D generalizes well across diverse model families. Table 2 validates held-out families including DriveTransformer, LEAD, and TransFuser++. Table 12 further tests AutoMoT and SparseDriveV2 without refitting, achieving SR MAEs of 0.85 and 0.83 percentage points. Table 6 extends validation to SR around 13–15% and above 90%, outside the 50–75% construction range. Rebuilding from 30 entirely independent records also yields comparable SR MAE, 2.37 versus 2.32 points, in Table 12.

(2) FastB2D identifies high-risk unobserved routes. After observing the 77 anchors, Eq. 3 predicts success probabilities for all 143 unobserved routes. Appendix F.6 directly validates these predictions in Table 13:

Risk-screening metricResult
Unobserved-route ROC AUC0.913
Recall of the top 25% most failure-prone routes97.2%, or 35/36
Recall of the top 50% most failure-prone routes94.4%, or 68/72
Recall of the top 75% most failure-prone routes97.2%, or 104/107

These results support targeted checks on routes with high predicted risk. Appendix C.8 also reports coverage of 40 of 44 scenario types, with the four absent types represented by behaviorally related retained scenarios in the evaluated records.

(3) Accurate estimates across different failure-rate patterns. We add results for BLUE [1] and FiveVLA [2], covering cases where failure rates are higher on either the anchor routes or the omitted routes. Estimation error remains low in both cases: across the four checkpoint groups, SR MAE ranges from 0.49 to 1.21 points, and DS MAE from 0.11 to 1.24 points. Low errors persist when unobserved routes fail more often.

Checkpoint groupFailure rate on 77 tested routesFailure rate on 143 unobserved routesSR MAE, pointsDS MAE, points
BLUE23.38%24.48%0.830.11
FiveVLA ckpt124.68%26.57%0.491.24
FiveVLA ckpt231.17%23.08%1.130.46
BridgeDrive-PDM29.87%23.78%1.210.17

(4) FastB2D is a validated proxy for efficient model development. Cross-model, distribution-shift, and unobserved-route tests support its practical reliability. For formal and safety-related conclusions, we recommend full Bench2Drive evaluation, as stated in Appendices F.1 and F.9. If a new model's anchor outcomes remain unchanged while failures emerge only on unobserved routes, the change cannot be reliably detected from anchor outcomes alone. This is an inherent limitation of partial observation, not a problem specific to FastB2D.

.

.

[1] BLUE: Toward Better Language Use in Efficient Vision-Language-Action Models for Autonomous Driving. EMNLP 2026.

[2] FIVE-VLA: Fast and EffectIVE Closed-Loop Autonomous Driving with Recurrent Action Memory. NeurIPS 2026.

Reviewer_4PkfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors focus on the route evaluation costs of the existing closed-loop autonomous driving benchmark and propose FastB2D by learning representations from historical model evaluations and selecting low-cost routes through clustering. The experimental results confirm that the proposed benchmark can reduce the evaluation time.

Critical strengths
  1. I think the motivation and the methodology are clear and easy to follow.
  2. The proposed benchmark achieves a considerable reduction in evaluation time and aligns well with the original benchmark.
Critical weaknesses
  1. My major concern is the novelty of the proposed approach for constructing the benchmark. The cost-aware selection in Eq. 2 is essentially a weighted combination of normalized cluster distance and evaluation cost.
  2. The diversity of the construction data is another concern, as most construction records are generated by modifying the training and inference configurations of only two base models, SimLingo and CriticVLA.
Questions for authors

Table 3 contains some relatively large errors in individual driving-skill scores despite low aggregate prediction errors. Could the authors clarify how these discrepancies affect the reliability of FastB2D for skill-level comparison and model diagnosis? How should practitioners interpret the risk of missing failures on removed scenarios?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4Pkf

We thank the reviewer for recognizing our clear motivation and methodology, substantial evaluation-time reduction, and strong alignment with the original benchmark. We appreciate your constructive suggestions and clarify the following points.

.
.

4Pkf q1 Beyond the simple weighted combination in Eq. 2, what is the novelty of FastB2D?

4Pkf a1 Thank you for this question. FastB2D turns structural benchmark inefficiencies into over 70% faster closed-loop evaluation. Our contributions extend beyond Eq. 2, as summarized in Appendix A:

  1. Structural insight. Route redundancy and cost imbalance recur across closed-loop AD benchmarks. Fig. 1 shows that the most expensive 20% of Bench2Drive routes account for nearly half the evaluation cost, while many subsets provide overlapping measurements. Appendix F.2 finds a similar pattern on EmBench.
  2. Counterintuitive finding. Routes with similar semantic labels can reveal very different model behaviors. FastB2D provides substantially more accurate full-benchmark estimates than the expert-selected DEV10, supporting response-based selection, as shown in Appendix C.6.
  3. Affordable evaluation protocol. FastB2D cuts evaluation time by 70.5%, from 38.7 to 11.4 hours. Three seeds take 34.2 hours, within the cost of one full run.
  4. Community resource. We will release over 200 complete evaluation records, representing approximately 8,700 A100 GPU-hours, to support benchmark research without regenerating this pool, as documented in Appendix C.7.
  5. Enabling richer benchmarks. Lower testing costs free evaluation budget for more scenarios and harder, nonredundant corner cases.

.
.

4Pkf q2 Is construction from two base-model families sufficiently diverse, or does FastB2D favor these families?

4Pkf a2 Thank you for raising this concern.

  1. FastB2D generalizes well to held-out model families. Table 2 and Appendix C.1 validate diverse families absent from construction, including DriveTransformer, LEAD, and TransFuser++. Appendix F.5 further tests AutoMoT and SparseDriveV2 without refitting, achieving SR MAEs of just 0.85 and 0.83 percentage points. AutoMoT uses cameras and LiDAR, extending validation across sensor modalities. Among the nine models in Table 2, SimLingo remains eighth and CriticVLA fifth under both full-benchmark and predicted SR rankings, giving no indication of preferential ranking for these families.
  2. The final release will span more model families. Public AD models supporting closed-loop Bench2Drive evaluation remain limited. We reserve more families for independent testing in this analysis version. As described in Appendix C.1, we will additionally release a version trained on all available records across more model families.

.
.

4Pkf q3 How do individual skill-score errors affect skill comparison and model diagnosis?

4Pkf a3 We appreciate this careful and accurate observation. FastB2D prioritizes overall Success Rate, the most widely adopted closed-loop metric, without a separate skill-specific objective, as stated in lines 159–161. Skill estimates can therefore vary in accuracy. These discrepancies do not undermine overall-SR reliability, consistently validated across families and performance ranges in Tables 2 and 6 and Appendix F.5.

Skill scores provide supplementary references for identifying weaknesses. Across the eleven models in Table 3:

Diagnostic measureResult
Mean within-model Spearman correlation across five skills0.955
Models with the same weakest skill10/11

For finer-grained assessment, selection and estimation can be optimized directly for the target skill.

.
.

4Pkf q4 Could FastB2D miss failures on the 143 unobserved routes, and which routes should users test further?

4Pkf a4 We appreciate this important concern. Actually FastB2D still retains visibility into unobserved routes. After observing the 77 anchor outcomes, Eq. 3 outputs a predicted success probability for every unobserved route.

FastB2D reliably identifies high-risk routes in the evaluated test set. Appendix F.6 reports the following results across all 143 unobserved routes in Table 13:

MetricResult
Unobserved-route ROC AUC0.913
Top 25% failure-prone route recall97.2%, or 35/36
Top 50% failure-prone route recall94.4%, or 68/72
Top 75% failure-prone route recall97.2%, or 104/107

The high AUC and recall show that FastB2D recovers nearly all of the most failure-prone routes, although perfect failure detection is not guaranteed. In practice, users can run the 77 anchors for routine evaluation, then perform targeted checks on unobserved routes with the highest predicted risk. Formal results should still use the full benchmark, as stated in Appendices F.1 and F.9.

Thanks for your reply. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.