acceptodds
Under review as a conference paper at ICLR 2027

CLOVER: Closed-Loop Value Estimation and Ranking for End-to-End Autonomous Driving Planning

23 views

Abstract

End-to-end autonomous driving planners are commonly trained by imitating a single logged trajectory, yet they are evaluated by rule-based planning metrics that measure safety, feasibility, progress, and comfort. This creates a training–evaluation mismatch: trajectories close to the logged path may still violate planning rules, while alternative trajectories farther from the demonstration can remain valid and high-scoring. The mismatch is especially limiting for proposal-selection planners, whose performance depends on both candidate-set coverage and scorer ranking quality. We propose CLOVER, a Closed-LOop Value Estimation and Ranking framework for end-to-end driving planning. CLOVER first expands single-trajectory imitation into set-level proposal coverage by constructing evaluator-filtered pseudo-expert trajectories. It then performs conservative closed-loop self-distillation: a trajectory-level scorer is fitted to true evaluator sub-scores on generated proposals, while the generator is refined toward teacher-selected top-k and vector-Pareto proposal targets with stability regularization. We also analyze when an imperfect scorer can improve the generator, showing that scorer-mediated refinement is reliable under local scorer accuracy, conservative updates, and selected-set enrichment. On NAVSIM, CLOVER achieves 94.5 PDMS and 90.4 EPDMS, establishing a new state-of-the-art performance. On the more challenging NavHard split, it obtains 48.3 EPDMS, matching the strongest reported result. Supplementary evaluations on nuScenes and Bench2Drive cover open-loop planning and reactive closed-loop driving, respectively: CLOVER achieves the lowest nuScenes L2 error and collision rate among compared methods and reaches a 70.5% success rate and an 89.2 Driving Score on Bench2Drive. Code will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, vision-language models, representation complementarity, vision-language-action models, fast-slow systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G49qOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CLOVER addresses a critical issue: simply imitating a single human trajectory during training does not equate to learning to generate a variety of safe, feasible behaviors with high planning scores. The method first expands the candidate coverage using pseudo-expert trajectories selected by an evaluator; it then alternately trains the scorer and uses the top-k or Pareto trajectories selected by the scorer to supervise the generator. The contribution lies in first providing the planner with more valid options, and then conservatively utilizing an imperfect scorer to improve the candidate set and the final selection.

Critical strengths

The paper keeps the original backbone and improves DrivoR's score on NAVSIM v1 by adjusting the training method without introducing any other information. The analysis of candidate trajectories is also quite thorough. The results show that Stage 1 does indeed improve the generated candidates, and it's not just about the scorer being more selective. Table 8 also illustrates the necessity of alternating updates: synchronous updates of the generator and scorer will crash, while alternating updates maintain stability.

Critical weaknesses

The result on NAVSIM v2 (EPDMS 90.4) requires further clarification. The core model achieves only 89.3, the 90.4 score relies on an additional, monomodal anchor trained to suppress trajectories that deviate too far from the 64 candidates. However, the paper does not clearly explain how the anchor’s weights were determined. Appendix D states that the search was conducted on a “validation set,” but NAVSIM does not have a commonly used validation set independent of navtest; therefore, it must be clearly specified which partition was used here. The role of the anchor also highlights a more direct issue: without the anchor, the EC is only 35.0, while other methods all exceed 75. The explanation provided in the paper is that the model generates a set of candidates with similar scores but different behaviors; however, since the highest-scoring candidate is selected independently for each frame, the model might move left in one frame and right in the next. This explanation does not resolve the issue; rather, it highlights the lack of continuity in frame-by-frame selection. A method named “closed-loop” yet clearly lagging behind existing methods on the very metric directly related to continuous decision-making is, in my view, a shortcoming that needs to be addressed head-on.

The differences between the two stages and existing methods have not yet been clearly explained. Stage 1 enumerates lateral offsets, speed, and acceleration/deceleration in the lane coordinate system, filters them using the PDM evaluator, and then uses a min-of-K loss to have the generator cover these “pseudo-expert” trajectories. Scoring by a rule-based teacher and then distilling the results to the model is already a practice adopted by the Hydra-MDP series; the change here is replacing a fixed vocabulary with network-generated candidates. The paper lacks a comparison using “the same backbone, the same scorer, and vocabulary distillation,” so it is impossible to determine whether the improvement stems from the learned candidates or from the supervision provided by the PDM. As I understand it, Stage 2 involves having the learned scorer select trajectories, with the generator fitting these trajectories while being constrained not to deviate too far from the previous version, and then refitting the scorer. This process is essentially consistent with methods such as expert iteration, ReST, and rejection-sampling fine-tuning, except that it is applied to trajectory generation. The paper devotes an entire section and a figure to emphasize that “this is not RL,” but what requires further explanation is: how exactly does it differ from “one-step offline RL combined with a learned reward” in terms of specific training objectives and update mechanisms?

Questions for authors
  1. On which dataset was the parameter search in Table 13 performed?

  2. Were anchors used in the NavHard results in Table 4? Please list both versions of EPDMS and EC with and without anchors.

  3. What is the inference latency with anchors? The 110 ms mentioned in the text refers to the case without anchors, correct?

  4. In most NAVSIM scenarios, the difference is observed only in EP. Does this mean the Pareto set degenerates into “the top EP”? Could this explain why EP outperforms humans and why EC collapses when anchors are not used?

  5. Eq. 97 breaks down the top-1 improvement into two components: an increase in the candidate pool upper bound and a reduction in ranking error, but the paper does not report specific values. According to Table 2, the Oracle@64 for Stage 2 decreased from 0.9976 to 0.9939, a drop of 0.37 percentage points; the gap narrowed from 0.0563 to 0.0491, a reduction of 0.72 percentage points. When these two figures are offset against each other, the result is exactly a 0.35 percentage point improvement in top-1. Please provide this breakdown and explain exactly how Stage 2 improved the candidate distribution itself.

  6. Does the “frozen teacher” in Stage 2 replicate a copy from the current student in each cycle, or does it remain unchanged throughout the entire 30 cycles? If the former, it effectively acts as a proximal constraint on the previous round, and the entire Stage 2 amounts to repeatedly performing filtered-BC; if the latter, the target set is already fixed, so I don’t understand why 60 epochs are still needed.

  7. What exactly is the configuration for “Non-iterative generator/scorer update” in Table 8? Does it mean the generator and scorer are backpropagated together in the same batch, or does it stop after just one round of scorer updates and one round of generator updates? What kind of trajectory does the model output when it crashes below 10 points?

  8. The range of PDMS for the three seeds in Table 21 is only 0.014, and the EPDMS is also 0.014 which is unusually small. Did these three runs use the same Stage 1 checkpoint? Please specify exactly what changes were made between the three runs.

7 comments

We thank the reviewer for the detailed analysis of our method and experiments. The questions raise important points regarding anchor configurations, Pareto supervision, the contributions of the two training stages, and the mechanisms underlying alternating optimization. We address the eight questions first, followed by the two major concerns raised in the Critical Weaknesses section.

Responses to Questions

Q1. Dataset Split Used for the Parameter Search in Table 13

We followed the train_logs / val_logs configuration provided in the NAVSIM training setup.

Specifically, the val_logs subset belongs to the NAVSIM navtrain data domain, rather than the independent navtest evaluation split. Since this subset was also included in model training, it is more accurately described as an internal subset used for parameter selection rather than a strictly independent held-out validation set.

This setup is consistent with the training configurations commonly adopted in NAVSIM-based planning methods.

We will clarify the dataset partition and parameter-search procedure in the revised manuscript.

Q2. Anchor Usage and Results Without Anchors on NavHard

Yes, the NavHard results in Table 4 use an anchor. The main results reported under the EPDMS metric use the corresponding anchor-assisted configuration, which was not sufficiently explicit in the original presentation.

We additionally evaluated CLOVER on NavHard without an anchor:

ConfigurationEPDMSStage 1 EPStage 1 ECStage 2 EPStage 2 EC
CLOVER + anchor48.375.756.074.257.3
CLOVER without anchor43.276.336.477.330.7

The results follow the same trend as the NAVSIM v2 navtest evaluation.

Without an anchor, CLOVER achieves higher EP in both stages, while EC decreases substantially. The anchor provides a more favorable trade-off between trajectory flexibility and temporal consistency.

We will explicitly label the anchor configurations in the revised tables.

Q3. Inference Latency with the Anchor

The reported 110 ms includes only the primary CLOVER model.

The default anchor uses the same base architecture with a separate set of weights. Since the two models perform largely independent inference, they can be executed concurrently.

The measured total inference latency with the default anchor is 137.2 ms on a single NVIDIA A100 GPU, compared with 110 ms for the primary CLOVER model alone.

This is substantially faster than fully sequential execution of the two models, which would take approximately 220 ms.

We will clarify this inference configuration and its latency in the revised paper.

Q4. Does the Pareto Set Degenerate into Selecting the Highest-EP Trajectories?

We believe the distinction between maximizing EP and selecting high-quality trajectories under multiple planning objectives is important.

First, the performance differences are not restricted to EP. In the NAVSIM v1 comparison (Table 1), CLOVER ranks first or second across all reported sub-metrics. On NAVSIM v2 navtest and NavHard, CLOVER does not achieve the highest EP among the compared methods, yet remains highly competitive in overall planning performance.

More generally, an appropriate autonomous driving objective should encourage progress toward the destination while satisfying safety, traffic-rule, and comfort requirements, rather than maximizing progress in isolation.

Regarding the Pareto set, its construction provides a direct mechanism for preserving these competing objectives.

From a multi-objective optimization perspective, Pareto selection operates on a vector of planning metrics rather than a single scalar score. A trajectory with lower EP can remain non-dominated if it provides improvements in other objectives that cannot be simultaneously matched by a higher-EP alternative.

Therefore, unless the different planning objectives admit no meaningful trade-offs, Pareto selection does not reduce to simply choosing the trajectories with the highest EP. Its purpose is precisely to preserve meaningful trade-offs that would otherwise be discarded by a single-objective selection rule.

Furthermore, NAVSIM's EP is normalized against a reference progress value and capped at 1. Once this upper bound is reached, traveling farther does not yield an additional EP reward.

A representative example is scene a1eab14904fc5807. Among the 64 candidate trajectories, we identified the following two non-dominated solutions under the true evaluator:

CandidateNCDACEPTTCComfortDDCPDMS
#25111.00000000.4167
#44110.94961110.9790

Candidate #25 achieves the maximum EP of 1.0, but performs poorly on TTC, comfort, and driving-direction compliance.

Candidate #44 sacrifices approximately 0.05 EP while substantially improving the other metrics, increasing its final PDMS from 0.4167 to 0.9790.

Both trajectories lie on the true Pareto frontier because neither dominates the other across all objectives. A selection rule based solely on maximum EP would discard candidate #44, despite its much higher overall planning quality.

This example directly illustrates why multi-objective Pareto supervision provides information beyond EP maximization.

Therefore, our results do not support the hypothesis that Pareto selection has degenerated into selecting only the highest-EP trajectories.

Regarding the relatively low EC without an anchor, we attribute this primarily to the interaction between independent cross-frame selection and the pseudo-closed-loop evaluation protocol, as discussed further in W1.

Q5. Numerical Decomposition of Eq. 97 and the Actual Contribution of Stage 2

We agree with the numerical decomposition provided by the reviewer.

According to Eq. 97:

ΔTop1=ΔOracle−ΔGap.\Delta \mathrm{Top1} = \Delta \mathrm{Oracle} - \Delta \mathrm{Gap}.

From Stage 1 to Stage 2:

  • Oracle@64: 0.9976 → 0.9939, a change of −0.0037.
  • Ranking gap: 0.0563 → 0.0491, a reduction of 0.0072.
  • Selected Top-1: 0.9413 → 0.9448, an improvement of 0.0035.

Thus,

ΔTop1=−0.0037+0.0072=+0.0035.\Delta \mathrm{Top1} = -0.0037 + 0.0072 = +0.0035.

This confirms that the improvement in the selected Top-1 trajectory is primarily reflected in a reduction in ranking regret, rather than a further increase in the already near-saturated Oracle@64.

However, we would like to distinguish this exact statistical decomposition from the independent contributions of the generator and scorer.

Because Stage 2 alternately optimizes both components, the proposal distribution and the scorer evolve together. The oracle score and ranking gap are therefore measured on different proposal distributions at different stages, making it difficult to attribute their changes to either component independently.

To investigate this interaction, we conducted an additional experiment by exchanging the generator and scorer checkpoints between Stage 1 and Stage 2:

GeneratorStage 1 ScorerStage 2 Scorer
Stage 1 Generator94.1393.6
Stage 2 Generator93.394.5

Neither replacing the generator alone nor replacing the scorer alone reproduces the final Stage-2 performance.

Using the Stage-2 generator with the Stage-1 scorer yields 93.3 PDMS, while using the Stage-1 generator with the Stage-2 scorer yields 93.6 PDMS.

Only the jointly refined Stage-2 generator and scorer achieve 94.5 PDMS.

This indicates that the central benefit of Stage 2 lies in the co-adaptation of the generator and scorer to the evolving proposal distribution, rather than an isolated improvement in either module.

Stage 2 also substantially improves the overall quality of the candidate distribution:

  • Mean@64: increases from 0.7374 to 0.8277.
  • Number of low-quality candidates (PDMS below 0.50): decreases from 12.21 to 6.83 per scene.
  • Pairwise ADE: remains at 5.20, substantially higher than the baseline value of 1.80.

These results clarify the complementary roles of the two stages.

Stage 1 primarily expands proposal coverage and improves the oracle upper bound. Stage 2 then refines this expanded distribution by removing low-quality candidates, improving overall proposal quality, and enabling the scorer to select better trajectories from the updated candidate set.

Q6. Frozen Teacher Updates and the Meaning of 60 Epochs

The former interpretation is correct. At the beginning of each alternating cycle in Stage 2, we create a frozen teacher by copying the current student generator.

During the generator-refinement phase, the teacher remains frozen and provides the top-k and Pareto distillation targets. The stability regularization term constrains the student's updates relative to the frozen teacher.

At the next cycle, the teacher is refreshed from the updated student. The scorer is also refitted on the current proposal distribution, producing updated guidance for subsequent generator refinement.

Therefore, the reviewer's interpretation of the frozen teacher as a proximal constraint relative to the preceding iteration is reasonable. Its purpose is to restrict the update magnitude and prevent excessive exploitation of scorer estimation errors.

Importantly, the teacher proposals, scorer, and selected targets evolve across cycles. Stage 2 is therefore not repeatedly fitting a single fixed set of teacher trajectories.

The training schedule consists of 30 alternating cycles, each containing:

  1. One epoch of scorer fitting.
  2. One epoch of generator refinement.

This corresponds to 30 × 2 = 60 training epochs.

The repeated scorer-fitting phases are necessary because generator updates continuously change the candidate distribution on which the scorer is expected to operate.

Reviewer_6ruSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces CLOVER, a closed-loop value estimation and ranking framework for selection-based end-to-end planners. Motivated by the mismatch between single-trajectory imitation and rule-based planning metrics, CLOVER first expands proposal coverage with evaluator-filtered pseudo-expert trajectories, then fits a trajectory scorer to ground-truth metric scores and refines the generator through conservative closed-loop self-distillation. CLOVER demonstrates strong planning and driving performance across NAVSIM, Navhard, and Bench2Drive, with competitive open-loop results on nuScenes as well.

Critical strengths
  1. The paper is well-motivated. It clearly identifies two issues of selection-based planners. The first issue is how to generate diverse high-quality trajectories, while the second one is how to refine and select them through a scorer. CLOVER addresses them by coordinating a generator with a scorer.

  2. The empirical results are impressive. CLOVER establishes state-of-the-art performance on NAVSIM and matches the strongest result on the challenging Navhard split, which indicates strong planning performance. The performance on closed-loop driving (Bench2Drive) is also state-of-the-art, but somehow this result is placed in the appendix rather than the main paper.

  3. The ablation study is rich. Tables on stage-wise contributions, iterative vs. non-iterative refinement, and teacher-set construction (real-PDMS top-k vs. vector-Pareto) provide useful evidence that Stage 1 coverage and Stage 2 conservative distillation are complementary components, both of which are essential to achieving the final performance.

Critical weaknesses
  1. Figure 2 is not very informative about some important details. In particular, it does not clearly illustrate how the trajectory generator produces multiple diverse candidates (e.g., the role of trajectory queries and the decoding process), nor where the pseudo-expert trajectories come from. Readers currently have to dig into the main text and appendix to recover these details.

  2. Relatedly, the main paper could better illustrate the construction of pseudo-expert trajectories in Section 3.2, rather than leaving most of the generation rules and examples to the appendix. Since Stage 1 is one of the two major pillars of CLOVER, a more detailed description in the main body would improve clarity.

  3. As mentioned in the Critical Strengths section, the state-of-the-art result on Bench2Drive should also be highlighted in the main paper rather than the appendix.

Questions for authors

I don't have other questions. Please refer to the Critical Weaknesses section.

2 comments

We sincerely thank the reviewer for the positive assessment of our motivation, empirical results, and ablation studies. We also appreciate the constructive suggestions regarding Figure 2, pseudo-expert trajectory construction, and the presentation of our Bench2Drive results.

During the preparation of the original submission, we considered including the complete Bench2Drive comparison and a more detailed explanation of pseudo-expert construction in the main paper. However, the strict main page limit made it difficult to accommodate these materials while preserving the completeness of the methodology, theoretical analysis, and primary experiments. We therefore placed the additional details in the appendix.

Fortunately, the revision policy permits one additional page in the main paper, allowing us to incorporate these important details more effectively.

We will make the following changes.

1. Improve the Illustration of Candidate Generation in Figure 2

We will clarify the roles of the trajectory queries and the decoder.

Specifically, the visual encoder extracts scene features from the multi-view camera inputs. The 64 learnable trajectory queries then interact with the scene representations through a Transformer decoder to generate diverse candidate trajectories. A trajectory-level scorer subsequently predicts planning-metric sub-scores for each candidate and ranks them for final selection.

2. Add a Pseudo-Expert Construction Diagram to Section 3.2

We will provide a more detailed illustration of the pseudo-expert construction pipeline.

Starting from route centerlines and future obstacle information available during training, we construct candidate trajectories by enumerating lateral offsets, speed profiles, and acceleration/deceleration behaviors. These candidates undergo lightweight geometric pre-filtering, PDM evaluator scoring, and coverage-aware selection to obtain the final pseudo-expert set.

The new figure will also explicitly distinguish the training-time pseudo-expert construction procedure from the neural trajectory generator used during inference.

3. Move the Bench2Drive Results into the Main Paper

We will move the Bench2Drive comparison currently presented in Appendix K, Table 28, into the main paper.

CLOVER achieves 70.5% SR and 89.2 DS, improving over Stage 1 by 1.9 percentage points in SR and 2.4 points in DS.

We believe these revisions will make the key methodological details and empirical contributions of CLOVER substantially clearer. We thank the reviewer again for these valuable suggestions.

Thanks for the reply. I do not have other concerns and I will keep my current rating.

Reviewer_mPiqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

authors propose CLOVER, a framework for end-to-end AV planning that generates multiple trajectories and learns to score them. the motivation is that a single logged trajectory does not capture all valid driving behaviors, and imitation does not directly optimize the eval metrics.

two stages: 1) generate pseudo-expert trajectories using maps and future agent information, filter with the evaluator, then train the generator to cover them. 2) alternate scorer training with generator refinement toward teacher-selected top-k and Pareto trajectories.

eval: NAVSIM v1/v2, NavHard, nuScenes + Bench2Drive. 94.5 PDMS on v1, 90.4 EPDMS on v2 with an extra anchor planner, 48.3 on NavHard. Bench2Drive success is 70.5%, driving score 89.2.

Critical strengths
  • learning from one expert trajectory is limiting. there can be several valid ways to drive in the same scene. authors address both generating these alternatives and choosing between them
  • good analysis of the proposals, not just the final selected trajectory. oracle score, mean score, low-score counts, diversity among high-quality proposals. stage 2 improves mean proposal score from 0.7374 to 0.8277 while keeping more diversity than the baseline
  • several ablations. baseline is 93.7 PDMS, diversity only 94.1, refinement only 93.8, both 94.5. comparisons against scalar top-k and distance suppression are useful
  • eval across several benchmarks, including reactive closed-loop Bench2Drive. stage 2 improves success from 68.6 to 70.5% and driving score from 86.8 to 89.2
Critical weaknesses
  • extra planner, unclear cost. the main NAVSIM v2 result uses an extra planner. this is my biggest concern. Table 3 just says CLOVER, but 90.4 EPDMS requires a separate anchor planner with the same base architecture. without it, CLOVER is 89.3. with it, it's only 0.3 above SparseDriveV2 (90.1). another planner does not sound cheap. does the reported 110ms include both? I'm having a hard time seeing how much improvement there is at comparable compute. why is this not marked in the main table?
  • inconsistent driving decisions. the extra trajectory diversity seems to cause a consistency problem. without the anchor, EC is only 35.0 authors explain that the planner can switch between similarly scored trajectories at successive planning steps. each trajectory may be fine on its own, but switching between different driving behaviors does not sound like good driving. having more options is useful, but the planner still needs to choose consistently. needing another planner to stabilize this seems like a major con, and it feels deceptive.
  • scores vs. driving quality. the evaluator is used to filter pseudo-experts, train the scorer, and refine the generaor. better metric scores make sense, but do they mean better driving? fixed rules can miss context, e.g. when to yield, stop, or go around an obstacle. Bench2Drive helps here, but I would like to understand what behaviors improve beyond the aggregate scores
  • no NavHard improvement. on NavHard, CLOVER ties DrivoR at 48.3. in the second eval stage, NC, DAC, and TTC are slightly worse. EC drops from 76.2 to 57.3. progress improves, but what failures are fixed? I don't see the extra diversity translating into better perf on the harder cases
  • unclear training budget comparison. CLOVER has 25 epochs of stage 1 + 60 refinement epochs, along with extra pseudo-expert supervision. do baselines get the same training budget and evaluator calls? What if we just train the baseline longer or keep updating its scorer? this is unclear
Questions for authors
  1. does 110ms include the anchor? total compute and memory? which tables use it?
  2. which NavHard scenarios improve or get worse? Is the anchor used here?
  3. are training budgets and evaluator calls matched? comparison with longer baseline training or scorer-only updates?

please see more in weaknesses.

I'm open to adjusting my rating depending on authors response to major weaknesses.

3 comments

We thank the reviewer for the careful evaluation and constructive feedback. We provide further clarifications regarding the computational cost of the additional planner, temporal consistency, driving behavior, and training budget.

1. Inference Cost and Fair Comparison with the Additional Anchor

The approximately 110 ms reported in the main paper refers to the inference latency of the primary CLOVER model on a single NVIDIA A100 GPU and does not include the optional anchor.

For our default configuration, the anchor is a separate checkpoint with the same base architecture. Since the two models perform largely independent inference, they can be executed concurrently.

Our measured total inference latency is 137.2 ms, with a total GPU memory footprint of approximately 3.1 GB, on a single NVIDIA A100, compared with approximately 110 + 110 ms for fully sequential execution.

We agree that the use of an anchor should be made explicit in the main comparison. In the revision, we will distinguish between CLOVER without an anchor and CLOVER with an anchor, together with the corresponding computational overhead.

2. Temporal Decision Consistency and Extended Comfort

We thank the reviewer for raising this concern. We would like to clarify an important aspect of the Extended Comfort (EC) metric and its interaction with NAVSIM's pseudo-closed-loop evaluation protocol.

Although EC is named a comfort metric, it primarily evaluates the kinematic consistency of trajectory predictions across consecutive planning frames, including changes in acceleration, jerk, and related motion quantities.

In NAVSIM, however, the history provided to each inference step comes from previously recorded or pre-generated scenes, rather than from the actual execution of the model's preceding decision.

This introduces an implicit approximation: consecutive evaluation scenes are assumed to represent sufficiently consistent states of the same continuously operating planner, so that their independently predicted trajectories can be compared to measure temporal consistency.

This approximation is generally reasonable when a model closely follows the logged driving trajectory. However, CLOVER introduces substantial multimodal pseudo-expert supervision while retaining the original imitation objective, allowing it to select reasonable driving behaviors that differ from the recorded trajectory.

For example, during a lane-change maneuver, the path selected by CLOVER may rapidly deviate laterally from the logged trajectory by approximately 3–4 meters. In such cases, the recorded history provided to the next inference step may differ substantially from the state that would have resulted from executing CLOVER's actual previous decision.

Consequently, the measured EC can be affected by both the consistency of the selected trajectories and the discrepancy between the supplied history and the model's actual planning behavior. This discrepancy can be particularly relevant to planners that preserve multiple valid behavioral modes.

A more direct solution is provided in Appendix D.1, “A Stateful Alternative.” By incorporating the trajectory selected at the preceding planning step into the current selection process, we obtain 90.6 EPDMS without an additional anchor, exceeding the 90.4 EPDMS of our default anchor-assisted configuration.

Furthermore, we did not observe pronounced repetitive decision switching in the reactive closed-loop Bench2Drive evaluation. In this benchmark, subsequent observations reflect the consequences of the model's executed actions, maintaining consistency between the input history and previous decisions. CLOVER exhibits stable planning behavior during continuous driving and achieves 70.5% Success Rate (SR) and 89.2 Driving Score (DS).

3. Behavioral Differences and Performance on Challenging Scenarios

We agree that examining individual driving behaviors provides useful insights beyond aggregate benchmark scores.

On Bench2Drive, we observed that CLOVER more frequently identifies feasible opportunities for proactive obstacle avoidance and overtaking.

For example, in scenario 25854, CLOVER chooses to change lanes and pass a cyclist ahead, whereas SparseDriveV2 and SimLingo tend to wait. We observed similarly effective planning behaviors in challenging scenarios such as HazardAtSideLaneTwoWays and ControlLoss.

On NavHard, we use the anchor-assisted configuration. Our scenario-level analysis suggests that CLOVER can identify more effective planning alternatives in several challenging situations involving potential collisions, vehicle cut-ins, and drivable-area boundaries. For example, CLOVER achieves nearly perfect EP in scenarios 4683eec43a1454ea9 and 87e87ae4d5581f967.

These results indicate that CLOVER provides stronger candidate coverage and progress in challenging situations, while EC remains an important limiting factor in the aggregate NavHard evaluation.

To provide a more systematic behavioral analysis, we will add a breakdown across all 44 Bench2Drive scenario categories, each containing five evaluation routes, for a total of 220 routes.

We will also include scenario-level visualizations and representative trajectory comparisons to illustrate the behavioral differences and performance advantages of CLOVER under different driving conditions.

4. Training Budget and Scorer-Only Improvements

We first clarify the training schedule. Stage 1 jointly trains the generator and scorer for 25 epochs. Stage 2 consists of 30 alternating cycles, each containing one epoch of scorer fitting and one epoch of generator refinement, for a total of 60 training epochs.

Our experiments use four NVIDIA A100 GPUs. As reported in Appendix F, Stage 1 requires approximately 30–40 hours, while Stage 2 requires approximately 60–80 hours.

Regarding longer baseline training, we previously extended both the DrivoR baseline and our CLOVER Stage-1 model to 40 epochs, but observed no meaningful performance improvement in either case.

This suggests that simply extending conventional training does not reproduce the benefits of the proposed training objectives and alternating refinement mechanism.

For scorer-only improvements, Appendix G, Table 20 provides a diagnostic analysis under a fixed pool of 64 candidate trajectories. With identical proposals, DINOv2-Small, DINOv2-Base, and DINOv3-Base achieve test scores of 92.4, 92.6, and 92.7, respectively. Even the substantially larger Wan2.2-5B feature backbone improves the test score only to 92.8, while exhibiting a larger train–test gap.

These results indicate that increasing scorer capacity alone provides relatively limited gains under the existing proposal distribution. The main benefit of CLOVER instead comes from iteratively coordinating the generator and scorer, allowing both components to adapt as the candidate distribution evolves.

Thank you again for your careful review and constructive feedback!

We have provided additional clarifications and experimental evidence addressing your major concerns. If any technical questions or concerns remain, we would be very happy to provide further clarification.

We sincerely appreciate your time and consideration!

Comments on the paper

3 comments

General Response: CLOVER's Contributions, Additional Evidence, and Planned Revisions

We sincerely thank the Area Chair and all three reviewers (mPiq, 6ruS, and G49q) for their thoughtful feedback and constructive discussions.

Throughout the rebuttal, we have provided additional experiments, clarified several aspects of our methodology and evaluation protocol, and identified concrete improvements for the revised manuscript.

Before summarizing the major concerns, we would like to briefly restate CLOVER's motivation and contributions to provide a unified context for the discussion.

Our Motivation and Core Contributions

Motivation. A fundamental limitation of conventional imitation-learning-based planning is that each driving scenario typically provides only one expert trajectory, although multiple safe and reasonable driving behaviors may exist. Moreover, for planners that generate multiple candidate trajectories and select one using a learned scorer, the final performance depends on two closely related factors:

  1. Candidate quality: Whether the generator produces sufficiently diverse, high-quality trajectories.
  2. Selection quality: Whether the scorer can reliably identify the best trajectory among the available candidates.

Existing approaches commonly train the generator to imitate expert trajectories and the scorer to predict planning scores, with relatively limited interaction between these two objectives.

CLOVER addresses both challenges through two complementary training stages:

  • Stage 1: Expanding high-quality trajectory coverage. We construct evaluator-filtered pseudo-expert trajectories using training-time scene information and simulator-based planning evaluation. These trajectories provide multiple valid supervision targets beyond the single recorded expert trajectory, allowing the generator to explore a broader range of feasible driving behaviors.

  • Stage 2: Iterative generator–scorer co-optimization. Rather than keeping the generator and scorer independently optimized, CLOVER establishes an alternating refinement process. The scorer is repeatedly fitted to true planning-metric sub-scores on the evolving candidate distribution. Its predictions then guide the selection of high-quality top-k and vector-Pareto trajectories from a frozen teacher generator, which are used to conservatively refine the student generator. The teacher is refreshed from the updated student in each iteration.

The central motivation of Stage 2 is that the quality of trajectory generation and the reliability of trajectory selection should improve together, rather than treating either component as fixed.

Overall Performance

Both stages contribute to the final performance while retaining the same underlying backbone on NAVSIM.

CLOVER also demonstrates strong reactive closed-loop performance on Bench2Drive. Compared with Stage 1, Stage 2 further improves SR from 68.6% to 70.5% and DS from 86.8 to 89.2.

Below, we summarize the three major topics raised during the discussion and the additional evidence we have provided.

G1. Why Is Iterative Generator–Scorer Co-Optimization Necessary? (G49q, mPiq)

A central question raised by G49q concerns the contribution of Stage 2, particularly whether repeatedly refitting the scorer provides benefits beyond iterative filtered behavior cloning with a fixed scorer.

This question closely relates to our original motivation.

When the generator is optimized using trajectories selected by an imperfect scorer, its proposal distribution gradually changes. A fixed scorer may incorrectly assign high scores to certain low-quality trajectories, especially near the boundaries of its learned distribution. Repeatedly distilling these incorrectly selected trajectories can progressively reinforce the scorer's prediction errors.

CLOVER addresses this issue by alternating generator refinement with scorer recalibration using the evaluator's true planning-metric sub-scores.

We conducted a new, strictly controlled experiment to directly examine this mechanism.

We repeated Stage 2 using exactly the same implementation, including all four loss components, identical λ weights, and the same training settings, with the only difference being that the Stage-1 scorer remained frozen.

Frozen-scorer training epoch13571016
PDMS94.2093.8693.8093.5592.8092.11

The frozen-scorer configuration achieves its best performance at epoch 1, followed by a clear overall degradation as generator refinement continues.

In contrast, the full alternating Stage 2 reaches 94.50 PDMS.

This result highlights two complementary benefits of alternating refinement:

  • The generator can learn from promising trajectories identified by the scorer.
  • Repeated scorer recalibration helps prevent the generator from progressively exploiting incorrect score predictions.

Regarding the training-budget concern raised by mPiq, we additionally extended both the DrivoR baseline and CLOVER Stage 1 to 40 epochs, observing no meaningful performance improvement in either case.

Together, these experiments provide additional evidence that CLOVER's gains arise from its training objectives and iterative coordination mechanism, rather than simply extending conventional training or increasing scorer capacity.

G2. Understanding Anchor Usage and Temporal Consistency in NAVSIM (mPiq, G49q)

Both mPiq and G49q raised important questions regarding the additional anchor planner, its computational overhead, and the relatively low Extended Comfort (EC) score without an anchor.

Anchor configuration and computational cost.

The main NAVSIM results reported using EPDMS in Tables 3 and 4 employ the anchor-assisted configuration.

The original 110 ms inference measurement refers to the primary CLOVER model alone. Our additional measurements show that the complete anchor-assisted configuration requires 137.2 ms and approximately 3.1 GB of GPU memory on a single NVIDIA A100.

The anchor is primarily introduced to improve the temporal consistency measured by EC.

Why does EC present a particular challenge?

NAVSIM uses a pseudo-closed-loop evaluation protocol in which consecutive planning predictions are generated from scene histories derived from recorded logs. These histories are not directly produced by executing the model's preceding decisions.

This distinction matters when a planner selects a reasonable action that differs from the recorded driving behavior.

For example, a planner may choose to change lanes while the recorded vehicle continues in its original lane. At the following evaluation step, the planner receives a history associated with the recorded trajectory rather than the state resulting from its actual lane-change decision.

Consequently, cross-frame consistency measurements can reflect both differences between the planner's selected trajectories and discrepancies between its decisions and the supplied scene histories.

This issue becomes particularly relevant for CLOVER because its multi-trajectory supervision encourages the exploration of alternative driving behaviors.

In Appendix D.1, we also investigated a stateful alternative that incorporates the preceding planning decision. This achieves 90.6 EPDMS without an additional anchor. However, because it introduces dependencies between consecutive inference steps, we retain the anchor-assisted configuration for the main comparisons under the standard NAVSIM evaluation protocol.

Importantly, we did not observe pronounced repetitive decision switching in the reactive closed-loop Bench2Drive benchmark, where subsequent observations reflect the consequences of the planner's executed actions.

We will explicitly clarify the anchor configurations, computational costs, and the distinction between pseudo-closed-loop and reactive closed-loop evaluation in the revised manuscript.

G3. Driving Behavior, Experimental Transparency, and Presentation (mPiq, 6ruS, G49q)

The reviewers also raised valuable questions about whether improvements in planning metrics translate into meaningful driving behaviors, whether experimental settings are sufficiently transparent, and whether the paper presents its core methodology clearly.

Driving behavior and scenario-level evidence (mPiq).

Beyond aggregate metrics, we examined representative driving behaviors on Bench2Drive and NavHard.

For example, on Bench2Drive, CLOVER can identify opportunities to change lanes and overtake a cyclist rather than waiting behind it. We also observed effective planning behaviors in challenging interaction scenarios, including HazardAtSideLaneTwoWays and ControlLoss.

On NavHard, we have clarified the trade-off between trajectory progress and temporal consistency, with EC remaining an important factor in the aggregate EPDMS score.

To provide a more systematic behavioral analysis, we will add per-category results for all 44 Bench2Drive scenario categories, covering 220 evaluation routes, together with additional visualizations and representative trajectory comparisons.

Architectural clarity and pseudo-expert construction (6ruS).

Reviewer 6ruS suggested making the candidate-generation architecture and Stage-1 pseudo-expert construction more explicit, and presenting the Bench2Drive results in the main paper.

We will revise Figure 2 to clarify how the 64 learnable trajectory queries interact with scene features through the Transformer decoder and how the scorer evaluates the generated candidates.

We will also add a dedicated illustration of pseudo-expert construction, including candidate generation, geometric filtering, evaluator scoring, and trajectory selection.

Furthermore, the revised manuscript will move the Bench2Drive comparison from the appendix into the main paper. The additional page permitted in the revision makes these improvements possible.

Experimental details and reproducibility (G49q, mPiq).

We have additionally clarified:

  • The dataset subset used for anchor parameter selection, which belongs to the NAVSIM navtrain domain.
  • The two alternating optimization phases in Stage 2 and the respective roles of the evaluator, scorer, and frozen teacher.
  • The Stage-2 training schedule and the setup of the random-seed experiments.
  • The distinction between multi-objective vector-Pareto selection and simply selecting trajectories with the highest progress score.

These clarifications will be incorporated into the corresponding methodology and experimental sections.

Summary of Planned Revisions

Based on the reviewers' feedback, we will:

  1. Improve the main architectural figure and add an illustration of the pseudo-expert construction pipeline.
  2. Move the Bench2Drive comparison into the main paper, highlighting reactive closed-loop driving performance.
  3. Add comprehensive Bench2Drive scenario-level results and visualizations, covering all 44 categories and 220 routes.
  4. Clarify the NAVSIM anchor configurations, their measured computational costs, and the interpretation of temporal consistency.
  5. Expand the description of Stage-2 alternating optimization and experimental settings, incorporating the additional controlled ablation and reproducibility details.

We sincerely appreciate the reviewers' engagement throughout the discussion. Their questions have helped us strengthen the experimental evidence, clarify CLOVER's methodological contributions, and identify concrete improvements to the manuscript.

We hope this consolidated response provides a clear overview of CLOVER's contributions and the additional evidence established during the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.