acceptodds
Under review as a conference paper at ICLR 2027

Harnessing Learned Planners: Efficient Reinforcement Fine-Tuning with Rank-Tree Sampling

Abstract

Learned multi-modal planners can be harnessed with inference-time search: wider search can recover useful alternatives, particularly on difficult cases, but increases inference latency. In this paper, we investigate how to transfer the gains from inference-time search back into the planner itself through reinforcement fine-tuning, while keeping the cheap deployment search configuration unchanged. The challenge is constructing informative rollout groups when planning involves both discrete continuation decisions and continuous proposal geometry. Naively enumerating initial proposals with greedy continuations can miss useful later decisions, while groups with identical returns provide no learning signal under group-relative optimization. We introduce group-based rank-tree sampling, which enumerates short top-K continuation-rank schedules under a frozen reference planner, combines them with independent geometry samples, and avoids global beam pruning during training. This provides structured coverage of discrete planning decisions while preserving variation in continuous proposal geometry. We further incorporate failure-aware reward shaping and a bounded imitation-gradient contribution to stabilize learning. On NAVSIM v1, fine-tuning improves the driving planner's PDMS Score from 93.7 to 93.9 under identical evaluation settings. On an internal benchmark of dead-end parking scenarios, fine-tuning improves the success rate from 84.7% to 92.2% using a fixed beam-width-8/top-6 evaluator, exceeding the 91.4% the original model reaches only with a wider beam-width-30/top-15 search. In zero-shot evaluation on the public ParkBench dataset, the fine-tuned planner also improves the success rate from 68.6% to 72.5%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Path planning, Reinforcement learning, Finetune, Harness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_M7hDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces reference-ranked continuation schedules to construct rollout groups for reinforcement fine-tuning of learned planners while retaining their deployment search. It reports improved parking success under fixed beam search and a small NAVSIM gain from a separate, single-step DrivoR specialization.

Critical strengths
  • Meaningful improvement under fixed deployment search. On the 359-scene dead-end suite, success increases from 84.68% to 92.20% with the same beam configuration, reaching the performance of a substantially wider-search baseline.

  • Concrete sampling design with supporting evidence. The matched 120-rollout probe improves successful-rollout coverage from 85.9% to 89.7%. The appendices clarify the independent geometry sampling, policy likelihoods, and collision-checking procedure.

Critical weaknesses
  • The final benefit of rank allocation remains insufficiently isolated. Table 2 provides an equal-budget pre-training probe, but the paper does not report final deployment performance for the expanded-legacy control with 120 rollouts per scene.
Questions for authors
  • Does the parking baseline include MultiPark’s original argmax fine-tuning with outcome-oriented losses, or only teacher-forced imitation training?

  • Could you clarify how the imitation-learning and RL training data are separated from the dead-end and Mini-5K evaluation suites, including any checks for overlapping scenes or closely related variants?

3 comments
AuthorsOpenReview ↗

Response for Reviewer M7hD — Part 1

Thank you for recognizing the improvement under fixed deployment search and the clarity of our sampling design and implementation details. We appreciate your request for a stronger training comparison and clearer descriptions of the starting model and data separation. We address the weakness first, followed by the two questions in the next part.

Response to the Weakness

To address the missing comparison, we have now completed end-to-end training of the 120-rollout expanded-legacy control. We also conducted a reduced-budget experiment with 28 rollouts per scene. These experiments provide final-model results beyond the original pre-training coverage probe.

W1. Final performance at matched and reduced rollout budgets.

120-rollout control. We trained an expanded-legacy control corresponding to the D=0D=0 specialization of our sampling framework. For each of the same 30 initial queries, we independently sample four versions of the motion geometry, giving 30×4=12030\times4=120 rollouts per scene.

At later decisions, every rollout selects the highest-ranked legal continuation while continuing to sample geometry. Thus, D=0D=0 does not mean that parking ends after one segment: the rollout can continue for up to eight segments, but it does not explore alternative continuation ranks. This provides the requested comparison with the four continuation-rank schedules used by the main K=2,D=2K=2,D=2 configuration.

Both 120-rollout runs use seed 42 and share the starting checkpoint, training data, reward shaping, failure-group gating and damping, imitation anchor, optimizer settings, batch size, and training duration.

28-rollout reduced-budget experiment. We also retain only the top seven initial queries while keeping K=2,D=2K=2,D=2. This gives

B=MKD=7×22=28B=MK^D=7\times2^2=28

rollouts per scene—slightly fewer than the original legacy configuration’s 30 rollouts. The training data, reward shaping, failure damping, imitation anchor, optimizer settings, and training duration remain unchanged from legacy.

Both additional experiments use seed 42 and predefined epoch-20 checkpoints. The main D=2D=2 result remains the paper’s epoch-15 checkpoint, selected using closed-loop evaluation on the training corpus. The test benchmarks were not used to select these checkpoints.

Results under unchanged deployment search. All models below use the same beam-width-8/top-6 evaluator and success and collision rules. Counts in parentheses are successful scenes.

Model / training allocationDead-end (359)Mini-5K (5,000)
Starting model84.68% (304)99.32% (4,966)
Legacy, 30 rollouts83.01% (298)99.34% (4,967)
Rank-tree, K=2,D=2K=2,D=2, 28 rollouts87.19% (313)99.38% (4,969)
Rank-tree, K=1,D=0K=1,D=0, 120 rollouts92.48% (332)99.36% (4,968)
Rank-tree, K=2,D=2K=2,D=2, 120 rollouts92.20% (331)99.34% (4,967)

What these experiments show.

The 28-rollout run improves the dead-end success count from 304 to 313: nine net additional successes, or +2.51 percentage points. This demonstrates an improvement in this run with fewer rollouts than the original legacy configuration. However, choosing seven initial queries also changes which starting proposals are explored. We therefore present it as reduced-budget evidence, rather than an isolated test of continuation-rank allocation.

At 120 rollouts, the D=0D=0 and D=2D=2 configurations give similar success rates, with the D=0D=0 run solving one more dead-end case. This suggests that deeper continuation exploration is not necessary for strong performance in this parking setting. We therefore treat continuation depth as a configurable allocation choice whose benefit should be assessed for the task and budget.

Together, these experiments extend the evidence beyond the pre-training probe. They also show why initial rollout coverage and final planning success should be reported separately: the higher pre-training coverage of D=2D=2 in Table 2 does not produce higher final success in this comparison.

We have added the 28-rollout ablation to the revised manuscript and will include the completed 120-rollout comparison alongside it, with the training settings, checkpoint-selection rules, and interpretation made explicit.

AuthorsOpenReview ↗

Response for Reviewer M7hD — Part 2

Answers to the Questions

Q1. Does the parking baseline include MultiPark’s argmax fine-tuning?

Yes. The starting parking checkpoint includes both training stages:

  1. Teacher-forced imitation learning, using demonstrated plans.
  2. MultiPark-style argmax fine-tuning with outcome-oriented losses.

Our RL fine-tuning starts after these two stages. The baseline result of 304/359 successes (84.68%) therefore comes from a model that has already received argmax fine-tuning; it is not a teacher-forcing-only baseline.

The original legacy run, the main rank-tree run, and the additional 120-rollout control start from the same checkpoint. Their comparison therefore does not reflect a difference in whether this earlier fine-tuning stage was included.

We will make the two-stage preparation explicit in the experimental setup and clarify the meaning of “imitation baseline” in the tables.

Q2. How are IL and RL training data separated from the evaluation suites?

For the main reported comparisons, the dead-end benchmark and Mini-5K come from a held-out test partition excluded from both imitation learning and RL fine-tuning.

Partitioning the source data. We reserve 10% of the raw scenario dataset for testing and a separate 10% for validation, with sampling intended to preserve the scenario distribution. The examples used for IL and RL parameter updates come from the training partition.

Constructing the RL dataset. We run the starting parking checkpoint on the training partition, collect 2,906 failure cases, and randomly select 3,094 successful cases. Together, these form the 6,000-scene RL dataset. The aim is to obtain an approximately balanced training set while reducing the amount of data needed for fine-tuning. These scenes are drawn from training data, not from either evaluation suite.

Constructing the evaluation suites. Mini-5K contains 5,000 scenes sampled from the held-out test partition. The 359-scene benchmark is manually selected from that partition to focus on difficult parking cases. Its overlap with the RL set in the type of scenario—such as dead-end parking—does not mean that the same scenarios are used for training.

Checkpoint selection. The main rank-tree checkpoint was selected using closed-loop evaluation on the training corpus. The new 120-rollout D=0D=0 control and the 28-rollout experiment use predefined epoch-20 checkpoints. Neither the dead-end benchmark nor Mini-5K was used to choose these checkpoints.

Closely related variants. The separation we can confirm is at the scenario level. We have not established cluster- or recording-level disjointness, so we cannot rule out closely related variants appearing across partitions. We will state this limitation explicitly rather than treating scenario-level separation as proof of independence at every level.

For completeness, Appendix E describes an earlier experiment trained directly on the 359-scene suite. This is a separate train-on-test diagnostic, not a held-out evaluation of the main 6K-trained model. We will keep that distinction explicit and exclude it from generalization claims.

We will revise the data description to state the partitioning procedure, the exact 6,000-scene RL composition, the sources of both evaluation suites, and the level of overlap checking that has been established.

AuthorsOpenReview ↗

Follow-up on Revisions and Clarifications

Dear Reviewer M7hD,

Thank you again for your thoughtful review and constructive suggestions. As the discussion period approaches its end, we would appreciate your feedback on whether our rebuttal and revised manuscript address your concern and questions.

Following your suggestions, we have:

  • Added the end-to-end 120-rollout expanded-legacy comparison, alongside the 28-rollout reduced-budget experiment, and revised the discussion of what these results establish about rollout budget and continuation-rank allocation.
  • Clarified that the starting parking model includes both teacher-forced imitation learning and MultiPark-style argmax fine-tuning with outcome-oriented losses.
  • Documented the training and evaluation splits, including the scenario-level separation of the dead-end and Mini-5K benchmarks from IL and RL training, while noting that cluster- and recording-level disjointness have not been established.

Changes in the revised manuscript are marked in red. Please let us know if any points remain unclear or require further clarification. We appreciate your time and the suggestions that helped us improve the paper.

Reviewer_q5cfOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Rank-Tree GRPO, an RL fine-tuning based approach for learned planners. The key idea is to explore combinations of the top-KK maneuvers across DD future steps during training, instead of greedily following the highest-ranked maneuver. The approach reports a gain of 0.2 (PDMS Score) on NAVSIM (with D=0), 7.5% success rate on an internal benchmark, and 3.9% success rate on ParkBench.

Critical strengths

This work addresses an important problem of improving learned planners without increasing inference time search costs. The approach is simple and can be integrated into existing frameworks without significant modifications to the deployment procedure. Additionally, the approach reports substantial improvements in planning success on the internal dead-end parking benchmark.

Critical weaknesses

While the method appears useful, based on empirical results, the novelty seems limited. Tree-based exploration for RL have already been explored [a, b]. While authors distinguish their approach through fixed continuation-rank schedules and independent geometric sampling, the significance of these differences remains unclear (especially due to the lack of direct comparisons with closely related methods.).

It appears this approach is evaluated primarily on parking benchmarks (one of which is internal and not released) despite the proposal being introduced as a general planning harness. If there’s some reason why parking ends up being a problem which is particularly well suited to display the benefits of this approach, it is not well motivated. Secondly, the paper’s NAVSIM experiments evaluate single-step trajectory generation, and do not test the paper’s central claim that exploring different continuation ranks improves multi-step planning. The authors should either narrow the introduction to this specific use case or conduct more exhaustive experiments to demonstrate the effectiveness of the approach beyond sequential parking.

“For example, a second-ranked reverse continuation in parking…” It would be useful for this example to be written in a better/more approachable manner. As it stands, I was not able to follow this at all, which is not great since it’s the primary example for helping understand the motivating hypothesis. In hindsight (reading the rest of the paper), it’s talking about parking a car, but the scenario is never established and key terminology is used without sufficient introduction or explanation. More broadly, beyond the motivating example, I believe this paper would greatly benefit from a dedicated preliminaries section as the repeated use of undefined or insufficiently explained terms harms the clarity of the paper.

As per my understanding, Tab. 1 does not establish a comparison under matched training rollout budgets, since Legacy GRPO uses one greedy continuation rollout per initial query whereas Rank-tree GRPO uses four continuation-rank schedules per initial query, making for a potentially unfair comparison. Tab. 2 shows equal budget comparisons, but only evaluates pre-training rollout coverage rather than the performance of the final fine-tuned models.

There’s a lack of ablation studies. The main approach is evaluated only for K=D=2K=D=2, there’s no comparison of other choices of (K,D) beyond a pre-training probe, no ablation for failure shaping, advantage damping, imitation anchor, etc.

mtm_t (maneuver mode at time t) and mxm_x (indicator for if rollout group for scene xx is retained for training) overloads mm and harms clarity.

References:

  • [a] Ding, et al. “Treegrpo: tree-advantage GRPO for online RL post-training of diffusion models.” ICLR (2026).
  • [b] Lian, et al. “Fine-tuning is not enough: A parallel framework for collaborative imitation and reinforcement learning in end-to-end autonomous driving.” CVPR (2026)
Questions for authors

Questions:

  • How does increasing the number of geometry samples in legacy GRPO compare?
  • Can wall clock measurements be provided to support the efficiency claims in the paper?
  • Could the authors provide ablation experiments or analyses described above?
  • Do the authors expect the approach to generalize well to scenarios beyond sequential parking?
6 comments
AuthorsOpenReview ↗

Response for Reviewer q5cf — Part 1

Thank you for recognizing the value of improving planning without increasing the deployment search budget, the ease of integrating our method, and the gains on difficult parking cases. We appreciate your suggestions for clearer explanations and stronger experiments.

Responses to the Weaknesses

W1. What is different from earlier tree-based methods?

We agree that tree-based exploration is established and cite TreeGRPO and PaIR-Drive. Our contribution concerns how to explore the choices already offered by an existing planner during training.

TreeGRPO builds trees over diffusion denoising steps, shares trajectory prefixes, and assigns credit to individual tree edges. We collect independent rollouts and compare their complete returns within each scene. Each rollout follows a short list of candidate ranks—for example, “take the first choice, then the second choice”—using a fixed reference planner to rank candidates at each reached state.

This matters because sampled motions reach different states, where the available choices can differ. We also avoid global beam pruning during collection, so a scheduled alternative is not discarded simply because its cumulative model score is low.

PaIR-Drive adds a neural sampler with offset and log-probability prediction heads, plus a reward model for selecting plans at inference. We reuse the existing planner, compute raw-action Gaussian likelihoods directly, and retain the original deployment search. We have not directly compared performance against those complete systems.

W2. Why parking, and what does NAVSIM demonstrate?

Parking is useful for studying sequential choices because each forward or reverse move changes the car’s position and the clearance available for its next move. A locally preferred move can leave too little room to finish parking.

In our NAVSIM experiment, DrivoR produces complete trajectory candidates in one planning decision. There are no later segment-selection decisions within that rollout, so D=0D=0. This tests generator fine-tuning under the same generate-then-score procedure. It does not test continuation branching, which we evaluate in sequential parking. We will state this scope clearly in the Introduction.

W3. A clearer example and definitions.

We will introduce the parking situation before discussing ranks:

Consider a car entering a parking space from a narrow dead-end aisle. It first moves forward, then reverses to make room for the final turn. The planner prefers one reverse path, but that path leaves too little clearance for the turn. Another reverse path leaves enough room. Small changes to the preferred path may still miss this alternative. Our sampler deliberately includes both choices during training.

We will expand Section 3 with preliminaries. A candidate is one proposed motion segment; a rollout is a sequence of segments; a rank is a candidate’s position in the reference planner’s ordering at the current state; and the deployment harness checks feasibility and searches for a complete plan. We will distinguish training settings M,K,DM,K,D from deployment settings W,kW,k.

W4. Comparing training with the same rollout budget.

We agree that comparing 30 legacy rollouts with 120 rank-tree rollouts does not isolate the effect of allocation. We have now trained a 120-rollout D=0D=0 specialization. For each of the 30 initial queries (starting motion proposals), we independently sample four versions of the geometry (curvature and length), giving 30×4=12030\times4=120 rollouts per scene. Each rollout then follows the highest-ranked legal continuation, with geometry sampled at every step. This is the expanded legacy allocation requested by the reviewer.

Both 120-rollout runs use seed 42, the same starting model, training data, reward and loss settings, optimizer settings, batch size, and training duration. The new D=0D=0 result uses the predefined epoch-20 checkpoint.

Model / training allocationDead-end (359)Mini-5K (5,000)
Starting model84.68%99.32%
Legacy, 30 rollouts83.01%99.34%
Rank-tree, K=2,D=2K=2,D=2, 28 rollouts87.19%99.38%
Rank-tree, K=1,D=0K=1,D=0, 120 rollouts92.5%99.36%
Rank-tree, K=2,D=2K=2,D=2, 120 rollouts92.2%99.34%

The 28-rollout experiment uses the top seven initial queries, seed 42, and the predefined epoch-20 checkpoint, with training settings unchanged from legacy. Compared with the starting model, it solves nine more dead-end cases (+2.51 percentage points). Using fewer initial queries also changes which starting proposals are explored, so this is a reduced-budget test rather than an isolated allocation comparison.

At 120 rollouts, D=0D=0 and D=2D=2 give similar results. This suggests that deeper continuation exploration is not needed for strong performance in this parking setting. This lets us choose an allocation suited to the task and budget.

AuthorsOpenReview ↗

Response for Reviewer q5cf — Part 2

W5. Allocation and component ablations.

The new experiments in W4 evaluate the planner after training, extending the original comparison of pre-training rollout coverage. We are also training the K=2,D=3K=2,D=3 configuration, which explores alternative choices at three continuation decisions instead of two. With M=30M=30, this doubles the number of rollouts per scene from 120 to 240. We will report the results if they become available during the discussion period.

We do not yet have separate results showing how much each supporting component contributes: failure shaping distinguishes better and worse unsuccessful attempts; advantage damping reduces the update strength when all rollouts fail; and the imitation anchor retains guidance from demonstrated behavior during fine-tuning.

Our reward design ensures that every successful rollout receives a higher reward than any failed rollout. However, this guarantee alone does not show how much shaping improves planning performance. The current results evaluate these components together; separate ablations are needed to identify their individual effects.

W6. Overloaded notation.

We agree that using mm for two different concepts is confusing. We will change the group-retention indicator from mxm_x to gxg_x, which indicates whether a scene’s rollout group is used for training. We will keep mtm_t for the maneuver mode at step tt and update the notation throughout the paper.

AuthorsOpenReview ↗

Response for Reviewer q5cf — Part 3

Answers to the Questions

Q1. How does increasing the number of geometry samples in legacy GRPO compare?

Please see W4 for the new end-to-end comparison. We keep the same 30 initial queries and increase sampling from one to four geometry samples per query, giving 120 rollouts per scene. Later decisions still follow the highest-ranked legal continuation, with geometry sampled at every step.

This run achieves 92.5% success on the dead-end benchmark and 99.36% on Mini-5K, compared with 83.01% and 99.34% for the original 30-rollout legacy run. Its performance is also close to the 120-rollout D=2D=2 configuration. These results suggest that deeper continuation exploration is not necessary for strong performance in this parking setting.

Q2. Can wall-clock measurements support the efficiency claims?

Thank you for highlighting this distinction. Our intended claim is that fine-tuning improves planning without requiring a wider deployment search. In parking, the fine-tuned model achieves 92.2% success with beam width 8/top-6, compared with 91.4% for the starting model using beam width 30/top-15. The corresponding nominal expansion budgets are 48 and 450 candidates per step. This is a search-budget comparison, not a measured speedup.

Our real-vehicle deployment workflow also includes quantization-aware training (QAT). A vehicle-latency comparison would therefore need to evaluate the deployed models under the same quantization settings and on the same target hardware. We do not report such measurements, or matched training-time measurements, in this paper.

Even with fixed search settings, runtime can vary because plans finish after different numbers of segments and require different amounts of collision checking. We will clarify the efficiency wording to distinguish the demonstrated search-budget benefit from unmeasured wall-clock savings.

Q3. Can the requested ablations be provided?

The new experiments in W4 evaluate two additional allocations: 28 rollouts using seven initial queries with K=2,D=2K=2,D=2, and 120 rollouts using four geometry samples per initial query with D=0D=0. These provide results after training, beyond the original pre-training coverage measurements.

The K=2,D=3K=2,D=3 experiment is still training. As explained in W5, separate results for failure shaping, advantage damping, and the imitation anchor are not yet available. The current experiments evaluate the combined training recipe; they do not tell us how much each supporting component contributes individually.

Q4. Do we expect the approach to generalize beyond sequential parking?

We expect the approach to apply when a planner offers several ranked candidate moves at each decision and allows us to evaluate sequences of those moves during training.

On public ParkBench, success increases from 35/51 (68.6%) to 37/51 (72.5%) without additional fine-tuning. This supports transfer across parking scenarios. NAVSIM also shows that the fine-tuning formulation can be applied to a different planner architecture, although its single-step setting does not test continuation branching.

Testing continuation branching in driving would require evaluating future states and other agents’ responses as the rollout progresses. Closed-loop driving evaluation, for example using Bench2Drive, is therefore an important next step and remains future work.

Reviewer_BhbnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies RL fine-tuning of learned multimodal planners that are deployed behind a fixed search harness. The goal is to recover the gains of wider inference-time search without changing the deployment configuration.

Critical strengths

It seems like this paper has a relatively weak contribution in methodology; however, the authors honestly confront the fact and do a good job in implementation.

  1. This paper tackles a practical problem. Internalizing the gains of wider search under a fixed deployment budget directly addresses real-time constraints. The evaluation target is defined as success under the deployed harness.

  2. Simple and harness-compatible. Only the construction of rollout groups changes. The method requires neither differentiation through beam search nor an additional categorical policy-gradient term.

3.It has clean likelihood designs. Gaussians are defined on raw actions and passed through a parameter-free deterministic map. This yields exact densities and well-defined KLs without Jacobian corrections.

  1. Guaranteed reward ordering. A margin check at construction time (maximum shaped failure −1 < worst success 1.4) ensures that every success outranks every failure.

  2. Honest equal-budget probe. Table 2 separates allocation from sample count at a matched 120-row budget and seed. It also acknowledges that most of the coverage gain comes from the larger budget.

Critical weaknesses
  1. Limited novelty. Relative to tree- and branch-based GRPO variants (TreePO, TreeRPO, BranchGRPO) and PaIR-Drive, the difference is a modest change in rollout allocation. The NAVSIM experiment uses D = 0 and therefore does not test the continuation-branching mechanism at all.

  2. Missing equal-budget training ablation. The legacy sampler uses 30 rows per scene versus 120 for rank-tree, and the paper does not state whether total rollout or update budgets were matched. Table 2 shows that budget explains most of the coverage gain (79.3% → 85.9%) while allocation adds only 2.2–3.8 points, so the headline 92.2% cannot be attributed to rank-tree allocation without end-to-end training of the 120-row geometry-only variant.

  3. Single training run per configuration. No seeds or confidence intervals are reported. Because evaluation is deterministic, all variance comes from training, and the regressing legacy run suggests that variance is not negligible. A 27-scene difference on 359 scenes needs multi-seed support and paired per-scene changes (rescued vs. broken).

  4. Unverified run configurations. Table 4 states that its values “specify the recipe, not a verified manifest for every reported run.” Appendix E also notes unestablished denominators and unaudited per-scene records, so it is unclear whether the compared runs are actually matched.

  5. Train/test separation is not documented. The 6K training corpus includes dead-end items, but the paper does not state whether it is disjoint from the 359-scene suite at the scene, cluster, or recording level. The header path in Figure 3 contains “/train/”, indicating that at least this qualitative example illustrates training fit rather than generalization.

Questions for authors

Please refer to the weaknesses above; besides, I have the following questions regarding the re-grading of the papers:

1 . Can you train the 120-row geometry-only allocation end-to-end under an otherwise identical configuration, and report training results for K = 2, D = 3?

2 . Can you report multiple training seeds for the base, legacy, and rank-tree runs, together with per-scene rescued/broken counts on the 359-scene suite?

3 . What were the exact hyperparameters of each reported run? Did the legacy runs use the same reward shaping, imitation anchor, gating rule, and number of epochs?

4 . Is the 359-scene suite disjoint from the 6K RL corpus and from the IL training data at the scene, cluster, and recording level? Which split do the qualitative figures come from?

5 . For the parking setting, can you report the coverage/conversion decomposition and the change in deployed mode rankings before and after fine-tuning?

6 . How many scenarios does ParkBench contain, and what success criteria and search settings were used? Can the NAVSIM results include multiple seeds?

I am more than willing to raise my score if my concerns are fully addressed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Bhbn — Part 1

Thank you for recognizing the practical value of improving planning under a fixed deployment search budget, the compatibility of our sampling design with existing planners, and the care in the raw-action likelihood and reward formulations. We also appreciate your recognition of the matched-budget probe and your constructive requests for stronger experimental controls and clearer reporting. Below, we address the weaknesses and then answer your specific questions.

Responses to the Weaknesses

W1. Limited novelty and the scope of the NAVSIM experiment

We cite and discuss TreePO, TreeRPO, BranchGRPO, and PaIR-Drive in Section 2, and agree that tree-structured exploration and configurable branching budgets are established. Our contribution is a training-only allocation strategy over an existing planner’s ranked continuations, designed to expose useful later decisions that greedy continuation sampling may miss. We prescribe short continuation-rank schedules under a frozen reference, interpreted at each rollout’s reached state, while sampling geometry independently throughout every rollout. The parameters M,K,DM,K,D explicitly control initial-query coverage, continuation-rank breadth, and exploration depth, yielding B=MKDB=MK^D allocated rollout rows. This provides systematic allocation across the selected rank combinations without maintaining a shared-prefix trajectory tree or applying global score-based pruning during collection.

Specifically, PaIR-Drive introduces a dedicated neural sampler with offset and log-probability prediction heads. Our method directly fine-tunes the existing planner, using its raw geometry outputs as Gaussian means and computing the corresponding log-likelihoods analytically. This requires no additional neural sampler or likelihood-prediction head and preserves the original deployment search.

For our NAVSIM experiment, our baseline, DrivoR, predicts complete four-second trajectory candidates in a single planning decision. Since DD counts continuation decisions after the initial proposal, this setting corresponds to D=0D=0. We agree that this experiment evaluates generator fine-tuning under an unchanged generate-then-score deployment procedure, while continuation branching is evaluated in sequential parking. Extending and evaluating continuation branching in closed-loop driving, for example on Bench2Drive, remains future work.

W2. Missing equal-budget end-to-end training comparison

We thank the reviewer for highlighting the unequal rollout budgets in the original comparison: 30 rollouts per scene for legacy GRPO versus 120 for rank-tree GRPO. We conducted an additional reduced-budget training experiment using the top seven initial queries with K=D=2K=D=2, yielding B=MKD=7×22=28B=MK^D=7\times2^2=28 rollouts per scene. The training data, reward shaping, failure damping, imitation anchor, optimizer settings, and training duration were unchanged from the legacy run.

Using seed 42, the results at the predefined epoch-20 checkpoint are:

ModelDead-end (359)Mini-5K (5,000)
Starting model84.68% (304)99.32% (4,966)
Rank-tree, 28 rollouts87.19% (313)99.38% (4,969)

We have added this reduced-budget ablation and its results to the revised manuscript. It yields nine net additional successes (+2.51 percentage points) on the dead-end benchmark, demonstrating an improvement in this run with fewer rollouts per scene than the original legacy configuration. However, selecting seven initial queries also changes root coverage, so this experiment provides complementary evidence rather than isolating rank allocation or replacing the requested 120-rollout geometry-only training control. Rollout counts also do not establish equal executed-segment counts or wall-clock training costs.

W3. Training variability and paired evaluation outcomes

We thank the reviewer for highlighting the need to assess training-seed variability. The originally reported results use seed 42. We have conducted an additional 120-rollout rank-tree training run with seed 21, evaluated on the same internal parking benchmarks:

Training seedDead-end (359)Mini-5K (5,000)
Seed 42 (original)92.20% (331)99.34% (4,967)
Seed 21 (additional)89.97% (323)99.36% (4,968)

Both runs improve over the starting model’s 304/359 successes (84.68%), yielding net gains of 27 and 19 scenes, respectively. The additional run supports that the improvement is not confined to seed 42, although two runs provide only a limited assessment of training variability. These aggregate counts represent net gains and do not separately identify rescued and regressed scenes.

AuthorsOpenReview ↗

Response to Reviewer Bhbn — Part 2

W4. Run configurations and experimental provenance

We thank the reviewer for the careful reading. For the original legacy-GRPO versus rank-tree comparison, only the sampling strategy changes. We also evaluated different deployment beams in separate search-budget experiments. The main fine-tuning comparison evaluates the starting and fine-tuned models under the same beam-width-8/top-6 configuration.

All training-set evaluations in Appendix E used the same 6,000-scene dataset. We constructed it by evaluating the original IL checkpoint on the full training partition, collecting 2,906 failure cases, and randomly selecting 3,094 successful cases to obtain an approximately balanced fine-tuning set. This accounts for the 3,094 baseline successes reported in Appendix E. These are training-set diagnostics; the held-out dead-end and Mini-5K evaluations contain 359 and 5,000 scenes, respectively. We will clarify these shared settings, run-specific differences, and exact evaluation denominators in Table 4 and Appendix E.

W5. Train/test separation and qualitative examples

We thank the reviewer for requesting clarification. We reserve 10% of the raw scenario dataset as a held-out test partition, excluded from both imitation learning and RL fine-tuning. The approximately 6K-scene RL dataset is constructed entirely from the training partition. Mini-5K is drawn from the held-out test partition, and the 359-scene benchmark is manually selected from that partition to focus on challenging parking cases.

The /train/ string in Figure 3 reflects the scene’s original source path in the raw dataset, rather than its assignment in our experimental split. The illustrated scene was not used for training. We will clarify this distinction in the figure caption and explicitly document the training and evaluation splits in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Bhbn — Part 3

Answers to the Questions

Q1. Can the 120-row geometry-only control and the K=2,D=3K=2,D=3 configuration be trained end-to-end?

We thank the reviewer for suggesting an end-to-end comparison at the same rollout budget. In addition to the reduced-budget experiment in W2, we conducted a 120-rollout experiment using the D=0D=0 specialization of rank-tree sampling. We draw four independent initial geometry samples for each of the 30 initial queries, yielding 30×4=12030\times4=120 rollouts per scene. With K=1,D=0K=1,D=0, subsequent decisions follow the legacy rule: selecting the highest-ranked legal query at each reached state while continuing to sample geometry. This implements the expanded legacy allocation requested by the reviewer.

Training allocationDead-end (359)Mini-5K (5,000)
Rank-tree, K=1,D=0K=1,D=0, 120 rollouts92.5%99.36%
Rank-tree, K=2,D=2K=2,D=2, 120 rollouts92.2%99.34%

The D=0D=0 specialization achieves comparable, slightly higher performance in this comparison. This illustrates the flexibility of our allocation framework and suggests that increasing continuation depth is not necessary for strong performance in this parking setting. We therefore view DD as a configurable exploration choice rather than a parameter that should always be increased.

We have also submitted the requested experiment with K=2,D=3K=2,D=3 and will report its results when available. Since the rollout count scales as B=MKDB=MK^D, this experiment will help assess whether deeper continuation exploration justifies its additional rollout budget.

Q2. Can multiple seeds and rescued/broken counts be reported?

Please see W3 for the main rank-tree results with seeds 42 and 21. Both runs improve over the starting checkpoint on the dead-end benchmark. The additional-seed comparison currently covers rank-tree training; the other configurations use seed 42. The reported aggregate improvements are net gains, rather than separate rescued/broken counts.

Q3. What were the exact settings, and were the legacy runs matched?

Please see W4 for the shared training settings and run-specific differences. For the original legacy and main rank-tree runs, initialization, training data, reward shaping, failure-group gating/damping, imitation anchor, optimizer settings, batch size, and training duration were identical. Both used the same beam-width-8/top-6 evaluator. Their sampling configurations differed, while the separately identified hard-case-reweighted variant additionally changed scene-sampling weights. We will make these distinctions explicit alongside the numerical hyperparameters in Table 4.

Q4. Are the evaluation suites disjoint from IL and RL training, and which splits do the figures use?

Please see W5 for the dataset construction and Figure 3 provenance. The 359-scene benchmark comes from the held-out test partition, excluded from both IL and RL training; the 6K RL corpus comes entirely from the training partition. Figure 3 also illustrates a held-out scene: /train/ records its original source path, not its experimental split. This clarification concerns scenario-level separation; cluster- and recording-level disjointness have not been established.

Q5. Can parking coverage/conversion and mode-ranking changes be reported?

Eq. (2) decomposes deployed success into proposal coverage and width/ranking conversion. Coverage measures whether a successful plan exists in the proposal tree without global beam-width pruning; conversion measures whether the deployed beam succeeds conditional on such a plan existing.

Parking success improves from 304/359 to 331/359 under the same width-8/top-6 beam. Unpruned-tree coverage was not measured before and after fine-tuning, so a numerical decomposition is unavailable. Table 2’s stochastic training-rollout coverage cannot supply this quantity.

The frozen reference sets training-time rank schedules; the deployed decoder remains trainable. Shared-parameter updates and the imitation objective can change mode rankings without a categorical policy-gradient term. We have not quantified these changes at matched states and legal-query sets.

Q6. What is the ParkBench protocol, and can NAVSIM include multiple seeds?

We evaluated all 51 ParkBench scenarios zero-shot, without additional fine-tuning or domain adaptation. Success requires reaching the target within 0.2 m and 3∘3^\circ, within eight maneuver segments and without collision under our evaluation rules. Both checkpoints use beam width 8, top-6 expansion, and the internal evaluation’s swept-footprint collision checks, including the terminal low-obstacle exemption.

Success increases from 35/51 (68.6%) to 37/51 (72.5%). We will clarify the scenario count and benchmark-specific tolerances in the revised manuscript.

NAVSIM currently uses seed 42 only; additional training seeds are unavailable. The seed-21 results in W3 concern parking.

Reviewer_BhbnOpenReview ↗

Thanks for your clarify

I thank the authors for the details. Many of my concerns are resolved. Based on my comments above, I still think borderline accept is the appropriate rank for this work. I wish the authors good luck and will offer my support in the later discussions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

Dear Reviewers and Area Chair,

Thank you for your careful reviews and constructive feedback. We have uploaded a revised manuscript incorporating the additional experiments and clarifications discussed in our rebuttal. New and revised text is highlighted in red to facilitate review.

The main updates include:

  • Additional experiments: a reduced-budget 28-rollout run, a matched-budget 120-rollout D=0D=0 control, and an additional parking training seed.
  • Clearer experimental reporting: clarification of the starting checkpoint, training/evaluation data separation, run configurations, checkpoint selection, and evaluation protocols.
  • Improved presentation: a clearer parking example, clarified terminology and notation, and updated discussions of related work, allocation ablations, and limitations.

The revised manuscript also clarifies the scope of the conclusions supported by the new results. Detailed responses to individual concerns are provided in our reviewer-specific rebuttals.

We appreciate your time in considering the revised manuscript and additional evidence.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.