- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates reward fine-tuning for diffusion-based autonomous driving planners. It combines shared and domain-specific LoRA adapters, differentiable DPM-Solver++ sampling, and reward diagnostics to analyze why higher training rewards do not translate into improved closed-loop driving performance.
Critical strengths
The paper investigates a relevant problem in diffusion planner fine-tuning and provides several diagnostic experiments.
Critical weaknesses
- Limited novelty and comparison.
The proposed SE-LoRA is essentially a combination of shared and domain-specific LoRA adapters, while differentiating through an unrolled diffusion solver is not fundamentally new. More importantly, the lack of comparison with any of the other methods on NuPlan makes the performance hard to appreciate. - Ambiguous experiment.
The paper demonstrates that reward fine-tuning changes the planner's behavior (Lines 408–413), but provides little evidence that these changes are beneficial. As shown in Table 2, no fine-tuned configuration (row 1) achieves a meaningful closed-loop improvement, and some introduce additional catastrophic failures. While the authors analyze why the aggregate score remains unchanged, it is unclear whether the observed adaptations represent useful driving capabilities rather than simply changes in model behavior. - Insufficient documentation of real-vehicle experiments.
The claimed real-vehicle deployment is also questionable due to the lack of experimental photographs and a sufficiently detailed testing setup. It remains unclear how the experiments were actually conducted and whether any real-world closed-loop driving evaluation was performed, making the claimed deployment difficult to assess or verify.
Questions for authors
See weakness
3 comments
I appreciate the authors' responses; I still have the following questions:
- After carefully reading, I'm still confused about the evaluation setting and metrics used in the paper. Could the authors provide the leaderboard results under NuPlan in a standard benchmarking setting and compare them with other methods? The new table is appreciated, but I didn't see it in the revised paper, and it is missing some of the comparisons, e.g., as you mentioned: DPPO, DiffusionDriveV2, PlannerRFT, Plan-R1.
- If it is only deployed on Autoware virtually, it should not be called a real-vehicle experiment; moreover, I would like to see more visualization results of common failure cases that previous methods may have, as well as a short video under Autoware.
- The writing of the paper is barely understandable; although it contains a lot of experiments, it is still really hard to read and seems like a "report" rather than a paper.
I raised my score given the author's current responses, while some of my concerns are still unresolved, as I mentioned.
Response to Reviewer Eic1's follow-up: Standard nuPlan Table, Real-Vehicle Setup, Failure Plots, and Writing
Thank you for raising your score and for the follow-up questions. The additions below are in the updated PDF, marked "Eic1 follow-up" in blue.
1. Standard nuPlan setting and comparison table
Our nuPlan closed-loop experiments use the official nuPlan non-reactive benchmark on the standard val14, Test14-random, and Test14-hard splits (plus one 169-scenario hold-out, App. C.13). Appendix G, Table 26 now provides the comparison on the standard 0–100 leaderboard scale, with published baselines quoted from the literature (sources are listed in Table 26):
| Planner | val14 | Test14-random | Test14-hard |
|---|---|---|---|
| Base planner, as published | 89.87 | 89.19 | 75.99 |
| Base planner in our pipeline, 3 runs | 89.77 to 89.83 | 88.57 to 89.26 | 75.08 to 75.68 |
| Fine-tuned, exact score operators, 3 seeds | 89.20 to 89.30 | 88.45 to 88.82 | 75.55 to 75.89 |
| PlannerRFT, as published | 89.96 | 90.76 | 77.16 |
| Plan-R1, as published | 88.98 | 91.23 | 77.45 |
| Plan-R1 with rule-based post-processing | 94.72 | 94.64 | 78.46 |
| PDM-Closed, rule-based | 92.84 | 90.05 | 65.08 |
| PLUTO w/o refinement | 88.89 | 89.90 | 70.03 |
| PlanTF | 84.27 | 85.62 | 69.70 |
DPPO is evaluated on robotics tasks and DiffusionDriveV2 on NAVSIM, so neither reports nuPlan scores; our own adapted DPPO run scores 89.60 on val14 with a flat reward. Rather than proposing a leaderboard planner, our paper studies rows 2 and 3: why a 30%–35% gain in held-out reward (up to 60% with our other rewards) leaves the closed-loop score flat or slightly lower.
2. Autoware and the real-vehicle test
Autoware itself is an open-source autonomous driving software stack, not a simulator: while our multi-map ablations use it without a vehicle, in Appendix D it connected directly to our physical drive-by-wire vehicle, processing live LiDAR and INS data to send real control commands. The base planner drove the physical car in closed loop over a 704 m route with a safety driver aboard. However, the fine-tuned planner never drove the car; its comparison with the base planner is an offline replay on sensor logs recorded on the vehicle. We agree that this offline replay should not be called a real-vehicle experiment, and we updated Appendix D's title and Appendix C.9 accordingly.
3. Failure case visualizations and video
We added bird's-eye-view plots of four zero-score failure cases (four of the six we simulated again) in Appendix H, Figure 5, covering traversing pickup dropoff, changing lane, and stopping with lead. Comparing both planners at the exact failure step shows the fine-tuned planner slightly further ahead along the route, where it clips a stopped vehicle, is touched by a pedestrian while still moving, or puts a corner outside the drivable area. We cannot record a video in the remaining hours and will add a supplementary Autoware video in the final version.
4. Writing and readability
Thank you for the candid feedback. Because this is a measurement study with many ablations, the initial draft read too much like a technical report. In the revised PDF, we reorganized the main text around one central question, rewrote the results subsection headings to state conclusions directly, and moved detailed protocols to the appendix. We will further trim secondary numbers from the body text in the final version.
Response to Reviewer Eic1: Measurement Positioning, nuPlan Baselines, Route-Progress Gains, and Real-Vehicle Setup
Thanks for your review.
W1. Novelty, positioning, and comparison with published nuPlan planners
DPPO,DiffusionDriveV2,PlannerRFT,Plan-R1), we analyze the flat-average case via scenario-by-scenario comparisons against repeated BC simulations and gate vs. component splits (Sec. 4.4, App. C.11). Unlike learned-router MoE-LoRA (MoLA,MixLoRA,MoLoRA), SE-LoRA routes by map so either half can be removed to measure what it carried (Sec. 4.3, Figure 2).Other Table 2 fine-tuned rows score
88.4to89.7on val14.W2. Are the changes beneficial, or only changes in behavior?
With exact score operators, the fine-tuned planners of Table 2 complete more of their route in every seed on val14 and on disjoint Test14-random, offset by added gate failures (Sec. 4.4, Table 6, App. C.11):
Nearly all newly perfect val14 scenarios (32 shared across all 3 seeds) gain via route progress alone, and Test14-random meets pre-fixed criteria (Test14-hard progress also rises in all seeds, CI > 0 in one; a 169-scene validation hold-out does not rise). Because a smaller group fails collision or drivable-area gates (min TTC shortens by
~0.03 s), val14 ends ~0.5 points lower (0 to 100 scale) and Test14 is indistinguishable from BC. Thus we claim a route-progress gain with no net score gain, and removed "safety improvement" wording.W3. Was a closed-loop real-vehicle drive performed, and how was it set up?