- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates supervision targets for trajectory selection when both the planner and the candidate pool are frozen. RethinkDrive trains a lightweight, attention-based selector using the candidate closest to the recorded human trajectory as its supervision target and, at inference, combines the learned scores with the planner’s scores. Controlled comparisons examine metric-derived targets, tie handling, and hard versus soft reference supervision. Experiments on DDv2 and DriveSuprim separately analyze target scores, learned rankings, and final fused performance.
Critical strengths
-
The controlled experiments substantially strengthen the credibility of the local findings. The main target comparisons hold the planner, candidate pool, selector architecture, and training recipe constant. Several plausible alternative explanations are addressed through log-disjoint retraining, matched-context controls, removal of score fusion and the auxiliary safety penalty, and validation-selected rule baselines. In particular, reference supervision still improves DDv2 without fusion, so its benefit cannot be attributed entirely to combining scores. The explicit reporting of negative results and behavioral costs further strengthens the study.
-
Separating learned rankings from fused execution offers a useful evaluation lesson. On DDv2, soft reference supervision performs better than hard reference supervision before fusion, but their ordering reverses after fusion. On DriveSuprim, fusion substantially compresses differences between learned selectors without establishing an improvement over the native planner. These comparisons illustrate how final aggregate performance alone can obscure meaningful differences between supervision targets.
Critical weaknesses
-
The demonstrated practical benefit remains tied to a local baseline whose unusually low temporal comfort is unexplained. In the primary DDv2 configuration, the approximately +3.93 pp EPDMS improvement decomposes into an EC contribution of +4.36 pp and a remaining contribution of −0.42 pp (Table 21). Reported progress falls from 91.2 to 87.8, and drivable-area compliance from 97.8 to 96.7. Although the comfort improvement is valuable, its significance depends on the representativeness of the starting point: local native EC is 34.0, and Appendix F does not resolve this discrepancy. On DriveSuprim, the improvement is only +0.29 pp, with a confidence interval spanning zero, and the refined score is numerically almost identical to that of coarse-only selection. Although validation-selected fusion improves the DDv2 trade-off, the evidence still leaves substantial uncertainty about the benefit on a planner that already performs well.
-
The explanatory contribution remains limited relative to the modest methodological novelty. Reference-based trajectory supervision and learned scoring are established ingredients; Hydra-MDP, for example, already combines distance-based imitation targets with metric supervision. The main contribution therefore rests on the controlled study. However, that study does not yet establish when a particular target should be preferred: tie handling and partial metric alignment leave much of the metric-target deficit unexplained, while the hard–soft comparison depends strongly on execution. These limitations do not invalidate the observed comparisons, but they leave the paper with a useful descriptive finding rather than sufficiently developed, transferable guidance for supervision design.
Questions for authors
-
What explains the discrepancy between the local DDv2 native EC of 34.0 and the published value discussed in Appendix F? Can the checkpoint provenance and evaluation differences be identified? Evidence that the refinement retains a meaningful benefit under a matched released-checkpoint configuration would materially strengthen my assessment of its practical significance.
-
Appendix O.4 notes that a common logit shift preserves the training loss but can change fused selection. Does the hard–soft reversal persist after controlling for this freedom? For example, through centered logits and validation-selected calibration or fusion weights for each target, using the reported evaluation criterion? This would help determine whether the result yields a robust supervision-design insight or primarily reflects the particular score parameterization.
Benefit remains on the matched released checkpoint, and the fusion rule, not the target, set the hard–soft order
Thank you sincerely for your careful and precise review, and we are truly sorry for our late answer. The reruns and the calibration control you proposed took several days, and we wanted each answer to come with its result and our reading.
In the revised PDF your changes are orange, and margin tag 3-n points to answer [3-n].
[3-1] Weakness 1
We agree with your reading, and we would like to add why the gain takes this form. The low extended comfort (EC) is documented in part and not reproduced (answer [3-3]). It comes from the publicly released weights, which we had to use as they are. Your decomposition holds, and the interval of the −0.42 is [−0.89, +0.04] (Table 21). Since EC is the only official term comparing successive plans and the training score has none, a target from one continuous human drive adds most where the planner's consistency is weakest. The lower progress is already present in the target (87.68 against 87.77). Of the drivable-area drop of −1.12 [−1.56, −0.69], 41% comes from the 5.0% of scenes where the target itself does not comply (Appendix Y).
On representativeness, the gain shrinks as native EC rises (34.0 / 41.7 / 55.5 / 77.3 gives +3.93 / +3.21 / +2.37 / +0.29, Table 36). On a further planner that already performs well, GTRS-Aug (native EC 78.2), the gain predicted in advance was +0.29, and we observed −0.37 [−0.78, +0.07], within its tolerance (Appendix X). At 91.0 the fitted line gives −0.76, and we claim no gain there. On DriveSuprim we claim no benefit (+0.29 [−0.18, +0.81]), and the refined score equals coarse-only selection. So where the benchmark leaves room, the gain is consistent and can be predicted.
Where to check. Section 5.6 (p. 9, lines 479–485) and limitations (p. 10, line 518).
[3-2] Weakness 2
Thank you, we agree that the ingredients are established, and our contribution is the controlled study. Hydra-MDP trains distance and metric heads jointly, whereas we ask which single target to train on with all else frozen. The study suggests three rules that we hope will carry over.
We agree that these are only first rules, from one benchmark and in open loop.
Where to check. Section 5.3 (p. 8, lines 392–394), Section 5.6 (p. 10, lines 511–517).
[3-3] Question 1
Where to check. Section 5.6 (p. 9, lines 479–485), Appendices P, B and F.
[3-4] Question 2
Where to check. Section 5.5 (p. 8, lines 428–431) and Table 4 (p. 10, lines 500–502).
If any question remains, we will answer it right away.