- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how human-reference forgiveness in NAVSIM affects the training labels used by GTRS-Dense for autonomous driving trajectory prediction. When the human reference fails a particular evaluation metric, the label generator assigns a passing score to every candidate trajectory for that metric, potentially removing meaningful distinctions between valid and invalid trajectories.
The authors reconstruct the original labels by disabling this overwrite and analyze all 103,288 NAVSIM training scenes. They find that forgiveness removes loss-relevant candidate distinctions in 10.88% of scenes, predominantly affecting lane keeping. They further demonstrate that released models have substantially weaker lane-keeping discrimination on affected scenes.
Through controlled checkpoint continuation experiments, the authors show that restoring the original labels improves lane-keeping discrimination and compliance. However, these improvements come with reduced temporal planning consistency and do not translate into a significant improvement in the aggregate EPDMS score.
The paper concludes that evaluation rules can introduce systematic supervision distortions when reused for training-label generation, highlighting an important distinction between evaluation robustness and effective learning supervision.
Critical strengths
-
Comprehensive and rigorous empirical analysis. The paper systematically investigates the effects of human-reference forgiveness across all 103,288 NAVSIM training scenes. The distinction between overwritten labels and genuinely erased candidate distinctions is particularly valuable. The authors also validate their reconstruction against the released labels, providing strong evidence for the reliability of their measurements (Sections 4-5).
-
Strong experimental evidence of supervision degradation. The authors demonstrate that forgiven labels substantially reduce the ability of trained models to distinguish valid from invalid trajectories. The evaluation includes released GTRS-Dense and Hydra-MDP checkpoints, matched control groups, held-out scenes, and controlled checkpoint continuation experiments (Sections 6 and 8). These complementary analyses strengthen the connection between the identified label issue and observed model behavior.
-
Important reproducibility findings. Section 7 identifies substantial hardware-dependent differences in regenerated training labels caused by processor-specific numerical kernels. The authors demonstrate that controlling the OpenBLAS configuration restores consistency with the released labels. This provides an additional practical contribution concerning reproducibility in simulation-based learning pipelines.
Critical weaknesses
-
Limited methodological novelty. The main contribution is an empirical investigation of an existing evaluation mechanism rather than a new learning methodology. Human-reference forgiveness was already documented by NAVSIM's authors, while Wei et al. (2026) investigated related evaluation-side issues. Although extending this analysis to training supervision is useful, the paper does not propose or validate a new general solution for addressing the identified problem. This limits its methodological contribution relative to ICLR expectations.
-
Insufficient validation through full model training. The controlled training experiments in Section 8 continue an existing GTRS-Dense checkpoint for only 300 optimizer steps, with an additional analysis at 530 steps, using three random seeds. Since the checkpoint was originally trained using forgiven labels, these experiments cannot establish whether the observed improvements persist under complete training from scratch or after convergence. The evaluation is also restricted to a single benchmark and training intervention, limiting generalizability.
-
Unclear improvement in overall planning quality. Although restoring pre-overwrite labels improves lane-keeping discrimination and compliance, the authors report reduced adjacent-frame planning consistency and no statistically distinguishable improvement in aggregate EPDMS (Section 8). Therefore, the results establish that forgiveness removes meaningful training distinctions but do not demonstrate that correcting these labels produces a better overall driving policy. This trade-off significantly limits the practical implications of the findings.
Questions for authors
-
Training from scratch: Would the observed improvements in lane-keeping discrimination and compliance persist if GTRS-Dense were trained from scratch using pre-overwrite labels? Could the authors provide longer training experiments or learning curves to establish whether the improvements remain after convergence?
-
Overall planning performance: How should the improvement in lane keeping be interpreted given the simultaneous reduction in adjacent-frame planning consistency and the absence of significant EPDMS gains? Is there evidence that restoring the original labels improves actual trajectory safety or planning quality rather than optimizing an individual evaluation component?
-
Alternative supervision strategies: The paper also considers masking forgiven columns instead of replacing their labels. Could the authors provide a more detailed comparison of masking, published forgiveness, and pre-overwrite supervision, particularly regarding overall planning performance? Would treating forgiven components as uncertain supervision provide a better balance between preserving informative distinctions and avoiding penalties for contextually justified human behavior?