acceptodds
Under review as a conference paper at ICLR 2027

When an Evaluation Rule Writes Training Labels: Measuring Human-Reference Forgiveness in NAVSIM

Abstract

When the human reference scores zero on a metric, the released GTRS-Dense label generator for NAVSIM marks every candidate trajectory in the scene as passing it. NAVSIM's authors introduced this human-reference forgiveness to avoid penalizing contextually justified maneuvers when scoring one trajectory, and warned that it could overlook important failures. In label generation it sets a whole column of 16,384 candidate targets to passing. To measure the consequences for supervision, we re-run the generator with the overwrite disabled and compare the pre-overwrite targets with the released labels on all 103,288 navtrain scenes. The rule erases a candidate distinction that the training loss reads on 11,237 of them (10.8793%). Firing usually changes most of a column: lane keeping carries 9,982 of the 13,042 forgiven loss columns, and its median forgiven column had 14,391 of 16,384 candidates failing before the overwrite. On held-out navtest scenes forgiven on lane keeping, the released lane-keeping head's median AUC against the pre-overwrite outcome is 0.7095; on unforgiven scenes matched on failing-candidate count it is 0.9807. For the Hydra-MDP checkpoint released with GTRS, whose configuration takes the same label file, the two values are 0.6627 and 0.9761. Continuing the released GTRS-Dense checkpoint for 300 optimizer steps with three paired seeds, we observe the forgiven-scene AUC 0.1086–0.1251 higher with pre-overwrite than with published targets, and a narrower gap between matched groups, still above zero. Scoring with forgiveness disabled, we observe lane keeping higher by 2.478–3.524 points on navtest scenes forgiven on any of five loss metrics, with lower adjacent-frame plan consistency. Both changes are larger there than on the rest. EPDMS, scored the same way, does not separate the two target sets.

Primary area:
datasets and benchmarks
Keywords:
benchmark evaluation, training labels, autonomous driving, NAVSIM, label quality, dataset audit, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bewiOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how human-reference forgiveness in NAVSIM affects the training labels used by GTRS-Dense for autonomous driving trajectory prediction. When the human reference fails a particular evaluation metric, the label generator assigns a passing score to every candidate trajectory for that metric, potentially removing meaningful distinctions between valid and invalid trajectories.

The authors reconstruct the original labels by disabling this overwrite and analyze all 103,288 NAVSIM training scenes. They find that forgiveness removes loss-relevant candidate distinctions in 10.88% of scenes, predominantly affecting lane keeping. They further demonstrate that released models have substantially weaker lane-keeping discrimination on affected scenes.

Through controlled checkpoint continuation experiments, the authors show that restoring the original labels improves lane-keeping discrimination and compliance. However, these improvements come with reduced temporal planning consistency and do not translate into a significant improvement in the aggregate EPDMS score.

The paper concludes that evaluation rules can introduce systematic supervision distortions when reused for training-label generation, highlighting an important distinction between evaluation robustness and effective learning supervision.

Critical strengths
  1. Comprehensive and rigorous empirical analysis. The paper systematically investigates the effects of human-reference forgiveness across all 103,288 NAVSIM training scenes. The distinction between overwritten labels and genuinely erased candidate distinctions is particularly valuable. The authors also validate their reconstruction against the released labels, providing strong evidence for the reliability of their measurements (Sections 4-5).

  2. Strong experimental evidence of supervision degradation. The authors demonstrate that forgiven labels substantially reduce the ability of trained models to distinguish valid from invalid trajectories. The evaluation includes released GTRS-Dense and Hydra-MDP checkpoints, matched control groups, held-out scenes, and controlled checkpoint continuation experiments (Sections 6 and 8). These complementary analyses strengthen the connection between the identified label issue and observed model behavior.

  3. Important reproducibility findings. Section 7 identifies substantial hardware-dependent differences in regenerated training labels caused by processor-specific numerical kernels. The authors demonstrate that controlling the OpenBLAS configuration restores consistency with the released labels. This provides an additional practical contribution concerning reproducibility in simulation-based learning pipelines.

Critical weaknesses
  1. Limited methodological novelty. The main contribution is an empirical investigation of an existing evaluation mechanism rather than a new learning methodology. Human-reference forgiveness was already documented by NAVSIM's authors, while Wei et al. (2026) investigated related evaluation-side issues. Although extending this analysis to training supervision is useful, the paper does not propose or validate a new general solution for addressing the identified problem. This limits its methodological contribution relative to ICLR expectations.

  2. Insufficient validation through full model training. The controlled training experiments in Section 8 continue an existing GTRS-Dense checkpoint for only 300 optimizer steps, with an additional analysis at 530 steps, using three random seeds. Since the checkpoint was originally trained using forgiven labels, these experiments cannot establish whether the observed improvements persist under complete training from scratch or after convergence. The evaluation is also restricted to a single benchmark and training intervention, limiting generalizability.

  3. Unclear improvement in overall planning quality. Although restoring pre-overwrite labels improves lane-keeping discrimination and compliance, the authors report reduced adjacent-frame planning consistency and no statistically distinguishable improvement in aggregate EPDMS (Section 8). Therefore, the results establish that forgiveness removes meaningful training distinctions but do not demonstrate that correcting these labels produces a better overall driving policy. This trade-off significantly limits the practical implications of the findings.

Questions for authors
  1. Training from scratch: Would the observed improvements in lane-keeping discrimination and compliance persist if GTRS-Dense were trained from scratch using pre-overwrite labels? Could the authors provide longer training experiments or learning curves to establish whether the improvements remain after convergence?

  2. Overall planning performance: How should the improvement in lane keeping be interpreted given the simultaneous reduction in adjacent-frame planning consistency and the absence of significant EPDMS gains? Is there evidence that restoring the original labels improves actual trajectory safety or planning quality rather than optimizing an individual evaluation component?

  3. Alternative supervision strategies: The paper also considers masking forgiven columns instead of replacing their labels. Could the authors provide a more detailed comparison of masking, published forgiveness, and pre-overwrite supervision, particularly regarding overall planning performance? Would treating forgiven components as uncertain supervision provide a better balance between preserving informative distinctions and avoiding penalties for contextually justified human behavior?

Reviewer_APVxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

NAVSIM is a data-driven, non-reactive autonomous vehicle simulation and benchmarking framework designed to evaluate vision-based end-to-end driving models. This paper studies how to improve NAVSIM’s human-reference filtering, by updating the label file and scoring methodology, which the authors argue have certain problems in the current NAVSIM system such as penalizing contextually justified maneuvers.

Critical strengths

This paper introduces a better set of labels and scoring to improve the NAVSIM evaluation framework.

Critical weaknesses

I could not really understand the paper at a meaningful level. Fundamental concepts of the paper like NAVSIM, “column”, “the loader” are never properly introduced nor contextualized. Maybe these concepts are standard in autonomous vehicles research (I do not work in that field), but regardless the paper overall and its contributions are extremely hard to understand.

Here is an example snippet that I could not parse (but there are many passages like this throughout the paper):

Its label generator replaces a component by passing labels throughout a scene’s vocabulary when the human reference fails that component. An overwrite can change a target level without removing a target distinction: a column that already assigns every candidate the same failing value had no candidate ordering to erase. We therefore distinguish a human-reference zero, an overwrite that changes a value, and an overwrite of a previously mixed column. The last predicate is evaluated after the training consumer’s target mapping and restricted to the columns its loss reads.

I recommend the authors share this work with several more colleagues, ask colleagues to explain back what they believe the contributions to be, and continue iterating on the writing until the value of the contribution is significantly more understandable. For a top-venue like ICLR, the paper’s setting and main contribution should at a high-level be understandable to most AI researchers, and this paper is far from that bar.

Other sections of the paper seem like an AI agent looked at the NAVSIM code and auto-generated some documentation about the code, rather than a rigorous scientific narrative intended for AI researchers to learn something. Example:

Population and human census. The denominator is the benchmark’s own navsim/planning/ script/config/common/train_test_split/scene_filter/navtrain.yaml token list, exactly 103,288 scenes. We use its complete original-scene population and the corresponding released GTRS-Dense labels. The human-zero mask is the label entry’s own human call, scored without history

I cannot understand this text. Why would the first sentence start with “the denominator” without describing any formula where a denominator is present? What is a “human call”?

Questions for authors

How much time did you spend asking for reviews of your manuscript / other AI researchers to read it?

Can you articulate your most basic ELI5 summary of why the contribution of this paper is meaningful and what is novel about it?

Reviewer_ijXROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate how the human-reference forgiveness rule used in NAVSIM affects the training labels of GTRS-Dense. When the human reference scores zero on a particular metric, the label generator assigns passing scores to all 16,384 candidate trajectories for that metric. The authors reconstruct the labels before this overwrite and compare them against the released labels across all 103,288 navtrain scenes. They find that this rule removes distinctions between candidate trajectories in 10.88% of the scenes, and most of the changes come from lane keeping. They also analyze the released GTRS-Dense and Hydra-MDP checkpoints and conduct additional training experiments using different supervision policies. Their results show that using pre-overwrite labels improves lane keeping but reduces planning consistency, without any consistent improvement in the overall EPDMS score.

Critical strengths
  • I find the full-corpus analysis of the released training labels the most noteworthy component of the paper. The authors reconstruct the labels before the forgiveness overwrite and measure how frequently the rule removes distinctions between candidate trajectories. The analysis covers all 103,288 navtrain scenes. The authors further perform several checks to verify that their reconstruction matches the released labels. The results show that the rule removes distinctions in 11,237 scenes, with lane keeping accounting for most of the affected columns. I also like that the authors distinguish between simply changing label values and removing distinctions that are actually used by the training loss.

  • I also like that the paper evaluates how these label changes affect the trained models, rather than just analyzing the labels themselves. The authors evaluate the released GTRS-Dense and Hydra-MDP checkpoints and conduct additional training experiments using different supervision policies. On held-out navtest scenes, using the pre-overwrite labels improves the lane-keeping head's ability to distinguish passing and failing trajectories. I think this is useful since it shows that label changes can affect what the model learns. The hardware-dependent label regeneration analysis is also interesting, especially that the default configuration on Sapphire Rapids fails to reproduce the complete published labels for all 4,992 tested scenes, and it identifies the numerical library configuration responsible for this difference.

Critical weaknesses
  • My main concern is the limited novelty and overall significance of the paper. The human-reference forgiveness mechanism was already introduced in NAVSIM, and its authors have themselves acknowledged that it could overlook important failures. More recent work by Wei et al. (2026) (referenced several times in the paper) also investigates reference-conditioned forgiveness and its effects on NAVSIM evaluation. The main contribution in this paper is extending the analysis to the released GTRS-Dense training labels and measuring how much information is removed from the candidate targets. While I appreciate how detailed the analysis is, I feel the underlying observation is quite straightforward once the label-generation procedure is examined. The authors also do not propose a way to resolve the trade-off introduced by the forgiveness rule. I understand that this is primarily an empirical analysis, but I believe the authors should better establish the broader significance of these findings beyond this particular training pipeline. The analysis also does not establish whether similar issues occur in other training pipelines or benchmarks that use reference-dependent evaluation rules.

  • I also have some key concerns about the experimental evaluation. The authors report a large difference in lane-keeping AUC between forgiven and unforgiven scenes. For example, on navtest, the released GTRS-Dense checkpoint achieves an AUC of 0.7095 on forgiven scenes compared to 0.9807 on matched unforgiven scenes. However, these groups are selected based on whether the human reference fails the metric, so the scenes may differ considerably in difficulty. While the authors match scenes based on the number of failing candidates and use the drivable-area head as a control, I do not think this completely accounts for the differences between these groups. The continued-training experiments provide more direct evidence of the effect of the labels, but they start from a checkpoint already trained using the published labels and continue for only 300 additional steps, with a supporting analysis at 530 steps, making it difficult to understand whether the same effects would be observed if the different supervision policies were used from the beginning of training.

  • Lastly, I feel the practical implications of the findings remain unclear. The paper shows that using pre-overwrite labels improves lane keeping by about 2.5–3.5 points on forgiven navtest scenes, but extended comfort decreases by about 6.3–7.6 points across the three seeds. The overall EPDMS also does not show any consistent improvement. While I appreciate that the authors report this trade-off rather than claiming an overall improvement, since the forgiveness mechanism was originally introduced to avoid penalizing potentially justified maneuvers, I feel it is unclear which supervision policy should actually be preferred.

Questions for authors
  • The paper builds on an already known human-reference forgiveness mechanism and the work by Wei et al. (2026), which also investigated its effects on NAVSIM evaluation. Could the authors clarify what they consider the main new insight beyond quantifying how this mechanism affects GTRS-Dense training labels? I am particularly interested in understanding whether these findings have implications beyond this particular training pipeline.

  • The paper reports that forgiveness removes distinctions in 10.88% of training scenes, but Section 5.2 also shows that the changed entries account for 1.4355% of all entries across the seven loss columns. Could the authors explain in more detail how these two measurements should be interpreted in terms of their actual effect on training? Is there any analysis (or can the authors run additional analysis) showing how much the affected scenes contribute to the training loss or gradients?

  • The continued-training experiments show improvements in lane keeping but reductions in extended comfort without any consistent improvement in EPDMS. Since neither supervision policy appears to be clearly better, could the authors clarify describe in more detail what implications they draw from these results? In particular, do the findings suggest any specific changes to the training-label generation procedure?

  • As I discussed in the weaknesses as well, the continued-training experiments start from the released checkpoint at step 4,800 and continue for 300 or 530 steps. Could the authors clarify how much the results might depend on starting from a model already trained using the published labels? I feel this is important for understanding whether these findings would hold if the models were trained with different supervision from the beginning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.