acceptodds
Under review as a conference paper at ICLR 2027

NavSafe-: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments

24 views

Abstract

End-to-end (E2E) driving policies have advanced rapidly on open-loop (OL) benchmarks, yet OL evaluation cannot reveal whether a policy can withstand compounding errors, recover from failures, or interact safely with surrounding actors. We introduce NavSafe-, a photorealistic closed-loop (CL) benchmark comprising 280 scenarios spanning 28 event types, each with success and failure criteria defined within a structured traffic-safety taxonomy, yielding category-level capability scores for Traffic Crashes, Vulnerable Road User Crashes, Traffic Violations, and Traffic Incidents. After evaluating 20 E2E policies, we find that OL gains do not reliably transfer to CL safety. Analysis of two common remedies reveals that (1) passive demonstration perturbation helps mainly when CL rollouts stay near their perturbed training states, and (2) OL reinforcement-learning fine-tuning exhibits reward hacking by trading safety margin for ego progress, which CL feedback amplifies into compounding safety-critical errors. Together, these results demonstrate the blind spot of OL benchmarks indicating CL safety success. The benchmark and an extensible toolbox for customizable event curation and policy diagnosis will be open-sourced to facilitate future research.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zCFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a photorealistic closed-loop driving benchmark comprising 280 scenarios across 28 safety-event types. Event-specific contracts distinguish successful handling of an event from generic route completion. The implementation combines log-based 3D Gaussian reconstructions, controllable actors, a diffusion image harmonizer, and a shared execution and scoring pipeline. Twenty learned policies are evaluated over three seeds. Their results show that strong NAVSIM performance does not ensure successful handling of these simulated events. Further case studies examine planning failures and the effects of demonstration perturbation and reward-based fine-tuning. A toolbox supports scenario construction and policy diagnosis.

Critical strengths
  1. Event completion is defined around the behavior that is actually required. For example, a compliant red-light hold can succeed without reaching a route endpoint, and a work-zone bypass receives a different contract in Table 5. This is consequential because an aggregate route score can otherwise penalize the intended safe behavior. The combination of these contracts with photorealistic observations is the most convincing contribution.

  2. The comparison is broad enough to be useful beyond one model family. Table 7 covers 20 learned policies, while the shared controller, matched post-training pairs, scenario provenance, and saved-trace scoring make several comparisons interpretable. The human reference is also appropriately presented as a solvability check rather than a population-level estimate. These choices give the benchmark a credible practical foundation.

Critical weaknesses
  1. The rendering validation does not yet establish the observation fidelity needed for the benchmark's central use case. Section C.4 evaluates the front camera along the logged ego trajectory, whereas evaluated policies depart from that trajectory and use additional cameras, including a camera not used in reconstruction. Section C.2 also switches independently reconstructed segments without blending, and Section C.3 applies the harmonizer independently to each frame. A failure after an ego deviation could therefore reflect a distorted or unstable safety cue as well as a planning error. Low aggregate FID, FVD, and feature distance on logged views do not distinguish these possibilities. Section G.2's shared actor state is a useful safeguard, but it does not establish that the rendered observation preserves that actor's visibility. I would want a bounded audit of critical actors and traffic controls in policy-induced views, particularly near segment switches. Perfect real-world fidelity is not necessary, but this narrower check matters before interpreting the failures as evidence about the policies.

  2. I do not think all of the proposed diagnoses identify the mechanism they name. The SimWAM example is particularly problematic. Section F.3 explicitly states that the decoded future frame does not enter the action head or change the executed trajectory. Its missing vehicle and the unsafe action are therefore two observed failures rather than evidence that losing the vehicle in that prediction removed a constraint from trajectory generation. Likewise, the perturbation coverage in Section F.4 measures only ego position at one timestamp. Two states at the same position can differ substantially in velocity and surrounding traffic, so proximity in that plot does not establish coverage of the policy's relevant inputs. These analyses are useful illustrations. They should be presented as hypotheses unless supported by evidence connecting the inspected representation to the action and the coverage proxy to the claimed state distribution.

Questions for authors
  1. In Section C.2, does the nearest logged-pose timestamp control only reconstruction selection, or also the appearance of dynamic objects and traffic controls? How is consistency with current simulator time maintained when the ego stops or slows relative to the log? This is a clarification about the evaluated implementation.

  2. Are there existing checks of critical-object visibility and traffic-control state in policy-induced views, including segment transitions and cameras not used in reconstruction? A concise description of existing checks could resolve much of the first concern. The bounded audit suggested above is a possible improvement, not a request for a new benchmark during rebuttal.

  3. Is there evidence beyond the displayed co-occurrence that connects the SimWAM prediction failure to its action representation? Is the positional coverage analysis supported by additional state variables? Otherwise, would the authors narrow these mechanism claims to qualitative hypotheses?

5 comments

W1 / Q2. Rendering fidelity in policy-induced views

"I would want a bounded audit of critical actors and traffic controls in policy-induced views, particularly near segment switches."

We thank the reviewer for suggesting a bounded audit. We ran this audit and report the results below.

Policy-induced views. We first measured how far the evaluated policies leave the logged trajectory (16 policies, 842k steps). The lateral offset is ≤ 2 m in 78% of steps and ≤ 5 m in 95% of steps. The heading offset is ≤ 15° in 94% of steps. For a controlled comparison, we render each scenario at every scoring timestamp at the logged pose, at lateral offsets of ±2 m and ±5 m, and at yaw offsets of ±15°. We use the front camera with the Harmonizer, i.e., the images that the policy receives. We use two scenario sets: the 94 scenarios that NavSafe-∞ shares with other simulators (Table A), and all 280 scenarios (Tables B and C).

Metrics.

  • Critical actors. We follow the NTA-IoU protocol of DriveDreamer4D (Zhao et al., CVPR 2025). For each critical actor, we project its 3D box into the render and compute the IoU with the nearest YOLO11x detection (Jocher & Qiu, 2024). If no detection lies within 10 px (center to center), the score is 0. We score only the actors that the detector finds in the real image (score ≥ 0.5), because the detector-based metric cannot score the others. Actor NTA-IoU is the mean score over these actors. A high value therefore means that the actors appear at the correct position and size.
  • Traffic controls. We use the mapped signal heads that the detector finds in the real image. Recall is the fraction of these heads that are also detected in the render, within 1 m at their depth. Signal state agreement is the fraction of recalled heads whose detected color matches the real image.

Comparison with other simulators. We ran the same audit on two other simulators: Odyssey (Kim et al., 2026; concurrent work, OmniRe 3DGS with Fixer) and DriveArena (Yang et al., ICCV 2025; generative). All three are rendered at the same offsets on the 94 shared scenarios (Table A). NavSafe-∞ scores highest on every metric and at every offset.

Table A. 94 shared scenarios.

0 m±2 m±5 m±15°
Actor NTA-IoUNavSafe-∞0.7030.6630.6130.675
Odyssey0.6880.6580.6040.655
DriveArena0.5920.5650.5410.391
Signal recallNavSafe-∞0.9280.8710.7900.910
Odyssey0.7100.6710.5690.700
DriveArena0.0220.0160.0080.012
Signal state agreement†NavSafe-∞0.9440.9300.9190.932
Odyssey0.8320.8500.8360.810
DriveArenan/a‡n/a‡n/a‡n/a‡

† Computed over the signal heads that each simulator recalls, so the set of heads differs between simulators. ‡ DriveArena recalls too few signal heads (≤ 2.2%) for a reliable state agreement.

Segment switches. To measure the effect of segment switches, we split the frames of all 280 scenarios into two groups: frames within 1 s (≤ 1 s) of a segment switch, and all other frames (> 1 s) (Table B). Actor NTA-IoU is almost the same in both groups; the largest difference is 0.027, at ±5 m. Near a switch, signal recall is 0.016–0.041 lower, and signal state agreement is 0.008–0.016 lower. A segment switch therefore adds only a small loss on top of the effect of the viewpoint offset.

Table B. NavSafe-∞, 280 scenarios, split by the time to the nearest segment switch.

0 m±2 m±5 m±15°
Actor NTA-IoU≤ 1 s0.6920.6250.5370.643
> 1 s0.6940.6250.5640.643
Signal recall≤ 1 s0.8740.8260.7320.862
> 1 s0.8910.8460.7730.878
Signal state agreement≤ 1 s0.9530.9450.9500.942
> 1 s0.9630.9540.9580.958

Cameras not used in reconstruction. We also render camera L2, which is not used in the reconstruction, on all 280 scenarios, and score the critical actors in the same way (Table C). At the logged pose, L2 scores 0.623, which is 0.091 below the real image, as expected for a view that the reconstruction never sees. The viewpoint offsets lower the L2 score by 0.027 at ±2 m, 0.086 at ±5 m, and 0.004 at ±15°.

Table C. NavSafe-∞, 280 scenarios, camera not used in reconstruction.

Real0 m±2 m±5 m±15°
Actor NTA-IoUL20.7140.6230.5960.5370.619

Harmonizer. We apply the Harmonizer to each frame independently because NuRec supported only this mode while we developed NavSafe-∞. NuRec first added a temporal mode in release 26.08 (September 2026). All results in Tables A–C use the per-frame version. We plan to support the temporal mode as an upgrade of the benchmark.

We will add this audit to the appendix of the revised paper.

Q1. What the nearest logged-pose timestamp controls

"Does the nearest logged-pose timestamp control only reconstruction selection, or also the appearance of dynamic objects and traffic controls? How is consistency with current simulator time maintained when the ego stops or slows relative to the log?"

The nearest logged-pose timestamp selects only the reconstruction segment that renders the static background. It does not control the state of dynamic objects or the scored signal state.

  • Dynamic objects. Each agent is placed at its logged state at the current simulator time, so the actors that the ego sees in the images always match the actor states used for scoring.
  • Traffic controls. Scoring and episode termination read the logged signal state at the current simulator time. The policies see signals only in the rendered images, where the signal color is part of the static background and follows the viewpoint.
  • Ego slower than the log. Segment selection follows the ego position, while agents and signal states advance with simulator time. The scored scene therefore stays consistent with simulator time.

Limitation: signal color in the image. The rendered signal color can differ from the scored state only when both conditions hold: a signal that faces the ego changes state during the clip, and the ego is ahead of or behind the log at that moment. This limitation applies to any reconstruction that stores signal heads as static background. We will state it in the revised paper and list time-consistent signal rendering as future work.


W2 / Q3a. SimWAM: link between the prediction failure and the action

"Is there evidence beyond the displayed co-occurrence that connects the SimWAM prediction failure to its action representation?"

We thank the reviewer for the careful reading and for viewing these analyses as "useful illustrations," which is exactly the role we intended. Sec. 4.2 is a failure-investigation case study (L326: "We further conduct case studies…"). It examines one representative event per safety category to offer qualitative insight into how mainstream E2E paradigms fail in closed loop. It is not intended to establish or test mechanistic hypotheses. To avoid overclaiming, we will revise the camera-ready version to make this scope clear.

We agree that the decoded future frame does not directly determine the executed trajectory, because SimWAM's isolated attention mask prevents action tokens from attending to future-frame tokens. The two observations are, however, not architecturally independent. At inference, the video expert still encodes the current frame, and the action expert attends to its layer-wise keys and values; the same encoding also conditions future-frame generation. A shared deficiency in how the current-frame encoding represents the oncoming vehicle is therefore a plausible common source of both observations. This reading is consistent with SimWAM's design, in which future prediction shapes the representation used for planning. We offer it as a qualitative insight from the case study, not as a verified mechanism.

More broadly, these case studies are intended to show that NavSafe-∞ can be used to study why policies fail, through qualitative analysis. How to turn such analyses into quantitative measures, for example of the discrepancy between the visual and action streams of a world-action model, remains an open question for the community. We hope that our benchmark and these qualitative studies are an effective step toward it.


W2 / Q3b. Perturbation coverage measured by ego position only

"Two states at the same position can differ substantially in velocity and surrounding traffic, so proximity in that plot does not establish coverage of the policy's relevant inputs."

We agree that ego position at a single timestamp does not establish coverage of the policy's inputs. For these cases we ran multiple rollouts and examined many states. The reported figure is a representative example, and the full set was omitted from the main text for space. As with Sec. 4.2, the analysis is intended as a case study that provides insight, not as a characterization of the full state distribution.

Our goal is to show that NavSafe-∞ can be used to study policy behavior through qualitative analysis. How to measure such effects quantitatively, for example the coverage of all policy-relevant inputs rather than ego position alone, remains an open question for the community. We hope that our benchmark and these qualitative studies are an effective step toward it.

Thank you for the additional rendering audit and for clarifying the scope of the case studies. The new audit measures how well rendered traffic signals match detections in the logged images under controlled viewpoint shifts. My main concern remains unresolved because the audit does not establish how often the signal shown to the policy disagrees with the state used for scoring during actual rollouts. A policy could proceed on a rendered green light and be penalized for running a red one, and the rebuttal leaves open whether this affected the reported results. I keep the original rate.

Reviewer_GrCfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NavSafe-∞, a photorealistic closed-loop benchmark for end-to-end driving policies. It reconstructs 280 nuPlan scenarios with 3DGS, groups them into 28 event types based on a traffic-safety taxonomy, and evaluates 20 NAVSIM-trained policies together with a human driver reference.

Critical strengths

S1. Closed loop evaluation for NAVSIM policies

Most recent E2E policies are trained and evaluated on NAVSIM, so having a closed-loop environment for NAVSIM is meaningful. Existing checkpoints can be used directly without retraining, and the rendering quality in the videos and the paper looks good.

S2. Scenario feasibility

Human drivers completed all scenarios, which shows that these scenarios are passable.

Critical weaknesses

W1. Overlap with existing benchmarks

Many photorealistic and closed-loop benchmarks already exist (e.g., navhard and HUGSIM), but the paper does not make the differences from them clear. For example, HUGSIM already provides 3DGS-based photorealistic closed-loop evaluation and even includes aggressive actors.

W2. Limited reactive traffic

Only following vehicles are reactive (IDM). Vehicles ahead, crossing vehicles, and pedestrians replay the logs (Appendix D.1). Navhard already makes all background vehicles reactive, so this setting is weaker for evaluating interactions with surrounding agents.

W3. Previously reported OL and CL relationship

The OL-CL relationship is not a new finding. Prior work already showed that displacement-based OL metrics correlate poorly with CL performance [1].

[1] F. Codevilla et al., “On offline Evaluation of Vision-based Driving Models”, ECCV 2018.

Questions for authors

Please see the weaknesses above. I also have the following points.

Q1. Correlation with other evaluation benchmarks

The paper only compares with the NAVSIM open-loop metric (Figure 13). What is the correlation with navhard (3DGS-based pseudo closed-loop evaluation) and with Bench2Drive SR/DS?

Q2. Evaluation runtime across models

Appendix H reports the per-episode runtime for DrivoR only. It would be good to report the total evaluation time for other models as well.

5 comments

W1. Difference from existing photorealistic closed-loop benchmarks

"Many photorealistic and closed-loop benchmarks already exist (e.g., navhard and HUGSIM), but the paper does not make the differences from them clear."

We thank the reviewer and agree that the submission did not state these differences clearly. We summarize them below.

navhard is a pseudo closed-loop benchmark. The ego vehicle does not follow real dynamics, and the scene does not respond to the policy's own actions.

HUGSIM supports 3DGS rendering and aggressive actors. Recent work (BridgeSim, Zhao et al., 2026; DreamStream, Leng et al., CoRL 2026) reports visible rendering artifacts in HUGSIM, which can affect how faithfully it evaluates E2E policies. In addition, its data domain differs from the nuPlan domain that the evaluated policies are trained on.

Although OOD behaviors are important to investigate, it covers up potential closed-loop failures with in-distribution driving scenarios.

In addition, HUGSIM does not define a systematic event taxonomy to evaluate different aspects of safety. On the other hand, NavSafe-∞ leverages clearly defined event-based evaluation protocols to benchmark closed-loop safety failures discussed in prior literature (RoadSafe365, Liu et al., 2026). Specifically, each safety event is curated based on annotated nuPlan scenario labels with our human annotators with sophisticated verification process. Such event-based protocol gives a more comprehensive evaluation and a finer-grained understanding of each policy's capabilities.

We will clarify this more in detail in the camera-ready version of the paper where we have a whole section discussing the differences compared with prior benchmarks.


W2. Limited reactive traffic

"Only following vehicles are reactive (IDM). Vehicles ahead, crossing vehicles, and pedestrians replay the logs."

The traffic setting is a deliberate part of our event-based protocol. It separates in-distribution interactions from out-of-distribution (OOD) ones, and each event type uses the setting that it is designed to test.

In-distribution behavior (218 natural scenarios). These scenarios replay unmodified nuPlan logs. All evaluated policies are trained on nuPlan, so the logged behavior of other agents is the distribution that they learned from. We therefore keep the surrounding traffic on its logs and make only following vehicles reactive (semi-reactive mode, ported from MetaDrive). This prevents not-at-fault rear-end collisions when the ego slows down, and it keeps the traffic behavior close to the logged distribution. A failure in these scenarios therefore reflects the policy’s own closed-loop behavior, not a shift in traffic behavior.

OOD behavior (62 intervened scenarios). These scenarios insert vehicles, VRUs, and objects that are absent from the log. The inserted actors run their authored controllers: IDM vehicles (22 scenarios), social-force pedestrians and cyclists that react to the ego (20), and static obstacles (20). The surrounding traffic stays semi-reactive.

Why not make all traffic reactive. We agree that broader reactive traffic is a useful extension. We chose the current setting for two reasons. Fully reactive traffic is itself OOD for log-trained policies. It also lets other agents yield to or avoid the ego, which can hide the ego’s own failures and makes it unclear whether a success comes from the policy or from the traffic model. For future extensions, the actor interface is modular, so learned traffic agents can be added as another controller.


W3. The OL–CL relationship was reported before

"Prior work already showed that displacement-based OL metrics correlate poorly with CL performance [Codevilla et al., ECCV 2018]."

We thank the reviewer for this reference and agree that the OL–CL gap was reported in prior work. This gap is the motivation of NavSafe-∞, not a finding that we claim, and we now cite Codevilla et al. (2018) in the introduction and Related Work. Our contribution is to measure and analyze this gap for current E2E policies: (1) a photorealistic CL benchmark in the policies’ own training domain — all scenes, including the intervened ones, are reconstructed from nuPlan logs, so NAVSIM-trained policies run directly without retraining, and a CL failure reflects the policy rather than a sensor or scene domain gap; (2) event-level evaluation that shows where the gap appears, by event type and category; and (3) a failure analysis across current E2E paradigms (diffusion policies, VLAs, world-action models), including why demonstration perturbation and OL RLFT do not reliably close the gap (Sections 4.2–4.3).

Q1. Correlation with navhard and Bench2Drive

"What is the correlation with navhard (3DGS-based pseudo closed-loop evaluation) and with Bench2Drive SR/DS?"

navhard (two-stage). We collected the published navhard two-stage EPDMS of our evaluated checkpoints from the original papers and project pages. 11 of the 20 learned policies report navhard score. Across these 11 policies, the Spearman ρ with NavSafe-∞ is 0.45 for DS and 0.55 for SR (0.57 and 0.66 when the privileged PDM-Closed is included). The two benchmarks agree on the top of the ranking (GTRS-Dense (SimScale), GTRS-Dense, and SimWAM). They disagree mainly on policies whose failures only appear over a full episode:

  • RAP ranks 3rd on navhard, but 8th by DS and 9th by SR on NavSafe-∞.
  • DriveLaW ranks 6th on navhard, but succeeds in only 1.9% of events.
  • For the passive-perturbation pairs, both LTF and DrivoR improve on navhard after SimScale training. On NavSafe-∞, however, the DS of LTF stays essentially unchanged, and the DS of DrivoR even drops. Section 4.3 explains this finding.

We expect a moderate correlation between these two complementary protocols on the same held-out logs. navhard scores two 4 s plans from pre-rendered states without considering policy induced state visitation, whereas NavSafe-∞ scores event handling over a full episode.

Bench2Drive. Only 4 of our 20 policies report Bench2Drive results (SparseDriveV2, AutoVLA, ReCogDrive, and RAP). All four use separately trained CARLA weights, and RAP and SparseDriveV2 also use different backbones. With n = 4 and different weights, a correlation is not informative (ρ = 0.8, p = 0.2).


Q2. Evaluation runtime for all models

"Appendix H reports the per-episode runtime for DrivoR only. It would be good to report the total evaluation time for other models as well."

We agree that this information is useful. The purpose of Appendix H is to report the resource use and runtime of the simulator itself, and we use DrivoR as a representative example. For other models, the total evaluation time differs only by the model's own inference latency. This latency is independent of the simulator's resource use and speed.

As a reference, we report the runtime of three additional policies below, and we will add the runtime of all 20 evaluated policies in the camera-ready version. Setup: 2× RTX 3090 per job (GPU 0 for rendering, GPU 1 for simulation and the policy), 10 scenarios, seed 0, visualization and diagnostic outputs disabled.

PolicyQuantityMedianp90MaxN
AutoVLAEpisode evaluation240.4 s578.5 s723.0 s10 scenarios
Steady-state frame1.636 s1.919 s2.871 s10 scenarios
GPU memory/worker (2 GPUs summed)19.0 GiB19.5 GiB19.8 GiB10
Full benchmark (280 × 3), estimated99 h on 1 worker, 10 h on 10 workers840 episodes
SimWAMEpisode evaluation170.6 s258.5 s271.7 s10 scenarios
Steady-state frame0.571 s0.767 s1.421 s10 scenarios
GPU memory/worker22.1 GiB22.2 GiB22.5 GiB10
Full benchmark, estimated53 h on 1 worker, 5.4 h on 10 workers840 episodes
SimWAM-RLEpisode evaluation150.7 s213.6 s238.1 s10 scenarios
Steady-state frame0.576 s0.635 s0.704 s10 scenarios
GPU memory/worker21.9 GiB22.3 GiB22.4 GiB10
Full benchmark, estimated43 h on 1 worker, 4.5 h on 10 workers840 episodes
AllRenderer startup50 s50 s50 s2 launches

Thank you for the answers. Despite the first answer, my concern on methodological novelty still remains. This work seems a good engineering work, combining a well-established event-based scenarios from RoadSafe365 and high-fidelity rendering pipelines from 3DGS and DiffusionHarmonizer, as the authors stated. The authors state that “OL gains do not reliably transfer to CL safety” as their finding, which is well-known issue from [A, B], so this is an over-claimed statement.

  • [A] JT Zhai et al., Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes, arXiv 2023
  • [B] Z. Li et al., Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?, CVPR 2024

While I acknowledge the practical value of the proposed benchmark to the community, I think this paper does not meet the quality of the ICLR as a top-tier conference, in terms of the technical novelty. So, I keep my original rating.

Reviewer_CH4GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a photorealistic closed-loop driving benchmark with 280 scenarios spanning 28 safety event types. It combines reconstructed scenes, editable actors, event-specific evaluation contracts, and adapters for 20 driving policies. Beyond the leaderboard, the authors analyze failures in planning, trajectory scoring, world prediction, and reasoning-action alignment, and investigate why demonstration perturbation and open-loop RL fine-tuning do not consistently improve closed-loop safety.

Critical strengths
  1. Substantial, task-relevant evaluation. Evaluating 20 policies under a shared controller and event protocol is a meaningful contribution. Also, the breakdown of failure attribution is interesting. The matched checkpoint comparisons are particularly useful for assessing post-training interventions.

  2. Detailed benchmark specification. The appendices document scenario provenance, train/test separation, actor interventions, termination rules, policy adapters, and saved traces. The human reference establishes solvability under the specified interface. These controls make the benchmark more interpretable and potentially reusable.

Critical weaknesses
  1. The rendering validation does not sufficiently cover the states on which the central conclusions depend. Table 1 and Appendix C.4 evaluate fidelity on the front camera along the logged ego trajectory. However, the benchmark is intended to diagnose policies after they deviate from that trajectory, including under actor edits and different camera rigs. Without a targeted robustness check, it remains difficult to separate closed-loop policy failures from failures induced by the observation pipeline. This matters directly to the interpretation of the policy rankings and failure diagnoses.

  2. The mechanistic explanations are stronger than the supporting evidence. In the SimWAM case, Appendix F.3 explicitly states that the decoded future frame does not enter the action head. Likewise, the perturbation analysis uses two scenarios selected from those exhibiting the desired policy-dependent effect, and measures coverage through ego position at one timestamp (Appendix F.4). This is useful illustration, but limited evidence for a general explanation based on training-distribution coverage. Finally, the RLFT comparisons show a safety/progress trade-off, but do not isolate reward exploitation from other consequences of post-training. These distinctions could be consequential because explanatory analysis, beyond constructing another simulator benchmark, is a main claimed contribution for this benchmark work.

Questions for authors
  1. Evaluation validity: How robust are the principal rankings and diagnosed failures under controlled changes to the rendering pipeline, particularly disabling the Harmonizer or examining segment boundaries and off-trajectory views? A targeted check on representative failures, together with policy-output agreement on matched real/rendered observations where available, would materially strengthen confidence in the benchmark.

  2. Mechanism clarification: What evidence links SimWAM's diagnostic video prediction to its action error, given the stated independence of the action head? Should this claim be restricted to correlated prediction and planning failures?

  3. Generality of the analysis: Does the relationship between perturbation coverage and improvement persist across the benchmark, including scenarios not selected for exhibiting that relationship? For the RLFT diagnosis, is there a matched reward-component control that supports attributing the degradation specifically to progress-driven reward exploitation?

3 comments

W1 / Q1. Rendering fidelity off the logged trajectory (1/2)

"Table 1 and Appendix C.4 evaluate fidelity on the front camera along the logged ego trajectory. However, the benchmark is intended to diagnose policies after they deviate from that trajectory, including [...] different camera rigs. [...] particularly [...] examining segment boundaries and off-trajectory views"

We thank the reviewer for asking for a targeted robustness check. We ran a bounded audit of the rendered observations in off-trajectory views and near segment switches, and we report the results below. We also explain why the evaluation keeps the camera rig fixed.

Off-trajectory views. We first measured how far the evaluated policies leave the logged trajectory (16 policies, 842k steps). The lateral offset is ≤ 2 m in 78% of steps and ≤ 5 m in 95% of steps. The heading offset is ≤ 15° in 94% of steps. For a controlled comparison, we render each scenario at every scoring timestamp at the logged pose, at lateral offsets of ±2 m and ±5 m, and at yaw offsets of ±15°. We use the front camera with the Harmonizer, i.e., the images that the policy receives. We use two scenario sets: the 94 scenarios that NavSafe-∞ shares with other simulators (Table A), and all 280 scenarios (Table B).

Metrics.

  • Critical actors. We follow the NTA-IoU protocol of DriveDreamer4D (Zhao et al., CVPR 2025). For each critical actor, we project its 3D box into the render and compute the IoU with the nearest YOLO11x detection (Jocher & Qiu, 2024). If no detection lies within 10 px (center to center), the score is 0. We score only the actors that the detector finds in the real image (score ≥ 0.5), because the detector-based metric cannot score the others. Actor NTA-IoU is the mean score over these actors. A high value therefore means that the actors appear at the correct position and size.
  • Traffic controls. We use the mapped signal heads that the detector finds in the real image. Recall is the fraction of these heads that are also detected in the render, within 1 m at their depth. Signal state agreement is the fraction of recalled heads whose detected color matches the real image.

Comparison with other simulators. We ran the same audit on two other simulators: Odyssey (Kim et al., 2026; concurrent work, OmniRe 3DGS with Fixer) and DriveArena (Yang et al., ICCV 2025; generative). All three are rendered at the same offsets on the 94 shared scenarios (Table A). NavSafe-∞ scores highest on every metric and at every offset.

Table A. 94 shared scenarios.

0 m±2 m±5 m±15°
Actor NTA-IoUNavSafe-∞0.7030.6630.6130.675
Odyssey0.6880.6580.6040.655
DriveArena0.5920.5650.5410.391
Signal recallNavSafe-∞0.9280.8710.7900.910
Odyssey0.7100.6710.5690.700
DriveArena0.0220.0160.0080.012
Signal state agreement†NavSafe-∞0.9440.9300.9190.932
Odyssey0.8320.8500.8360.810
DriveArenan/a‡n/a‡n/a‡n/a‡

† Computed over the signal heads that each simulator recalls, so the set of heads differs between simulators. ‡ DriveArena recalls too few signal heads (≤ 2.2%) for a reliable state agreement.

W1 / Q1. Rendering fidelity off the logged trajectory (2/2)

Segment switches. To measure the effect of segment switches, we split the frames of all 280 scenarios into two groups: frames within 1 s (≤ 1 s) of a segment switch, and all other frames (> 1 s) (Table B). Actor NTA-IoU is almost the same in both groups; the largest difference is 0.027, at ±5 m. Near a switch, signal recall is 0.016–0.041 lower, and signal state agreement is 0.008–0.016 lower. A segment switch therefore adds only a small loss on top of the effect of the viewpoint offset.

Table B. NavSafe-∞, 280 scenarios, split by the time to the nearest segment switch.

0 m±2 m±5 m±15°
Actor NTA-IoU≤ 1 s0.6920.6250.5370.643
> 1 s0.6940.6250.5640.643
Signal recall≤ 1 s0.8740.8260.7320.862
> 1 s0.8910.8460.7730.878
Signal state agreement≤ 1 s0.9530.9450.9500.942
> 1 s0.9630.9540.9580.958

We will add this audit to the appendix of the revised paper.

Camera rigs. We agree that the configurable camera rig makes this a natural question. All evaluated policies are trained on NAVSIM, i.e., on nuPlan logs recorded with the nuPlan camera rig. In the evaluation, each policy therefore receives the nuPlan cameras that it was trained on, with their original intrinsics and extrinsics. This keeps the observations in the policies' own training domain, so that a closed-loop failure reflects the policy rather than a sensor domain gap. The configurable rig in our formulation (Sec. 3) is a toolbox feature for future policies that are trained on other rigs. We do not change the rig in the reported results, and we will state this explicitly in the revised paper.

Robustness of the diagnosed failures. Regarding the suggested targeted check on representative failures, the four case studies in Sec. 4.2 already allow such a check, because in each of them the policy perceives the hazard before it fails. LTF detects the crossing vehicles at 29 m. DrivoR's candidate set contains a safe stopping trajectory, which it ranks 43rd. The oncoming vehicle is visible in SimWAM's rendered observation. AutoVLA's reasoning identifies the crossing pedestrians and calls for a stop in both states. If the renderer had caused these failures, the policies would not have perceived the hazards. The diagnosed failures therefore lie in planning, scoring, prediction, and reasoning, after the observation.

W2 / Q2 / Q3. Strength of the explanatory analyses

"The mechanistic explanations are stronger than the supporting evidence. [...] These distinctions could be consequential because explanatory analysis, beyond constructing another simulator benchmark, is a main claimed contribution for this benchmark work."

We thank the reviewer for the close reading of Appendices F.3 and F.4. We would like to clarify that the analyses in Secs. 4.2 and 4.3 explain effects that NavSafe-∞ measures, and that each explanation is supported by the policies' own outputs. We address each case below.

SimWAM (Q2). We agree that the decoded future frame does not enter the action head. We decode it only to inspect what SimWAM's representation contains, so it does not need to lie on the action path. SimWAM co-trains the two experts with future-frame prediction as supervision, and at inference the video expert still encodes the current observation, which the action expert uses through shared attention (Zhao et al., 2026). The same current-frame encoding therefore conditions both the action and the future-frame generation. Both outputs of this shared encoding agree: the decoded frame omits the oncoming vehicle, and the executed plan passes through the space that the vehicle occupies (Fig. 6). This supports the claim in Sec. 4.2 that latent prediction errors can undermine planning in world-model-based policies.

Perturbation coverage (Q3a). The policy-dependent effect is measured on all 280 scenarios and three seeds (Table 2). SimScale co-training raises the DS of the anchor-based DiffusionDrive (44.77 → 52.19) and GTRS-Dense (64.27 → 73.54), but does not raise that of the query-based policies: the DS of DrivoR drops (64.74 → 57.66), and that of LTF stays essentially unchanged (53.04 → 52.65). This split follows from the training objective. The perturbed data are constructed independently of the policy, so they cannot follow the states that each policy visits in CL. Anchor-based policies keep their candidates close to expert trajectories, and training mainly improves their scorer, which keeps the selected rollouts near the perturbed states. Query-based policies drift away from these states in CL. Fig. 7 measures coverage in ego position because SimScale defines its perturbations in this space (up to 20 m longitudinally and 2 m laterally), over 30 paired seeds for each policy pair.

RLFT (Q3b). Fig. 8(a) shows the reward exploitation directly in OL. MTDrive raises PDMS mainly through progress, while its safety sub-scores drop. RLFT as applied in ReCogDrive instead leaves the safety sub-scores largely intact. The CL outcomes follow this difference. ReCogDrive's OL gains transfer to CL, in both nominal and safety-critical scenarios. MTDrive's trade-off persists in nominal CL (Fig. 8(b)) and is amplified in safety-critical scenarios, where collisions terminate rollouts early and its DS drops sharply (Fig. 8(c)). The CL degradation thus appears in the policy that shows the OL signature of reward exploitation, and not in the policy that does not.

Contribution. These analyses show that NavSafe-∞ can localize a CL failure to planning, scoring, prediction, or reasoning, beyond aggregate scores. How to turn such qualitative analyses into quantitative measures, for example of the discrepancy between the visual and action streams of a world-action model, remains an open question for the community. We hope that our benchmark and these studies are an effective step toward it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

**Summary for the Area Chair**

Dear Area Chair,

Thank you for handling our submission. We summarize our contributions and how the rebuttal addressed the reviewers' concerns.

Contributions

NavSafe-∞ is a benchmark paper (primary area: datasets and benchmarks). It contributes:

  • A closed-loop safety benchmark in the policies' training domain. NavSafe-∞ has 280 photorealistic scenarios over 28 event types, all reconstructed from nuPlan logs. NAVSIM-trained policies therefore run unchanged, and a failure reflects the policy, not a domain gap.
  • Event-level safety scoring. Each event type has its own success and failure criteria within a traffic-safety taxonomy. A human driver completes every scenario.
  • Findings on 20 current E2E policies. NAVSIM and navhard scores still miss closed-loop failures. Qualitative case studies (Sec. 4.2) localize representative failures to planning, trajectory scoring, world prediction, and reasoning–action alignment. Two common remedies, demonstration perturbation and open-loop RL fine-tuning, do not reliably close this gap (Sec. 4.3).

Concerns addressed in the rebuttal

  1. Rendering fidelity away from the logged trajectory (CH4G, zCFG). We ran the requested audit on all 280 scenarios. The offsets (±2 m, ±5 m, ±15°) cover 94–95% of the deviations of the policies. At the logged pose, ±2 m, ±5 m, and ±15°, actor NTA-IoU is 0.69, 0.63, 0.54–0.56, and 0.64, and signal recall is 0.87–0.89, 0.83–0.85, 0.73–0.77, and 0.86–0.88. Signal state agreement stays at 0.94–0.96. A segment switch changes each metric by at most 0.041. Against two other simulators (Odyssey, DriveArena), NavSafe-∞ scores highest at every offset.

  2. Strength of the mechanism claims (CH4G, zCFG). We clarified that the case studies in Sec. 4.2 are qualitative illustrations, not verified mechanisms, and we will revise the text to make this scope clear. The effect of demonstration perturbation itself is measured on all 280 scenarios with three seeds. zCFG acknowledged this clarification.

  3. Difference from prior benchmarks (GrCf). navhard is pseudo closed-loop. HUGSIM uses a different data domain and has no event taxonomy. As a new result, the Spearman ρ with navhard across 11 shared policies is 0.45 (DS) and 0.55 (SR). For example, RAP ranks 3rd on navhard and 8th on NavSafe-∞.

  4. Reactive traffic (GrCf). The 218 natural scenarios use semi-reactive traffic: only following vehicles are reactive (IDM), and the other agents replay the log. This keeps the traffic close to the training distribution of the policies, and it prevents not-at-fault rear-end collisions. The 62 intervened scenarios also insert actors with their own controllers.

  5. The OL–CL gap was reported before (GrCf). We agree that the gap itself is known, and we will state it as motivation in the revised paper. The cited works study displacement-based open-loop metrics on earlier policies. Our findings concern current policies and the newer NAVSIM and navhard metrics, which still miss closed-loop failures.

We will add the audit and these clarifications to the revised paper.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.