- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a photorealistic closed-loop driving benchmark comprising 280 scenarios across 28 safety-event types. Event-specific contracts distinguish successful handling of an event from generic route completion. The implementation combines log-based 3D Gaussian reconstructions, controllable actors, a diffusion image harmonizer, and a shared execution and scoring pipeline. Twenty learned policies are evaluated over three seeds. Their results show that strong NAVSIM performance does not ensure successful handling of these simulated events. Further case studies examine planning failures and the effects of demonstration perturbation and reward-based fine-tuning. A toolbox supports scenario construction and policy diagnosis.
Critical strengths
-
Event completion is defined around the behavior that is actually required. For example, a compliant red-light hold can succeed without reaching a route endpoint, and a work-zone bypass receives a different contract in Table 5. This is consequential because an aggregate route score can otherwise penalize the intended safe behavior. The combination of these contracts with photorealistic observations is the most convincing contribution.
-
The comparison is broad enough to be useful beyond one model family. Table 7 covers 20 learned policies, while the shared controller, matched post-training pairs, scenario provenance, and saved-trace scoring make several comparisons interpretable. The human reference is also appropriately presented as a solvability check rather than a population-level estimate. These choices give the benchmark a credible practical foundation.
Critical weaknesses
-
The rendering validation does not yet establish the observation fidelity needed for the benchmark's central use case. Section C.4 evaluates the front camera along the logged ego trajectory, whereas evaluated policies depart from that trajectory and use additional cameras, including a camera not used in reconstruction. Section C.2 also switches independently reconstructed segments without blending, and Section C.3 applies the harmonizer independently to each frame. A failure after an ego deviation could therefore reflect a distorted or unstable safety cue as well as a planning error. Low aggregate FID, FVD, and feature distance on logged views do not distinguish these possibilities. Section G.2's shared actor state is a useful safeguard, but it does not establish that the rendered observation preserves that actor's visibility. I would want a bounded audit of critical actors and traffic controls in policy-induced views, particularly near segment switches. Perfect real-world fidelity is not necessary, but this narrower check matters before interpreting the failures as evidence about the policies.
-
I do not think all of the proposed diagnoses identify the mechanism they name. The SimWAM example is particularly problematic. Section F.3 explicitly states that the decoded future frame does not enter the action head or change the executed trajectory. Its missing vehicle and the unsafe action are therefore two observed failures rather than evidence that losing the vehicle in that prediction removed a constraint from trajectory generation. Likewise, the perturbation coverage in Section F.4 measures only ego position at one timestamp. Two states at the same position can differ substantially in velocity and surrounding traffic, so proximity in that plot does not establish coverage of the policy's relevant inputs. These analyses are useful illustrations. They should be presented as hypotheses unless supported by evidence connecting the inspected representation to the action and the coverage proxy to the claimed state distribution.
Questions for authors
-
In Section C.2, does the nearest logged-pose timestamp control only reconstruction selection, or also the appearance of dynamic objects and traffic controls? How is consistency with current simulator time maintained when the ego stops or slows relative to the log? This is a clarification about the evaluated implementation.
-
Are there existing checks of critical-object visibility and traffic-control state in policy-induced views, including segment transitions and cameras not used in reconstruction? A concise description of existing checks could resolve much of the first concern. The bounded audit suggested above is a possible improvement, not a request for a new benchmark during rebuttal.
-
Is there evidence beyond the displayed co-occurrence that connects the SimWAM prediction failure to its action representation? Is the positional coverage analysis supported by additional state variables? Otherwise, would the authors narrow these mechanism claims to qualitative hypotheses?
5 comments
Q1. What the nearest logged-pose timestamp controls
"Does the nearest logged-pose timestamp control only reconstruction selection, or also the appearance of dynamic objects and traffic controls? How is consistency with current simulator time maintained when the ego stops or slows relative to the log?"
The nearest logged-pose timestamp selects only the reconstruction segment that renders the static background. It does not control the state of dynamic objects or the scored signal state.
- Dynamic objects. Each agent is placed at its logged state at the current simulator time, so the actors that the ego sees in the images always match the actor states used for scoring.
- Traffic controls. Scoring and episode termination read the logged signal state at the current simulator time. The policies see signals only in the rendered images, where the signal color is part of the static background and follows the viewpoint.
- Ego slower than the log. Segment selection follows the ego position, while agents and signal states advance with simulator time. The scored scene therefore stays consistent with simulator time.
Limitation: signal color in the image. The rendered signal color can differ from the scored state only when both conditions hold: a signal that faces the ego changes state during the clip, and the ego is ahead of or behind the log at that moment. This limitation applies to any reconstruction that stores signal heads as static background. We will state it in the revised paper and list time-consistent signal rendering as future work.
W2 / Q3a. SimWAM: link between the prediction failure and the action
"Is there evidence beyond the displayed co-occurrence that connects the SimWAM prediction failure to its action representation?"
We thank the reviewer for the careful reading and for viewing these analyses as "useful illustrations," which is exactly the role we intended. Sec. 4.2 is a failure-investigation case study (L326: "We further conduct case studies…"). It examines one representative event per safety category to offer qualitative insight into how mainstream E2E paradigms fail in closed loop. It is not intended to establish or test mechanistic hypotheses. To avoid overclaiming, we will revise the camera-ready version to make this scope clear.
We agree that the decoded future frame does not directly determine the executed trajectory, because SimWAM's isolated attention mask prevents action tokens from attending to future-frame tokens. The two observations are, however, not architecturally independent. At inference, the video expert still encodes the current frame, and the action expert attends to its layer-wise keys and values; the same encoding also conditions future-frame generation. A shared deficiency in how the current-frame encoding represents the oncoming vehicle is therefore a plausible common source of both observations. This reading is consistent with SimWAM's design, in which future prediction shapes the representation used for planning. We offer it as a qualitative insight from the case study, not as a verified mechanism.
More broadly, these case studies are intended to show that NavSafe-∞ can be used to study why policies fail, through qualitative analysis. How to turn such analyses into quantitative measures, for example of the discrepancy between the visual and action streams of a world-action model, remains an open question for the community. We hope that our benchmark and these qualitative studies are an effective step toward it.
W2 / Q3b. Perturbation coverage measured by ego position only
"Two states at the same position can differ substantially in velocity and surrounding traffic, so proximity in that plot does not establish coverage of the policy's relevant inputs."
We agree that ego position at a single timestamp does not establish coverage of the policy's inputs. For these cases we ran multiple rollouts and examined many states. The reported figure is a representative example, and the full set was omitted from the main text for space. As with Sec. 4.2, the analysis is intended as a case study that provides insight, not as a characterization of the full state distribution.
Our goal is to show that NavSafe-∞ can be used to study policy behavior through qualitative analysis. How to measure such effects quantitatively, for example the coverage of all policy-relevant inputs rather than ego position alone, remains an open question for the community. We hope that our benchmark and these qualitative studies are an effective step toward it.
Thank you for the additional rendering audit and for clarifying the scope of the case studies. The new audit measures how well rendered traffic signals match detections in the logged images under controlled viewpoint shifts. My main concern remains unresolved because the audit does not establish how often the signal shown to the policy disagrees with the state used for scoring during actual rollouts. A policy could proceed on a rendered green light and be penalized for running a red one, and the rebuttal leaves open whether this affected the reported results. I keep the original rate.
W1 / Q2. Rendering fidelity in policy-induced views
We thank the reviewer for suggesting a bounded audit. We ran this audit and report the results below.
Policy-induced views. We first measured how far the evaluated policies leave the logged trajectory (16 policies, 842k steps). The lateral offset is ≤ 2 m in 78% of steps and ≤ 5 m in 95% of steps. The heading offset is ≤ 15° in 94% of steps. For a controlled comparison, we render each scenario at every scoring timestamp at the logged pose, at lateral offsets of ±2 m and ±5 m, and at yaw offsets of ±15°. We use the front camera with the Harmonizer, i.e., the images that the policy receives. We use two scenario sets: the 94 scenarios that NavSafe-∞ shares with other simulators (Table A), and all 280 scenarios (Tables B and C).
Metrics.
Comparison with other simulators. We ran the same audit on two other simulators: Odyssey (Kim et al., 2026; concurrent work, OmniRe 3DGS with Fixer) and DriveArena (Yang et al., ICCV 2025; generative). All three are rendered at the same offsets on the 94 shared scenarios (Table A). NavSafe-∞ scores highest on every metric and at every offset.
Table A. 94 shared scenarios.
† Computed over the signal heads that each simulator recalls, so the set of heads differs between simulators. ‡ DriveArena recalls too few signal heads (≤ 2.2%) for a reliable state agreement.
Segment switches. To measure the effect of segment switches, we split the frames of all 280 scenarios into two groups: frames within 1 s (≤ 1 s) of a segment switch, and all other frames (> 1 s) (Table B). Actor NTA-IoU is almost the same in both groups; the largest difference is 0.027, at ±5 m. Near a switch, signal recall is 0.016–0.041 lower, and signal state agreement is 0.008–0.016 lower. A segment switch therefore adds only a small loss on top of the effect of the viewpoint offset.
Table B. NavSafe-∞, 280 scenarios, split by the time to the nearest segment switch.
Cameras not used in reconstruction. We also render camera L2, which is not used in the reconstruction, on all 280 scenarios, and score the critical actors in the same way (Table C). At the logged pose, L2 scores 0.623, which is 0.091 below the real image, as expected for a view that the reconstruction never sees. The viewpoint offsets lower the L2 score by 0.027 at ±2 m, 0.086 at ±5 m, and 0.004 at ±15°.
Table C. NavSafe-∞, 280 scenarios, camera not used in reconstruction.
Harmonizer. We apply the Harmonizer to each frame independently because NuRec supported only this mode while we developed NavSafe-∞. NuRec first added a temporal mode in release 26.08 (September 2026). All results in Tables A–C use the per-frame version. We plan to support the temporal mode as an upgrade of the benchmark.
We will add this audit to the appendix of the revised paper.