acceptodds
Under review as a conference paper at ICLR 2027

World4Scorer: Outcome-Grounded World Modeling for Autonomous Driving

159 views

Abstract

Autonomous driving requires choosing a safe and efficient plan as surrounding traffic evolves. Generate-and-select planners propose multiple trajectories and score them for execution, and they have outperformed representative directprediction baselines on NAVSIM. Their scorer must compare plans that were never executed. Driving logs record the future of only the executed trajectory, so matching the logged future can leave predictions for the alternatives unconstrained; a simulator, in contrast, can label the outcome of every candidate. We introduce World4Scorer, which builds the scorer as a trajectory-conditioned JEPAstyle predictor: it predicts a state for each candidate and reads the candidate’s scores from that state. Simulator outcome labels supervise the states of all candidates, and the observed future of the executed trajectory anchors the predictor to real scene evolution. Because one predictor produces every candidate’s state, the anchor can constrain shared parameters used to score unexecuted plans, while the future itself is needed only during training. Generated candidates mostly score well, so a scene-matched bank adds low-scoring plans to the outcome supervision; framewise choices can conflict, so inertial re-ranking keeps consecutive selections consistent. World4Scorer achieves state-of-the-art NAVSIM-v2 performance and a strong adapted-system result on closed-loop Bench2Drive. With the LeWM world model and planning budget fixed, outcome-based scoring also improves manipulation planning on the OGBench-Cube benchmark.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, World Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QVvnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes World4Scorer to address the supervision gap in generate-and-select planners for end-to-end autonomous driving, specifically the lack of future supervision when the scorer evaluates unexecuted trajectories. Experiments demonstrate state-of-the-art performance on NAVSIM benchmarks, along with strong generalization in closed-loop Bench2Drive evaluations and cross-domain manipulation tasks.

Critical strengths
  1. Clear problem formulation and efficient method design. To address the lack of supervision for unexecuted trajectories, the paper integrates simulator outcome labels with observed future visual features. This mechanism effectively constrains the learning of latent states without incurring the computational overhead of video generation during inference, offering a computationally efficient trade-off.

  2. Comprehensive experiments, ablation studies and visualization analysis. For instance, visualizing the future features for different trajectory intentions and demonstrating the specific impact of state interventions on trajectory selection intuitively and effectively validates the physical plausibility of the latent state representations.

Critical weaknesses
  1. Potential reliance on simulator rules. The core supervision signals for the scorer depend on simulator-generated labels. If the simulator's physical rules or collision detection mechanisms deviate from the real world (Sim-to-Real gap), the scorer may produce misjudgments in complex, long-tail scenarios not adequately covered by the simulator.

  2. Decoupling of the post-processing module from the backbone network. Inertial re-ranking serves as a non-learnable post-processing module, meaning it cannot optimize the backbone network via gradient backpropagation. If the scorer's predicted scores have minor deviations, the re-ranking stage might frequently reject high-scoring trajectories, potentially leading to sub-optimal selections.

Questions for authors

Please refer to the Weaknesses.

3 comments
AuthorsOpenReview ↗

Thank You for the Positive Review: Re-Ranking Changes 16% of Choices, Repairing 1,599 Consecutive Pairs and Breaking 10

Dear Reviewer QVvn,

Thank you for your valuable feedback. The revision adds a test of re-ranking under noisy scores in a new Appendix F at the end of the paper (F.6), and the Conclusion now states the limitation of non-reactive agents. The changes made for you are shown in orange.

In short:

  • Our outcome labels simulate only the ego vehicle's motion, and Bench2Drive tests the planner against reactive traffic (W1).
  • Re-ranking overrides a high-scoring plan only to keep the motion continuous, and it keeps its gain when the scores are noisy (W2).

W1. How much do the labels rely on simulator rules?

Only through the ego vehicle's own motion: the scene, the other agents and the map in our outcome labels come from real driving logs, and the future target is a real camera frame. Each candidate is tracked through a recorded nuPlan scene by an LQR controller with a kinematic bicycle model, and then checked for collisions against the logged agent boxes and for drivable-area compliance against the HD map (Appendix B.1).

Because imitation makes most generated candidates safe, few of them show rare outcomes. The scene-matched bank adds labelled unsafe plans, such as collisions and drivable-area violations (Figure 4).

The remaining gap is that background agents follow their logs and do not react to our plan. Bench2Drive tests the planner in CARLA, where traffic reacts. There, World4Scorer reaches a Driving Score of 73.27 over 220 routes (Table 3), and 72.13 when its ego state comes from GNSS, IMU and speedometer estimates (Appendix F.7).


W2. Can re-ranking reject high-scoring plans when the scores are slightly off?

Only when the top plan breaks continuity: candidates that continue the previous plan receive no penalty, so their learned order stands.

Effect of re-ranking (fixed weights and candidates)ValueSource
Choices changed1,601 of 10,040 consecutive frames (16%)App. E.2
Consecutive pairs repaired / broken (extended comfort)1,599 / 10App. E.2
Ego progress90.0 → 89.8Table 2c
Extended comfort77.0 → 92.8Table 2c
EPDMS range over an eight-fold change of λ_state92.42–92.49App. E.3, Coverage variant

On the repaired pairs, re-ranking also reduces heading jitter by 34%, and among the changed selections, the distance to the logged plan drops from 1.98 to 1.61 m (Appendix E.2).

Re-ranking keeps its gain when the learned scores are noisy. In a new test (Appendix F.6), we added Gaussian noise to the scores before selection, with a standard deviation of up to twice the per-frame spread of the 64 candidate scores. Re-ranking raised EPDMS at every noise level, by +1.58 to +2.59 points, including levels at which the noise had already changed most framewise choices. Without noise, when re-ranking changes the choice, the plan it picks scores a median of 0.0066 below the top-scoring plan.

Re-ranking sits outside the network on purpose: it needs no labels and no retraining, so it works with any scorer. With DrivoR's weights and candidates fixed, it raises EPDMS from 90.66 to 92.27 (Appendix E.3).


Thank you again for your positive assessment. We hope these results address both concerns, and we would be grateful if you would consider raising your score.

Best regards,

Authors

Dear Reviewer QVvn,

Thank you again for your positive review. The discussion period ends on November 18 (AoE), and we have not heard back from you yet. If you have any follow-up questions, an early reply would leave us enough time to answer them before the deadline.

Following your review, the revision tests re-ranking under noisy scores (Appendix F.6), and the Conclusion now states that the logged agents behind our NAVSIM labels do not react to our plan. Could you let us know whether this addresses your concerns? If it does, we would be grateful if you could take it into account in your final rating.

Best regards, Authors

Thank you for the rebuttal. I appreciate the authors' efforts to address my concerns. The revisions have improved the completeness of the paper. Based on my overall assessment of the paper's novelty and contributions, I will maintain my rating.

Reviewer_JF2iOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes the World4Scorer, which use world model to predict the scores for outcome-based supervision in autonomous driving. A trajectory generator produces candidate plans, which are evaluated using a shared latent world model and scoring heads. The model is jointly trained with simulator-derived outcome labels for candidate trajectories and a JEPA-style future prediction objective for the executed trajectory. An additional candidate bank provides more diverse training examples, and inertial re-ranking improves temporal consistency. Experiments on NAVSIM and Bench2Drive demonstrate competitive planning performance, with an additional evaluation on robotic manipulation.

Critical strengths
  • The idea to include predicted but unexecuted trajectories in the training is interesting. The way to use simulator-derived outcome labels is a reasonable approach to addressing the inaccessible supervision for unexecuted trajectories.

  • The method achieves competitive results on NAVSIM, including 94.0 PDMS and 93.0 EPDMS, and reports improvements over existing methods on Bench2Drive.

Critical weaknesses
  • The presentation of this paper can be further improved, particularly the abstract and the training framework part. The paper shows clear signs of AI-assisted writing but has not been carefully polished. The relationships among trajectory generation, latent state prediction, outcome scoring, and future supervision are not clearly stated. The key contributions are not sufficiently emphasized. My suggestion is that the important part should be stated first, followed by the less important ones. Figure 2 is also not very clear. The 'fire' icon and the stop gradient could be very confusing.

  • The idea to introduce outcome-supervised scoring is good. Except this, other modules like JEPA-style future feature prediction and candidate-bank sampling are more like the combination of existing components. Thus, the novelty of this paper could be limited.

  • The experiments do not adequately study the contributions of the different training objectives. There are four losses in the overall loss. The ablation study of each part is not enough.

  • There are four different losses to optimize different parts of the framework. The stability of the proposed method needs further verification. Do the four jointly optimized losses introduce gradient conflicts in the shared modules?

Questions for authors
  • Could the authors provide more comprehensive ablations of the training objectives, particularly the interaction between future prediction, future feature prediction, and outcome supervision, as well as the individual contribution of the candidate bank?

  • Why are PDMS and several other metrics missing from the last row of Table 1?

  • What is the specific advantage of using JEPA in this framework? What benefits does it provide over other auxiliary prediction objectives?

8 comments
AuthorsOpenReview ↗

Thank You for the Review: Every Removable Objective Is Now Ablated, and the Four Losses Show No Gradient Conflict (1/2)

Dear Reviewer JF2i,

Thank you for your valuable feedback, and for finding outcome-supervised scoring a good idea. The revision adds what you asked for: a bank-loss ablation (F.3) and a gradient-conflict test (F.5) in a new Appendix F at the end of the paper, and an abstract and Section 3 that now put the shared predictor first. The changes made for you are shown in blue.

In short:

  • Removing the future target costs 0.58 PDMS at seed 3 and 0.61 over three seeds, and removing the bank loss costs 0.33 (point 3).
  • No two losses that share a module have a negative mean gradient cosine (point 4).

1. Presentation, AI-assisted writing and Figure 2

As you suggest, the abstract and Section 3 now put the core first: one predictor that every candidate passes through. The generator proposes 64 trajectories, the shared predictor turns each one into a state, and the score heads read that candidate's scores from the state.

In Figure 2, the fire icon marks the LoRA-tuned DINOv2 encoder, and the stop-gradient symbol marks where the future path is detached from it; the revised caption now explains both.

Our AI-use statement on page 11 describes how we used language models, as ICLR requires. The claims, the experimental design and the interpretation are the authors' own.


2. Is the method more than a combination of existing components?

Yes. What is new is the scorer: one trajectory-conditioned predictor produces the state from which each candidate is scored, and outcome labels supervise that state for all 64 candidates while the observed future supervises it for the executed trajectory. Because both signals train the same predictor, the observed future also shapes how the other candidates are scored (Proposition 1), at no extra cost at inference.

The JEPA objective and the bank are borrowed, and the paper says so: "Following JEPA" in Section 3.3 and "Following CLOVER" in Section 3.4.


3. Are the training objectives, including the bank, ablated?

Yes. Table 2 already ablates the future target and the outcome labels, and the revision adds a run without the bank loss and repeats the future-target ablation over three seeds.

LossWhat it trainsTest
Imitation (WTA)the trajectory generatornone possible: removing it removes the candidates
Outcome, generated candidatesthe shared predictor and score headsTable 2b: shuffling all outcome labels drops ordering accuracy from 64.2% to 51.5%
Outcome, bankthe same, on low-scoring plansTable 2b: shuffling only the bank labels gives 55.6%; Appendix F.3 removes the loss
Futurethe shared predictor, through the executed trajectoryTable 2a: no target, current-frame target, observed future; Appendix F.2 repeats the first and last with three seeds

Future prediction and future feature prediction are one objective: predicting the DINOv2 feature of the future frame.

In the new run (Appendix F.3), we trained the full model without the bank loss at seed 3, keeping the schedule and last-epoch evaluation of Table 2a.

Removed at seed 3PDMS
nothing (reported model)93.96
bank loss93.63
future target93.38
AuthorsOpenReview ↗

Response to Reviewer JF2i (2/2)

4. Do the four losses introduce gradient conflicts in the shared modules?

No. On the shared modules, every pair of losses has a non-negative mean gradient cosine, from +0.01 to +0.26.

In a new test (Appendix F.5), we computed each loss's gradient separately on the reported checkpoint, over 64 batches of training scenes, and took the cosine between each pair on the two shared modules, the predictor and the image encoder.

Loss pair (mean ± s.d.)Shared predictorImage encoder
Outcome vs. bank+0.26 ± 0.39+0.13 ± 0.23
Outcome vs. future+0.05 ± 0.14future loss does not reach it
Future vs. bank+0.01 ± 0.13future loss does not reach it
Imitation vs. outcomeimitation does not reach it+0.03 ± 0.23
Imitation vs. bankimitation does not reach it+0.04 ± 0.16

Single batches sometimes conflict, as the standard deviations show. Imitation and the future loss share no module. By design, stop-gradients block the outcome loss from the generator and the future loss from the encoder.


5. Why are PDMS and other metrics missing from the last row of Table 1?

The last row is a NAVSIM-v2 result, as Section 3.5 states: "We report both framewise and re-ranked results on NAVSIM-v2." Re-ranking uses NAVSIM-v2's two-frame extended-comfort check, which NAVSIM-v1 does not score, so the row leaves the NAVSIM-v1 columns (EP¹, TTC¹, C and PDMS) empty. The revised caption of Table 1 now says so.


6. What does JEPA offer over other auxiliary prediction objectives?

JEPA is the cheapest way to let the observed future train the scorer: its target is a feature of the future frame, so it needs no decoder, no annotation and nothing at inference. Pixel or video prediction would add a decoder and heavy training compute, and BEV or occupancy targets built from the logged boxes would repeat what the outcome labels already use. With only the target changed (Appendix F.4, seed 3), per-candidate agent boxes reach 93.73 PDMS, below the image target. Section 3.3 now opens with this reason.


Your suggestions on the abstract, Figure 2 and the ablations have improved the paper. We hope this addresses your concerns and respectfully ask you to reconsider your score.

Best regards,

Authors

Dear Reviewer JF2i,

Thank you again for your review. The discussion period ends on November 18 (AoE), and we have not heard back from you yet. If you have any follow-up questions, an early reply would leave us enough time to answer them before the deadline.

Following your review, the revision rewrites the abstract and Section 3 to put the core idea first, revises Figure 2, ablates every removable training objective (Table 2, Appendix F.3), and measures how well the gradients of the four losses agree (Appendix F.5). Could you let us know whether this addresses your concerns? If it does, we would be grateful if you could take it into account in your final rating.

Best regards, Authors

Reviewer_CStxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes World4Scorer for generate-and-select planners on NAVSIM. The key idea is to treat DrivoR's scorer as a trajectory-conditioned predictor and to train it with two signals: simulator outcome labels for all candidates, and the observed future of the executed trajectory (the front-camera DINOv2 feature 2 s ahead). It further adds a bank of low-scoring candidates and an inertial re-ranking step that accounts for compatibility with the previous plan. The paper reports 94.094.0 PDMS on NAVSIM-v1 and 93.093.0 EPDMS on NAVSIM-v2, along with strong closed-loop results on Bench2Drive and improved manipulation on OGBench-Cube.

Critical strengths
  1. Strong performance with a lightweight model. The future-prediction loss is used only during training, so it adds no inference cost to the model. With a lightweight ViT-S backbone, the method achieves the best results among the compared methods: 94.094.0 PDMS on NAVSIM-v1 and 93.093.0 EPDMS on NAVSIM-v2 with re-ranking. Even without re-ranking, its 91.491.4 EPDMS remains the best in Table 1.

  2. Broad evaluation. Beyond NAVSIM-v1 and v2, the paper evaluates closed-loop driving on Bench2Drive and robotic manipulation on OGBench-Cube, and provides a wide range of ablations and analyses.

Critical weaknesses
  1. Limited novelty and an overstated motivation. Training every candidate with simulator outcome labels is already standard practice for scorers such as the Hydra-MDP line [1], GTRS, and DrivoR. SafeDrive [2] already builds trajectory-conditioned per-candidate world representations and trains them with per-agent, per-timestep collision labels. Moreover, the paper's own outcome labels apply the logged futures of surrounding agents to every candidate under NAVSIM's non-reactive assumption (App. B.1). The claimed gap, that unexecuted candidates have no observed future, therefore stems from choosing an ego-view image as the target rather than from the data.

  2. The effect of the future loss, the core contribution, is not sufficiently demonstrated. The gain over DrivoR is only +0.3+0.3 PDMS. The +0.95+0.95 in Table 2a comes from a single seed, and the no-target baseline (93.0193.01) is itself below DrivoR (93.793.7), so the gain cannot be distinguished from seed variance. Proposition 1 holds for any added loss, so it does not explain why the future target in particular should help.

References

[1] Z. Li et al., "Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation," arXiv:2406.06978, 2024.

[2] J. Kim et al., "SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse World," CVPR 2026.

Questions for authors
  1. The outcome labels are computed by applying the logged futures of surrounding agents to every candidate under NAVSIM's non-reactive assumption (App. B.1). Under the same assumption, agent-level futures (e.g., future agent boxes) could also supervise every candidate. Why was an ego-view image, which applies only to the executed trajectory, chosen as the target instead? Have the authors compared the two targets?

  2. Excluding re-ranking, the gain over DrivoR is small (+0.3+0.3 PDMS, +0.7+0.7 EPDMS). Most of the NAVSIM-v2 gain (1.61.6 of 2.32.3 EPDMS points) comes from re-ranking, which yields the same gain when applied to DrivoR. What evidence shows that the improvement comes from the proposed future loss (e.g., Table 2a repeated over multiple seeds)?

  3. On Bench2Drive, the future loss is heavily down-weighted (weight 0.10.1, with its gradient scaled by 0.050.05), while ground-truth ego state and route-based rules are added. Could the authors provide a component-wise ablation showing where the performance gain comes from?

  4. Please discuss the relationship to Hydra-MDP [1] and SafeDrive [2] in the main text. In particular, SafeDrive trains trajectory-conditioned per-candidate world representations with per-agent, per-timestep collision labels; please clarify how World4Scorer differs from it.

References

[1] Z. Li et al., "Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation," arXiv:2406.06978, 2024.

[2] J. Kim et al., "SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse World," CVPR 2026.

4 comments
AuthorsOpenReview ↗

Thank You for a Careful and Constructive Review: The Observed-Future Target Raises PDMS on All Three Seeds (+0.61), and Our Lead over DrivoR Holds under the Same Re-Ranking (+0.71 EPDMS) (1/2)

Dear Reviewer CStx,

Thank you for your valuable feedback. The revision adds what you asked for in a new Appendix F at the end of the paper: Table 2a repeated over three seeds (F.2), the agent-box target (F.4), a component ablation on Bench2Drive (F.7) and a comparison with Hydra-MDP and SafeDrive (F.1). Sections 1, 2 and 4.3 summarize them in the main text, and the changes made for you are shown in purple.

In short:

  • With the observed-future target, every seed scores above DrivoR (W2).
  • Under the same re-ranking, World4Scorer still leads DrivoR (Q2), and an agent-box target does no better than the image (Q1).
  • On Bench2Drive, the two decision-time rules add 14.25 Driving Score (Q3).

W1. Is the novelty limited and the motivation overstated?

There may be a misunderstanding here: the paper does not claim outcome labels as new, and the logged agent futures that the review points to are exactly how our outcome labels reach every candidate. Section 2 credits outcome labels for every candidate to GTRS, CLOVER and DrivoR, and now to the Hydra-MDP line as well. What World4Scorer adds is the scorer itself: one trajectory-conditioned predictor produces the state from which every candidate is scored, outcome labels supervise these states for all candidates, and the observed future trains the same predictor through the executed trajectory, at training time only.

The gap discussed in Section 1 concerns world models trained by matching the logged future, which a driving log records for the executed trajectory alone. Under NAVSIM's non-reactive assumption, the logged agents are replayed against every alternative (Appendix B.1), and the revised Section 1 now says so.

SafeDrive, by contrast, supervises its per-candidate sparse worlds with collision-free and drivable-area labels, and none of its losses uses future sensor data. It also adds LiDAR and pretrains on detection and BEV segmentation, while World4Scorer uses cameras alone and no perception labels. Without re-ranking, World4Scorer scores higher on both NAVSIM versions (Appendix F.1).

PDMSEPDMS (framewise)
SafeDrive (cameras + LiDAR)91.687.5
World4Scorer (cameras)94.091.4

W2. Can the future-loss gain be told apart from seed variance?

Yes. Every run with the observed-future target scores above every run without it, by at least 0.51 PDMS, while the runs without it span only 0.07. With the target, all three seeds also score above DrivoR's released checkpoint (93.69 in our evaluation, 93.7 as DrivoR reports).

In new runs (Appendix F.2), we trained the no-target model of Table 2a with seeds 1, 2 and 3 and the observed-future model with seeds 1 and 2, keeping the code, schedule, bank and last-epoch evaluation unchanged. The observed-future run with seed 3 is the submitted model, and Table 2a now lists the seed-3 run of both models.

TargetSeed 3Seed 1Seed 2Mean
None93.3893.4593.3993.41
Observed future (+2 s)93.9694.1293.9894.02
Difference+0.58+0.67+0.59+0.61

Proposition 1 describes the channel through which the future loss acts, and Table 2a shows which target to use. A loss on the executed trajectory can narrow the range within which two unexecuted candidates swap ranks only through parameters it shares with them, and the proposition holds for any such loss by design. In Table 2a, the same objective with a current-frame target scores 92.70, below even the no-target run, so what the target contains matters.

AuthorsOpenReview ↗

Response to Reviewer CStx (2/2)

Q1. Why the image over agent-level futures, and have the two been compared?

We now compare them: a per-candidate agent-box target reaches all 64 candidates, yet it scores below the observed image.

In a new run (Appendix F.4), the model predicts, for each of the 64 generated candidates, an occupancy grid of the logged vehicles, pedestrians and cyclists two seconds ahead, drawn in that candidate's ego frame at its +2 s pose. The run uses seed 3, the weight and stop-gradients of the image loss, and the schedule, bank and last-epoch evaluation of Table 2a.

Future targetCandidates it reachesPDMS
Nonenone93.38
Logged agent boxes, per candidateall 6493.73
Observed front view (+2 s)executed trajectory93.96

All three image runs under W2 also score above the box run. The image is the simplest target that adds information: it needs no annotation, so every driving log provides it, while an agent target needs 3D box labels. The agent boxes already enter training through the outcome labels, and the image adds what they leave out, such as lane markings, signals and unannotated objects.


Q2. Does most of the gain come from re-ranking, and what shows the future loss helps?

Yes, re-ranking gives DrivoR the same gain as it gives World4Scorer, +1.61 against +1.58 EPDMS, and with re-ranking applied to both, our checkpoint still leads. We fix DrivoR's released weights and candidates, and both checkpoints use the same selection weights and evaluation protocol (Appendix E.3). Ours is the submitted checkpoint (seed 3), and each interval is a paired bootstrap over the 136 test logs.

EPDMSFramewiseWith re-ranking
DrivoR90.6692.27
World4Scorer91.4092.98
Difference (95% CI)+0.74 [+0.44, +1.05]+0.71 [+0.40, +1.02]

W2 answers the second part of the question, over three seeds (Appendix F.2).


Q3. Where does the Bench2Drive gain come from?

The two decision-time rules add 14.25 Driving Score without retraining, and the route point adds 2.98. Reading the ego state from sensors costs 1.14, less than the 2.46 between two runs of the reported checkpoint, and removing the future loss from the route-point stage has no measurable effect. Every row covers all 220 routes with the official evaluator (Appendix F.7). The route-blind checkpoint is trained on the 1000 clips without the route-point input, and the reported model is warm-started from it with that input added (Appendix E.5).

CheckpointDecision-time rulesGPUDriving ScoreSuccess rate (%)
Route-blindnoH20056.0420.91
Route-blindyesH20070.2943.64
ReportedyesH20073.2746.36
Reported, estimated ego stateyesH20072.1346.36
Reported, rerunyesRTX 409070.8142.73
Reported, route re-ranking offcommand retention onlyRTX 409059.1228.18
Reported, command retention offroute re-ranking onlyRTX 409069.4841.82
Route-point stage retrained without the future lossyesRTX 409071.6548.18

In the estimated-ego-state evaluation, one route ran on an RTX 4090, and in the evaluation with route re-ranking off, 22 routes ran on an H200.

Against the reported configuration rerun on the same RTX 4090 cards, turning route re-ranking off costs 11.69 Driving Score, and turning command retention off costs 1.33, within the run-to-run difference.

The last row tests whether the route-point stage still needs the future loss once the route-blind stage has trained with it: we retrained the route-point stage from the same route-blind checkpoint, with the same data and schedule and the future loss off. Against the RTX 4090 rerun, the retrained model differs by +0.84 Driving Score (paired 95% interval over routes [−2.00, +3.66]), while the reported checkpoint's two runs differ by more, −2.46 [−5.48, +0.46]. The retrained model's success rate is 5.45 points higher than the rerun's: 25 routes succeed only without the future loss and 13 only with it (exact sign test, p = 0.07), while the reported checkpoint's two runs differ in success on 46 routes.


Q4. How does World4Scorer differ from Hydra-MDP and SafeDrive?

Section 2 now contrasts both methods with World4Scorer, and Appendix F.1 gives the full comparison table.

Hydra-MDP learns PDM sub-scores for a fixed vocabulary of 4,096 or 8,192 trajectories, and W1 describes SafeDrive. Neither uses future observations, which World4Scorer adds at training time.


We hope this addresses your concerns and respectfully ask you to reconsider your score.

Best regards,

Authors

Dear Reviewer CStx,

Thank you again for your careful and constructive review. The discussion period ends on November 18 (AoE), and we have not heard back from you yet. If you have any follow-up questions, an early reply would leave us enough time to answer them before the deadline.

Following your review, the revision discusses Hydra-MDP and SafeDrive in Section 2 and Appendix F.1, repeats the future-target comparison over three seeds (Appendix F.2), compares it with the agent-box target you asked about (Appendix F.4), and breaks the Bench2Drive result down by component (Appendix F.7). Could you let us know whether this addresses your concerns? If it does, we would be grateful if you could take it into account in your final rating.

Best regards, Authors

Thank you for the additional experiments and clarifications. The multi-seed results address my concern about the consistency of the gains from observed-future supervision.

However, my main concern remains the incremental contribution beyond existing outcome-supervised methods such as SafeDrive and Hydra-MDP. Future supervision provides a consistent but modest improvement, while much of the headline NAVSIM-v2 gain comes from rule-based re-ranking, which similarly benefits DrivoR. The Bench2Drive ablations also show a substantial contribution from route re-ranking, without a measurable benefit from retaining the future loss during the route-point training stage.

I will therefore maintain my original rating. Nevertheless, I recognize the empirical strengths of the overall system and would not oppose acceptance if the AC and other reviewers find the contribution sufficient.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Our Responses

We thank the reviewers. All three note our NAVSIM results (94.0 PDMS, 93.0 EPDMS). CStx and QVvn value the breadth of the evaluation and the training-only future target, which adds no cost at inference. JF2i finds outcome-supervised scoring of unexecuted candidates a good idea.

The revision adds a new Appendix F at the end of the paper and colours each change by the reviewer who asked for it: purple for CStx, blue for JF2i, orange for QVvn, and teal where several reviewers asked. A November 19 update also defines the scorer in Section 3 and states in the Introduction what existing scorers lack.

All three reviewers have now replied, and none disputes a result. CStx finds that the multi-seed results address the consistency of the future-supervision gains and recognizes "the empirical strengths of the overall system". JF2i writes, "I have never questioned your contributions or your additional results". QVvn finds that "the revisions have improved the completeness of the paper". The table lists each concern and how we resolved it.

Concern (raised by)How we resolved it
1. Is the future-loss gain larger than seed variance? (CStx W2, Q2; JF2i 3)Yes. The target adds 0.61 PDMS on average over three seeds, and every run with it beats every run without it (Appendix F.2)
2. Why an image target, and how do agent boxes compare? (CStx Q1; JF2i 6)The image target scores higher, even though agent boxes supervise all 64 candidates (Appendix F.4)
3. What is new beyond Hydra-MDP and SafeDrive? (CStx W1, Q4, follow-up; JF2i 2, follow-up)The scorer: one predictor scores every candidate and learns from both outcome labels and the observed future. Hydra-MDP and SafeDrive use no future observations (Appendix F.1)
4. Does each loss help, and do the losses conflict? (JF2i 3, 4)Yes. Every removable objective is ablated and each one helps, and no pair of losses conflicts on average (Table 2; Appendices F.3, F.5)
5. Is the gain from re-ranking, and does it cope with noisy scores? (CStx Q2, follow-up; QVvn W2)No. Re-ranking gives DrivoR the same gain, so our lead holds with and without it. It still helps when the scores are noisy (Appendices E.3, F.6)
6. What do the labels simulate, and what drives the Bench2Drive gain? (QVvn W1; CStx Q3, follow-up)Only the ego motion, as the Conclusion now states. On Bench2Drive, the gain comes from the decision-time rules and the route point (Appendix F.7)
7. Can the presentation be clearer? (JF2i 1, 5, follow-up)We revised the abstract, the Introduction, Section 3, Figure 2 and Table 1

Our reply to each reviewer gives the details.

Dear Reviewer JF2i,

Before the discussion closes, we summarize its focus: each question you raised in your review and your follow-up, and where it stands.

Your questionWhere it stands
Are the abstract and training framework clear, is the key contribution stated first, and is Figure 2 clear? (W1)Revised. The abstract, Section 3 and the first contribution now state the predictor first, and the Figure 2 caption explains the flame and the stop-gradient.
Is the method more than a combination of existing components? (W2)Yes. The paper credits JEPA and the bank to prior work, and what is new is the scorer: one predictor trained by outcome labels and the observed future.
Is each loss ablated, including the bank? (W3, Q1)Yes. Every removable objective is ablated: the future target over three seeds, the bank loss and the outcome labels (Table 2; Appendices F.2, F.3).
Do the losses conflict in the shared modules? (W4)No. No pair of losses has a negative mean gradient cosine (Appendix F.5).
Why are metrics missing from the last row of Table 1? (Q2)It is the re-ranked NAVSIM-v2 row, as Section 3.5 and the revised caption state.
What does JEPA offer over other auxiliary objectives? (Q3)It needs no decoder, no labels and nothing at inference, and a per-candidate agent-box target scores lower (Appendix F.4).
Are the scorer and the predictor the same thing? (follow-up 1)Yes. They are one network in two roles, and the updated revision defines both at the opening of Section 3.
Outcome labels are standard, so why does the third paragraph of the Introduction discuss them? (follow-up 2)The paper credits outcome labels to prior work. The revised paragraph now also states that none of the existing scorers trains its candidate states on future sensor data.
Why is your scorer better than existing scorers? (follow-up 3)It also learns from the observed future, which Hydra-MDP and SafeDrive do not use, and this raises PDMS on every seed at no cost at inference.

After two rounds, the focus has narrowed to one question of presentation: how strongly the Introduction emphasizes the scorer. The updated revision answers it in the Introduction and in Section 3, and your follow-up disputes no result.

We hope this summary helps, and we respectfully ask you to reconsider your score.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.