acceptodds
Under review as a conference paper at ICLR 2027

RoleDrive: Scene Representations with Distinct Planning Roles for End-to-End Autonomous Driving

Abstract

End-to-end autonomous driving offers a promising route to safe and efficient navigation by learning to predict driving trajectories directly from sensor observations. However, many existing end-to-end planners mix different types of planning-relevant information within a unified representation, without explicitly differentiating their roles in trajectory planning, potentially limiting planning performance. To address this problem, we propose RoleDrive, an intermediate scene representation framework that explicitly organizes scene information into three complementary representations with distinct planning roles. Specifically, RoleDrive constructs Context Tokens for global scene context modeling and Map Tokens for fine-grained local spatial reasoning from shared visual features, while Evolution Tokens capture recent scene changes through residuals between aligned historical and current Map representations. These complementary representations are then progressively integrated through hierarchical trajectory reasoning. Without using any additional training data and with only 36.9M parameters, RoleDrive achieves a PDMS of 94.9 on NAVSIM-v1, surpassing the reported human-driver baseline, while also attaining state-of-the-art performance on NAVSIM-v2 and the photorealistic closed-loop HUGSIM benchmark. Code will be publicly released upon acceptance at https://github.com/K1ngAO/RoleDrive

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_UrnUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoleDrive, which categorizes scene representations for end-to-end autonomous driving into three types: Context, which captures global scene context; Map, which provides local geometric structure; and Evolution, which represents temporal changes through residuals of historical features aligned with the ego vehicle's motion. Planning queries first attend to Context and then jointly attend to Map and Evolution, enabling staged interaction with different types of information. The authors evaluate the method on NAVSIM and HUGSIM, reporting a PDMS of 94.9 on NAVSIM v1, an EPDMS of 91.5 on the NAVSIM v2 navtest single-stage evaluation, and an HD-Score of 34.2 on HUGSIM. The paper also includes ablations of representation combinations, temporal fusion, and interaction order.

Critical strengths
  1. The architecture is coherent and addresses a practically important problem. Global scene information, local geometric constraints, and historical dynamics correspond to distinct information requirements in trajectory planning. Organizing these elements into representations that can be directly used for planning is sensible. The explicit motion alignment used to reduce ego-motion effects in temporal residuals is also well motivated.

  2. The ablations provide consistent, although relatively modest, evidence for the proposed components. In Table 4, the Context + Map configuration achieves a validation PDMS of 92.5, which increases to 92.9 after adding Evolution. Aligned residuals outperform unaligned residuals (92.7), and progressive interaction outperforms joint interaction (92.5). These results suggest that the gains are not attributable solely to the final configuration. However, the stability and precise source of these gains require further verification, as discussed below.

Critical weaknesses
  1. The paper does not yet isolate the contribution of its proposed division of roles. The main innovation appears to be a combination of existing mechanisms. ResWorld (https://arxiv.org/html/2602.10884v1) already uses motion-aligned temporal residuals, while DiffRefiner employs staged global and local interactions. The authors should clarify what capabilities are enabled specifically by the three-way division into Context, Map, and Evolution and by the proposed interaction pattern, beyond what could be obtained by combining these prior mechanisms. The improvements in the key ablations are small (approximately 0.2–0.5 PDMS), and all experiments appear to use a single random seed. The paper also does not clarify whether Joint versus Progressive and Fusion versus Evolution are matched in parameter count, decoder depth, or computational budget. In addition, the residual-weighted visualization of Evolution and the CKA analysis only establish statistical differences between representations; they do not by themselves demonstrate a semantic division of labor. Finally, the paper does not discuss CLOVER (https://arxiv.org/abs/2605.15120, May 2026), which uses a DrivoR-style planning framework and reports a PDMS of 94.5 and a navtest EPDMS of 90.4 under a revised protocol. This is an important reference point for assessing the actual improvement.

  2. The evaluation protocols raise concerns about the validity of the main conclusions. Appendix B states that both K=2 and K=3 obtain a validation score of 92.9, but K=3 was selected because the test-set results favored it. If the test set was used to select the default configuration, the reported test performance is no longer independent of model selection. The HUGSIM comparison also contains important confounding factors. RoleDrive is initialized with the V0 policy for 1.5 seconds before taking control, whereas DrivoR starts driving from the first step. The two methods also use different treatments of extrinsic parameters and lateral-acceleration inputs. More importantly, the original DrivoR paper reports RC/HD-Score values of 49.8/35.7, whereas the reproduction in this paper obtains 46.4/31.0, compared with 48.0/34.2 for RoleDrive. Until these reproduction discrepancies are explained and the system configurations are standardized, the observed closed-loop difference cannot be reliably attributed to the proposed architecture. The EPDMS of 91.5 should also be explicitly contextualized as a single-stage navtest result and should not be directly compared with navhard or full two-stage evaluation results.

Questions for authors
  1. Configuration selection: Was K=3 fixed before the navtest results were first accessed? Please describe the exact experimental sequence leading to the statement that “test set results favor K=3,” including how many configurations were evaluated or selected using test-set performance.

  2. Closed-loop comparison: Why does the reproduced DrivoR HD-Score decrease from the original 35.7 to 31.0? Does RoleDrive retain an advantage when initialization policy, extrinsic-parameter handling, vehicle-state inputs, and simulator corrections are standardized across both methods?

  3. Verification of the claimed contributions: Are model capacity and computational budget matched between Fusion and Evolution and between Joint and Progressive? Are the reported gains stable across multiple random seeds? If the interaction order is reversed, or if representational interventions are designed around different information needs, do the claimed roles and division of labor remain valid?

10 comments
AuthorsOpenReview ↗

Response to W1 (Part 1/2): Novelty and Comparison with Prior Work

Ego-motion alignment, temporal differencing, and staged attention are established techniques. Prior work has paid less attention to how the roles of different types of scene information in trajectory planning should guide their representation. RoleDrive’s contribution is to assign explicit planning roles to intermediate scene representations and use those roles to determine how each representation is constructed and when the planner reads it. This design reflects how drivers assess the overall traffic situation while paying particular attention to the road they are about to enter. Context first guides the trajectory queries with an overview of the scene. During refinement, the planner retains Context and reads Map and Evolution at corresponding locations in the region ahead. Map provides the current local geometry, while Evolution captures recent changes, allowing the planner to use both “what is here now” and “what has recently changed here” to refine the trajectory.

Comparison with ResWorld. Compared with ResWorld, RoleDrive gives the planner more direct access to recent changes at specific locations ahead and saves the computation required to predict future BEV features. ResWorld pools aligned BEV features across the full grid into sparse scene queries using a shared spatial attention mask. Its world model uses differences between queries at adjacent time steps to predict future BEV features for trajectory refinement. RoleDrive retains an overview of the visible scene through Context. On the forward local grid, it subtracts each aligned historical Map from the current Map at cells with valid overlap and preserves each cell’s location during temporal aggregation. Figure 3 of our paper illustrates the benefit during a turn. Map immediately provides the geometry of a region that has just come into view. The validity mask excludes missing historical observations from differencing, so entering the field of view does not itself register as a scene change. Where coverage overlaps, the planner reads current geometry from Map and recent change from Evolution at the same cell. Our paper already cites ResWorld in its discussion of future scene prediction. We will make this comparison explicit in the revision.

ComparisonResWorldRoleDrive
Temporal representationSparse queries pooled over the aligned BEV gridForward local Map grid
Residual computationQuery differences at adjacent time stepsCurrent minus each aligned historical Map, cell by cell
Spatial organizationQuery-level residualsOne Evolution Token per Map cell
Historical coverageNot compared hereMask missing history; retain current geometry in Map
Use in planningConstruct future BEV features for refinementRead Evolution alongside the current Map

Comparison with DiffRefiner. Compared with DiffRefiner, RoleDrive gives the planner separate access to overall scene context, current local geometry, and recent changes, so each can guide trajectory refinement according to its planning role. In DiffRefiner’s semantic interaction module, global and local attention draw from the same pool of BEV-derived features. In RoleDrive, the three inputs already differ in construction and physical structure before attention. Context summarizes the scene without fixed BEV locations. Map and Evolution share a local grid, allowing the planner to first use Context for an overall view, then retain it while reading current geometry and recent changes at corresponding cells to refine the trajectory. We will cite DiffRefiner and clarify these architectural differences in the revision.

Comparison with CLOVER. CLOVER reports 94.5 PDMS on NAVSIM-v1 and 90.4 EPDMS on NAVSIM-v2 navtest under the revised protocol. RoleDrive achieves 94.9 and 91.5, respectively. We have added CLOVER to both comparison tables.

CLOVER uses evaluator-filtered pseudo-expert trajectories to expand supervision beyond a single logged trajectory, then refines the generator through conservative self-distillation. We particularly appreciate how ranking feedback helps the generator produce better candidates. The approaches are complementary. CLOVER focuses on generating and ranking candidate trajectories. RoleDrive focuses on assigning distinct planning roles to intermediate scene representations and using those roles to guide trajectory refinement. We will cite CLOVER and discuss this relationship in the revision.

AuthorsOpenReview ↗

Response to W1 (Part 2/2): Multi-Seed Ablations, Model Capacity, and Functional Evidence

Multi-seed ablations and controlled comparisons. We repeated two ablations with training seeds 0 and 1 in addition to the original seed 2. All runs used the same data split, 25-epoch training budget, and checkpoint-selection rule based on validation PDMS. The full Progressive model is the common reference. Each gain below is computed between runs with the same seed.

Adding Evolution Tokens (validation PDMS)

SeedContext + MapContext + Map + Evolution (Progressive)Gain
092.59893.023+0.425
192.68093.133+0.453
292.53692.869+0.333
Mean92.60593.008+0.404

Progressive versus Joint interaction with all three token groups (validation PDMS)

SeedJointProgressiveGain
092.45393.023+0.570
192.49293.133+0.641
292.50092.869+0.369
Mean92.48293.008+0.527

Both gains were positive for all three seeds. The paired gains were 0.404 ± 0.063 PDMS points for adding Evolution and 0.527 ± 0.141 for Progressive over Joint (mean ± sample standard deviation across seeds).

VariantTotal parametersTrainable parameters
Full Joint36,853,79113,810,719
Full Progressive (with Evolution)36,853,79113,810,719
Direct Fusion36,517,66313,474,591

Joint and Progressive match in parameter count and decoder depth. Both have five trajectory interaction layers. Joint reads all three token groups in every layer. Progressive reads Context in the first two layers and all three groups in the final three.

Direct Fusion and Evolution each use five trajectory interaction layers and supply 5,120 spatial tokens in addition to 48 Context Tokens. Direct Fusion aggregates aligned historical Maps. Evolution first computes differences between the current and aligned historical Maps. In the original ablation, their validation PDMS scores were 92.4 and 92.9, respectively. Direct Fusion has 336,128 fewer parameters (0.91% of Evolution's total), so this comparison is close in capacity but not exactly matched. We will report measured computational costs under the same conditions. Additional multi-seed ablations are underway, and we will report their per-seed results alongside the two comparisons above in the supplementary material.

Functional evidence for role separation. To test whether the planner uses the information carried by each representation, we froze a model trained on navtrain. We replaced one token group at a time with its position-wise mean computed on navtrain and evaluated the model on navval. When replacing Context, we retained the visual initialization of the trajectory queries. We evaluated all 18,179 validation scenarios and the subset of 8,237 curved-driving scenarios in which the maximum absolute deviation of future ego heading from the current heading exceeds 10°.

Replaced tokensΔPDMS, all scenariosΔPDMS, curved-driving subset
Context−0.98−1.43
Map−32.15−39.27
Evolution−4.41−6.11

Each change is relative to the unmodified model on the same subset. Replacing any group reduced PDMS, with larger drops in the curved-driving subset. The interventions show that the trained planner relies on scene-specific information from each group. The visualizations provide evidence about what the groups attend to, and CKA measures differences in their learned feature structure. Together, these findings support the proposed complementary roles of global context, current local geometry, and recent change. We will add the intervention results to the main ablation section and document the intervention settings and curved-driving analysis in the supplementary material.

AuthorsOpenReview ↗

Response to W2: Configuration Selection and Evaluation Protocols

(1) History-length selection and independence of test evaluation. Thank you for asking us to clarify this choice. We chose K=3 because it retains a longer history window at little additional cost, while both K=2 and K=3 achieve a validation PDMS of 92.9 at the reported precision. With historical Maps already cached, K=3 increased measured inference latency by only 0.04 ms, peak GPU memory by 0.003 GB, and cache size by 2.62 MB relative to K=2. We fixed K=3 before first inspecting the navtest results.

  1. We compared K=0, 1, 2, and 3 on the validation set.
  2. We selected and fixed K=3 based on validation performance and resource use.
  3. For this history-length study, we evaluated only K=3 on navtest. Test performance did not influence the choice.
MetricK=2K=3Change
Validation PDMS92.992.90.0
Parameters (M)36.85436.8540.000
Forward latency with cached historical Maps (ms)78.6978.73+0.04
Peak GPU memory (GB)0.4140.417+0.003
Historical Map cache (MB)5.247.86+2.62

During continuous driving, each historical Map can be reused once encoded. Each planning step still retrieves and aligns the cached Maps. The reported latency includes these operations and excludes the initial encoding of a newly added historical frame. We will revise the appendix statement that “test-set results favor K=3” to describe the validation-based selection procedure and include the comparison above alongside the history-length ablation.

(2) HUGSIM reproduction and updated comparison. We aligned the initialization policy, camera extrinsic conventions, vehicle-state processing, controller settings, simulator corrections, and scoring protocol between RoleDrive and our DrivoR reproduction. Both take control from the first simulation step without V0 warmup. RoleDrive uses three forward-facing cameras, whereas DrivoR uses four cameras, including a rear camera. The other evaluation settings are consistent across our runs.

To handle missing history at startup, we fine-tuned RoleDrive for five epochs using only NAVSIM-v1 training data. In each batch, we randomly assigned samples to four equally sized groups retaining the most recent 0, 1, 2, or 3 historical frames, respectively. Every sample retained the current frame. Validity masks excluded missing observations from temporal differencing and aggregation.

Our earlier DrivoR reproduction obtained 46.4 RC and 31.0 HD-Score. After aligning the evaluation settings, our updated reproduction yielded scores close to the original report. Because several settings changed together, we cannot attribute the earlier discrepancy to a single factor. We therefore cite DrivoR’s published scores of 49.8 RC and 35.7 HD-Score as the reference below.

MethodCamera viewsAverage RCAverage HD-Score
DrivoR (original paper)449.835.7
RoleDrive (updated evaluation)351.636.6
Gain over published DrivoR scores—+1.8+0.9

With one fewer camera, RoleDrive exceeded the published DrivoR scores by 1.8 points in average RC and 0.9 points in average HD-Score. We will update the HUGSIM table and document the shared settings, camera views, missing-history fine-tuning procedure, and score sources in the revised manuscript.

(3) Evaluation protocol for 91.5 EPDMS. We obtained 91.5 EPDMS on single-stage NAVSIM-v2 navtest by averaging the corrected evaluator’s scores over all 12,146 scenarios. Navhard and full two-stage evaluation use different protocols.

As stated in the Table 2 footnote, rescoring the same RoleDrive predictions yielded 87.8 before the evaluator fix and 91.5 after it. We evaluated our DrivoR reproduction under the corresponding pre-fix and corrected protocols. For literature comparisons, we retain the reported settings and identify the evaluator version where known. We will make the single-stage navtest scope explicit in the main text and table caption and retain the evaluator-version details in the table notes.

Reviewer_CTVqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RoleDrive, which organizes scene information into Context, Map, and Evolution Tokens to represent global context, local geometry, and recent changes. Evolution Tokens are constructed from differences between current and aligned historical BEV features. Trajectory queries progressively incorporate these representations to generate candidate trajectories, which are scored to select the final output. The method reports competitive results on NAVSIM-v1/v2 and includes closed-loop evaluation on HUGSIM.

Critical strengths
  1. The representation design is well motivated. Compact tokens summarize global context, BEV features preserve local geometry, and temporal residuals provide cues about recent changes. Aligning historical features before subtraction is a sensible way to reduce differences caused by ego motion.
  2. RoleDrive achieves competitive results on NAVSIM-v1/v2 with approximately 36.9M parameters. The ablations cover the main design choices, and the comparison between direct historical fusion, unaligned residuals, and aligned residuals provides useful evidence for the temporal module.
Critical weaknesses
  1. The results support the method’s effectiveness, though the evidence for distinct planning roles remains limited. CKA and the residual-weighted visualization provide indirect support; clarifying whether Joint and Progressive have matched depth and parameter counts would strengthen this analysis.
  2. The acknowledged differences in HUGSIM initialization and input settings limit attribution of the gains to the architecture. Multiple training seeds would also help confirm the smaller ablation improvements of 0.2–0.4 PDMS.
  3. Reproducibility would benefit from clarification of positional embedding handling during alignment and subtraction, historical Map computation during training, and gradient propagation through the historical branch.
Questions for authors
  1. Section 3.2 adds positional embeddings to the Map features. Are these embeddings included in the subsequent warping and subtraction? Since the same physical location has different coordinates in the historical and current ego frames, how is the difference between its positional embeddings handled?
  2. During training, are historical Maps computed online or retrieved from a cache? Are gradients propagated through the historical branch? If cached features are retained across parameter updates, how are they kept comparable with current features? The appendix specifies precomputed historical Maps for inference timing, but does not clarify the training procedure.
  3. The appendix states that test-set results favor 𝐾=3. Was the history length fixed before inspecting test results, or did test performance influence the choice of the reported configuration?
4 comments
AuthorsOpenReview ↗

Response to W1: Evidence for Role Separation and Matched Model Capacity

The results support the method’s effectiveness, though the evidence for distinct planning roles remains limited. CKA and the residual-weighted visualization provide indirect support; clarifying whether Joint and Progressive have matched depth and parameter counts would strengthen this analysis.

Response.

(1) Evidence for role separation. Thank you for highlighting the need for more direct functional evidence. We have added representational intervention experiments to examine whether the trained planner actually relies on all three token groups. These experiments directly assess their contributions to planning performance, complementing the existing analyses of what they attend to and whether their learned representations differ.

Specifically, we use a model trained exclusively on navtrain and freeze all model parameters. We independently replace Context, Map, or Evolution Tokens with their position-wise training-set means, reducing scene-specific information while keeping the remaining inputs unchanged. All interventions are evaluated on the held-out navval split, which was not used for model training. We evaluate all 18,179 validation scenarios and further analyze a subset of 8,237 curved-driving scenarios, defined by a maximum absolute future ego-heading deviation from the current heading exceeding 10°. Replacement means are computed exclusively from training samples, without using validation data.

Replaced representationAll validation scenarios (N=18,179)Curved-driving validation scenarios (N=8,237)
Context−0.98−1.43
Map−32.15−39.27
Evolution−4.41−6.11

Values are ΔPDMS in points relative to the unmodified model on the same validation subset. Negative values indicate degradation.

Replacing any of the three representations reduces planning performance, demonstrating that the planner uses scene-specific information from all three token groups. The larger drops in curved-driving scenarios further indicate greater sensitivity to these representations under those conditions. The visualizations show attention patterns consistent with the intended roles, CKA reveals differences in the learned feature structures, and the interventions show that each representation contributes to planning performance. Specifically, Context provides global scene context to help the planner understand the overall driving environment. Map represents current local geometry, including the road layout and the spatial relationships among surrounding objects. Evolution provides spatially aligned information about recent changes, supplying temporal cues beyond the current frame. Together, these findings support complementary planning roles centered on global context, current geometry, and recent changes.

We will add the validation-set mean-replacement results to the ablation section of the main paper and include a comparison between the full validation set and the curved-driving subset in the supplementary material. We will also explicitly document the intervention locations, computation of training-set means, and scenario-selection criteria for reproducibility.

(2) Matched interaction depth and model capacity. Joint and Progressive both contain five trajectory interaction layers, organized into two consecutive modules of two and three layers, and have 36.85M parameters. Each layer consists of self-attention, cross-attention, and a feed-forward network. Progressive attends to Context Tokens in the first two layers and jointly attends to Context, Map, and Evolution Tokens in the subsequent three layers. Joint attends to all three representations throughout all five layers. The two configurations therefore have the same interaction depth and parameter count and differ in when each representation enters trajectory reasoning. We will clarify these settings in the ablation section of the main paper by specifying the layer allocation, representations accessed at each stage, and parameter counts for both variants.

AuthorsOpenReview ↗

Response to W2: Updated HUGSIM Results and Multi-Seed Ablations

The acknowledged differences in HUGSIM initialization and input settings limit attribution of the gains to the architecture. Multiple training seeds would also help confirm the smaller ablation improvements of 0.2–0.4 PDMS.

Response.

(1) Updated HUGSIM evaluation. Thank you for pointing out the differences in evaluation settings. We aligned the initialization policy, camera extrinsic conventions, vehicle-state processing, controller settings, simulator corrections, and scoring protocol between RoleDrive and our DrivoR reproduction. Both methods take control at the first simulation step with no V0 warmup. RoleDrive uses three forward-facing cameras, while DrivoR uses four cameras, including a rear camera. The other evaluation settings are consistent across the two runs.

To handle missing history at startup, we fine-tuned RoleDrive for five epochs using only NAVSIM-v1 training data. Within each batch, we randomly assigned samples to four equally sized groups that retained zero, one, two, or three of the most recent historical frames, respectively. Every sample retained the current frame. Validity masks excluded missing historical observations from temporal differencing and aggregation.

Our updated DrivoR reproduction yielded scores close to those reported in the original paper. We therefore use DrivoR’s published scores of 49.8 RC and 35.7 HD-Score as the reference for the comparison below.

MethodCamera viewsAverage RCAverage HD-Score
DrivoR (original paper)449.835.7
RoleDrive (updated evaluation)351.636.6
Gain over published DrivoR scores—+1.8+0.9

With one fewer camera, RoleDrive exceeded the published DrivoR scores by 1.8 points in average RC and 0.9 points in average HD-Score. In the revised manuscript, we will update the HUGSIM comparison table and document the shared evaluation settings, the camera views used by each method, the fine-tuning procedure for missing history, and the source of each score.

(2) Robustness across training seeds. We repeated two ablations with training seeds 0 and 1 in addition to the original seed 2. For each seed, we compared the full Progressive model with the Context + Map configuration to measure the gain from adding Evolution Tokens. We then compared Progressive with Joint to measure the effect of introducing the representations in stages. The Joint and Progressive variants use all three token groups and have identical interaction depth and parameter count. All runs used the same data split, training settings, 25-epoch budget, and checkpoint-selection rule based on validation PDMS.

Effect of adding Evolution Tokens on validation PDMS

Training seedContext + MapContext + Map + Evolution (Progressive)ΔPDMS
092.59893.023+0.425
192.68093.133+0.453
292.53692.869+0.333
Mean92.60593.008+0.404

Effect of Progressive versus Joint interaction on validation PDMS

Training seedJointProgressiveΔPDMS (Progressive − Joint)
092.45393.023+0.570
192.49293.133+0.641
292.50092.869+0.369
Mean92.48293.008+0.527

Both gains were positive for all three training seeds. Adding Evolution Tokens improved validation PDMS by 0.404 ± 0.063 points, and Progressive outperformed Joint by 0.527 ± 0.141 points. Each value gives the mean and sample standard deviation of the three within-seed gains. We will summarize both comparisons in the main ablation section. Additional multi-seed ablations are underway, and we will report their per-seed results alongside these two comparisons in the supplementary material.

AuthorsOpenReview ↗

Response to W3: Reproducibility of Temporal Representation Construction

Reproducibility would benefit from clarification of positional embedding handling during alignment and subtraction, historical Map computation during training, and gradient propagation through the historical branch.

Response.

We add spatial embeddings both before and after temporal differencing, using independently parameterized embeddings at the two stages. During training, we compute historical Maps online on every forward pass using the current model parameters. Historical encoding runs under torch.no_grad(), so gradients do not propagate through the historical branch. The shared encoder receives gradients through the current-frame branch.

(1) Positional embeddings before and after temporal differencing. Before differencing, we add Map positional embeddings to both current and historical Map features and then apply layer normalization. We warp the historical features, including their positional information, into the current ego frame. At each location with valid historical coverage, we subtract the aligned historical feature from the current feature.

Alignment establishes correspondence between physical locations across observations. The features being compared still encode each location’s position relative to the ego vehicle at the time of observation. For example, as the ego vehicle passes a parked car, the car shifts from ahead of the vehicle to its side in the ego frame. This change in relative position can contribute to the residual even though the car remains stationary.

After differencing, we add a separate spatial coordinate embedding to each residual feature to identify its location in the current planning grid. We also add embeddings for relative ego pose, time offset, and frame identity before temporal aggregation.

To assess the effect of including Map positional embeddings in the differencing path, we will compare the default model with a variant that omits these embeddings from the features used to compute the residuals. Both variants will use the same layer normalization, retain the positional embeddings in the current Map supplied directly to the planner, and use the same spatial, pose, time, and frame embeddings after differencing. Given the limited time and computational resources available during the rebuttal period, we will conduct this ablation after the rebuttal and report the results in the supplementary material.

(2) Historical Map computation and gradient propagation. On each training forward pass, the shared visual encoder and geometric lifting module recompute historical Maps from the historical images. This ensures that the historical features reflect the current model parameters after each update. We use precomputed historical Maps only for the reported inference-efficiency measurement.

Computing historical Maps under torch.no_grad() reduces GPU memory usage. Gradients from the planning loss propagate through the temporal aggregation module and stop at the historical Map features. The shared visual encoder and geometric lifting module receive gradients through the current-frame branch. We will document both spatial encoding stages, the training-time computation of historical Maps, and the gradient paths in the revised manuscript.


AuthorsOpenReview ↗

Response to Q1–Q3: Positional Embeddings, Historical Maps, and History-Length Selection

Q1. Positional embeddings during alignment and subtraction. We retain the historical positional encoding during warping. We neither replace it with the current-frame encoding nor explicitly cancel the difference before subtraction. In each frame, we add each grid cell’s positional embedding to its Map feature and apply layer normalization. For a current cell with valid historical coverage, we use the relative ego pose to locate the corresponding coordinate in the historical grid, bilinearly sample the encoded historical features there, and subtract the sampled feature from the encoded current feature.

Warping determines where to sample. The sampled feature still reflects the position assigned to it in the historical ego frame. For example, as the ego vehicle passes a parked car, the car shifts from ahead of the vehicle to its side. Alignment brings the same physical location into correspondence, but the two ego-relative coordinates differ. This difference can contribute to the residual even for a stationary car. Because layer normalization follows the addition of the positional embedding, subtraction operates on the encoded features. We do not separately subtract the positional vectors.

After subtraction, we add an independently parameterized spatial coordinate embedding to identify the residual’s location in the current planning grid. We also add embeddings for relative ego pose, time offset, and frame identity before temporal aggregation.

Q2. Historical Map computation, gradients, and inference timing. During training, the shared visual encoder and geometric lifting module recompute historical Maps from historical images on every forward pass. After a parameter update, both current and historical features are therefore computed with the updated weights. Historical encoding runs under torch.no_grad() to avoid storing computation graphs for multiple historical frames. Gradients stop at the historical Map features. The shared encoder and lifting module receive gradients through the current-frame branch, and the temporal aggregation module remains trainable.

During continuous inference, model weights are fixed. Once the historical branch has encoded an observation, its Map can be cached and reused at later planning steps. The cache stores historical-branch outputs. Current-frame encoding includes trajectory queries, whereas historical encoding does not. An observation newly entering the history window therefore requires one historical encoding unless its Map is already cached. In a sequential-input replay benchmark, the model produced identical outputs with and without caching.

The appendix reports forward latency with historical Maps already available. It includes current-frame encoding, retrieval and alignment of cached Maps to the current ego frame, temporal aggregation, and planning. It excludes the initial computation of historical Maps. In the supplementary material, we will report both cache-ready forward latency and continuous-inference latency including the historical encoding of newly added observations.

Q3. Selection of history length before test evaluation. We fixed K=3 before first inspecting navtest results. We compared K=0, 1, 2, and 3 on the validation set. K=2 and K=3 both achieved 92.9 validation PDMS at the reported precision. We selected K=3 because it retained one additional historical frame at little extra cost. We then evaluated only the fixed K=3 configuration on navtest for this history-length study.

MetricK=2K=3Change
Validation PDMS92.992.90.0
Parameters (M)36.85436.8540.000
Forward latency with historical Maps cached (ms)78.6978.73+0.04
Peak GPU memory (GB)0.4140.417+0.003
Historical Map cache (MB)5.247.86+2.62

As described in Q2, subsequent planning steps reuse previously encoded historical Maps. Under this cache-ready timing protocol, the extra frame increased measured forward latency by 0.04 ms, peak GPU memory by 0.003 GB, and cache size by 2.62 MB.

The appendix statement that “test-set results favor K=3” inaccurately describes the selection procedure. We will revise it to clarify that K=3 was selected on validation performance and resource use and fixed before test evaluation. We will include the table above alongside the history-length ablation in the appendix.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to W1: Distinction from Conventional Fusion and Reliability of the Improvements

(1) How RoleDrive differs from prior scene representation methods. As illustrated in Fig. 1(a) of our paper, representative methods have focused primarily on the form of the scene representation, such as compact latent tokens or a metric BEV grid. Less attention has been paid to the roles that different types of scene information play in trajectory planning and how those roles should guide the construction of representations. RoleDrive’s contribution is to give intermediate scene representations explicit planning roles and to reflect those roles in how each representation is constructed and how the planner uses it.

Context summarizes the visible scene across cameras. Map preserves current geometry on a local grid ahead of the vehicle. Evolution compares the current Map with each ego-motion-aligned historical Map at cells with valid overlap. Fig. 3 illustrates the distinction. When a region first comes into view during a turn, Map already provides its geometry. Where historical coverage exists, Evolution also captures what recently changed at the same cell. Context guides the initial trajectory queries, and all three groups guide refinement.

We froze the model and replaced one group at a time with its position-wise training-set mean. Values are changes in PDMS relative to the unmodified model on the same scenarios.

Replaced groupAll validation scenarios (18,179)Curved-driving subset (8,237)
Context−0.98−1.43
Map−32.15−39.27
Evolution−4.41−6.11

Replacing any group reduced PDMS. Replacing Context preserved the visual initialization of the trajectory queries, so its drop reflects the contribution beyond initialization. These interventions complement the attention visualizations and CKA analysis by testing whether the planner uses scene-specific information from each group.

(2) Controlled comparisons. We tested how temporal information is represented and when it enters trajectory reasoning.

ComparisonVariantValidation PDMSTotal parameters
Temporal representationDirect Fusion92.436,517,663
Temporal representationEvolution92.936,853,791
Interaction scheduleJoint92.536,853,791
Interaction scheduleProgressive92.936,853,791

Direct Fusion aggregates aligned historical Map features. Evolution first computes their differences from the current Map. Both use five trajectory interaction layers and supply 5,120 spatial tokens and 48 Context Tokens. Direct Fusion has 336,128 fewer parameters (0.91% of Evolution’s total). We will add a fusion baseline matched in capacity and auxiliary supervision and report computational costs under the same measurement conditions.

Joint and Progressive have identical parameter counts and five interaction layers. Joint reads all three groups in every layer. Progressive reads Context in the first two and all three groups in the final three, testing the interaction schedule at matched depth and capacity.

(3) Stability across training seeds. We repeated two ablations with seeds 0 and 1 in addition to the original seed 2. All runs used the same data split, 25-epoch budget, and checkpoint selection based on validation PDMS.

ConfigurationSeed 0Seed 1Seed 2Mean ± SD
Context + Map92.59892.68092.53692.605 ± 0.072
Full model, Joint92.45392.49292.50092.482 ± 0.025
Full model, Progressive93.02393.13392.86993.008 ± 0.133

Each gain below is the difference in validation PDMS within the same seed. SD is the sample standard deviation across seeds.

Paired gainSeed 0Seed 1Seed 2Mean gain ± SD
Full Progressive − Context + Map+0.425+0.453+0.333+0.404 ± 0.063
Full Progressive − Full Joint+0.570+0.641+0.369+0.527 ± 0.141

Both gains were positive for all three seeds. Additional multi-seed ablations are underway. We will report their per-seed scores and paired gains alongside these results in the supplementary material and summarize the findings in the main ablation section.

AuthorsOpenReview ↗

Response to W2: HUGSIM Closed-Loop Performance and Evaluation Settings

(1) HUGSIM performance in Extreme scenarios and overall. Thank you for raising this concern. We would like to clarify that Table 3 reports an Extreme route completion (RC) score of 17.8 for UniAD, whereas the review cites 47.8. RoleDrive’s originally reported Extreme RC of 40.8 therefore already exceeded the UniAD reference. In our updated evaluation, RoleDrive’s Extreme RC increases further to 46.7.

MethodAverage RCAverage HD-ScoreExtreme RC
UniAD (published reference)45.932.717.8
RoleDrive (original evaluation)48.034.240.8
RoleDrive (updated evaluation)51.636.646.7
Updated RoleDrive − published UniAD+5.7+3.9+28.9

In the original evaluation, RoleDrive exceeded UniAD by 2.1 points in average RC and 1.5 points in average HD-Score. In the updated evaluation, these margins increase to 5.7 and 3.9 points, respectively. Both comparisons use the published UniAD scores cited in Table 3.

(2) Evaluation settings and comparison with DrivoR. We aligned the initialization policy, camera extrinsic conventions, vehicle-state processing, controller settings, simulator corrections, and scoring protocol between RoleDrive and our DrivoR reproduction. Both methods take control from the first simulation step with no V0 warmup. RoleDrive uses three camera views (front, front-left, and front-right). DrivoR uses these three views plus a rear view. All other evaluation settings are consistent across our runs.

To handle missing history at startup, we fine-tuned RoleDrive for five epochs on NAVSIM-v1, using no additional datasets or HUGSIM training data. Within each training batch, we randomly divided samples into four equally sized groups that retained the most recent 0, 1, 2, or 3 historical frames, respectively. Every sample retained the current frame. Validity masks excluded missing observations from temporal differencing and aggregation.

Our updated DrivoR reproduction yielded scores close to those reported in the original paper. We therefore use DrivoR’s published scores of 49.8 average RC and 35.7 average HD-Score as the benchmark reference below.

MethodCamera viewsAverage RCAverage HD-Score
DrivoR (original paper)449.835.7
RoleDrive (updated evaluation)351.636.6
RoleDrive − published DrivoR—+1.8+0.9

With one fewer camera view, RoleDrive exceeds the published DrivoR reference by 1.8 points in average RC and 0.9 points in average HD-Score. These HUGSIM results measure the closed-loop performance of the complete system, including its adaptation to missing history at startup. The controlled NAVSIM ablations in W1 examine the individual contributions of temporal differencing and progressive interaction. We will update Table 3 and document the shared evaluation settings, camera views, startup adaptation, and source of each score in the revised manuscript.

AuthorsOpenReview ↗

Response to Q1–Q4: Role Separation, Evaluation Settings, and Robustness

Q1. How does role separation differ from conventional fusion under matched conditions? RoleDrive lets the planner read “what is here now” from Map and “what recently changed here” from Evolution at the same grid cell. Evolution computes differences only where the current and aligned historical Maps overlap. When a region first comes into view, Map still provides its current geometry. Context supplies the overall scene information used to guide the initial trajectory queries.

Direct Fusion aggregates aligned historical features. Evolution computes temporal differences before aggregation. Both use five trajectory interaction layers and supply 5,120 spatial tokens plus 48 Context Tokens. Evolution scores 92.9 validation PDMS versus 92.4 for Direct Fusion, with a 0.91% difference in total parameters. Joint and Progressive both have 36,853,791 parameters and five interaction layers. Joint reads all three groups throughout. Progressive reads Context in the first two layers and all three groups in the last three, scoring 92.9 versus 92.5 for Joint. W1(2) gives the full comparison. We will add a historical-fusion baseline matched in token count, model capacity, auxiliary supervision, and planner interaction settings.

Q2. Which HUGSIM settings were matched in the DrivoR comparison? RoleDrive and our DrivoR reproduction both take control from the first simulation step with no V0 warmup. We aligned camera extrinsic conventions, vehicle-state processing, controller settings, simulator corrections, and scoring. RoleDrive uses three views (front, front-left, and front-right). DrivoR also uses a rear view. To handle missing history at startup, we fine-tuned RoleDrive for five epochs using only NAVSIM-v1 data. W2(2) describes the adaptation and evaluation settings.

Our DrivoR reproduction yielded scores close to those in the original paper, so we use DrivoR’s published 49.8 average RC and 35.7 average HD-Score as the reference. With one fewer camera, RoleDrive scores 51.6 and 36.6, respectively (+1.8 and +0.9). This comparison measures closed-loop performance of the complete systems. The architectural controls are reported in W1(2).

Q3. Are the ablation gains consistent across seeds, and how much do they vary? Yes. Both gains were positive for each of seeds 0, 1, and 2. All runs used the same data split, 25-epoch training budget, and checkpoint selection based on validation PDMS.

Paired gain in validation PDMSSeed 0Seed 1Seed 2Mean ± sample SD
Full Progressive − Context + Map+0.425+0.453+0.333+0.404 ± 0.063
Full Progressive − Full Joint+0.570+0.641+0.369+0.527 ± 0.141

The first row measures the gain from adding Evolution Tokens. The second measures the gain from Progressive interaction with all three token groups present. We computed each difference within the same seed, then calculated the sample standard deviation across the three differences. W1(3) reports the individual model scores. Additional multi-seed ablations are underway, and we will include their results in the supplementary material.

Q4. How does RoleDrive compare with UniAD, particularly in Extreme scenarios? We would like to clarify that Table 3 reports an Extreme RC of 17.8 for UniAD, whereas the review cites 47.8. RoleDrive’s original Extreme RC of 40.8 already exceeded that reference. The updated evaluation raises it to 46.7, a margin of 28.9 points over published UniAD.

The original margins over UniAD were 2.1 points in average RC and 1.5 points in average HD-Score. In the updated evaluation, these increase to 5.7 points in average RC (51.6 vs. 45.9) and 3.9 points in average HD-Score (36.6 vs. 32.7). These are comparisons with the published UniAD scores. W2(1) gives the full table, and W2(2) specifies which settings were aligned for our DrivoR reproduction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.