acceptodds
Withdrawn from ICLR 2027

Ego-Centric Joint Supervision for Planning-Oriented Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

End-to-end (E2E) planners predict the future trajectories of the ego and the surrounding agents, yet recent methods supervise each agent marginally against its own trajectory. However, scenes in our training data and real-world driving data often require the planner to consider ego–agent interaction, which a marginal objective does not supervise explicitly. To address this, we propose CastAD, which trains a planning-oriented E2E planner under ego-centric joint supervision. Unlike conventional joint forecasting objectives, which treat all actors symmetrically, this supervision centers the scene on the ego, whose future is a decision in planning. Only the ego and its interaction-relevant agents, those with any predicted future that overlaps the spatial trajectory the ego followed, are supervised as one scene under the mode closest to the ego's spatial trajectory. A policy optimization stage then refines the stochastic ego policy on the joint-mode ego embeddings, which this supervision trains to encode ego–agent interaction, with rewards scored against map context and logged agent futures. CastAD reaches state-of-the-art results on closed-loop Bench2Drive (Driving Score 87.53, Success Rate 71.82) and on NAVSIM (PDMS 91.1). Video demos are available at https://castad-anon.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
end-to-end autonomous driving, joint prediction, planning, closed-loop driving, reinforcement learning, motion forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Js9iOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CastAD, an ego-centric joint supervision framework for planning-oriented end-to-end autonomous driving. Built on HiP-AD, the method adds a joint prediction branch, uses the ego’s ground-truth spatial trajectory to select interaction-relevant agents, and selects a shared joint mode based on the ego spatial trajectory. Under this mode, the ego and the relevant agents are jointly supervised. A GRPO-based policy optimization stage is then applied on the joint-mode ego embeddings, with rewards based on a PDMS decomposition. Experiments are conducted on Bench2Drive and NAVSIM, reporting closed-loop metrics, driving-ability breakdowns, and ablations.

Critical strengths
  1. The motivation is reasonable: marginal supervision can produce interaction-inconsistent futures, while all-actor joint supervision can be dominated by irrelevant agents.

  2. The method is relatively complete: Agent-Mode Attention, ego-centric mode selection, interaction-relevant agent selection, and GRPO form a full training pipeline.

  3. The experiments are fairly comprehensive: both Bench2Drive closed-loop and NAVSIM are reported, with multi-ability breakdowns and ablations. The ablation suggests that most of the gain comes from ego-centric joint supervision rather than GRPO.

Critical weaknesses
  1. Limited novelty / incremental contribution. Joint supervision is not new in motion forecasting; it has been studied in Scene Transformer, M2I, JointMotion, RetroMotion, etc. CastAD essentially changes all-actor winner-takes-all to ego-only mode selection and uses the GT spatial trajectory for collision-based agent filtering. This is close to a simple variant of conditional joint prediction or ego-as-influencer modeling.

  2. GRPO is not a new method. GRPO was introduced in DeepSeekMath/R1, and RL or preference optimization has already been used in autonomous driving, e.g., DiffusionDriveV2. This paper mainly applies GRPO to joint-mode ego embeddings, which is an implementation-level change.

  3. The SOTA claim is outdated. For example, on Bench2Drive, there are numerous works that achieved far better results (as reference https://github.com/autonomousvision/Bench2Drive-Leaderboard).

  4. Key details and inference mechanism are unclear. The Ego-Centric Joint Prediction Head is not used for planning, yet policy optimization is performed on joint-mode embeddings. The masked trajectory distance, collision criterion, and GRPO mode probability definition are not clearly specified.

Questions for authors
  1. Limited novelty / incremental contribution. Joint supervision is not new in motion forecasting; it has been studied in Scene Transformer, M2I, JointMotion, RetroMotion, etc. CastAD essentially changes all-actor winner-takes-all to ego-only mode selection and uses the GT spatial trajectory for collision-based agent filtering. This is close to a simple variant of conditional joint prediction or ego-as-influencer modeling.

  2. GRPO is not a new method. GRPO was introduced in DeepSeekMath/R1, and RL or preference optimization has already been used in autonomous driving, e.g., DiffusionDriveV2. This paper mainly applies GRPO to joint-mode ego embeddings, which is an implementation-level change.

  3. The SOTA claim is outdated. For example, on Bench2Drive, there are numerous works that achieved far better results (as reference https://github.com/autonomousvision/Bench2Drive-Leaderboard).

  4. Key details and inference mechanism are unclear. The Ego-Centric Joint Prediction Head is not used for planning, yet policy optimization is performed on joint-mode embeddings. The masked trajectory distance, collision criterion, and GRPO mode probability definition are not clearly specified.

Reviewer_Hi2vOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper is concerned with end-to-end autonomous driving, where a learned network predicts ego and other agent trajectories. The novelty is a training objective that in addition to marginals, also trains for an ego-plan oriented mode of the joint ego and relevant-agent trajectory distribution. The method is applied to a specific base architecture, and is shown to improve over this base model consistently in established closed-loop training benchmarks, and achieve near-SOTA performance in general.

Critical strengths
  1. The problem is well motivated and relevant
  2. The approach makes sense and the technical implementation seems sound
  3. The method is evaluated in established closed-loop driving benchmarks, and it improves over the base model
  4. The paper is generally well written and easy to follow
Critical weaknesses
  1. The SOTA claim on Bench2Drive and NAVSIM are not substantiated. The reported tables are missing a number of entries that in fact have reported higher scores on the same benchmark. Non-exhaustive list of examples for Bench2Drive: AlignDrive (Wu et al., 2026) reports DS 89.07; SparseDriveV2 (Sun et al., 2026) reports DS 89.15; and for NAVSIM: DriveSuprim (Yao et al., 2026), DrivoR (Kirby et al., 2026), SafeDrive (Kim et al., 2026) all report higher scores. It would be important to apply the method to some of the higher scoring base models to claim SOTA.
  2. The improvement over the base method HiP-AD is demonstrated, but given the small difference and small number of underlying scenes, it would be important to include variance and statistical significance.
  3. It would be useful to more explicitly separate the contribution of ideas wrt joint prediction methods (e.g. diffusion models), and isolate the contribution of ego-based mode assignment and selective agent supervision.
  4. The works DTPP (Huang et al. 2024) and RAP (Tang et al 2026) are highly relevant yet not cited. I believe they are not exactly the same method but they may reduce the novelty, especially DTPP. The difference should be discussed and potentially compared experimentally if possible.

A strong rebuttal would be needed to raise my score to weak accept.

Questions for authors
  • The path-based relevant agent selection rule misses on some cases of ego-agent interactions that do not result in crossing paths. What would be a possible way to include such cases?
Reviewer_4cAEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CastAD, an end-to-end autonomous driving framework based on ego-centric joint supervision. The method selects interaction-relevant agents whose predicted futures overlap the logged ego spatial path and supervises them under a shared mode selected by ego trajectory error. Agent-Mode Attention exchanges information between marginal and joint-mode embeddings. A subsequent policy optimization stage refines the ego policy using rewards computed against map context and logged agent futures. CastAD reports a Driving Score of 87.53 and a Success Rate of 71.82% on Bench2Drive, and a PDMS of 91.1 on NAVSIM v1. The results suggest practical benefits, but the interpretation of interaction modeling and the attribution of gains require further evidence.

Critical strengths

The paper addresses an important limitation of actor-wise supervision: individually plausible ego and agent predictions may form an inconsistent joint future. Organizing supervision around the ego maneuver is a relevant idea for planning, and the distinction between all-actor forecasting objectives and ego-centered planning objectives is clearly motivated.The evaluation includes closed-loop Bench2Drive experiments and planning evaluation on real-world NAVSIM logs. Improvements in merging and yielding are consistent with the intended focus on interactive driving. The comparisons of spatial and temporal selection cues, together with the policy optimization ablations, provide useful initial evidence for the proposed framework.

Critical weaknesses

The connection between the supervision rule and reciprocal interaction modeling remains only partially established. Spatial overlap may include agents that traverse the same region at unrelated times, while a single logged joint future directly constrains only the selected mode. Moreover, policy optimization evaluates alternative ego rollouts against fixed logged agent futures. These choices leave uncertainty about whether the learned modes capture plausible agent responses to different ego behaviors. The sources of the performance gains are not sufficiently disentangled. Relative to the marginal baseline, the method introduces additional mode embeddings, attention computation, and auxiliary supervision. The comparison with all-actor joint supervision also changes both the mode-selection rule and the set of agents sharing that mode. More controlled ablations are needed to quantify the independent contributions of joint supervision, ego-based mode selection, and agent filtering.

Questions for authors

1.Using spatial trajectory overlap alone to identify future interactions may be insufficient, as interaction relevance generally depends on both spatial and temporal overlap. Could this criterion identify too many agents as interaction-relevant and consequently lead to overly conservative ego behavior?

2.Modeling interactive behavior should account for the mutual influence between the ego vehicle and surrounding agents. However, agent behavior remains fixed during policy optimization in this work, you should explain the rationale for this setting and discuss its limitations.

3.Each training scene contains only one observed ego–agent joint future, and supervision primarily constrains the mode selected by the ego trajectory. How are agent responses in the remaining modes effectively constrained?

4.You should provide additional ablation studies to quantify the individual contributions of key design choices, including joint supervision, best-mode selection, and interaction-relevant agent selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.