- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes learning end-to-end (E2E) camera driving policies without expert demonstrations by decoupling the problem into two phases:
- Phase 1 (Self-Play): Training a vector-state policy with PPO via multi-agent self-play in a vectorized simulator using DeepSets road/partner encoders, an ego MLP with randomized reward weights, and a discrete action set.
- Phase 2 (Vision Alignment): Freezing the planning head and ego encoder while replacing road and partner encoders with linear adapters on a frozen DINOv3 backbone. These adapters are trained offline on fixed paired (image, scene-state) frames using a forward-KL action-distribution loss and a batch-relational structural loss matching cosine-similarity Gram matrices within the teacher's top- PCA subspace, without supervision from logged trajectories.
Across 436 HUGSIM scenarios, the student achieves 0.548 CL / 0.528 native HD-Scores, outperforming ECO (0.427/0.344) and approaching teacher-level performance (0.562/0.542), supported by extensive ablations and strong teacher results on nuPlan, InterPlan, and WOSAC.
Critical strengths
Important Objective and Efficient Alignment Design: The paper addresses a compelling goal—transferring demonstration-free self-play driving competence to camera policies without expensive online rendering loops—and introduces an economical offline alignment design that reuses frozen planning heads at ~80 A100 hours.
Valuable Empirical Trends and Component Ablations: The provided loss and rank ablations (Tables 2–3) offer meaningful insights into the internal dynamics of the teacher's feature space, demonstrating that the proposed structural loss and spectrum-guided rank choices contribute constructively to the distillation process.
Critical weaknesses
W1. In-environment training exposure on test scenes: The student is trained on 300K paired frames sampled directly from HUGSIM reconstructions of the evaluation split, giving it direct exposure to 3DGS renders of the evaluated test environments; without these frames, performance drops significantly (from 0.636 to 0.574 on nuScenes), indicating the headline gains stem partly from scene exposure rather than pure policy generalization.
W2. Non-standard and author-defined evaluation protocol: The headline Closed-Loop (CL) HD-Score uses an author-defined constructed stopping plan rather than the model's actual predicted trajectory, leading to strong divergence from native scoring (e.g., ECO drops from 0.427 CL to 0.344 native) and making comparisons against official zero-shot baselines like-for-like invalid.
W3. Incomplete manuscript with missing referenced appendices: Crucial sections including Appendices A.3 (rewards), A.4 (action space), A.5 (controller), B (scorers), C (backbone ablations), and D (per-split results) are completely absent from the submission PDF, making it impossible to evaluate or verify the primary claims regarding teacher performance across external benchmarks.
W4. Flawed and under-specified imitation baseline: The imitation baseline in Table 3 (0.206) omits fundamental setup details—such as target mapping, trained parameters, and resource allocation within its 1,616 A100-hour budget—preventing a fair isolation of the self-play paradigm against imitation pre-training.
W5. Opaque simulation environment and agent state setup: Key components of the experimental setup—including the underlying self-play simulator, map sources, initial state distributions, and the exact test-time composition of the ego state vector (goal, route, and reward weights )—are inadequately specified, severely hindering reproducibility.
Questions for authors
Q1: How sensitive is the structural alignment loss to the choice of the spectrum-defined cutoffs (), and what happens to feature geometry if the projection space captures noisy tail directions instead of the dominant singular vectors?
Q2: Since the forward-KL action loss and the batch-relational structural loss operate concurrently, how do they balance or interact when conflicting gradients arise between action-matching and feature-geometry preservation?
Q3: What motivated the choice of keeping the DINOv3 backbone entirely frozen and relying solely on lightweight linear adapters for the road and partner encoders, and did you explore any partial or full fine-tuning of the vision backbone during Phase 2?
We appreciate the reviewer's efforts in reviewing the paper and asking insightful questions. Most answers are already in the appendices available in the original supplementary material download link (a zip file).
W1. Training exposure on test scenes
Please see our response to 1Vj6 W2 for detail.
W2. Author-defined evaluation protocol
We would like to clarify a misunderstanding.
W3. Missing appendices
We would like to clarify that all appendix sections are present in the pdf file available from the official download link for "Supplementary Material" (a zip file). These include
W4. Imitation baseline
We will expand the setup in Appendix. The imitation baseline uses the student's architecture (frozen DINOv3 + road/partner adapters + policy head) and the same 1.83M nuPlan paired frames. The only change is the supervision: each logged nuPlan ego trajectory is mapped to the nearest discrete jerk action via inverse dynamics and trained with cross-entropy, in place of the teacher's distribution. Trainable parameters are the same. Compute: with no self-play phase, the full 1,616 A100-hour budget goes to imitation,
W5. Simulator and ego-state setup
Appx. A specifies the simulator details.
Q1. Sensitivity to k_p = 13, k_r = 9
In terms of retained spectral energy (Fig. 4), the rank-ablation rows correspond to 70%, 80%, 90% and 100% cumulative energy for both encoders. Across 70% to 90% (a 4× range in rank) CL HD-Score varies by at most 0.031 (0.517–0.548), so the structural loss is not sensitive to the exact cutoff. The 80% rule picks the peak, but nearby thresholds work comparably. Including the tail is what hurts: the last 38/46 partner/map directions carry only 7% to 9% of the energy, yet matching them lowers the score by 0.077 (0.533 to 0.456), consistent with the tail being noise the image cannot identify (Section 4.3). We will add the energy column to Table 2.
Q2. Interaction of action KL and structural loss
Their gradients are also complementary rather than competing: across 500 training batches throughout training, the cosine similarity between the action-KL and structural-loss gradients on the trainable adapters averages +0.03 (± 0.05, minimum −0.05), so the structural term adds directions the action loss does not supply without opposing it.
Q3. Frozen DINOv3 backbone
We follow the DINOv3 authors' recommended usage, keeping the backbone frozen and reading its patch features through simple linear adapters, so the closed-loop score reflects what the alignment objective transfers rather than added backbone capacity. We did not explore partial or full fine-tuning of the backbone; because the pipeline is end-to-end differentiable, it is complementary and can be trained with the same objective. This is left for future work.