- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Cartender, a language-conditioned autonomous driving framework that addresses the limitations of learning from single demonstrated trajectories. The authors introduce Counterfactual Branching to generate alternative instruction–trajectory pairs from real driving logs without relying on simulation. The proposed framework also incorporates scene–language compatibility gating and reward-based trajectory selection to improve instruction following while avoiding infeasible requests. Experiments on WOD-E2E demonstrate improvements in instruction-following performance and trajectory quality, supported by several ablation studies.
Critical strengths
-
I think the paper addresses an important limitation of existing language-conditioned driving approaches. The observation that matching recorded trajectories does not necessarily demonstrate instruction following is well motivated. Evaluating different instructions under the same driving scene is a meaningful way to investigate whether a model actually responds to language.
-
The experimental evaluation is reasonably convincing. I particularly appreciate the comparisons between matched-only imitation, same-scene imitation, and the full framework, which help distinguish the benefits of counterfactual supervision from those of the proposed model components. The improvement in maneuver-paired success and the ablation studies on compatibility gating and preference learning provide useful evidence for the effectiveness of the approach.
Critical weaknesses
The alternative trajectories are generated using a VLM and validated through rule-based checks, which cannot fully establish their feasibility in complex driving environments. Although the authors provide encouraging RFS results and acknowledge this limitation, the experiments remain primarily based on offline driving data. It is therefore unclear how well the approach would perform in interactive driving scenarios, particularly when other traffic participants respond to the ego vehicle's alternative actions.
The quality of the figures needs to be improved. Without zooming in, I found them a bit difficult to read due to the small font size.
Questions for authors
-
How sensitive is the performance of Cartender to the quality of the generated counterfactual trajectories? In particular, have the authors investigated whether lower-quality but rule-admitted trajectories negatively affect instruction following or driving performance?
-
The compatibility gating mechanism substantially reduces the execution of infeasible instructions, but the paper also reports a decrease in feasible-request following in one evaluation. Could the authors further explain this trade-off and whether the gating mechanism can be adjusted to improve feasible-request fulfillment without significantly compromising safety?
2 comments
Response to Q1 and Q2: Teacher-quality sensitivity and the compatibility-gating trade-off
Q1. Sensitivity to teacher quality.
Cartender preserves over 90% of the proxy-passing control’s performance on both maneuver and qualifier paired success in our controlled continuation experiment. With 8.19% of A/B teachers failing the action proxy, maneuver and qualifier paired success reach 52.00% and 32.43%, respectively, corresponding to decreases of 5.12 and 2.50 percentage points relative to the control.
Both models continue from the same Cartender checkpoint, using 342 rule-admitted A/B teachers per group. We hold the scene–instruction pairs fixed and replace 28 teacher trajectories in the mixed group with rule-admitted alternatives that fail the action proxy. The remaining targets, training budgets, and evaluation settings are unchanged. The proxy checks whether a teacher trajectory satisfies the criterion for its requested action.
Teacher trajectories supervise generator flow matching and provide distance-based soft targets for base-ranker imitation. GRPO then updates only the language residual, with the generator and base ranker frozen.
| Training teachers | Action-proxy failures | Maneuver paired success ↑ | Qualifier paired success ↑ | RFS ↑ | Coverage ↑ |
|---|---|---|---|---|---|
| Proxy-passing | 0% | 57.12% | 34.93% | 7.10 | 97.72% |
| Mixed | 8.19% | 52.00% | 32.43% | 6.577 | 92.55% |
The proxy-passing control also achieves higher RFS and coverage, supporting action-consistency checks as a useful complement to rule-based admission.
We will add this comparison to Section 4.5 and document the teacher-group construction, trainable parameters and update counts for each stage, and action-proxy criterion in Appendix C.3.
Q2. Compatibility-gating trade-off.
Our original gate balances feasible-request fulfillment with strong suppression of infeasible execution: it achieves 43.80% feasible success and 14.20% infeasible execution. Scaling the gate to 0.5g raises feasible success to 57.35%, but also increases infeasible execution to 26.87%. This comparison clarifies the trade-off behind our original setting, which prioritizes suppressing scene-incompatible execution.
All three settings use the same evaluation scenes and feasible-success criterion, requiring both instruction completion and passing the collision checks. The constant gate is set to 0.409559, the training-set mean of the learned gate. For the g versus 0.5g comparison, the checkpoint, inputs, and initial sampling noise are fixed, with candidates regenerated. We apply g' = 0.5g in both generation and ranking, halving the generation language coefficient and reducing the ranking language-residual coefficient to one quarter of its original value.
| Gate setting | Feasible success ↑ | Infeasible execution ↓ | Coverage ↑ |
|---|---|---|---|
| Constant gate, 0.409559 | 51.00% | 27.30% | — |
| Original learned gate g | 43.80% | 14.20% | 89.14% |
| Scaled learned gate 0.5g | 57.35% | 26.87% | 86.15% |
Coverage was not reported for the constant-gate control.
Relative to g, scaling to 0.5g increases feasible success by 13.55 percentage points and infeasible execution by 12.67 points, while coverage decreases by 2.99 points.
At comparable rates of infeasible execution, our scaled learned gate achieves higher feasible success than constant gating: 57.35% versus 51.00%, with infeasible execution of 26.87% and 27.30%, respectively. These results characterize the joint effect of changing language weighting in generation and ranking; they do not isolate the contribution of either stage.
We will add this comparison to Section 4.6 and specify the gate transformations in generation and ranking, candidate regeneration, and the shared evaluation criteria in Appendix D.2.
Response to W1 and W2: Counterfactual feasibility, interaction, and figure readability
We thank the reviewer for recognizing our same-scene evaluation and learning controls. Our audit supports the quality of the actual training supervision: 600 of 660 teachers pass the specified instruction and geometric checks, yielding a joint pass rate of 90.91%. Cartender also achieves the highest final EPDMS of 51.46 among the evaluated adapted systems in NavHard's two-stage pseudo-closed-loop evaluation with reactive traffic. We further discuss teacher-quality sensitivity and gate trade-offs, and provide visual examples of navigation and instruction responses.
W1. Counterfactual feasibility and interaction.
Our audit supports the quality of the actual training supervision: 600 of 660 teachers pass the specified instruction and geometric checks, yielding a joint pass rate of 90.91%. Cartender also achieves the highest final EPDMS of 51.46 among the evaluated adapted systems on NavHard with reactive traffic.
We randomly sampled 660 A/B trajectories actually used for training: 594 A and 66 B teachers, spanning 493 temporal anchors across 293 episodes.
Road-edge checks test assumed ego footprints against Waymo Perception v1.4.3 validation road-edge polylines. They detect boundary intersections rather than establish full drivable-area containment. Actor-overlap checks use time- and coordinate-aligned logged boxes at approximately 10 Hz. These checks use recorded actor motion and do not simulate reactions to the alternative ego trajectory.
A/B RFS is recomputed with the official implementation and saved human-rated references at the corresponding anchors, providing a separate trajectory-quality measure. All audited failures remain counted, including unattainable fixed speed reductions at near-zero initial speeds and cases with insufficient starting motion. We will add the teacher-specific audit to Table 2 and document the random sampling procedure and evaluation criteria in Appendix C.3.
For driving with traffic responses, we use NavHard's official two-stage pseudo-closed-loop protocol, including reactive traffic and perturbed follow-up observations. All methods output four-second trajectories and are evaluated on the same 5,912 observations organized into 225 groups. Cartender achieves the highest Stage 1 EPDMS of 84.17 as well as the highest final EPDMS in this comparison.
The three external baselines start from their officially released pretrained weights. All four methods use the same 92,853 training examples and 10,435 development examples from navtrain, with comparable adaptation budgets and a common evaluation protocol.
AutoMoT uses rigidly transformed logged LiDAR scans as approximate inputs for synthetic observations; SimLingo uses the command-navigation interface. We will document Cartender's and the baselines' adaptations and input approximations in Appendix C.5.
The teacher audit evaluates supervision against the recorded scene, while NavHard evaluates the adapted policy under traffic responses. The fixed-observation tests separately assess instruction control. We will specify the NavHard protocol in Section 4.1 and report both evaluation stages in Section 4.5.
W2. Figure readability.
Thank you for pointing this out. We have enlarged the text in Figures 1–3 and reorganized the technical workflow diagram in Figure 3 to improve readability and make the method flow easier to follow.