acceptodds
Under review as a conference paper at ICLR 2027

Cartender: Verifiable Language-Conditioned Driving by Learning the Drives Not Taken

Abstract

Every driving scene is logged exactly once, yet driving is inherently one-to-many. The scene decides which behaviors are admissible and the driver's intent which one to take: vision alone cannot choose among them, and language alone cannot tell what the scene permits. But the log keeps only the drive that was taken, and its instruction merely describes it. Language-conditioned driving policies are reported to follow instructions, yet on such data this cannot be established: a policy that ignores the instruction and one that obeys it regardless of the scene fit the logs equally well, and neither is distinguishable from one that weighs both. Trained only on such logs, driving vision–language–action models (VLAs) and world–action models (WAMs) receive no direct supervision on actions that violate their instructions or instructions the scene rules out. We present Cartender, which learns from the drives not taken. To construct matched and mismatched instruction–action pairs from real recordings without a simulator, we propose Counterfactual Branching, which treats each logged trajectory as one branch among several candidate futures: a vision–language model proposes the others and the instruction that selects each, and instructions the scene cannot support become negative labels. Cartender then weighs both explicitly: a learned scene–language compatibility regulates language conditioning during trajectory generation, and a reward-trained language residual re-ranks the candidates over a scene-based score. Branches serve training; instruction following is evaluated by changing instructions under identical recorded inputs, using behavior criteria rather than generated trajectory targets. On WOD-E2E, the best of four proposals reaches an RFS of 8.642 across 3,603 feasible instruction conditions. Across all 8,974 admitted proposals, mean RFS is 8.203 versus 7.906 for their matched logged trajectories, and 67% score higher than their logged counterpart. Relative to matched-only imitation with the same backbone, Cartender more than doubles the rate of fulfilling both instructions of a same-scene pair (57.00% vs. 23.67%) while RFS rises from 6.32 to 7.10; its gate cuts execution of infeasible instructions from 53.36% to 14.18%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Vision-Language-Action Models, Language-Guided Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_t49YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Cartender, a language-conditioned autonomous driving framework that addresses the limitations of learning from single demonstrated trajectories. The authors introduce Counterfactual Branching to generate alternative instruction–trajectory pairs from real driving logs without relying on simulation. The proposed framework also incorporates scene–language compatibility gating and reward-based trajectory selection to improve instruction following while avoiding infeasible requests. Experiments on WOD-E2E demonstrate improvements in instruction-following performance and trajectory quality, supported by several ablation studies.

Critical strengths
  1. I think the paper addresses an important limitation of existing language-conditioned driving approaches. The observation that matching recorded trajectories does not necessarily demonstrate instruction following is well motivated. Evaluating different instructions under the same driving scene is a meaningful way to investigate whether a model actually responds to language.

  2. The experimental evaluation is reasonably convincing. I particularly appreciate the comparisons between matched-only imitation, same-scene imitation, and the full framework, which help distinguish the benefits of counterfactual supervision from those of the proposed model components. The improvement in maneuver-paired success and the ablation studies on compatibility gating and preference learning provide useful evidence for the effectiveness of the approach.

Critical weaknesses

The alternative trajectories are generated using a VLM and validated through rule-based checks, which cannot fully establish their feasibility in complex driving environments. Although the authors provide encouraging RFS results and acknowledge this limitation, the experiments remain primarily based on offline driving data. It is therefore unclear how well the approach would perform in interactive driving scenarios, particularly when other traffic participants respond to the ego vehicle's alternative actions.

The quality of the figures needs to be improved. Without zooming in, I found them a bit difficult to read due to the small font size.

Questions for authors
  1. How sensitive is the performance of Cartender to the quality of the generated counterfactual trajectories? In particular, have the authors investigated whether lower-quality but rule-admitted trajectories negatively affect instruction following or driving performance?

  2. The compatibility gating mechanism substantially reduces the execution of infeasible instructions, but the paper also reports a decrease in feasible-request following in one evaluation. Could the authors further explain this trade-off and whether the gating mechanism can be adjusted to improve feasible-request fulfillment without significantly compromising safety?

2 comments
AuthorsOpenReview ↗

Response to W1 and W2: Counterfactual feasibility, interaction, and figure readability

We thank the reviewer for recognizing our same-scene evaluation and learning controls. Our audit supports the quality of the actual training supervision: 600 of 660 teachers pass the specified instruction and geometric checks, yielding a joint pass rate of 90.91%. Cartender also achieves the highest final EPDMS of 51.46 among the evaluated adapted systems in NavHard's two-stage pseudo-closed-loop evaluation with reactive traffic. We further discuss teacher-quality sensitivity and gate trade-offs, and provide visual examples of navigation and instruction responses.

W1. Counterfactual feasibility and interaction.

Our audit supports the quality of the actual training supervision: 600 of 660 teachers pass the specified instruction and geometric checks, yielding a joint pass rate of 90.91%. Cartender also achieves the highest final EPDMS of 51.46 among the evaluated adapted systems on NavHard with reactive traffic.

We randomly sampled 660 A/B trajectories actually used for training: 594 A and 66 B teachers, spanning 493 temporal anchors across 293 episodes.

Teacher auditResult
Instruction criterion passed629/660; 95.30%
Road-edge intersection detected15/660; 2.27%
Sampled logged-actor footprint overlap detected16/660; 2.42%
Joint pass: instruction criterion and both geometric checks600/660; 90.91%
Mean RFS, A / B8.202 / 8.360

Road-edge checks test assumed ego footprints against Waymo Perception v1.4.3 validation road-edge polylines. They detect boundary intersections rather than establish full drivable-area containment. Actor-overlap checks use time- and coordinate-aligned logged boxes at approximately 10 Hz. These checks use recorded actor motion and do not simulate reactions to the alternative ego trajectory.

A/B RFS is recomputed with the official implementation and saved human-rated references at the corresponding anchors, providing a separate trajectory-quality measure. All audited failures remain counted, including unattainable fixed speed reductions at near-zero initial speeds and cases with insufficient starting motion. We will add the teacher-specific audit to Table 2 and document the random sampling procedure and evaluation criteria in Appendix C.3.

For driving with traffic responses, we use NavHard's official two-stage pseudo-closed-loop protocol, including reactive traffic and perturbed follow-up observations. All methods output four-second trajectories and are evaluated on the same 5,912 observations organized into 225 groups. Cartender achieves the highest Stage 1 EPDMS of 84.17 as well as the highest final EPDMS in this comparison.

MethodStage 1 EPDMS ↑Weighted Stage 2 EPDMS ↑Final EPDMS ↑
AutoVLA74.0462.4346.20
AutoMoT51.8955.0928.59
SimLingo76.0266.4450.51
Cartender84.1761.0751.46

The three external baselines start from their officially released pretrained weights. All four methods use the same 92,853 training examples and 10,435 development examples from navtrain, with comparable adaptation budgets and a common evaluation protocol.

AutoMoT uses rigidly transformed logged LiDAR scans as approximate inputs for synthetic observations; SimLingo uses the command-navigation interface. We will document Cartender's and the baselines' adaptations and input approximations in Appendix C.5.

The teacher audit evaluates supervision against the recorded scene, while NavHard evaluates the adapted policy under traffic responses. The fixed-observation tests separately assess instruction control. We will specify the NavHard protocol in Section 4.1 and report both evaluation stages in Section 4.5.

W2. Figure readability.

Thank you for pointing this out. We have enlarged the text in Figures 1–3 and reorganized the technical workflow diagram in Figure 3 to improve readability and make the method flow easier to follow.

AuthorsOpenReview ↗

Response to Q1 and Q2: Teacher-quality sensitivity and the compatibility-gating trade-off

Q1. Sensitivity to teacher quality.

Cartender preserves over 90% of the proxy-passing control’s performance on both maneuver and qualifier paired success in our controlled continuation experiment. With 8.19% of A/B teachers failing the action proxy, maneuver and qualifier paired success reach 52.00% and 32.43%, respectively, corresponding to decreases of 5.12 and 2.50 percentage points relative to the control.

Both models continue from the same Cartender checkpoint, using 342 rule-admitted A/B teachers per group. We hold the scene–instruction pairs fixed and replace 28 teacher trajectories in the mixed group with rule-admitted alternatives that fail the action proxy. The remaining targets, training budgets, and evaluation settings are unchanged. The proxy checks whether a teacher trajectory satisfies the criterion for its requested action.

Teacher trajectories supervise generator flow matching and provide distance-based soft targets for base-ranker imitation. GRPO then updates only the language residual, with the generator and base ranker frozen.

Training teachersAction-proxy failuresManeuver paired success ↑Qualifier paired success ↑RFS ↑Coverage ↑
Proxy-passing0%57.12%34.93%7.1097.72%
Mixed8.19%52.00%32.43%6.57792.55%

The proxy-passing control also achieves higher RFS and coverage, supporting action-consistency checks as a useful complement to rule-based admission.

We will add this comparison to Section 4.5 and document the teacher-group construction, trainable parameters and update counts for each stage, and action-proxy criterion in Appendix C.3.

Q2. Compatibility-gating trade-off.

Our original gate balances feasible-request fulfillment with strong suppression of infeasible execution: it achieves 43.80% feasible success and 14.20% infeasible execution. Scaling the gate to 0.5g raises feasible success to 57.35%, but also increases infeasible execution to 26.87%. This comparison clarifies the trade-off behind our original setting, which prioritizes suppressing scene-incompatible execution.

All three settings use the same evaluation scenes and feasible-success criterion, requiring both instruction completion and passing the collision checks. The constant gate is set to 0.409559, the training-set mean of the learned gate. For the g versus 0.5g comparison, the checkpoint, inputs, and initial sampling noise are fixed, with candidates regenerated. We apply g' = 0.5g in both generation and ranking, halving the generation language coefficient and reducing the ranking language-residual coefficient to one quarter of its original value.

Gate settingFeasible success ↑Infeasible execution ↓Coverage ↑
Constant gate, 0.40955951.00%27.30%—
Original learned gate g43.80%14.20%89.14%
Scaled learned gate 0.5g57.35%26.87%86.15%

Coverage was not reported for the constant-gate control.

Relative to g, scaling to 0.5g increases feasible success by 13.55 percentage points and infeasible execution by 12.67 points, while coverage decreases by 2.99 points.

At comparable rates of infeasible execution, our scaled learned gate achieves higher feasible success than constant gating: 57.35% versus 51.00%, with infeasible execution of 26.87% and 27.30%, respectively. These results characterize the joint effect of changing language weighting in generation and ranking; they do not isolate the contribution of either stage.

We will add this comparison to Section 4.6 and specify the gate transformations in generation and ranking, candidate regeneration, and the shared evaluation criteria in Appendix D.2.

Reviewer_o2G1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a framework for augmenting real driving logs with multimode behavior. The augmentation is done via a vision language model queried to produce candidate trajectories. These are subsequentially verified leveraging heuristic rules. Open-loop results for a trained language-conditioned trajectory planner show gains (e.g., on WOD-E2E).

Critical strengths

[S1] Motivation: The paper analyzes a known but important limitation of conventional matched-pair evaluation, where low trajectory prediction error does not necessarily indicate that the model follows language instructions.

[S2] Counterfactual supervision: The paper studies a simple approach to constructing alternative instruction + trajectory pairs from the same recorded scene, however, these alternatives may be unsafe or unfeasible in the current method.

Critical weaknesses

[W1] Limited Novelty: The differentiation with established trajectory generation / scoring is limited. Other methods (e.g., AutoVLA, AlphaDrive) also explore GRPO-based fine-tuning, such that the components leveraged are not particularly novel. Some components are also unclear within the proposed framework, as mentioned next.

[W2] Inconsistent Results: Some of the components in the framework have unclear contribution, and the results are not well studied / explained. For instance, the exact expected reward optimization slightly outperforms GRPO, e.g., on maneuver paired success (58.17% vs. 57.00%), yet performs substantially worse on qualifier paired success (4.86% vs. 35.07%). Although both methods use the same frozen candidate set and reward, their optimization objectives differ. The large performance discrepancy is unexplained.

[W3] Insufficient Validation: Applicability to real-world driving remains concerning, particularly around the counterfactual branching and the lack of collision avoidance or drivable-area verification. It remains unclear whether the alternatives are genuinely feasible tracks. Although the authors evaluate generated trajectories using RFS, this metric does not verify instruction correctness or collision safety. The reported RFS statistics also aggregate different proposal types rather than specifically evaluating the A/B trajectories used for training.

[W3.1] Closed-loop: Limited closed-loop results, and comparisons with existing methods such as SimLingo, which reports closed-loop performance in CARLA.

[W3.2] Limited improvement: The proposed method does not demonstrate a clear improvement over SimLingo in Table 1b. Maneuver success is within the reported standard deviation of SimLingo, while trajectory shift is worse. Without further statistical analysis, it is difficult to determine whether the differences are meaningful. Why was Cartender not included in Table 1a?

[W3.3] Insufficient baselines: The paper compares Cartender with a limited set of existing methods, while other VLA models, such as AutoMoT, AutoVLA, and AlphaDrive, could provide additional comparisons (where their evaluation settings are compatible). [W3.4] Missing inference analysis? The proposed method generates 24 candidate trajectories followed by constraint filtering and learned ranking, but does not report end-to-end inference latency, throughput, or memory consumption.

Minor Weaknesses:

[MiW1] Poor presentation: The paper’s writing and presentation could be improved, e.g., some font sizes in figures are small.

[MiW2] Compatibility gating tradeoff: The compatibility gate substantially reduces infeasible-instruction following from 53.4% to 14.2%, but also decreases feasible-instruction following from 61.6% to 43.8% (Table 9). This tradeoff raises concerns about whether the gate is overly conservative and suppresses valid instructions.

[MiW3] Unclear inference sensor configuration: Counterfactual Branching uses front-camera observations, while the driving policy relies on frozen U1 scene features. Can the authors clarify the inference time sensor configuration?

Questions for authors
  • Explain inconsistent results
  • Provide sufficient validation
  • Why was the method not tested in closed-loop
  • Address missing baselines
  • Clarify inference configuration
5 comments
AuthorsOpenReview ↗

Response to W1 and W2: Novelty and the GRPO–Exact discrepancy

Thank you for the careful review. Our core contribution is to construct and learn from same-scene alternative instruction–behavior supervision under the partial observations available in real driving recordings. To address your concerns about the supervision’s value and validity, we report same-backbone learning controls and an audit of the A/B training teachers. We also explain the GRPO–Exact discrepancy, clarify driving evaluation and inference, and provide visual examples of navigation and instruction responses.

W1. Limited novelty.

Cartender studies how to construct and learn from instruction supervision beyond the behavior demonstrated in real driving logs. A logged trajectory supervises the executed behavior; it provides no direct target for a request that requires a different behavior. Yet real logs do not reveal the outcomes of unexecuted instructions or how other actors would respond. Our goal is therefore to learn policies that use instructions to select among the behaviors a scene permits, and verify this control beyond matching the recorded trajectory.

Counterfactual Branching uses partial observations from real recordings and the recorded future route to construct route-relative instruction–trajectory alternatives without a complete scene-state oracle or simulator rollouts. Admitted A/B alternatives provide behavior supervision, while applicability labels train compatibility to regulate language conditioning. These are constructed training targets, not observed counterfactual ground truth. W3 reports the checks on their validity.

AutoVLA already generates trajectories and uses GRPO, including experiments on real driving data; AlphaDrive uses GRPO for meta-action planning. Our contribution is the learning approach and its empirical demonstration; we use GRPO and standard trajectory-generation components as established tools within it. SimLingo’s Action Dreaming constructs same-scene alternatives through offline simulation. Cartender focuses on constructing and learning from alternative supervision in real recordings under these observational limits, and measuring its contribution to instruction control.

With the backbone, imitation procedure, and optimization budget fixed, adding same-scene supervision raises maneuver paired success from 23.67% to 39.50%. Cartender reaches 57.00% on the same expanded training set. These controls show gains from the added supervision and from the full learning procedure. Evaluation holds observations fixed and requires both feasible instructions in a pair to pass behavioral criteria, without using generated teachers as targets. We will clarify this contribution and its scope in the Introduction and Section 3.1, and the controls in Section 4.5.

W2. Explaining the GRPO–Exact discrepancy.

Our additional control identifies within-group reward standardization as a major contributor to GRPO’s stronger qualifier control. Although the candidates and raw rewards are shared, standardization changes how each scene–instruction group contributes to training. We test its effect by applying it to Exact while retaining exact expected-reward optimization.

Raw Exact and Standardized Exact differ only in within-group reward standardization; candidates, raw rewards, initialization, and the remaining training settings are held fixed.

Qualifier paired success (%), averaged over three seeds:

MethodSpeedTimingOverall
Raw Exact6.253.135.21
Standardized Exact37.5018.7531.25
GRPO42.1921.8835.42

Overall weights speed and timing by 2/3 and 1/3, respectively. Standardization raises Exact from 5.21% to 31.25%, reducing its gap to GRPO's 35.42% from 30.21 to 4.17 percentage points. This closes 86.2% of the qualifier gap while preserving exact optimization.

With fixed candidate rewards and group statistics, subtracting the group mean contributes a constant to the exact reward expectation and leaves its gradient unchanged. Dividing by the group standard deviation rescales each group's reward gradient relative to other groups and the unchanged regularization terms. Thus, preserving within-group reward ordering does not preserve its influence on optimization.

Maneuver and qualifier success also measure different requirements: qualifiers test the requested speed magnitude and timing within a maneuver, so similar maneuver-level success need not imply similar qualifier control. The control shows that much of the qualifier discrepancy can be reduced by changing reward scaling alone. We will clarify this optimization effect in Section 4.5 and document the control's experimental settings in Appendix D.5.

AuthorsOpenReview ↗

Response to W3, W3.1, and W3.3: Teacher validation, driving evaluation, and additional baselines

W3. Validation of the training teachers.

The audit directly supports the validity of our A/B training supervision under the specified instruction and geometric checks. Among 660 randomly sampled trajectories actually used for training, 629 (95.30%) satisfy the instruction criteria, and 600 (90.91%) also pass both geometric checks. These results assess the actual training teachers separately from the pooled proposal-level RFS statistics. The sample contains 594 A and 66 B teachers, spanning 493 temporal anchors across 293 episodes. Kinematic and comfort constraints were already enforced during teacher admission.

Audit itemResult
Instruction criteria passed629/660 (95.30%)
Road-edge intersection detected15/660 (2.27%)
Sampled overlap with logged actor footprints detected16/660 (2.42%)
Joint pass: instruction criteria and both geometric checks600/660 (90.91%)

Instruction-criterion pass rates are 570/594 (95.96%) for A and 59/66 (89.39%) for B. All audited failures remain in the denominator, including longitudinal cases with unattainable fixed speed reductions at near-zero initial speeds or insufficient starting motion.

Road-edge checks test assumed ego footprints against Waymo Perception v1.4.3 validation road-edge polylines, detecting boundary intersections rather than verifying full drivable-area containment. Actor-overlap checks use time- and coordinate-aligned logged boxes at approximately 10 Hz; they assess overlap with recorded actor motion, without simulating responses to the alternative trajectory.

Teacher-specific mean RFS is 8.202 for A and 8.360 for B, recomputed with the official implementation and corresponding human-rated references. These scores provide a separate measure of trajectory quality alongside the instruction and geometric checks.

The audited alternatives vary progress, speed profiles, and lateral offsets relative to the recorded future route. We will add the teacher-specific audit summary to Table 2 and make the random sampling procedure and references to the existing evaluation criteria explicit in Appendix C.3.

W3.1 & W3.3. Driving evaluation and additional baselines.

With shared adaptation data and comparable training budgets, Cartender achieves the highest Stage 1 EPDMS of 84.17 and final EPDMS of 51.46 among the evaluated adapted systems on NavHard. This extends our evaluation beyond fixed observations to reactive traffic and perturbed follow-up observations. We use the official two-stage pseudo-closed-loop protocol on 5,912 observations organized into 225 groups, with four-second trajectory outputs.

MethodStage 1 EPDMS ↑Weighted Stage 2 EPDMS ↑Final EPDMS ↑
AutoVLA74.0462.4346.20
AutoMoT51.8955.0928.59
SimLingo76.0266.4450.51
Cartender84.1761.0751.46

Cartender exceeds SimLingo by 8.15 points in Stage 1 and 0.95 points in final EPDMS. These results add evidence of driving performance under reactive traffic; the fixed-observation tests remain our direct evaluation of instruction control.

The three baselines start from their officially released pretrained weights. All four methods use the same 92,853 training examples and 10,435 development examples from navtrain, with comparable adaptation budgets.

AutoVLA retains its native action decoding and is adapted with LoRA; AutoMoT and SimLingo retain their original prediction heads. All three baselines use the provided NavHard RGB observations without additional image rendering. AutoMoT uses logged point clouds for original observations and rigidly transformed scans as approximate inputs for synthetic observations; its image-history interval is adjusted from approximately 0.25 s to 0.5 s. SimLingo uses the command-navigation interface in place of its default route-target-point interface.

We also provide qualitative examples, separating standard navigation cases from added-command diagnostics. We will clarify the evaluation protocol in Section 4.1, report both stages in Section 4.5, and document Cartender's adaptation modules and training supervision, together with model-specific inputs and input approximations, in Appendix C.5.

AuthorsOpenReview ↗

Response to W3.2 and W3.4: SimLingo comparison, Table 1 diagnostics, and inference efficiency

W3.2. Comparison with SimLingo and Table 1 diagnostics.

On the same scenes and instructions, Cartender follows the requested direction of change more often than SimLingo, achieving 83.09% direction consistency versus 62.50%, a gain of 20.59 percentage points. Reported maneuver success is numerically close: 73.83% versus 72.80%.

Metric (%)SimLingoCartenderDifference (pp)
Maneuver success ↑72.8073.83 ± 2.601.03
Direction consistency ↑62.5083.09 ± 7.4020.59

The ± values are Cartender's sample standard deviations across three selector-training seeds; they are not confidence intervals for the between-model difference.

Trajectory shift measures displacement between predictions before and after an instruction swap, not error against a reference trajectory. Cartender's larger shift (1.666 m versus 0.89 m) indicates a greater response magnitude. The higher direction consistency shows that its responses more often align with the requested direction. We therefore interpret shift together with direction consistency and maneuver fulfillment.

Table 1a originally focused on released policies to illustrate that low reference-trajectory error does not establish instruction following. We now report Cartender's corresponding removal, shuffling, and swapping diagnostics:

ModelRef. ADE (m) ↓Removal shift (m)Shuffle ΔL_actionSwap paired success (%) ↑
Qwen-Drive1.4130.064—21.13
SimLingo0.8422.511+0.232 / +1.28950.00
LMDrive0.2030.178+0.0113.20
Cartender (added)0.7921.825+0.11757.00

Table 1a reports within-model interventions under each model's evaluation protocol; the direct same-scene comparison with SimLingo is in Table 1b. Shuffle losses use native objectives, so their magnitudes are not directly comparable across models. SimLingo's two values correspond to route and speed-waypoint losses, respectively.

For Cartender, removing instructions changes predictions, and shuffling increases the action loss, supporting instruction dependence. Its 57.00% paired success additionally tests whether both requests in a pair satisfy their behavioral and hard-constraint criteria. This complements the individual-request success in Table 1b by checking whether the policy fulfills both alternative instructions under the same observation.

We will complete Cartender's diagnostics in Table 1a and Section 4.2, specify its shuffled-loss definition in Appendix C.1, and clarify the metric definitions and seed-variation annotations in Table 1, Section 4.3, and Appendix C.2.

W3.4. Inference efficiency.

Cartender's full 24-candidate pipeline achieves 8.19 requests/s on one NVIDIA L20X at batch size 1, with a mean latency of 118.39 ms per request. We profiled 100 WOD-E2E validation scenes using eight sampling steps.

Timing begins with raw image bytes and ego states already resident in CPU memory and includes image decoding and preprocessing, U1 encoding, candidate generation, constraint filtering, and learned ranking. Throughput is measured over the complete 100-request run.

Mean latencyP95 latencyThroughputPeak memory
118.39 ms122.89 ms8.19 requests/s4.25 GiB

Filtering and ranking the candidate set averaged 2.03 ms per returned trajectory in separate profiling. We will add these profiling results and document the timing boundaries and throughput calculation in Appendix B.6.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to W1/Q1 and W2/Q2: Teacher validation and selector comparison

We thank the reviewer for recognizing our instruction-control diagnostics and learning controls. Our audit supports the quality of the actual A/B training teachers: among 660 randomly sampled teachers, 95.30% meet the instruction-completion criteria and 90.91% jointly pass the instruction and specified geometric checks. On the same frozen candidates and hard mask, the full learned selector improves instruction control and selection speed over direct reward ranking, gaining 8.02 and 11.53 percentage points in maneuver and qualifier paired success, respectively, and reducing median selection latency from 10.95 ms to 1.42 ms. We also provide visual examples of navigation and instruction responses.

W1 / Q1. Validation of A/B training teachers.

Our audit provides direct evidence supporting the quality of the A/B supervision actually used to train Cartender. Among 660 randomly sampled training teachers, 629 satisfy the instruction-completion criteria, yielding 95.30%, and 600 jointly pass the instruction and specified geometric checks, yielding 90.91%. These teacher-specific results establish high instruction consistency and a high joint pass rate within the audited sample, separately from the pooled A–D proposal statistics.

The sample spans 493 temporal anchors across 293 episodes. Instruction completion is 95.96% for A teachers, with 570 of 594 passing, and 89.39% for B teachers, with 59 of 66 passing. The breakdown by instruction category is:

Instruction categoryPassed / AuditedCompletion ↑
Speed change, stop, wait, start286/31092.26%
Lane keeping207/21098.57%
Lane change13/1586.67%
Turning20/2290.91%
Following72/72100%
Obstacle avoidance17/17100%
Yielding14/14100%
Overall629/66095.30%

Road-edge checks compare assumed ego footprints with Waymo Perception v1.4.3 validation road-edge polylines, with a pass rate of 97.73%. They assess boundary intersections rather than full drivable-area containment. Actor-overlap checks use time- and coordinate-aligned logged boxes at approximately 10 Hz, with a pass rate of 97.58%. All audited failures remain in the reported denominators.

As a separate reference-based trajectory-quality measure, teacher-specific RFS is 8.202 for A and 8.360 for B, recomputed with the official implementation and saved human-rated references at the corresponding anchors.

We will add the teacher-specific results and category breakdowns to Table 2, and document the random sampling, instruction-completion criteria, and geometric and RFS evaluation protocols in Appendix C.3.

W2 / Q2. Learned selection versus direct reward ranking.

Cartender’s full learned selector achieves higher instruction-following success and faster selection than direct reward ranking on the same frozen candidates and hard mask. Maneuver and qualifier paired success increase by 8.02 and 11.53 percentage points, respectively, with higher RFS at identical coverage. Median selection latency falls from 10.95 ms to 1.42 ms, a 7.7× speedup at the selection stage.

MethodManeuver paired success ↑Qualifier paired success ↑RFS ↑Coverage ↑Selection-stage P50 ↓
Direct reward ranking48.98%23.54%6.96197.70%10.95 ms
Cartender selector57.00%35.07%7.1097.70%1.42 ms

Direct reward ranking selects arg⁡max⁡i∈Ar(τi,ℓ)\arg\max_{i\in\mathcal{A}} r(\tau_i,\ell), where A\mathcal{A} contains the candidate indices retained by the hard mask and the fixed reward measures trajectory–instruction agreement. Cartender’s score in Eq. (3) combines a scene-aware base with a compatibility-gated, reward-trained language residual. It uses scene context and the instruction to rank candidates without evaluating the reward at inference.

The Exact comparison concerns expected-reward optimization during training; this experiment directly tests trajectory selection at inference.

We will report this comparison in Section 4.5 and document the direct-reward baseline, shared-candidate setup, and selection-time measurement protocol in Appendix D.5.

AuthorsOpenReview ↗

To all reviewers and ACs: Clarification of the Core Contribution and Additional Evidence

We would like to clarify the central insight of Cartender and how the paper demonstrates it.

1. Core insight: driving policies can learn effectively from unexecuted trajectories and instructions in real-world scenes.

A real log records the behavior that was executed, but unexecuted instructions and alternative behaviors can also provide useful supervision. Cartender demonstrates how to construct this supervision from partial real observations and use it for scene-grounded language–action learning. It couples scene–instruction compatibility with language–action reward learning: feasible alternatives teach instruction-dependent behavior, while unsupported requests teach when to limit language’s influence. Our contribution is this learning approach and its empirical demonstration; we do not claim contribution in GRPO itself or in standard trajectory-generation components.

2. The key challenge and novelty is constructing useful alternative supervision for real scenes without the privileged access to complete scene state that simulators provide.

Simulators such as CARLA provide direct access to ground-truth scene state, including road geometry and surrounding actors’ poses and velocities, for constructing instruction–action alternatives. Our setting instead requires constructing supervision from partial real sensor observations and recorded ego motion, without an equivalent scene-state oracle or simulator rollouts. This is not merely a sim-to-real gap: the supervision itself must first be constructed without complete knowledge of the surrounding world.

We construct alternative trajectories paired with free-form language instructions, together with scene–instruction applicability labels, from real recordings and the logged ego future. Teacher audits and same-backbone comparisons then test both the quality of this supervision and its effectiveness in improving instruction control.

3. The experiments demonstrate both supervision quality and learning benefits.

Same-backbone maneuver paired success rises from 23.67% with matched-only imitation to 39.50% with same-scene imitation and 57.00% with Cartender. In an audit of 660 randomly sampled A/B teachers actually used for training, 95.30% satisfy instruction-completion criteria, and 90.91% jointly pass the instruction and specified geometric checks. Additional experiments cover teacher-quality sensitivity, selector and optimization controls, gate trade-offs, NavHard’s two-stage pseudo-closed-loop evaluation, and inference efficiency. We have improved figure readability and will clarify the contribution and experimental protocols in the manuscript.

Together, these results provide a concrete demonstration of learning beyond the recorded drive. Visual examples of navigation and instruction responses are available on our anonymous project page.

Thank you for the detailed rebuttal and additional experiments. The authors have addressed my main concerns and strengthened the evaluation. I appreciate their efforts and will maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.