- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Counterfactual-Agent, a planning method for closed-loop driving on Bench2Drive. A small VLM with Qwen2-0.5B backbone generates seven ego trajectories, one for each predefined speed behavior. For each trajectory, the model predicts how nearby vehicles would respond using an interaction graph and a response predictor. A hand-designed cost then scores each predicted future and the car follows the lowest-cost trajectory. The backbone is finally adapted with LoRA so that the selected trajectory matches the expert path. The method reaches a driving score of 87.47 and a success rate of 72.73%, which results in a 9.43% improvement over MindDrive.
Critical strengths
- This paper reaches strong closed-loop results with only a 0.5B language backbone. It does particularly well on merging, overtaking, giving way and traffic signs, where reacting to other vehicles matters most.
- This paper tackles a practical question in interactive driving, which is how other cars would respond if the ego car acted differently. Predicting a separate future for each candidate is an intuitive way to answer it and also makes it easy to see why the final trajectory was chosen.
Critical weaknesses
- It is not clear that the improvement comes from modeling how other vehicles respond. Without the interaction graph the Driving Score drops by only 1.2%, which could easily be run-to-run variance. Most of the improvement over MindDrive is already there before this part is added but the paper does not say how its base model differs from MindDrive. It would help to see the same seven candidates scored without response prediction. Some evidence that the predicted responses actually change from one candidate to another would also make the claim more convincing.
- Some strong prior work is missing from the comparison. SimLingo, HiP-AD and BridgeDrive all score between 85% and 87% on Bench2Drive but are not included in Table 1. Compared with them, the reported lead shrinks from 9.4% to less than 1%. Planning that accounts for how other vehicles react to the ego car has also been studied extensively in autonomous driving. Discussing this line of work would help justify the contribution.
Questions for authors
- Could you describe the base model and training of the ablation models and how they differ from MindDrive?
- Could you report results for the same seven candidates scored with ego-only cost terms or with forecasts that ignore the ego candidate?
- Could you show how much the predicted trajectories of other vehicles change across candidates and how often this changes the final choice?
- Could you report results over several runs and add the missing Bench2Drive baselines to Table 1?
4 comments
Response to Reviewer 3 (part 2)
Q1: Base model and training of ablation variants
Thank you. Our framework combines a camera-based scene encoder providing visual and map-related features, a Qwen2-0.5B language backbone and an ego-trajectory planner. These pretrained base components are initialized from checkpoints provided with the MindDrive code repository. The added response module integrates agent-history and ego-candidate encoders, interaction-graph aggregation (IA), and parallel trajectory and meta-action (speed/path) heads. A fixed risk scorer evaluates candidate-specific futures and selects the ego trajectory.
Two-stage training. Stage I learns surrounding-agent response prediction through trajectory and meta-action supervision. The trajectory and meta-action heads operate in parallel; meta-action predictions are not sequential inputs to trajectory decoding. Stage II adapts the backbone through LoRA, using an imitation loss between the risk-selected ego trajectory and the recorded expert trajectory. The closed-loop Full results use the final checkpoint after both stages. During closed-loop inference, generated ego candidates condition surrounding-agent response prediction.
Ablation construction. Unified and Full without IA use the final two-stage Full checkpoint without retraining. Unified reuses one surrounding-agent forecast across candidates; Full without IA disables interaction analysis during inference. Candidate generation, scoring terms and weights, controller and evaluation protocol remain fixed. The original baseline retains its pretrained checkpoint and original prediction/planning pipeline.
Independent prediction diagnostic. Tables 1 report a separate response-only experiment. The pretrained scene encoder, language backbone and ego planner are frozen; newly initialized feature projections and the response module are trained using cached visual/map features, agent histories and recorded ego futures. Recorded surrounding-agent futures provide trajectory supervision and derived speed/path labels.
This experiment uses 203,810 retained frames from 950 scenes, trained for 10 epochs with AdamW, a learning rate of 0.001 and a global batch size of 16:
SmoothL1 is averaged over valid future coordinates; speed/path classification uses complete future trajectories. Recorded ego futures are supplied during both training and validation. No additional ego-planner, backbone or LoRA update occurs in this diagnostic.
Q2: Ego-only costs and candidate-independent forecasts
Thank you. We report the candidate-independent forecast control and clarify its distinction from ego-only scoring.
Candidate-independent forecasts. As reported under **Reviewer 1 **, Unified reuses one surrounding-agent forecast across the seven ego candidates. Ego-trajectory-conditioned instead supplies candidate-specific forecasts, while retaining the same checkpoint, candidate generator, scoring terms and weights, controller and evaluation protocol.
DS increases from 73.76 to 87.47 (+13.71 points), SR from 47.27% to 72.73% (+25.46 percentage points), and mean ability from 44.48% to 82.00%. Efficiency and comfort decrease from 148.07% to 139.15% and 12.96% to 10.51%, respectively. These results support the planning utility of conditioned forecasts within the fixed planner, with an efficiency/comfort trade-off.
Ego-only costs and component analysis. An ego-only control would exclude costs involving surrounding-agent forecasts. Reviewer 3, W1 (Table 11) illustrates the full scorer's trade-off: compared with Unified Candidate 0, Candidate 6 has a higher progress penalty (8.090 versus 0.000), but zero collision and interaction penalties and a lower TTC-related penalty (3.396 versus 7.942). Its recorded total falls from 45.935 to 30.395, the lowest among the candidates, slightly below Candidate 4 (30.515).
Response to Reviewer 3 (part 3)
Q3: Response differences across candidates and decision impact
Thank you. We provide the visualization through an external link: Figure 4: Ego candidates, predicted responses and risk profiles.
Response variation. In panel (c), the surrounding-vehicle trace on the left varies in longitudinal extent, particularly in the third and fifth candidate columns, while the right-side vehicle remains comparatively stable. This suggests agent-specific sensitivity of predicted responses to ego candidates. The accompanying risk profiles illustrate evaluation of the resulting joint futures.
Decision example. Panels (a)–(b) illustrate risk-based selection: candidate 5 is selected with a score of 8.30, compared with 18.43 for candidate 6. These are planning scores, not trajectory-displacement measurements.
Alternative Candidate Selection Rate. On one evaluated route, non-original ego candidates were selected in 7 of 126 saved decision snapshots (5.56%), including stopping, deceleration and lateral adjustments. Such flexibility is particularly relevant to long-tail, safety-critical interactions, where an ill-timed maneuver or failure to yield can cause a collision and route failure.
Q4: Multiple runs and missing Bench2Drive baselines
Thank you. We address both requests with a three-seed evaluation and an expanded baseline comparison.
Results over multiple runs. A full 220-route evaluation takes over 20 hours on eight RTX 5090 GPUs; limited GPU availability during the 13-day rebuttal period therefore restricted repeated evaluations to a fixed subset of 22 routes. We appreciate your understanding of these computational constraints. As discussed in Reviewer 1, Q3, we evaluated both forecast variants using seeds 0, 1, and 2 on the same fixed subset of 22 routes selected for broad scenario coverage. We reproduce the results below for convenience.
| Method | DS seed 0 | DS seed 1 | DS seed 2 | Mean DS | DS SD | Mean SR (%) | SR SD (pp) |
|---|---|---|---|---|---|---|---|
| Unified | 50.08 | 75.06 | 65.03 | 63.39 | 12.57 | 68.18 | 12.03 |
| Ego-trajectory-conditioned | 61.58 | 68.53 | 65.33 | 65.14 | 3.48 | 74.24 | 5.25 |
Ego-trajectory-conditioned achieves higher mean DS/SR (65.14/74.24% versus 63.39/68.18%) with lower observed variability across the three seeds on this fixed subset.
Missing Bench2Drive baselines. The expanded comparison in our response to Reviewer 3, W2 (Table 15) includes SimLingo, HiP-AD, and BridgeDrive, grouped into Non-VLA and VLA methods, with explicit source versions and author-reported results. To avoid duplication, we refer to that table here. Our method leads the listed VLA methods in DS (87.47), SR (72.73%), and mean ability (82.00%), while SimLingo leads in Emergency Brake. Non-VLA BridgeDrive v4 (PDM-Lite) achieves higher DS/SR (87.99/74.99%). VLA methods support language-conditioned planning, whereas the listed Non-VLA methods lack native free-form instruction interfaces. We have incorporated these baselines, group-wise best-value highlighting, and updated discussion into the revised manuscript’s Table 1.
Baseline summary and outlook. Beyond benchmark scores, BridgeDrive highlights a related design principle: it refines data-derived behavioral anchors through a diffusion bridge, while our method uses discrete speed meta-actions to generate ego candidates and explicitly models their surrounding-agent responses for risk-based selection. Both exploit behavioral structure through different mechanisms. Building on this connection, we plan to explore meta-action-conditioned diffusion to refine candidate trajectories and improve the diversity and resolution of speed and passage timing beyond the current seven hypotheses.
Thank you for your detailed rebuttal. My concerns have been addressed and will raise the score to 3.
Response to Reviewer 3 (part 1)
W1: Contribution of response modeling and interaction graph
The ablation in Reviewer 1, Table 2 separates response modeling from interaction analysis (IA).
Response modeling. With IA enabled, both variants retain identical model weights, candidate generation, scoring terms/weights, controller and protocol. Ego-trajectory-conditioned improves DS from 73.76 to 87.47 (+13.71), SR from 47.27% to 72.73% (+25.46 percentage points), and mean ability from 44.48% to 82.00% over Unified. This supports conditioned forecasts within the fixed planner.
IA. Adding IA with conditioned forecasts raises DS from 86.26 to 87.47 and SR from 70.45% to 72.73%, with the largest category gains in Merging (+5.00 percentage points) and Traffic Signs (+7.37). The IA increment has no established statistical significance. The 22-route, three-seed comparison (Reviewer 1, Table 6) compares forecast variants and does not isolate IA.
Paper Figure 3 illustrates the process; Table 11 reports its costs.
Table 11. Logged component costs and scorer totals.
Relative to Unified, Candidate 6 lowers the recorded total cost 45.935→30.395. Collision and interaction penalties both fall to zero (from 1.579 and 5.406); TTC-related, comfort and Rule penalties fall 7.942→3.396, 1.258→0.716 and 24.032→3.968, respectively, despite the higher progress penalty (8.090 vs 0.000). Candidate 6 is selected for its minimum total, just below Candidate 4 (30.515), reflecting combined-cost selection.
W2: Missing baselines and interaction-aware planning literature
Table 12 adds SimLingo, HiP-AD and BridgeDrive.
Table 12. Expanded benchmark comparison.
Ours leads listed VLAs in DS/SR and mean ability. Non-VLA BridgeDrive v4 exceeds our DS/SR by 0.52/2.26 points; its LEAD-trained results use separate data. VLA enables language-conditioned planning, whereas listed Non-VLAs lack native free-form instruction interfaces.
Shared principle and contribution. Both methods use behavioral information to guide trajectory generation: BridgeDrive refines data-derived anchors through a diffusion bridge; ours conditions ego candidates on discrete speed meta-actions. This supports behavior-guided planning. Our framework additionally predicts surrounding-agent responses for each candidate and selects through explicit interaction-risk evaluation. Inspired by BridgeDrive, future work will explore meta-action-guided diffusion to refine ego candidates and expand speed/timing coverage beyond our seven discrete hypotheses.
PiP [5] conditions forecasts on ego plans; DIPP [6] integrates prediction with differentiable planning and learned costs. We model ego–agent meta-action dependencies to supplement sparse trajectory information and use an interaction graph to analyze surrounding-agent interactions and responses to each ego candidate.
[1] SimLingo.
[2] HiP-AD.
[3] BridgeDrive v1.
[4] BridgeDrive v4.
[5] PiP.
[6] DIPP.