acceptodds
Under review as a conference paper at ICLR 2027

Learning Interactive Counterfactual Futures for Risk-Aware Embodied Planning

Abstract

Many existing embodied planning methods learn direct mappings from observations and instructions to actions through imitation of recorded demonstrations. Such mappings do not explicitly account for surrounding agents' responses to alternative ego actions, potentially leaving interaction risks unanticipated. We propose a framework that learns interactive counterfactual futures for risk-aware embodied planning. Given multimodal observations and language instructions, a vision-language backbone generates candidate ego trajectories. For each candidate, a learned interaction graph encodes inter-agent relationships, and a meta-action-guided response predictor models surrounding agents' speed and path behaviors before decoding them into future trajectories. The resulting candidate-specific future scenes enable explicit risk assessment and trajectory selection. Training proceeds in two stages: task module pretraining first learns interaction relationships from recorded trajectories; LoRA adaptation then aligns perceptual representations with the planning objective by supervising the risk-aware ego trajectory against the recorded ego future. On Bench2Drive, our method with a Qwen2-0.5B language backbone achieves a Driving Score of 87.47 and a Success Rate of 72.73%, exceeding the strongest compared baseline by 9.43 points and 17.64 percentage points, respectively. The full framework also improves aggregate planning scores over the variant without interaction analysis, with modest reductions in efficiency and comfort.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Safety-critical planning, Risk aware, Interaction analysis, Counterfactual reasoning, Embodied navigation problem

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vVUZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Counterfactual-Agent, a planning method for closed-loop driving on Bench2Drive. A small VLM with Qwen2-0.5B backbone generates seven ego trajectories, one for each predefined speed behavior. For each trajectory, the model predicts how nearby vehicles would respond using an interaction graph and a response predictor. A hand-designed cost then scores each predicted future and the car follows the lowest-cost trajectory. The backbone is finally adapted with LoRA so that the selected trajectory matches the expert path. The method reaches a driving score of 87.47 and a success rate of 72.73%, which results in a 9.43% improvement over MindDrive.

Critical strengths
  1. This paper reaches strong closed-loop results with only a 0.5B language backbone. It does particularly well on merging, overtaking, giving way and traffic signs, where reacting to other vehicles matters most.
  2. This paper tackles a practical question in interactive driving, which is how other cars would respond if the ego car acted differently. Predicting a separate future for each candidate is an intuitive way to answer it and also makes it easy to see why the final trajectory was chosen.
Critical weaknesses
  1. It is not clear that the improvement comes from modeling how other vehicles respond. Without the interaction graph the Driving Score drops by only 1.2%, which could easily be run-to-run variance. Most of the improvement over MindDrive is already there before this part is added but the paper does not say how its base model differs from MindDrive. It would help to see the same seven candidates scored without response prediction. Some evidence that the predicted responses actually change from one candidate to another would also make the claim more convincing.
  2. Some strong prior work is missing from the comparison. SimLingo, HiP-AD and BridgeDrive all score between 85% and 87% on Bench2Drive but are not included in Table 1. Compared with them, the reported lead shrinks from 9.4% to less than 1%. Planning that accounts for how other vehicles react to the ego car has also been studied extensively in autonomous driving. Discussing this line of work would help justify the contribution.
Questions for authors
  1. Could you describe the base model and training of the ablation models and how they differ from MindDrive?
  2. Could you report results for the same seven candidates scored with ego-only cost terms or with forecasts that ignore the ego candidate?
  3. Could you show how much the predicted trajectories of other vehicles change across candidates and how often this changes the final choice?
  4. Could you report results over several runs and add the missing Bench2Drive baselines to Table 1?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (part 1)

W1: Contribution of response modeling and interaction graph

The ablation in Reviewer 1, Table 2 separates response modeling from interaction analysis (IA).

Response modeling. With IA enabled, both variants retain identical model weights, candidate generation, scoring terms/weights, controller and protocol. Ego-trajectory-conditioned improves DS from 73.76 to 87.47 (+13.71), SR from 47.27% to 72.73% (+25.46 percentage points), and mean ability from 44.48% to 82.00% over Unified. This supports conditioned forecasts within the fixed planner.

IA. Adding IA with conditioned forecasts raises DS from 86.26 to 87.47 and SR from 70.45% to 72.73%, with the largest category gains in Merging (+5.00 percentage points) and Traffic Signs (+7.37). The IA increment has no established statistical significance. The 22-route, three-seed comparison (Reviewer 1, Table 6) compares forecast variants and does not isolate IA.

Paper Figure 3 illustrates the process; Table 11 reports its costs.

Table 11. Logged component costs and scorer totals.

Prediction methodCandidateCol.Prog.TTCDmn/Inter.Com.RuleRecorded total
Unified prediction01.5790.0007.9425.4061.25824.03245.935
Candidate-conditioned11.7860.6017.8905.6221.22923.31548.354
20.0000.4605.7450.7631.23023.74731.184
30.0000.0006.2571.5581.27325.41732.523
40.0002.4394.4580.0001.04918.94730.515
51.5080.3977.6065.3101.19321.32043.420
6 (selected)0.0008.0903.3960.0000.7163.96830.395

Relative to Unified, Candidate 6 lowers the recorded total cost 45.935→30.395. Collision and interaction penalties both fall to zero (from 1.579 and 5.406); TTC-related, comfort and Rule penalties fall 7.942→3.396, 1.258→0.716 and 24.032→3.968, respectively, despite the higher progress penalty (8.090 vs 0.000). Candidate 6 is selected for its minimum total, just below Candidate 4 (30.515), reflecting combined-cost selection.

W2: Missing baselines and interaction-aware planning literature

Table 12 adds SimLingo, HiP-AD and BridgeDrive.

Table 12. Expanded benchmark comparison.

MethodLLMRef.Avg. L2 ↓DS ↑SR (%) ↑MOEBGWTSMean
Non-VLA methods
HiP-AD—[2]0.6986.7769.0950.0084.4483.3340.0072.1065.98
BridgeDrive (v1)—Liu et al., 2025 [3]—86.8772.2763.5058.8988.3450.0088.9569.93
BridgeDrive (v4, PDM-Lite)—Liu et al., 2026 [4]—87.9974.9969.9266.6790.0050.0089.4773.15
VLA methods
SimLingoQwen2-0.5B-Instruct[1]—85.0767.2754.0157.0488.3353.3382.4567.03
MindDriveQwen2-0.5BFu et al., 2026—78.0455.0932.8975.5668.3350.0057.8956.94
OursQwen2-0.5BThis work1.0987.4772.7361.2597.7878.3380.0092.6382.00

Ours leads listed VLAs in DS/SR and mean ability. Non-VLA BridgeDrive v4 exceeds our DS/SR by 0.52/2.26 points; its LEAD-trained results use separate data. VLA enables language-conditioned planning, whereas listed Non-VLAs lack native free-form instruction interfaces.

Shared principle and contribution. Both methods use behavioral information to guide trajectory generation: BridgeDrive refines data-derived anchors through a diffusion bridge; ours conditions ego candidates on discrete speed meta-actions. This supports behavior-guided planning. Our framework additionally predicts surrounding-agent responses for each candidate and selects through explicit interaction-risk evaluation. Inspired by BridgeDrive, future work will explore meta-action-guided diffusion to refine ego candidates and expand speed/timing coverage beyond our seven discrete hypotheses.

PiP [5] conditions forecasts on ego plans; DIPP [6] integrates prediction with differentiable planning and learned costs. We model ego–agent meta-action dependencies to supplement sparse trajectory information and use an interaction graph to analyze surrounding-agent interactions and responses to each ego candidate.

[1] SimLingo.
[2] HiP-AD.
[3] BridgeDrive v1.
[4] BridgeDrive v4.
[5] PiP.
[6] DIPP.

AuthorsOpenReview ↗

Response to Reviewer 3 (part 2)

Q1: Base model and training of ablation variants

Thank you. Our framework combines a camera-based scene encoder providing visual and map-related features, a Qwen2-0.5B language backbone and an ego-trajectory planner. These pretrained base components are initialized from checkpoints provided with the MindDrive code repository. The added response module integrates agent-history and ego-candidate encoders, interaction-graph aggregation (IA), and parallel trajectory and meta-action (speed/path) heads. A fixed risk scorer evaluates candidate-specific futures and selects the ego trajectory.

Two-stage training. Stage I learns surrounding-agent response prediction through trajectory and meta-action supervision. The trajectory and meta-action heads operate in parallel; meta-action predictions are not sequential inputs to trajectory decoding. Stage II adapts the backbone through LoRA, using an imitation loss between the risk-selected ego trajectory and the recorded expert trajectory. The closed-loop Full results use the final checkpoint after both stages. During closed-loop inference, generated ego candidates condition surrounding-agent response prediction.

Ablation construction. Unified and Full without IA use the final two-stage Full checkpoint without retraining. Unified reuses one surrounding-agent forecast across candidates; Full without IA disables interaction analysis during inference. Candidate generation, scoring terms and weights, controller and evaluation protocol remain fixed. The original baseline retains its pretrained checkpoint and original prediction/planning pipeline.

Independent prediction diagnostic. Tables 1 report a separate response-only experiment. The pretrained scene encoder, language backbone and ego planner are frozen; newly initialized feature projections and the response module are trained using cached visual/map features, agent histories and recorded ego futures. Recorded surrounding-agent futures provide trajectory supervision and derived speed/path labels.

This experiment uses 203,810 retained frames from 950 scenes, trained for 10 epochs with AdamW, a learning rate of 0.001 and a global batch size of 16:

Lresponse=LSmoothL1+0.1LspeedCE+0.1LpathCE.L_{response}=L_{SmoothL1}+0.1L_{speedCE}+0.1L_{pathCE}.

SmoothL1 is averaged over valid future coordinates; speed/path classification uses complete future trajectories. Recorded ego futures are supplied during both training and validation. No additional ego-planner, backbone or LoRA update occurs in this diagnostic.

Q2: Ego-only costs and candidate-independent forecasts

Thank you. We report the candidate-independent forecast control and clarify its distinction from ego-only scoring.

Candidate-independent forecasts. As reported under **Reviewer 1 **, Unified reuses one surrounding-agent forecast across the seven ego candidates. Ego-trajectory-conditioned instead supplies candidate-specific forecasts, while retaining the same checkpoint, candidate generator, scoring terms and weights, controller and evaluation protocol.

DS increases from 73.76 to 87.47 (+13.71 points), SR from 47.27% to 72.73% (+25.46 percentage points), and mean ability from 44.48% to 82.00%. Efficiency and comfort decrease from 148.07% to 139.15% and 12.96% to 10.51%, respectively. These results support the planning utility of conditioned forecasts within the fixed planner, with an efficiency/comfort trade-off.

Ego-only costs and component analysis. An ego-only control would exclude costs involving surrounding-agent forecasts. Reviewer 3, W1 (Table 11) illustrates the full scorer's trade-off: compared with Unified Candidate 0, Candidate 6 has a higher progress penalty (8.090 versus 0.000), but zero collision and interaction penalties and a lower TTC-related penalty (3.396 versus 7.942). Its recorded total falls from 45.935 to 30.395, the lowest among the candidates, slightly below Candidate 4 (30.515).

AuthorsOpenReview ↗

Response to Reviewer 3 (part 3)

Q3: Response differences across candidates and decision impact

Thank you. We provide the visualization through an external link: Figure 4: Ego candidates, predicted responses and risk profiles.

Response variation. In panel (c), the surrounding-vehicle trace on the left varies in longitudinal extent, particularly in the third and fifth candidate columns, while the right-side vehicle remains comparatively stable. This suggests agent-specific sensitivity of predicted responses to ego candidates. The accompanying risk profiles illustrate evaluation of the resulting joint futures.

Decision example. Panels (a)–(b) illustrate risk-based selection: candidate 5 is selected with a score of 8.30, compared with 18.43 for candidate 6. These are planning scores, not trajectory-displacement measurements.

Alternative Candidate Selection Rate. On one evaluated route, non-original ego candidates were selected in 7 of 126 saved decision snapshots (5.56%), including stopping, deceleration and lateral adjustments. Such flexibility is particularly relevant to long-tail, safety-critical interactions, where an ill-timed maneuver or failure to yield can cause a collision and route failure.

Q4: Multiple runs and missing Bench2Drive baselines

Thank you. We address both requests with a three-seed evaluation and an expanded baseline comparison.

Results over multiple runs. A full 220-route evaluation takes over 20 hours on eight RTX 5090 GPUs; limited GPU availability during the 13-day rebuttal period therefore restricted repeated evaluations to a fixed subset of 22 routes. We appreciate your understanding of these computational constraints. As discussed in Reviewer 1, Q3, we evaluated both forecast variants using seeds 0, 1, and 2 on the same fixed subset of 22 routes selected for broad scenario coverage. We reproduce the results below for convenience.

MethodDS seed 0DS seed 1DS seed 2Mean DSDS SDMean SR (%)SR SD (pp)
Unified50.0875.0665.0363.3912.5768.1812.03
Ego-trajectory-conditioned61.5868.5365.3365.143.4874.245.25

Ego-trajectory-conditioned achieves higher mean DS/SR (65.14/74.24% versus 63.39/68.18%) with lower observed variability across the three seeds on this fixed subset.

Missing Bench2Drive baselines. The expanded comparison in our response to Reviewer 3, W2 (Table 15) includes SimLingo, HiP-AD, and BridgeDrive, grouped into Non-VLA and VLA methods, with explicit source versions and author-reported results. To avoid duplication, we refer to that table here. Our method leads the listed VLA methods in DS (87.47), SR (72.73%), and mean ability (82.00%), while SimLingo leads in Emergency Brake. Non-VLA BridgeDrive v4 (PDM-Lite) achieves higher DS/SR (87.99/74.99%). VLA methods support language-conditioned planning, whereas the listed Non-VLA methods lack native free-form instruction interfaces. We have incorporated these baselines, group-wise best-value highlighting, and updated discussion into the revised manuscript’s Table 1.

Baseline summary and outlook. Beyond benchmark scores, BridgeDrive highlights a related design principle: it refines data-derived behavioral anchors through a diffusion bridge, while our method uses discrete speed meta-actions to generate ego candidates and explicitly models their surrounding-agent responses for risk-based selection. Both exploit behavioral structure through different mechanisms. Building on this connection, we plan to explore meta-action-conditioned diffusion to refine candidate trajectories and improve the diversity and resolution of speed and passage timing beyond the current seven hypotheses.

Thank you for your detailed rebuttal. My concerns have been addressed and will raise the score to 3.

Reviewer_v7csOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of risk-aware planning in autonomous driving. Specifically, it focuses on how surrounding vehicles may react to the ego vehicle's actions. Specifically, the method consists of three steps: 1) generating candidate ego trajectories corresponding to predefined speed behaviors, 2) using a learned graph-based interaction model to predict surrounding vehicle trajectories conditioned on each ego trajectory, and 3) selecting the ego trajectory that minimizes a manually designed cost function involving collision risk, progress, and motion smoothness. A policy is then trained using supervised learning on the trajectories by finetuning a VLM backbone. Evaluations are conducted in Bench2Drive, showing improvements over existing baselines.

Critical strengths
  • Accounting for responses from other agents is an important topic especially in safety-critical applications such as autonomous driving.
  • The proposed method strongly outperforms baselines in Bench2Drive according to the evaluations.
Critical weaknesses
  • Lack of ablations for counterfactual predictions: Despite the emphasis on "counterfactual futures", the model is trained entirely on recorded trajectories, where only one ego action and its corresponding surrounding-agent responses are observed. It seems that there is no supervision for how other agents would have responded to alternative ego actions from the same initial state. While this is a common challenge for action-conditioned dynamics learning, the paper does not provide convincing evidence that the model actually learns these interactions rather than simply predicting plausible future trajectories. For example, evaluating prediction accuracy under alternative ego actions in simulation would help establish the central claim.
  • Limited ablations to support the proposed contributions: The largest performance improvement comes from introducing candidate-conditioned trajectory prediction and planning, but the corresponding ablation changes multiple components simultaneously, making it difficult to isolate the contribution of modeling agent responses. Furthermore, adding the proposed interaction graph only provides limited improvement. It is unclear whether the additional architectural complexity is justified.
  • Heavily engineered planning procedure and limited generality: The planning procedure relies on seven predefined speed behaviors and a manually designed weighted cost function. These design choices make it difficult to understand how much of the improvement comes from the learned model versus the hand-engineered planner. Additionally, evaluations are limited to autonomous driving in CARLA, leaving the generality of the proposed framework to other embodied planning problems unclear while the title implies the generality of the method rather than specifically designed for applications in autonomous driving.
Questions for authors

See weaknesses section above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2

W1: Lack of ablations for counterfactual predictions

Thank you. We distinguish the intended prediction inputs from the completed diagnostic and planning evaluations.

The intended pipeline generates ego candidates from scene observations and agent histories. Recorded ego futures supervise ego meta-action and trajectory outputs; the surrounding-agent predictor receives generated candidates and is supervised by recorded surrounding-agent futures. Its trajectory and meta-action heads are parallel. Unified shares one forecast across candidates; Ego-trajectory-conditioned generates a forecast for each candidate. Recorded episodes do not supply response labels for every unexecuted alternative.

Table 9. Recorded-ego-conditioned diagnostic, corresponding to Reviewer 1 Table 1.

MethodADE (m) ↓FDE (m) ↓
Unified1.50222.4492
Ego-trajectory-conditioned1.28122.2785
Relative error reduction14.71%6.97%

This completed diagnostic used the recorded ego future as the response condition during training and validation. It evaluates 10,986 frames and 73,588 complete agent futures with identical matching/masks, reducing ADE/FDE by 14.71%/6.97%. It is not an evaluation of the intended generated-candidate conditioning scheme or an isolated conditioning ablation.

Reviewer 1 Q1 reports a separate nine-state CARLA execution diagnostic. Its references reflect reactive simulator controllers, and its table reports trajectory errors rather than ranking accuracy. The fixed-planner comparison below addresses planning utility. These evidence types do not by themselves establish accurate responses to arbitrary alternative actions.

W2: Limited ablations to support the proposed contributions

Thank you. The Unified control separates forecast replacement from the incremental interaction-analysis (IA) comparison. CC denotes candidate-conditioned response prediction.

Table 10. Main-evaluation ablations, matching Reviewer 1 Table 2.

Candidate-ConditionedInteraction AnalysisDS ↑SR (%) ↑Eff. (%) ↑Comf. (%) ↑MergingOvertakingEmergency BrakeGive WayTraffic SignsMean
××78.0455.09154.8912.6132.8975.5668.3350.0057.8956.94
×✓73.7647.27148.0712.9630.3860.0058.3340.0033.6844.48
✓×86.2670.45140.6010.9456.2595.5676.6780.0085.2678.75
✓✓87.4772.73139.1510.5161.2597.7878.3380.0092.6382.00

Forecast replacement. The two IA-enabled rows retain the same weights, candidate generator, cost terms/weights, controller and protocol. Replacing Unified with candidate-conditioned forecasts adds 13.71 DS points and 25.46 SR percentage points. Original MindDrive lacks the same added candidate-evaluation pipeline, so comparisons against it also change planning.

IA increment. With CC enabled, IA adds 1.21 DS points, 2.28 SR percentage points and 3.25 mean-ability percentage points. Efficiency/comfort decrease from 140.60%/10.94% to 139.15%/10.51%. We therefore present an incremental benefit with trade-offs; computational overhead remains to be quantified.

Reviewer 1 Q3 reports seed sensitivity on the same scenario-coverage-oriented 22-route subset: DS is 63.39 ± 12.57 versus 65.14 ± 3.48, and SR 68.18% ± 12.03 pp versus 74.24% ± 5.25 pp.

W3: Heavily engineered planning procedure and limited generality

Autonomous driving and robot navigation among pedestrians share an interaction-planning problem: choosing ego motion while anticipating others’ responses. Merging into traffic and entering pedestrian flows both require coordinating passage timing, yielding, safety, and progress.

Our framework transfers this candidate–response–evaluation structure, while replacing driving-specific behaviors with feasible robot motions, retraining response prediction on robot–pedestrian interactions, and adapting costs to robot geometry and navigation objectives.

We plan experiments in narrow corridors and crossing pedestrian flows, measuring collision rate, navigation success, traversal time, and deadlock frequency. This remains future work; current empirical evidence is limited to autonomous driving.

Dear Reviewer v7cs,

We hope that we have addressed your concerns.

With less than 5 hours remaining before the discussion deadline (November 18), we would be grateful if you would incorporate these revisions into your final assessment, or ask any remaining questions for us to clarify.

Thank you again for your time and consideration!

To view the comment, click here https://openreview.net/forum?id=7iRbzeH6ia&noteId=N5Uptuxhnl

Reviewer_8mg5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Counterfactual-Agent, a risk-aware driving planner that generates seven ego trajectory candidates conditioned on predefined speed meta-actions. For each candidate, it constructs a candidate-conditioned interaction graph, predicts surrounding-agent meta-actions and trajectories, and scores the resulting future scene with proximity, TTC-proxy, progress, comfort, and lateral-motion costs. Training uses factual recorded ego trajectories to supervise interaction and response modules, followed by LoRA adaptation using imitation loss on the risk-selected ego output. On Bench2Drive, the paper reports improved closed-loop Driving Score, Success Rate, and multi-ability results relative to the listed baselines, and includes an ablation for candidate-conditioned response prediction and interaction analysis.

Critical strengths

1.The proposed formulation explicitly connects ego candidates, predicted surrounding-agent responses, and a trajectory-selection cost. The use of candidate-specific interaction graphs and discrete surrounding-agent speed/path meta-actions provides a clear, interpretable decomposition that is more explicit than directly mapping the current scene to one ego trajectory.

2.The paper evaluates the full system in closed loop on Bench2Drive rather than relying only on open-loop trajectory error. It reports Driving Score, Success Rate, five ability categories, an ablation of interaction analysis, and qualitative planning visualizations. The reported full model improves over MindDrive in Driving Score, Success Rate, and mean multi-ability score, while Table 2 shows an additional gain when interaction analysis is added to candidate-conditioned responses.

Critical weaknesses

1.The central counterfactual mechanism is not directly validated. During training, surrounding-agent response prediction is conditioned on the recorded factual ego trajectory; at inference, this factual trajectory is replaced by generated and unexecuted ego candidates. Thus, the key claim requires generalization from factual interactions to counterfactual reactions. The paper does not compare predicted responses under alternative ego candidates with simulator rollouts in which those candidates are actually executed. The conclusion explicitly acknowledges that responses to unexecuted candidates remain unvalidated. Without this evidence, the closed-loop gain cannot be attributed specifically to accurate candidate-conditioned response prediction rather than to candidate generation, heuristic costs, or other architectural changes.

2.The ablation and safety evidence do not isolate the proposed mechanism sufficiently. The paper states that the comparison between MindDrive and the candidate-conditioned-response variant does not isolate the effect of response prediction. The remaining interaction-analysis gain is relatively small (1.21 Driving Score and 2.28 Success-Rate points) and no seed variation or confidence intervals are reported. Moreover, the risk cost uses point-trajectory proximity, a TTC proxy that Appendix A.2 states is not a physical TTC in seconds, and a coordinate-based lateral-motion heuristic rather than map geometry or traffic rules. Table 3 reports nearly unchanged mean TTC and only one highlighted critical case. These results support improved task performance, but not a robust general claim of improved interaction safety.

Questions for authors

1.Can the authors validate counterfactual response predictions in CARLA by restoring a common state, executing multiple ego candidates, and comparing the predicted surrounding-agent trajectories or meta-actions with the observed simulator responses? Please report prediction error and candidate ranking accuracy, ideally stratified by interaction scenario.

2.Can the authors add an ablation that holds candidate generation and the risk scorer fixed while removing, shuffling, or replacing candidate-conditioned response predictions? This is needed to isolate the contribution of the proposed response model from the effects of additional modules and heuristic trajectory costs.

3.Please report variability across training seeds and closed-loop evaluation runs for the full model and the Table 2 variants. Given the modest increment from interaction analysis, confidence intervals or paired route-level statistics would clarify whether this component is robust.

4.Please clarify whether the TTC (s) values in Table 3 are an external simulator metric or the Appendix A.2 TTC proxy. If they are distinct, define the Table 3 metric explicitly. In either case, please report aggregate collision/infraction and safety-margin outcomes over all relevant safety-critical scenarios rather than emphasizing a single illustrative case.

5.The seven ego candidates correspond to predefined speed hypotheses. Please quantify their actual trajectory diversity, including lateral/path variation, and explain how this candidate set covers interaction-relevant alternatives such as yielding, changing path, or rerouting.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (part 1)

W1: Counterfactual mechanism validation

Unified shares one forecast across candidates; Ego-trajectory-conditioned predicts one per candidate. The intended pipeline uses generated candidates as response inputs and recorded ego futures as planner labels. The diagnostic below instead uses recorded ego conditions during training/validation.

Table 1. Recorded-ego-conditioned diagnostic.

MethodADE (m) ↓FDE (m) ↓
Unified1.50222.4492
Ego-trajectory-conditioned1.28122.2785
Relative reduction14.71%6.97%

This tests recorded-ego forecasting, not generated-candidate accuracy or an isolated conditioning effect. The control below tests planning utility; Q1 reports simulator errors.

W2: Ablation and safety evidence

Table 2. Main-evaluation ablations. ✓/× indicate enabled/disabled. M/O/EB/GW/TS denote Merging/Overtaking/Emergency Brake/Give Way/Traffic Signs; Mean averages these five scores.

Candidate-ConditionedInteraction AnalysisDS ↑SR (%) ↑Eff. (%) ↑Comf. (%) ↑MOEBGWTSMean
××78.0455.09154.8912.6132.8975.5668.3350.0057.8956.94
×✓73.7647.27148.0712.9630.3860.0058.3340.0033.6844.48
✓×86.2670.45140.6010.9456.2595.5676.6780.0085.2678.75
✓✓87.4772.73139.1510.5161.2597.7878.3380.0092.6382.00

Within the fixed planner, conditioned forecasts add 13.71 DS points/25.46 SR percentage points over Unified. With CC enabled, IA adds 1.21 DS points, 2.28 SR and 3.25 mean-ability percentage points, with lower efficiency/comfort; significance is not established.

Table 3. Safety margins. TTC means use finite values; D is minimum center distance (m). Definitions are under Q4.

CCIAMean TTC (s) ↑Mean D (m) ↑Critical mean TTC (s) ↑Critical D (m) ↑
××0.9322.6000.800.30
×✓1.0602.8210.890.51
✓×1.0593.0210.822.50
✓✓1.4764.1041.323.35

CC+IA leads all four indicators: mean TTC/D rise from 0.932 s/2.600 m to 1.476 s/4.104 m, and safety-critical TTC/D from 0.80 s/0.30 m to 1.32 s/3.35 m versus (×,×). These are margins, not collision rates.

Q1: CARLA candidate-execution validation

CARLA vehicles follow routes but react to ego state. We compare predictions with matched responses observed during candidate execution at nine states.

Table 4. Simulator-rollout prediction errors (m).

ScenarioADE ↓FDE ↓
Following/braking1.3712.984
Merging1.6042.405
Intersection0.5850.688
Yielding0.9360.979
Low-interaction control1.8231.820
Overall, reported aggregate1.3422.248

Overall ADE/FDE=1.342/2.248 m; intersections have the lowest errors and following/braking the highest FDE (2.984 m).

Q2: Controlled forecast ablation

The IA-enabled rows of Table 3 fix weights, candidate generation, costs/weights, controller and protocol; only forecasts change. This tests planning utility, not response accuracy. Paper Figure 3 visualizes candidate forecasts/costs.

Q3: Variability and statistical robustness

A full 220-route evaluation takes over 20 hours on eight RTX 5090 GPUs. Given limited GPU availability during the 13-day rebuttal period, we therefore assess seed variability on a fixed subset of 22 routes selected for broad scenario coverage. Both methods use the same routes under seeds 0, 1, and 2 to keep the comparison controlled.

Table 5. Three-seed DS/SR on the fixed 22 routes.

MethodDS seed 0DS seed 1DS seed 2Mean DSDS SDMean SR (%)SR SD (pp)
Unified50.0875.0665.0363.3912.5768.1812.03
Ego-trajectory-conditioned61.5868.5365.3365.143.4874.245.25

Ego-trajectory-conditioned achieves higher mean DS/SR (65.14/74.24% versus 63.39/68.18%) with lower observed variability across the three seeds on this fixed subset.

Table 6. Category success rates on the same 22 routes across three seeds (mean±SD; 0–1 scale).

CCIAMOEBGWTS
××0.32±0.050.75±0.060.68±0.060.50±0.150.57±0.05
×✓0.30±0.050.60±0.070.58±0.060.40±0.150.33±0.04
✓×0.56±0.050.95±0.030.76±0.050.80±0.120.85±0.03
✓✓0.61±0.050.97±0.020.78±0.050.80±0.120.92±0.02

With CC enabled, adding IA increases mean success rates from 56% to 61% in Merging, 95% to 97% in Overtaking, 76% to 78% in Emergency Brake, and 85% to 92% in Traffic Signs, while Give Way remains at 80%.

AuthorsOpenReview ↗

Response to Reviewer 1 (part 2)

Q4: TTC and aggregate safety

All configurations use the same safety evaluator, distinct from the planning-cost proxy. Samples are frames with a 3 s window at 0.5 s intervals. Circle radii and clearance are

ri=12Li2+Wi2,gi,h=∥pi,h−p0,h∥2−ri−r0.r_i=\tfrac12\sqrt{L_i^2+W_i^2},\qquad g_{i,h}=\|p_{i,h}-p_{0,h}\|_2-r_i-r_0.

Minima span valid vehicles/future samples. Mean D averages per-sample minimum center distances, not clearances. Critical screening uses minimum clearance ≤0 m (approximate footprint overlap) or minimum finite TTC ≤1.5 s. TTC means exclude non-finite values; critical means use the selected subset. The 3 s window does not itself imply TTC censoring.

Records contain 12,806 open-loop samples, 1,465 CF replacements and 1,439 paired safety records, not 1,439 verified critical samples. Table 3 reports margins, not actual collision/infraction rates; conditional TTC means cannot establish reduced collision frequency.

Q5: Candidate diversity and coverage

We evaluate 12,806 frames, 50 scenes and 11 towns: seven candidates/frame give 89,642 trajectories, with six positions at 0.5 s intervals. Ego-local x points left, y forward. At t=3 s, per-frame spreads are candidate coordinate ranges and pairwise distance is maximum Euclidean separation.

Table 7. Endpoint diversity (m).

MetricMedianP90
Longitudinal spread2.9215.74
Lateral spread0.1341.90
Maximum pairwise distance2.966.29

Lateral spread exceeds 1 m in 19.7% of frames.

Table 8. Endpoint diversity by shared path token (m). Lat/Long denote lateral/longitudinal spread; Pair denotes maximum pairwise distance.

PathFramesMedian LatP90 LatMedian LongMedian Pair
Lane following11,4900.121.472.842.85
Right turn5062.147.652.644.98
Straight3790.502.376.626.62
Left turn3201.846.173.574.54
Left lane change720.522.214.574.67
Right lane change391.352.233.894.11

Candidates share a path token; 89.7% of frames are lane following. Speed/timing alternatives can express yielding. Lateral differences on curves may reflect progress along one path; candidates do not explicitly cover alternative path maneuvers or rerouting.

Dear Reviewer 8mg5,

We hope that we have addressed your concerns.

With less than 5 hours remaining before the discussion deadline (November 18), we would be grateful if you would incorporate these revisions into your final assessment, or ask any remaining questions for us to clarify.

Thank you again for your time and consideration

To view the comment, click here https://openreview.net/forum?id=7iRbzeH6ia&noteId=aY1W98UES8

Thank you for the detailed response and clarifications. Based on the rebuttal and the original paper, I believe my main concerns have been adequately addressed and the novelty of the work is better supported, so I am willing to raise my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area chair

We sincerely thank the Area Chair for coordinating the review of our submission. We are grateful for the opportunity to clarify our work during the discussion period.

Our contribution. To address interaction risks caused by ignoring other agents’ responses to alternative ego actions, we propose a framework that learns interactive counterfactual futures for risk-aware embodied planning.

Reviewer standing. In the rebuttal, we clarify the interaction modeling, training procedure, and risk assessment, and provide additional baseline comparisons, response-prediction diagnostics, multi-seed evaluations, and candidate-diversity analysis to strengthen the evidence for our approach. Two reviewers have raised their ratings from Weak Reject (2) to Weak Accept (3), increasing the average rating from 2.00 to 2.67.

ReviewerInitial → CurrentPost-rebuttal assessment
8mg52 → 3States that the main concerns have been adequately addressed and the novelty is better supported.
v7cs2No follow-up response posted so far.
vVUZ2 → 3States that the concerns have been addressed and explicitly raises the score to 3.

The updated assessments are available in the comments from Reviewer 8mg5 and Reviewer vVUZ.

Additional experiments addressing the reviewers’ main concerns.

ConcernAdditional evidence and key results
Additional baseline comparisonsAdded SimLingo, HiP-AD and BridgeDrive. Our method achieves 87.47 DS / 72.73% SR, ranking first in both metrics among the compared VLA methods.
Response-prediction diagnosticsFactual prediction ADE/FDE decrease from 1.5022/2.4492 m to 1.2812/2.2785 m (14.71%/6.97% reductions). Executed-candidate CARLA diagnostics yield 1.342/2.248 m across nine scenarios.
Multi-seed evaluationsEvaluated the same 22 routes across three seeds. Unified versus conditioned prediction achieves DS 63.39 ± 12.57 vs. 65.14 ± 3.48, and SR 68.18 ± 12.03% vs. 74.24 ± 5.25%, showing higher means and lower observed variability.
Candidate-diversity analysisAnalyzed 12,806 frames from 50 scenes across 11 towns. Median longitudinal endpoint spread is 2.921 m; 19.7% of frames have lateral spread above 1 m, quantifying spatial and timing variation among candidates.

Key experimental conclusion. In the controlled forecast-replacement experiment, candidate generator, scoring function, controller and evaluation protocol, candidate-conditioned forecasts improve DS from 73.76 to 87.47 (+13.71 points) and SR from 47.27% to 72.73% (+25.46 percentage points). This supports their planning utility within the fixed planner. We distinguish this evidence from prediction accuracy under arbitrary unexecuted actions and acknowledge the limited scale of the simulator diagnostics and repeated evaluations.

In addition, we discuss how the shared requirements for response-aware interaction planning in driving and robot navigation supports extending our framework through domain-specific adaptation.

We respectfully ask the AC to consider the additional evidence and clarified scope, together with the two reviewers’ explicit score increases and statements that their concerns have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.