acceptodds
Withdrawn from ICLR 2027

SIC-Drive: Sparse Instruction Conditioning for Closed-Loop End-to-End Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

End-to-end autonomous driving has achieved significant progress by learning a direct mapping from sensory observations to motion planning. However, future autonomous systems require not only the ability to drive autonomously, but also the ability to adapt their behaviors according to human intentions in open and dynamic environments. Natural language provides a flexible interface for specifying high-level driving objectives, such as safety-related interventions, and temporary behavioral adjustments, which are difficult to infer solely from visual observations. Despite recent advances in language-conditioned driving, incorporating language inputs does not necessarily guarantee instruction-controlled behaviors. Existing approaches often focus on integrating language representations with driving policies or generating language-conditioned trajectories, while lacking explicit mechanisms that connect human intent with planning decisions. As a result, it remains unclear whether and how language instructions actually affect closed-loop driving behaviors. In this work, we study language-conditioned controllability for autonomous driving, aiming to enable driving systems to reliably translate natural-language instructions into behavioral changes. We propose SIC-Drive, an explicit instruction grounding framework that decomposes driving instructions into lateral and longitudinal semantics and injects them into trajectory planning through path and velocity candidate scoring. Furthermore, we construct a language-augmented Bench2Drive dataset with temporally aligned driving instructions and introduce DIF-Benchmark to evaluate whether driving agents can follow instructions beyond conventional driving performance metrics. Extensive closed-loop experiments demonstrate the effectiveness of our method, highlighting the importance of explicit language grounding for controllable autonomous driving.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Driving, Language-guided end-to-end

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PKXUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Enabling autonomous vehicles to follow passenger instructions is an important and practical research problem. SIC-Drive offers an interesting approach by integrating FiLM into SparseDriveV2's candidate scoring mechanism, allowing language instructions to influence planning decisions.

However, I have several concerns that prevent me from giving a higher score.

C1. Lack of key baseline comparisons. Instruction following is the paper's main objective, yet SIC-Drive is not compared with existing methods on Instruction Success Rate (ISR). Table 2 mainly reports conventional driving metrics, which do not establish superiority in instruction following. Comparisons with other language-conditioned driving methods, including VLA models, on DIF-Benchmark would be valuable.

C2. Unclear necessity of language augmentation. The contributions of Navigation Augmentation and Risk Warnings Augmentation are not adequately evaluated. In Table 3, Correct and Mute show relatively small differences in DS and SR. Since the instructions are derived largely from existing navigation and scene annotations, it remains unclear whether these augmentations provide meaningful additional information. Separate ablations would help clarify their contributions.

C3. Potential ambiguity in instruction success evaluation. A vehicle may slow down or avoid an obstacle even without receiving a warning. Should such behavior count as successful instruction following? Although DIF-Benchmark uses matched scenes, additional no-instruction controls would help distinguish instruction-induced behavior from default driving behavior.

C4. Execution of distance-constrained instructions. For instructions such as "Maintain your current direction of travel for [x] meters," SIC-Drive does not appear to include an explicit mechanism for tracking traveled distance. How does it ensure accurate execution of such constraints? Experiments with varying distance requirements would help.

C5. Instruction completion and termination. The benchmark uses predefined conditions to activate instructions and determine their completion. However, it is unclear how SIC-Drive would independently detect instruction completion and deactivate the instruction during real-world operation.

C6. Clarity of terminology. The paper is generally clear, but CARLA-specific fields such as "Command Far," "Command Near," and "affects_ego" are introduced without sufficient explanation. Brief definitions would improve accessibility for readers unfamiliar with CARLA.

Critical strengths

See above.

Critical weaknesses

See above.

Questions for authors

See above.

Reviewer_n5YbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SIC-Drive, a language-conditioned driving framework built on SparseDriveV2. It uses a frozen mT5 encoder and two learnable attention queries to extract lateral and longitudinal instruction representations. These representations modulate path and velocity candidate features through FiLM, producing residual score adjustments before Top-K trajectory selection. The authors also augment Bench2Drive-Base with 2,522 temporally aligned instructions and introduce DIF-Benchmark, which evaluates instruction completion in matched closed-loop driving scenarios. SIC-Drive achieves a 60.29% instruction success rate on DIF-Benchmark and a Driving Score of 86.96 on Bench2Drive.The paper makes a useful distinction between language-conditioned trajectory prediction and actual instruction execution. DIF-Benchmark evaluates whether the vehicle reaches the instructed exit, lane, or speed target in closed loop, rather than relying on trajectory imitation error. The matched-scene design, which fixes the initial conditions while varying instructions, is a reasonable approach to measuring behavioral controllability.

Critical strengths

The paper makes a useful distinction between language-conditioned trajectory prediction and actual instruction execution. DIF-Benchmark evaluates whether the vehicle reaches the instructed exit, lane, or speed target in closed loop, rather than relying on trajectory imitation error. The matched-scene design, which fixes the initial conditions while varying instructions, is a reasonable approach to measuring behavioral controllability.

Critical weaknesses
  1. The methodological novelty is limited. SIC-Drive essentially combines a pretrained text encoder, attention-based semantic pooling, FiLM modulation, and additive score residuals within the existing SparseDriveV2 candidate-ranking architecture. These are established techniques, and the lateral/longitudinal decomposition largely follows the baseline planner's existing path–velocity factorization. The paper does not demonstrate why this particular design is preferable to simpler instruction-conditioned candidate scoring or other existing language-conditioning approaches such as NudgeVAD.

  2. The ablations in Table 4 only compare text encoders and fusion methods. They do not isolate the contributions of the semantic classification losses, residual supervision, or separate lateral/longitudinal representations.

  3. The evidence for instruction controllability is weak. The correct/mute/shuffled experiment in Table 3 primarily demonstrates that providing instructions consistent with the original navigation improves performance, while conflicting instructions degrade it. This does not establish that the model can reliably execute alternative feasible maneuvers. DIF-Benchmark addresses this issue more directly, but its overall instruction success rate is only 60.29%, with lateral success at 67.65% compared with 92.65% for longitudinal instructions.

  4. Furthermore, the benchmark contains only 136 instruction episodes across 60 decision points, without uncertainty estimates or repeated-seed analysis. These limitations make it difficult to assess the robustness of the reported improvements.

Questions for authors
  1. Can the authors provide ablations that independently remove the lateral/longitudinal semantic supervision, residual score supervision, and branch-specific instruction representations? In particular, how does SIC-Drive compare with a single language-conditioned residual scorer applied directly to the final trajectory candidates?

  2. How does SIC-Drive compare with NudgeVAD or another language-conditioned planning baseline on DIF-Benchmark under identical training data and evaluation conditions? Table 2 compares general driving performance, but does not establish the relative instruction-following capability of these methods.

  3. Can the authors report per-instruction-category success rates and variability across random seeds? In particular, it would be useful to distinguish failures caused by incorrect semantic interpretation from failures caused by insufficient candidate coverage in the underlying path–velocity planner.

Reviewer_yvhmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SIC-Drive, an explicit instruction grounding framework that decomposes driving instructions into lateral and longitudinal semantics and injects them into trajectory planning through path and velocity candidate scoring. The authors also augment Bench2Drive-Base with ~2.5k templated, temporally aligned instructions and propose DIF-Benchmark. Evaluations show that SIC-Drive reaches 60.29% ISR on DIF and marginal improvement on Driving Score.

Critical strengths
  • Well-motivated problem: clearly separates "language as input" from "language-controlled behavior" and argues the distinction convincingly
  • DIF-Benchmark's matched-scene, closed-loop design that checks realized behavior (exit branch, lane, speed) rather than trajectory error is a useful evaluation contribution
Critical weaknesses
  • No baselines on the proposed benchmark
  • Missing ablations of the core design choices
  • Training does not address the supervision gap the paper itself identifies
Questions for authors

Whether instructions are grounded to navigation context is a good research question raised by this paper. However, the proposed approach reduces to where and how language is injected, which is less distinctive compared to conventional conditioning than I would expected where the introduction is leading towards. Also, that choice is never compared against existing alternatives. At least a substantial gain from existing language conditioning baselines are expected: e.g. the same backbone with language injected conventionally (e.g., cross-attention or concatenation into the ego/scene query). From table 1, the DS and SR has only marginal improvement compared to SimLingo and other baselines.

The lateral/longitudinal decomposition is not ablated. This is presented as the key design element, but there is no comparison against a single shared instruction representation applied to both branches, or language applied to only one branch.

Minor

  • Table 2 column header "w/o Instructions" with ✓/× marks is confusing; consider "Uses instructions"
  • Broken equation reference in A.1 ("In equation ??"); should be Eq. 8
  • Section 4.2, paragraph beginning "The evaluation result of SIC-Drive on DIF-Benchmark, which..." is a sentence fragment

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.