acceptodds
Under review as a conference paper at ICLR 2027

Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Driving

Abstract

Autonomous driving is an important and safety-critical task, and recent advances in LLMs/VLMs have opened new possibilities for reasoning and planning in this domain. However, large models demand substantial GPU memory and exhibit high inference latency, while conventional supervised fine-tuning (SFT) often struggles to bridge the capability gaps of small models. To address these limitations, we propose Drive-KD, a framework that decomposes autonomous driving into a “perception–reasoning–planning” triad and transfers these capabilities via knowledge distillation to improve VLM performance on comprehensive driving tasks. Through a systematic study of distillation design for autonomous driving, we identify layer-specific attention as the distillation signal to construct capability-specific single-teacher models. Moreover, we unify these single-teacher settings into a multi-teacher distillation framework and introduce asymmetric gradient projection to mitigate cross-capability gradient conflicts. Experiments show that our distilled InternVL3-1B model, with less GPU memory and higher throughput, achieves better overall performance than the pretrained 78B model from the same family on DriveBench, and surpasses GPT-5.1 on the planning dimension. It further substantially outperforms its SFT counterpart in closed-loop evaluation on Bench2Drive-VL, achieving 84.50 Route Completion. Further attention analysis shows that Drive-KD restores the capability-specific visual grounding suppressed by SFT.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, multimodal large language models, knowledge distillation, multi-teacher distillation, autonomous driving, post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_e93MOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Drive-KD studies knowledge distillation for small vision-language models (VLMs) in autonomous driving. It applies distinct attention supervision to perception, reasoning, and planning, and integrates these objectives through multi-teacher training with a two-stage gradient projection scheme. The paper evaluates InternVL3 and Qwen2.5-VL on DriveBench and Drive-P2D, and reports closed-loop improvements on ten Bench2Drive-VL Dev10 routes. Ablations on layers, supervision signals, and gradient handling support the design.

Critical strengths

The gains over unified SFT are meaningful and consistent across both model families, and the closed-loop improvements in Table 2 further show that the training recipe is useful beyond isolated QA evaluation.

The component ablations provide useful empirical guidance. Tables 4, 13, and 14 compare layer choices, attention versus hidden-state supervision, and gradient-handling variants, while the matched single-capability CE baselines further support an advantage for attention distillation beyond task specialization alone.

Critical weaknesses
  1. The main justification for choosing attention over hidden states is questionable. Section 3.2.2 compares answer-final hidden states with attention over a shared image/system-prompt prefix. Under causal attention, that prefix cannot depend on the later question, so its low dispersion may follow from the mask rather than indicate a more stable capability representation. The ablations support the recipe empirically, but do not resolve this issue in the analysis motivating it.

  2. The paper does not clearly explain how the three teachers are obtained. Also, the “Single” row aggregates results from three separately trained capability experts, whereas SFT and Multi are unified models (§4.2; Appendix F.1). This is useful as an expert comparison, but it does not isolate the benefit of teacher specialization.

Questions for authors
  1. I’m a bit confused by the dispersion calculation in §3.2.2. Are you only comparing attention over the shared image and system-prompt tokens? If so, wouldn’t causal masking already make that attention unchanged across different questions? I’m also not sure how $(m=\min(L_i,L_j))$ picks out the shared prefix.

  2. Where do the three teachers come from, exactly? Are they separately trained specialists, or the same model providing different attention targets? Did you also try using just one teacher with the same data, attention losses, and AGP? That would help me understand what the multiple teachers are adding.

  3. Could you say a bit more about the data splits? How did you keep the training and pre-study scenes separate from those in DriveBench and Drive-P2D? I also noticed the Bench2Drive-VL adaptation mentioned in Appendix J. What data did that use, how was it kept separate from the evaluation routes, and did SFT get the same adaptation as Drive-KD?

Reviewer_6JbmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Drive-KD, a knowledge distillation framework for autonomous-driving vision-language models. It separates driving capabilities into perception, reasoning, and planning, and assigns different attention-distillation targets to these capabilities based on a layer-wise analysis. The framework combines three capability-specialized teachers with asymmetric gradient projection to address conflicting training objectives. Experiments across InternVL3 and Qwen2.5-VL include open-loop evaluation on DriveBench and Drive-P2D, closed-loop evaluation on Bench2Drive-VL, and additional distillation baselines and ablations. The results show improvements over supervised fine-tuning while retaining the inference footprint of the small student models.

Critical strengths
  1. The improvements extend beyond open-loop question answering. Table 2 reports improvements over SFT in closed-loop driving for both model families; for InternVL3-1B, Driving Score increases from 37.36 to 62.41. This provides meaningful evidence that the training approach can improve executable driving behavior.

  2. The empirical investigation covers several important design choices. Table 4 and Appendix I examine distillation signals, layer selection, attention targets, and gradient handling, while Appendix H supplements the qualitative attention examples with quantitative measurements. These analyses make the paper more informative than a framework supported only by final benchmark scores.

Critical weaknesses
  1. The distinct methodological contribution is insufficiently isolated. The framework combines attention matching, layer-group matching adopted from prior work, and gradient projection, but the experiments do not establish whether three specialized teachers are necessary compared with one generalist teacher supervising all capabilities under the same joint-training and gradient-handling setup. Table 1’s “Single” results aggregate separately trained capability-specific students, and although Table 9 includes recent KD baselines, the cited AMMKD is absent, leaving the specific advantage of the multi-teacher formulation insufficiently established.

  2. The signal-selection analysis may compare fundamentally different information. Section 3.2.2 compares answer-token hidden states with attention submatrices described as covering shared vision/system-prefix tokens; under causal attention, an identical prefix cannot depend on subsequent questions, so low dispersion need not demonstrate a superior capability-specific distillation signal. Moreover, defining the aligned region by the minimum sequence length does not itself establish semantic token alignment, weakening a central justification for choosing attention over hidden-state supervision.

Questions for authors
  1. Clarification of Section 3.2.2: Does the attention-dispersion calculation include question- or answer-token query rows, or only the shared vision/system prefix? Please explain how the stated minimum-length rule identifies corresponding tokens and avoids measuring question-independent prefix attention.

  2. The multi-teacher contribution: How are the three teachers specialized, and do any existing results compare them against a single generalist teacher using the same complete training data, capability-specific objectives, and AGP? Such evidence would help distinguish the benefit of teacher specialization from that of joint training and gradient handling.

  3. The baseline implementations: Which compatibility modifications were made to EPIC/PCD, Align-KD, and Beta-KD in Table 9, and what teacher configurations and optimization budgets were used? Please also explain the omission of the cited multimodal multi-teacher method AMMKD.

Reviewer_ucKrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work introduces Drive-KD, which distills small driving VLMs. Driving is broken down into perception, reasoning, and planning, and the small model is trained to copy the large model's attention patterns at the layers where each capability is most strongly represented. The three teachers are combined in a multi-teacher setup, with an asymmetric gradient projection method to reduce loss conflicts. Experiments on InternVL3 and Qwen2.5-VL students show the distilled 1B model beats the larger model on DriveBench and clearly outperforms SFT in closed-loop driving on Bench2Drive-VL.

Critical strengths
  1. The distilled 1B model outperforms the 78B model of the same family on DriveBench, which shows the effectiveness of the design. The closed-loop driving on Bench2Drive-VL also shows strong results.
  2. The paper analyzes how representations and attention change across layers in InternVL3 and Qwen2.5-VL models of several sizes. This analysis is insightful and directly motivates the distillation design, i.e., which layers and signals to distill for each capability.
Critical weaknesses
  1. Attention map distillation has been done by many existing papers (e.g., TinyBERT[1], MiniLM[2]), and Drive-KD uses a similar layer-group matching as CompoDistill [3]. Multi-teacher distillation has also been done by previous works such as AMMKD [4]. The asymmetric gradient projection (AGP) is similar to the one in [5], especially stage 2. This makes the paper sound like a combination of various previous works, which limits the paper's novelty.
  2. The distilled 1B model outperforms the teacher model. It is unclear whether the teacher models are fine-tuned on the driving data. If they are not, it is hard to see how the teacher could transfer driving capability it does not have. Also, the gain over SFT may come from the attention loss acting as a regularizer that prevents visual attention from collapsing during fine-tuning (Table 11 shows SFT causes the student to largely stop attending to the image), rather than from distilling the teacher's driving capability. These possible reasons conflict with the paper's main claim that Drive-KD transfers perception, reasoning, and planning capabilities from the teachers to the student.

[1] Jiao, Xiaoqi, et al. "Tinybert: Distilling bert for natural language understanding." Findings of the association for computational linguistics: EMNLP 2020. 2020.

[2] Wang, Wenhui, et al. "Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers." Advances in neural information processing systems 33 (2020): 5776-5788.

[3] Kim, Jiwan, et al. "Compodistill: Attention distillation for compositional reasoning in multimodal llms." International Conference on Learning Representations. Vol. 2026. 2026.

[4] Li, Yuqi, et al. "Ammkd: Adaptive multimodal multi-teacher distillation for lightweight vision-language models." arXiv preprint arXiv:2509.00039 (2025).

[5] Yu, Tianhe, et al. "Gradient surgery for multi-task learning." Advances in neural information processing systems 33 (2020): 5824-5836.

Questions for authors
  1. Could you clarify whether the teacher models are fine-tuned on driving data before distillation? If not, how can the teacher transfer driving capabilities it does not have? In addition, how does the distilled 1B model compare with a larger model (e.g., InternVL3-8B) fine-tuned on the same driving dataset?
  2. Have you tried running the same attention distillation but using a frozen copy of the pre-SFT student as the teacher for comparison? This would show whether the improvement over SFT comes from keeping the student's attention from drifting during fine-tuning or from transferring the teacher's capabilities.
  3. Could you clarify the novelty given the similarity to PCGrad and CompoDistill?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.