- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Drive-KD studies knowledge distillation for small vision-language models (VLMs) in autonomous driving. It applies distinct attention supervision to perception, reasoning, and planning, and integrates these objectives through multi-teacher training with a two-stage gradient projection scheme. The paper evaluates InternVL3 and Qwen2.5-VL on DriveBench and Drive-P2D, and reports closed-loop improvements on ten Bench2Drive-VL Dev10 routes. Ablations on layers, supervision signals, and gradient handling support the design.
Critical strengths
The gains over unified SFT are meaningful and consistent across both model families, and the closed-loop improvements in Table 2 further show that the training recipe is useful beyond isolated QA evaluation.
The component ablations provide useful empirical guidance. Tables 4, 13, and 14 compare layer choices, attention versus hidden-state supervision, and gradient-handling variants, while the matched single-capability CE baselines further support an advantage for attention distillation beyond task specialization alone.
Critical weaknesses
-
The main justification for choosing attention over hidden states is questionable. Section 3.2.2 compares answer-final hidden states with attention over a shared image/system-prompt prefix. Under causal attention, that prefix cannot depend on the later question, so its low dispersion may follow from the mask rather than indicate a more stable capability representation. The ablations support the recipe empirically, but do not resolve this issue in the analysis motivating it.
-
The paper does not clearly explain how the three teachers are obtained. Also, the “Single” row aggregates results from three separately trained capability experts, whereas SFT and Multi are unified models (§4.2; Appendix F.1). This is useful as an expert comparison, but it does not isolate the benefit of teacher specialization.
Questions for authors
-
I’m a bit confused by the dispersion calculation in §3.2.2. Are you only comparing attention over the shared image and system-prompt tokens? If so, wouldn’t causal masking already make that attention unchanged across different questions? I’m also not sure how $(m=\min(L_i,L_j))$ picks out the shared prefix.
-
Where do the three teachers come from, exactly? Are they separately trained specialists, or the same model providing different attention targets? Did you also try using just one teacher with the same data, attention losses, and AGP? That would help me understand what the multiple teachers are adding.
-
Could you say a bit more about the data splits? How did you keep the training and pre-study scenes separate from those in DriveBench and Drive-P2D? I also noticed the Bench2Drive-VL adaptation mentioned in Appendix J. What data did that use, how was it kept separate from the evaluation routes, and did SFT get the same adaptation as Drive-KD?