acceptodds
Under review as a conference paper at ICLR 2027

ARDrive: Adaptive Retrieval-Augmented Generation for Autonomous Driving

Abstract

Retrieval-augmented generation (RAG) has recently been introduced to enhance the reasoning ability of vision-language models (VLMs) for autonomous driving (AD) by incorporating external driving knowledge. However, existing RAG-based AD systems typically perform retrieval indiscriminately, assuming that external knowledge is always beneficial. This assumption does not always hold. RAG can be redundant in simple scenarios and may even introduce misleading information when retrieval is imperfect, leading to extra latency and hallucinated reasoning. To address this limitation, we propose ARDrive, a RAG-aware driving framework that formulates retrieval usage as a learnable, context-dependent decision rather than a fixed inference procedure. ARDrive introduces a learnable [RAG] token as a dynamic retrieval gate and trains it through a two-stage RAG-aware pipeline. First, supervised fine-tuning (SFT) establishes initial retrieval awareness using performance-gain-driven pseudo labels generated by comparing RAG and No-RAG reasoning outcomes with an LLM-based semantic scorer. Second, RAG-aware reinforcement learning (RL) further refines the retrieval policy by directly optimizing the trade-off between reasoning accuracy and retrieval cost. Experiments on public AD benchmarks show that ARDrive reduces reasoning latency by 14% and RAG activation by 34%, while improving driving reasoning performance over conventional always-RAG baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonomous Driving, Vision-Language Models, Adaptive Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vMLHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ARDrive, an adaptive retrieval-augmented generation (RAG) framework for autonomous driving. Instead of always using external knowledge, ARDrive learns when retrieval is needed through a [RAG] token.

The authors design a two-stage training pipeline. First, supervised fine-tuning (SFT) uses pseudo labels generated by comparing RAG and No-RAG outputs with an LLM-based semantic scorer. Then, reinforcement learning (GRPO) further optimizes reasoning quality while penalizing retrieval cost.

Experiments on BDD-X and DriveLM show improvements on several reasoning and prediction tasks. On BDD-X, ARDrive reduces RAG usage by 34 percentage points and average inference latency by 14% compared with Always-RAG. The authors also provide ablation studies, human evaluations, and closed-loop driving results.

Critical strengths
  1. A well-motivated and practical research problem.

The paper addresses an important limitation of RAG-based driving systems. External knowledge is not always helpful, and unnecessary retrieval increases latency. Learning when to retrieve is a reasonable way to balance reasoning quality and efficiency. This makes the research problem meaningful and relevant to practical driving systems.

  1. A clear method supported by multiple evaluations.

The two-stage training framework is clearly designed. The authors evaluate the method on multiple datasets and tasks. They also provide comparisons with simpler adaptive retrieval strategies, ablation studies, human evaluations, and closed-loop driving experiments. These results provide useful evidence for the effectiveness of the proposed approach.

Critical weaknesses
  1. Limited evidence that the learned gate accurately identifies retrieval needs.

The main contribution is learning when RAG is beneficial. Appendix Table 13 compares ARDrive with several adaptive retrieval baselines, which is valuable. However, these methods have different RAG activation rates. This makes it difficult to separate retrieval decision quality from retrieval frequency.

Moreover, the human evaluation in Appendix A.4.2 reports only 64.2% agreement between the learned gate and human judgments on non-neutral cases. This suggests that the gate captures some retrieval utility, but its decision quality remains limited.

  1. The contribution of RL to adaptive retrieval is not fully clear.

Table 5 shows that adding RL improves action and justification performance. However, RAG usage decreases only from 67.8% to 66.0%. This raises the question of whether RL mainly improves retrieval decisions or the model's general generation ability.

The additional Always-RAG + GRPO comparison in Appendix Table 7 helps address this concern. However, the contribution of improved gating decisions could be further isolated.

  1. The performance gains are not consistent across driving tasks.

ARDrive improves several metrics, but it does not consistently outperform Always-RAG. For example, on BDD-X, course prediction RMSE increases from 3.85 to 4.14.

The closed-loop evaluation in Appendix Table 15 provides encouraging results. However, the overall success rates remain low. These results support the potential of the method, but its practical benefits across different driving tasks are not yet fully established.

Questions for authors
  1. Could the authors compare ARDrive with random or threshold-based retrieval strategies under the same RAG activation rate? This would help determine whether the gains come from better retrieval decisions rather than retrieval frequency.

  2. Could the authors provide a more detailed analysis of the learned gate against human judgments? In particular, how often does it correctly avoid retrieval when external knowledge would be harmful?

  3. How much of the improvement from RL comes from better retrieval decisions rather than better response generation? Could the authors further separate these two effects?

  4. In the Bench2Drive evaluation, ARDrive activates RAG on only 3.806% of decision steps. How much of the closed-loop improvement can be attributed to these retrieval decisions?

5 comments
AuthorsOpenReview ↗

Response by Authors (1)

We sincerely thank the reviewer for recognizing ARDrive's well-motivated research problem and clear method supported by multiple evaluations. We appreciate the constructive feedback and address the concerns below. All the revisions in the main manuscript are marked in blue.

Weakness 1 & Question 1: Retrieval decision quality

Thanks for raising this important concern. We would like to clarify that ARDrive learns useful retrieval policies beyond simply adjusting retrieval frequency.

(1) Matched-budget Random-Gate comparison. We compare ARDrive with Random-Gate, which replaces the learned retrieval decisions with randomly assigned RAG/No-RAG decisions while keeping the same ARDrive model and a matched 66.0% RAG ratio. The three random seeds represent independent random retrieval assignments. The results are summarized below:

SettingAction B4 ↑Justification B4 ↑Speed RMSE ↓Course RMSE ↓RAG Ratio
Random-Gate (Seed 1)34.08.20.644.1566.0%
Random-Gate (Seed 2)34.58.60.644.1566.0%
Random-Gate (Seed 3)34.18.70.634.1466.0%
ARDrive (Learned Gate)38.910.10.624.1466.0%

Under identical retrieval budgets, ARDrive consistently outperforms all three random assignments on Action and Justification, while achieving better Speed RMSE and comparable Course RMSE. This demonstrates that the gains come from more effective retrieval allocation rather than retrieval frequency. We have added the results and corresponding analysis to Table 7 at Page 10 in the main manuscript.

(2) Human agreement and retrieval decision quality. Beyond the aggregate agreement of 64.2%, our additional analyses provide direct evidence that the gate makes useful retrieval decisions. The matched-budget comparison in (1) shows that these learned decisions improve task performance over random assignments with the same generator and retrieval frequency. Together, these results demonstrate that the gate captures useful retrieval opportunities and delivers measurable benefits. Its practical utility is therefore supported by controlled task-performance evidence.

(3) Practical benefits over always-on retrieval. Across BDD-X and DriveLM, ARDrive demonstrates improved overall task performance with selective retrieval compared with Always-RAG. This advantage also extends to closed-loop driving. On Bench2Drive, ARDrive achieves the best driving performance while activating retrieval on only 3.806% of decision steps. These results demonstrate the benefits of selective retrieval across multiple autonomous-driving benchmarks and evaluation settings.

AuthorsOpenReview ↗

Response by Authors (2)

Weakness 2 & Question 3: The contribution of RL

Thanks for raising this important concern. We investigate the contribution of RL from three complementary aspects, i.e., overall task performance, semantic response quality, and retrieval decision quality. Our results further separate response generation and adaptive retrieval. We have added the results and corresponding analysis to Table 10 at Page 18, and Table 11 at Page 18 in the main manuscript.

(1) Overall task performance. As shown in Table 5, compared with SFT, RL improves Action B4 from 36.9 to 38.9, Justification B4 from 9.0 to 10.1, and Speed RMSE from 0.64 to 0.62. Meanwhile, the RAG activation ratio only decreases from 67.8% to 66.0%. These results demonstrate the overall benefits of RL.

(2) Semantic response quality. To further assess whether RL improves response generation independently, we evaluate SFT checkpoints and SFT+RL checkpoints on all 2,123 test samples using GPT-6.1 Sol High as the independent semantic evaluator. The results are:

TaskSFTSFT + RL
Action0.860.88
Justification0.520.53

Both tasks show consistent semantic improvements, providing evidence that RL enhances the quality of generated responses.

(3) Retrieval decision quality. Human Gate Agreement on non-neutral cases increases from 61.7% for SFT to 64.2% for RL, indicating improved alignment with human retrieval judgments. More importantly, to isolate the contribution of the learned gate, we conduct a controlled gate-replacement experiment by fixing the Generator and replacing its Gate. We sample the SFT Gate decisions to match the same 66.0% RAG ratio. The results are:

SettingAction B4 ↑Justification B4 ↑Speed RMSE ↓Course RMSE ↓RAG Ratio
SFT Generator + SFT Gate36.28.60.634.0666.0%
RL Generator + SFT Gate37.79.10.644.1666.0%
RL Generator + RL Gate38.910.10.624.1466.0%

With the SFT Gate decisions and retrieval budget fixed, replacing the SFT Generator with the RL Generator improves Action B4 and Justification B4, demonstrating improved language generation. Furthermore, with the RL Generator and retrieval budget fixed, replacing the SFT Gate with the RL Gate improves all four task metrics. These controlled comparisons show that RL contributes through both improved semantic response generation and more effective retrieval decisions, with the optimized gate providing additional benefits beyond the generator update.

Weakness 3: The performance across driving tasks

Thanks for raising this concern. We would like to clarify that ARDrive has demonstrated the benefits of adaptive retrieval across different autonomous-driving tasks, from action understanding and motion prediction to closed-loop driving. Our results show that selective retrieval improves task performance while reducing unnecessary retrieval, with benefits extending beyond the BDD-X evaluation setting.

(1) Task-dependent performance and cross-task generalization. The Course RMSE degradation on BDD-X is closely related to its task formulation. Specifically, Speed and Course are jointly generated within the same reasoning round and share a single retrieval gate, despite having different retrieval preferences. Speed prediction tends to benefit from No-RAG, whereas Course prediction consistently benefits more from RAG (Appendix A.4.6). This creates conflicting retrieval preferences during joint policy optimization, causing the learned gate to suppress retrieval in some Course-sensitive scenarios. Consistent with this analysis, cases where Speed improves but Course deteriorates exhibit 6% lower RAG activation and 10% fewer turning scenarios than the opposite cases. This issue is specific to the joint prediction setting in BDD-X rather than a general limitation of ARDrive. Indeed, on DriveLM, ARDrive improves all three motion prediction metrics over Always-RAG. Furthermore, Bench2Drive demonstrates that the learned retrieval policy also generalizes to closed-loop driving evaluation.

(2) Closed-loop transfer. ARDrive is trained exclusively in the open-loop setting, without closed-loop rollouts or optimization. Therefore, its Bench2Drive results evaluate generalization to a different evaluation paradigm rather than performance after dedicated closed-loop training. As contextual references, open-loop-trained AD-MLP and TCP-ctrl report Driving Scores of 18.05 and 30.47, and Success Rates of 0% and 7.27%, respectively. ARDrive achieves a higher Driving Score of 35.795 and a Success Rate of 5.455%. ARDrive achieves the highest Driving Score among these open-loop-trained references. Moreover, ARDrive outperforms No-RAG and Always-RAG on Bench2Drive with only a 3.806% Decision-step RAG Ratio. These results demonstrate that adaptive retrieval generalizes well on closed-loop evaluation.

AuthorsOpenReview ↗

Response by Authors (3)

Question 2: More detailed analysis of the learned gate.

Thanks for this helpful suggestion. We further analyze the learned gate against human judgments to assess its ability to avoid unnecessary and potentially harmful retrieval. The results demonstrate that ARDrive exhibits selective retrieval behavior aligned with human assessments. Specifically, among samples where human annotators judge retrieval to be harmful, ARDrive selects No-RAG in approximately 54.1% of cases, compared with an overall No-RAG rate of 34.0%. This higher avoidance rate indicates that the learned gate preferentially suppresses potentially harmful retrieval rather than uniformly skipping retrieval across samples. We have added the results and corresponding analysis to Appendix A.4.2(1) at Page 23 in the main manuscript.

Question 4: Contribution of retrieval decisions to closed-loop evaluation.

Thanks for raising this important question. To isolate the contribution of learned retrieval decisions from retrieval frequency, we conduct an additional matched-budget Random-Gate experiment. Random-Gate uses the same trained ARDrive generator, retrieval engine, and low-level controller, replacing only the learned gate with random retrieval decisions while matching ARDrive's 3.806% decision-step RAG ratio. We have added the results and corresponding analysis to Table 20 at Page 27 in the main manuscript.

The results are:

MethodDriving Score ↑Route Completion ↑Infraction Penalty ↑Success Rate ↑RAG Ratio
Random-Gate30.30243.520%0.7545.455%3.806%
ARDrive35.79544.108%0.8415.455%3.806%

Under the same retrieval budget, ARDrive outperforms Random-Gate, improving Driving Score by 5.493, Route Completion by 0.588 percentage points, and Infraction Penalty by 0.087, while maintaining the same Success Rate. With the generator, retrieval engine, and controller fixed, these results isolate the benefit of learned retrieval allocation and demonstrate that the closed-loop gains cannot be explained by retrieval frequency alone.

Reviewer_rAjkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

ARDrive learns when to retrieve external knowledge for multimodal driving tasks. It first constructs retrieval labels by comparing outputs with and without retrieval. It then applies SFT and GRPO to optimize task performance and retrieval cost. The paper reports a better performance–efficiency trade-off than fixed retrieval policies.

Critical strengths

The paper addresses a practical question: when does retrieval actually help driving tasks? The experiments on BDD-X and DriveLM show benefits from selective retrieval. On BDD-X, the method reduces retrieval usage and latency while improving several task metrics. These results support the practical value of adaptive retrieval.

Critical weaknesses
  1. The novelty of the core mechanisms appears limited. SKR [1] already constructs retrieval supervision by comparing answers generated with and without retrieval. Repoformer [2] combines performance-gain labels with a special token that controls retrieval. Its labeling rule uses a zero gain threshold, closely matching Eq. (4). The application to multimodal driving is useful, but the paper does not clearly establish the methodological advance over these precedents.

  2. The method is not sufficiently compared with established tool-use approaches. An LLM can decide when to call a retrieval tool, which also provides adaptive retrieval. A stronger teacher could supply retrieval decisions and reasoning traces for distillation. This may provide richer supervision than binary performance-gain labels. Appendix A.4.4 includes adaptive baselines, but neither alternative is tested. I would be happy to raise my rating if ARDrive shows a better performance–cost trade-off than a tool-calling policy trained on stronger-model demonstrations under comparable settings.

  3. I could not find an anonymous repository, code, checkpoints, or the generated supervision data in the submission. The appendix provides implementation details, but these do not fully specify the pipeline. The missing artifacts make it difficult to reproduce the labeling process and verify the reported improvements. This limits my confidence in the experimental evidence.

References

[1] Yile Wang, Peng Li, Maosong Sun, and Yang Liu. Self-Knowledge Guided Retrieval Augmentation for Large Language Models. Findings of EMNLP, 2023.

[2] Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. Repoformer: Selective Retrieval for Repository-Level Code Completion. ICML, 2024.

Questions for authors
  1. How exactly is the [RAG] gate implemented? Is the decision read from generated tokens or a separate prediction head? Please specify the output format, decision rule, and parameters updated during SFT and RL.

  2. What are the response targets during SFT: ground-truth answers or outputs from the selected RAG/No-RAG model? For multi-turn samples, are the two modes evaluated under identical dialogue histories when generating retrieval labels?

  3. During training and pseudo-label construction, are the query sample and other samples from the same source video excluded from the retrieval database? For the additional 3,000/447 DriveLM split, are the 447 held-out samples excluded from both the knowledge base and all training stages?

  4. How are speed and course errors combined in the numerical reward? Are they normalized before applying the exponential kernel in Eq. (5)? Please also specify the RMSE-to-score conversion used for SFT labeling.

  5. Appendix A.1.2 selects the highest-scoring candidate to construct the next dialogue turn during RL training. How are responses selected at test time without ground-truth rewards? Does the reported latency include all candidate generation and selection costs?

  6. Do independent judges or human evaluations confirm improvements from the SFT checkpoint to the RL checkpoint? Table 11 compares the final model with No-RAG. A direct SFT-versus-RL comparison would help assess whether optimizing the Qwen-Max reward improves answer quality and retrieval decisions.

  7. See weaknesses.

5 comments
AuthorsOpenReview ↗

Response by Authors (1)

We sincerely thank the reviewer for recognizing ARDrive's contributions and practical value. We appreciate the constructive feedback and address the concerns below. All the revisions in the main manuscript are marked in blue.

Weakness 1 & Question 7: Novelty.

Thanks for raising this important concern. We would like to clarify that our contribution is not to introduce these individual components, but to establish an effective learning framework beyond pseudo-label supervision for adaptive retrieval in autonomous driving. We have added discussion about Adaptive Retrieval-Augmented Generation to the Related Work Section in the main manuscript to further clarify our contribution.

(1) ARDrive advances adaptive retrieval from pseudo-label supervision to outcome-driven policy optimization. Prior work establishes adaptive retrieval as a general concept, while also showing that its effective formulation for a specific task can constitute a meaningful contribution. ARDrive develops a driving-specific training framework that goes beyond learning retrieval decisions from offline supervision (SKR and Repoformer). Specifically, SFT provides an initial retrieval policy, while RL further optimizes this policy using task-performance rewards and an explicit retrieval cost. This enables the model to refine its retrieval decisions according to actual driving-task outcomes and retrieval overhead. Our SFT-to-RL comparisons confirm this improvement, i.e., RL improves both answer quality and retrieval decisions.

(2) ARDrive challenges the always-on retrieval paradigm in autonomous driving. Prior AD-RAG methods, such as RAG-Driver and SafeAuto, adopt always-on retrieval, implicitly assuming that retrieved knowledge is beneficial at every reasoning step. ARDrive challenges this assumption by showing that retrieval utility depends on the driving context and that selective retrieval can achieve a better performance-efficiency trade-off. This is supported by our matched-budget Random-Gate experiment (see response to Question 6(2)), where ARDrive consistently outperforms random retrieval decisions under the same 66.0% RAG ratio. More importantly, in the closed-loop Bench2Drive evaluation, ARDrive achieves the best Driving Score, Route Completion, and Success Rate, while activating retrieval on only 3.806% of decision steps. These results demonstrate that adaptive retrieval is both meaningful and practically beneficial for autonomous driving.

AuthorsOpenReview ↗

Response by Authors (2)

Weakness 2 & Question 7: Comparison with teacher-based adaptive retrieval approaches.

Thanks for this valuable suggestion. We implement a strong-teacher distillation baseline using Qwen3.8-Flash to generate retrieval decisions, with and without reasoning traces, and distill them into our 7B student model. Our experiments show that stronger teacher supervision and richer reasoning traces do not necessarily lead to more effective retrieval policies, whereas ARDrive achieves better driving-task performance with substantially lower retrieval costs. We attribute this difference to the distinct supervision objectives, i.e., external judgments of retrieval relevance do not necessarily reflect the actual task utility of retrieval for the target model. We have added the results and corresponding analysis to Table 8 at Page 10 in the main manuscript.

(1) Strong-teacher distillation under comparable settings. We use Qwen3.8-Flash, with 125B main-model parameters, 51B additional N-gram embedding parameters, and 512 experts, as the teacher for our 7B student. All methods share the same student backbone, data splits, retrieval corpus, retriever, Top-k settings, and BDD-X ground-truth response targets. The teacher receives the same initial visual observations, ego-state information, and task prompt as the student. A system prompt encourages retrieval in potentially uncertain scenarios and requests a RAG/No-RAG decision with a brief rationale. We distill either the decision alone or both the decision and rationale. In the latter variant, the student generates the rationale after the gate token, and the rationale tokens are included in the training loss.

(2) Performance–cost comparison. The results on BDD-X test set are:

MethodAction B4 ↑Justification B4 ↑Speed RMSE ↓Course RMSE ↓RAG Ratio ↓
Teacher Distillation (w/o Reasoning)32.98.00.654.7198.8%
Teacher Distillation (w/ Reasoning)32.77.40.674.6193.8%
ARDrive (SFT)36.99.00.643.9667.8%
ARDrive (SFT+RL)38.910.10.624.1466.0%

Neither teacher-distilled variant surpasses even the original SFT model. Moreover, adding reasoning traces does not consistently improve task performance and only reduces RAG usage from 98.8% to 93.8%. In contrast, ARDrive improves all four driving metrics over both distilled variants while reducing RAG usage by 27.8–32.8 percentage points.

Besides, the high RAG ratios of Teacher Distillation results are consistent with the teacher's labeling behavior, which recommends RAG for about 80% Justification turns. Since dialogue-level RAG usage counts a dialogue as retrieval-enabled whenever any turn activates RAG, this task-specific preference explains the high overall retrieval rates of the distilled students.

(3) Why does stronger teacher supervision not translate into better retrieval policies? The key distinction lies in the supervision objectives. Human annotators or stronger teachers can judge whether external knowledge appears relevant or potentially useful in a given driving scenario. However, such judgments do not necessarily indicate whether retrieval will actually improve the target model's task outputs. Even with reasoning traces, teacher distillation primarily transfers the teacher's retrieval preferences and explanations, rather than directly optimizing the model-specific utility of retrieval. In contrast, ARDrive derives retrieval supervision from actual downstream task performance and further optimizes the retrieval policy through RL with task-performance rewards and retrieval costs. This enables the gate to learn when retrieval provides sufficient benefit to the target model.

Weakness 3 & Question 7: Reproducibility.

Thanks for raising this concern. We agree that providing implementation details and reproducibility artifacts is essential for verifying our experimental results. To facilitate inspection and reproduction, we provide an anonymized code package in the supplementary materials, including the retrieval supervision files, SFT and GRPO training implementations, evaluation scripts, and corresponding configurations. The original datasets can be obtained from their official sources. Due to the substantial size, model checkpoints are not included in the supplementary package and will be released upon acceptance. We will also make the complete implementation and associated artifacts publicly available upon acceptance.

AuthorsOpenReview ↗

Response by Authors (3)

Question 1: Implementation details of the [RAG] gate.

Thanks for raising this question. We clarify that ARDrive implements the retrieval gate through autoregressively generated control tokens. The retrieval decision is directly integrated into the VLM's generation process and optimized through our SFT and RL training. We have added the clarification to Appendix A.1.2 at Page 15 in the main manuscript.

(1) The retrieval gate is implemented through generated control tokens. ARDrive uses task-specific RAG/No-RAG markers at the beginning of the generated response. For example, the action prediction task uses [RAG_ACTION] and [NO_RAG_ACTION], while justification and control-signal prediction use their corresponding markers. These tokens indicate the retrieval decision and are followed by the task response.

(2) Retrieval is triggered directly by the generated decision. At inference, the model first generates a RAG/No-RAG marker conditioned on the driving observations, ego-state, and task prompt. If the marker indicates RAG, the system invokes the retrieval engine and incorporates the retrieved knowledge into response generation. Otherwise, the model generates the response without external retrieval.

(3) Parameters updated during SFT and RL. During SFT, we use full-parameter fine-tuning. The newly added control-token embeddings and VLM parameters are jointly optimized through the autoregressive language modeling objective. During RL, we further optimize the same VLM using GRPO based on the probabilities of the RAG/No-RAG control tokens. The generated responses provide task-reward feedback, while policy gradients update the shared VLM parameters responsible for retrieval decisions.

Question 2: SFT response targets and multi-turn dialogue histories.

Thanks for raising this question. We clarify that SFT uses ground-truth answers as response targets. Both retrieval modes share identical dialogue histories during retrieval label generation. We have added the clarification to Appendix A.1.2 at Page 15 in the main manuscript.

(1) Ground-truth answers are used as SFT response targets. Our data construction pipeline compares the Always-RAG and No-RAG model outputs against the original ground-truth answers to determine the retrieval gate labels. These comparisons provide pseudo-labels for RAG/No-RAG decisions, while the original ground-truth answers are retained as the response-generation targets.

(2) Both retrieval modes share identical dialogue histories during multi-turn label generation. For each turn, the RAG and No-RAG conditions use the same preceding ground-truth dialogue history and current query. The only difference is whether retrieved driving experiences are provided for the current turn.

Question 3: Retrieval data leakage and held-out split isolation.

Thanks for raising this concern. We would like to clarify that ARDrive strictly separates query samples, source videos, and held-out evaluation data from the retrieval database and training pipeline, preventing retrieval leakage.

(1) Source-video-level exclusion during retrieval. During both training and pseudo-label construction, we exclude not only the query sample itself but also all other samples from the same source video from its retrieval candidates. This prevents the retrieval of overlapping or highly similar driving experiences from the same video. We have added the clarification to Appendix A.1.2 at Page 16 in the main manuscript.

(2) Strict isolation of the 447 held-out DriveLM samples. The 447 held-out samples are completely excluded from the retrieval knowledge base, pseudo-label generation, retrieval encoder/projector training, SFT, and RL. They are reserved exclusively for final evaluation, while the remaining 3,000 samples are used for training and knowledge-base construction. We have added the clarification to Appendix A.2 at Page 17 in the main manuscript.

Reviewer_9DZzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ARDrive studies adaptive retrieval-augmented generation for vision-language models in autonomous driving by introducing a learned gate that decides whether to retrieve external driving experiences. The model is trained through supervised fine-tuning with pseudo-labels derived from LLM-scored comparisons of RAG and No-RAG outputs, followed by GRPO using a reward that balances task quality and retrieval cost. Experiments on BDD-X and processed DriveLM, supplemented by Bench2Drive closed-loop evaluation, show that ARDrive can reduce retrieval usage while improving several task metrics. On BDD-X, it reduces dialogue-level retrieval usage from 100% to 66% and average inference time from 3.32 to 2.85 seconds compared with Always-RAG, although course prediction accuracy deteriorates.

Critical strengths
  1. This work learns when to retrieve external driving knowledge, reducing retrieval usage and inference latency while improving several task metrics over Always-RAG.

  2. This work supports its findings with adaptive retrieval baselines, human assessments, and closed-loop driving evaluation.

Critical weaknesses
  1. The paper does not sufficiently establish whether ARDrive’s gains stem from better retrieval decisions, as adaptive baselines are compared at different retrieval budgets and the contribution of improved gating versus stronger response generation remains unclear.

  2. ARDrive degrades course prediction, particularly on the safety-critical subset where RMSE increases from 2.44 to 3.19 compared with Always-RAG, without sufficiently explaining the implications of this deterioration for the overall driving performance–efficiency trade-off.

Questions for authors

Can the authors compare ARDrive with the adaptive retrieval baselines at matched retrieval usage or inference latency to clarify whether its gains come from more effective retrieval decisions?

3 comments
AuthorsOpenReview ↗

Response by Authors

We sincerely thank the reviewer for recognizing ARDrive's contributions. We appreciate the constructive feedback and address the concerns below. All the revisions in the main manuscript are marked in blue.

Weakness 1 & Question 1: Effectiveness of retrieval decisions under matched retrieval budgets.

Thanks for raising this concern. To directly disentangle the effects of retrieval decisions, retrieval budget, and generator capability, we conduct a controlled comparison that varies only the retrieval gate. The results confirm that with the generator and retrieval budget fixed, ARDrive's advantage over Random-Gate directly demonstrates the contribution of learned retrieval decisions. We have added the results and corresponding analysis to Table 7 at Page 10 in the main manuscript.

(1) ARDrive consistently outperforms random retrieval at the same budget. We compare ARDrive with Random-Gate, which replaces the learned retrieval decisions with randomly assigned RAG/No-RAG decisions while keeping the same ARDrive model and a matched 66.0% RAG ratio. The three random seeds represent independent random retrieval assignments. The results are summarized below:

SettingAction B4 ↑Justification B4 ↑Speed RMSE ↓Course RMSE ↓RAG Ratio
Random-Gate (Seed 1)34.08.20.644.1566.0%
Random-Gate (Seed 2)34.58.60.644.1566.0%
Random-Gate (Seed 3)34.18.70.634.1466.0%
ARDrive (Learned Gate)38.910.10.624.1466.0%

Under the same retrieval budget, ARDrive consistently improves Action B4 and Justification B4 over random retrieval across all three seeds, while achieving better or comparable speed and course prediction accuracy.

(2) Learned retrieval decisions provide gains with the generator fixed. By keeping the generator and retrieval budget identical, this comparison isolates the effect of retrieval allocation. ARDrive consistently outperforms random gating, demonstrating that its learned gate selects more beneficial retrieval opportunities rather than merely increasing retrieval frequency.

Weakness 2: Implications of the performance–efficiency trade-off.

Thanks for raising this concern. We would like to clarify the task-specific implications and strategies for the safety-critical subset. Importantly, this localized degradation does not undermine ARDrive's overall performance–efficiency advantages, as supported by its strong performance on other driving benchmarks.

(1) The course prediction degradation is specific to the task formulation and data characteristics of BDD-X. As analyzed in Appendix A.4.6, on the BDD-X full test set, speed and course are jointly generated in the same reasoning round despite having different retrieval preferences. Their conflicting signals can cause under-activation on apparently simple but course-sensitive samples. However, ARDrive achieves the best results on DriveLM and Bench2Drive, showing that this degradation is localized rather than a general failure of ARDrive's motion prediction.

(2) The safety-critical subset needs task-specific prediction and control strategies. Following the BDD-X-specific limitation discussed in (1), we further discuss its implications for the safety-critical subset. ARDrive improves action, justification, and speed prediction while reducing retrieval usage, but incurs a degradation in Course RMSE. The trade-off suggests a dataset-specific configuration, i.e., using ARDrive for action, justification, and speed prediction, while using No-RAG for course prediction, which achieves the lowest Course RMSE on this subset. Notably, BDD-X follows an open-loop evaluation protocol. Thus, the open-loop accuracy trade-off does not directly establish its impact on actual driving performance. If course prediction errors affect closed-loop steering performance, potential safeguards include constrained model predictive control (MPC) to enforce steering, lateral-acceleration, and road-boundary constraints, together with a verified fallback controller for unsafe predictions. This discussion has been added to the last paragraph of the Appendix A.4.5.

(3) ARDrive maintains strong overall performance across different driving benchmarks. The course prediction degradation observed on BDD-X is specific to its task setting and does not reflect a general limitation of ARDrive. On DriveLM, ARDrive achieves the best ADE, FDE, and MR@2m, demonstrating strong motion prediction capability. More importantly, in the closed-loop Bench2Drive evaluation, ARDrive achieves the best driving score, route completion, and success rate, while activating RAG on only 3.806% of decision steps. These results demonstrate that ARDrive retains strong driving performance while substantially reducing retrieval overhead, supporting its favorable overall performance–efficiency trade-off.

AuthorsOpenReview ↗

Follow-up on Our Responses

Thank you again for your thoughtful feedback. As the discussion period is nearing its end, we would appreciate your thoughts on whether our matched-budget experiments and clarifications address your concerns. We would be happy to provide any further clarification.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 9DZz,

With only approximately two hours remaining before the discussion period closes, we hope you can understand our eagerness to hear your thoughts.

We would be truly grateful for any further feedback you may have. Thank you again for your time and efforts in reviewing our work. We wish you all the best!

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response by Authors (1)

We sincerely thank the reviewer for the continued discussion.

Novelty of Contributions 2 and 3.

Thank you for this important clarification. This prompted us to carefully revisit SKR and Repoformer and reconsider how we position our contributions. We acknowledge that the basic principles of autoregressive token-based retrieval gating and automatic labeling have been explored in prior work. We therefore believe our contributions can be better positioned around a driving-specific training framework that integrates task-specific retrieval utility with progressive policy optimization.

(1) A driving-specific SFT-to-RL framework for adaptive retrieval. We sincerely appreciate the reviewer's feedback, which prompted us to reconsider the positioning and organization of our contributions. Our contribution is to develop a dedicated training framework that progressively learns and optimizes adaptive retrieval policies for autonomous driving. SFT serves as a cold start, jointly supervising retrieval decisions and ground-truth driving responses to establish basic retrieval awareness. More importantly, RL is the core policy optimization stage. It moves beyond imitating fixed retrieval pseudo-labels by sampling and executing retrieval decisions, evaluating their actual downstream task performance, and incorporating explicit retrieval costs into policy optimization. Unlike SKR, which relies on self-knowledge elicitation, and Repoformer, which learns from fixed retrieval labels, ARDrive enables the driving model to actively explore retrieval strategies, learning from the actual outcomes of its own decisions.

The effectiveness of this framework is supported by our experiments. Compared with SFT, SFT+RL improves Action B4 by 5.4% and Justification B4 by 12.2%, while reducing Speed RMSE by 3.1%. Meanwhile, RAG usage decreases from 67.8% to 66.0%. Controlled generator/gate replacement experiments further identify contributions from both response generation and retrieval decisions. These results establish the complementary contributions of our framework. Moreover, the reward design is also tailored to autonomous driving. Together, these components enable the framework to achieve effective adaptive RAG for autonomous driving.

(2) Task-specific retrieval-utility supervision. We use LLM-based semantic scores for language outputs and prediction errors for numerical outputs to construct task-specific retrieval-utility signals. These signals support SFT initialization and subsequent RL optimization within our proposed framework. The main contribution is not the scoring methods, but their integration into the task-specific training framework to enable outcome-driven adaptive retrieval in autonomous driving.

(3) Manuscript revisions. Rather than presenting token-based gating and performance-based labeling as methodological novelties, we have reorganized our contributions on Page 2 to clarify the overall research progression from problem formulation to method development and empirical validation. The revised contributions are:

Contribution 1 — Problem formulation:

We make the first attempt to study adaptive retrieval-augmented generation (RAG) for autonomous driving (AD). Different from existing always-RAG frameworks, ARDrive formulates retrieval usage as a learnable, context-dependent decision and enables driving agents to determine when external knowledge is necessary.

Contribution 2 — Methodology:

We develop a driving-specific two-stage training framework for adaptive retrieval. SFT initializes the retrieval policy through joint supervision of retrieval decisions and ground-truth driving responses, while RL enables the model to actively explore and execute retrieval strategies, optimizing downstream task rewards with explicit retrieval costs. Together, these components enable the framework to achieve effective adaptive RAG for autonomous driving.

Contribution 3 — Empirical validation:

Extensive experiments on public benchmarks (BDD-X and DriveLM) show that ARDrive reduces reasoning latency by 14% and RAG usage by 34%, while improving driving reasoning performance over conventional always-RAG baselines, achieving a better accuracy-efficiency trade-off. Further evaluation on Bench2Drive demonstrates the benefits of adaptive retrieval in closed-loop driving.

These three contributions now form a coherent progression. Contribution 1 establishes the need for adaptive retrieval in autonomous driving. Contribution 2 develops the task-specific training framework to achieve it, and Contribution 3 validates its effectiveness through extensive open-loop and closed-loop experiments. We believe this revised positioning more accurately reflects the actual contributions of ARDrive and clarifies its distinction from existing adaptive retrieval approaches.

AuthorsOpenReview ↗

Response by Authors (2)

Reproducibility of the teacher-distillation experiments.

Thank you for raising this. We have added the teacher annotation script and the distilled datasets to data/teacher_diss/ in the supplementary package. The script includes the teacher prompts, multimodal input construction, generation parameters, and output parsing.

(1) Training and evaluation configurations. Both distillation variants use the same student SFT training script, hyperparameters, and evaluation scripts already provided in the supplementary package (scripts/). No separate training or evaluation procedure is introduced. The student backbone, data splits, retrieval corpus, retriever, and Top-k settings remain unchanged.

For decision-only distillation, the student learns the teacher’s gate decision followed by the original ground-truth response. For distillation with reasoning, the teacher’s brief rationale is inserted after the gate token and included in the autoregressive training loss. Ground-truth driving responses remain the response targets in both variants.

(2) Teacher prompts. For direct inspection, we reproduce the system prompt below:

You are an autonomous-driving retrieval-gate annotator.
The retrieval database contains historical driving experiences and their associated
questions and answers. Decide independently whether retrieving similar experiences
is likely to improve each current task.

Use these task-specific criteria:
- ACTION: RAG when the maneuver is ambiguous or partially observable and similar
  experiences may help disambiguate it. NO_RAG when the action is directly evident.
- JUSTIFICATION: RAG when the cause of the maneuver is not directly observable or
  may require contextual evidence. NO_RAG when the cause is already clear.
- CONTROL_SIGNAL: RAG when the current motion state is insufficient or ambiguous
  and similar motion experiences may help predict the next frame. NO_RAG when the
  current state provides sufficient evidence for prediction.

Do not assume retrieval is always useful or always unnecessary. Do not force any
predefined RAG ratio. Do not generate the final task answers. Do not use the target
answer of the task being evaluated or any future dialogue turns. Earlier assistant
responses are allowed only when they appear in that task's dialogue history.

Return ONLY valid JSON with exactly these keys: action, justification, control_signal.
Each value must contain exactly two fields: gate (RAG or NO_RAG) and reason.
Each reason must be ONE short sentence of AT MOST 20 words. State only the key
observable cue and its retrieval implication. Do not provide chain-of-thought,
step-by-step analysis, alternatives, or background explanation.

The user message supplies the three task-specific dialogue prefixes together in one request, followed by sampled video frames. Its text template is:

The images below are uniformly sampled from the CURRENT driving video, in chronological order.

Evaluate each task using ONLY the dialogue history supplied for that task.
The target response for that task is intentionally omitted. Earlier assistant
responses are included only when they are part of that task's preceding dialogue history.

=== ACTION TASK HISTORY ===
{action_context}

=== JUSTIFICATION TASK HISTORY ===
{justification_context}

=== CONTROL SIGNAL TASK HISTORY ===
{control_signal_context}

Return one independent RAG/NO_RAG decision and one short rationale for each task.

Each task-specific prefix ends at its current query and retains preceding ground-truth dialogue responses. The prompt instructs the teacher to judge each task using only its corresponding prefix. The complete serialization of these prefixes and video frames is provided in the annotation script.

Although I still find the revised second contribution somewhat limited and the [RAG] method kind of outdated, I appreciate the additional experiments and your efforts to address my concerns. I have raised my rating.

AuthorsOpenReview ↗

Response by Authors

We sincerely thank you for the thoughtful reassessment and recognition of our additional experiments and efforts to address the concerns. Your constructive feedback has helped us clarify ARDrive’s contributions and strengthen the experimental evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.