acceptodds
Under review as a conference paper at ICLR 2027

Vision-Language-Action Autonomous Driving Agent with Language-based Memory

Abstract

Vision-Language-Action (VLA) foundation models have recently emerged as one of the prevailing solutions for autonomous driving, as they can utilize knowledge acquired during vision-language pretraining for accurate and interpretable driving. However, VLAs can take only a limited number of frames as visual input due to the high token cost of an image, which is problematic for memory-dependent tasks such as determining the arrival order at all-way stops and long-horizon driving scene understanding. Existing solutions use latent vector memories accessed through cross-attention, which are neither interpretable nor portable. In this paper, we propose AD-Memo, a general-purpose VLA driving agent with language-based memory. The agent outputs memory as an extension of its Chain-of-Thought (CoT) to record surrounding objects critical to driving; this memory becomes part of the agent's future input. We curate memory-based datasets and train VLAs with a two-stage recipe: Supervised Fine-Tuning (SFT) and Da Capo, a novel semi-closed-loop Reinforcement Learning (RL) algorithm which uses trajectory-level advantage for memory and step-level advantage for driving, leading to better credit assignment. Across scenarios such as all-way stops and general driving, AD-Memo improves driving quality, enables better question answering on driving scenes, and produces plug-and-play memory for other models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, autonomous driving, memory, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_t4keOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AD-Memo, a VLA-based autonomous driving framework that uses language-based memory to retain historical driving information. The model periodically generates textual descriptions of important objects and events, which are reused as inputs at subsequent steps. The authors also introduce Da Capo, a semi-closed-loop RL method that optimizes memory generation using future driving and VQA rewards. Experiments on all-way-stop and general driving datasets evaluate trajectory prediction, scene understanding, and memory transferability.

Critical strengths

The paper proposed to handle the problem in long-horizon driving by utilizing text language for history representation. The Da Capo training strategy looks interesting, since it is trained on the driving reward. The paper is generally easy to follow and understand.

Critical weaknesses
  1. From the perspective of the reviewer, language memory works better than latent memory is because that language memory comes from well-pretrained LLMs (e.g., chatgpt), but latent memory comes from small memory module trained by small amount of data (e.g., Orion). Therefore, it is hard to say whether your memory mechanism works or the LLMs are too powerful. So the reviewer think this paper is similar to GPTDriver, which describe the scene in language. The reviewer does not think using language itself for memory representation is a good idea, since it loses a lot of information, such as grounding, spatial, geometric, motion, etc. Suggestions: you can even use a graph for such a memory representation, or you can research how to utilize LLMs great powers for better latent memory representation (like ChatGPT boosted latent representation).

  2. A big problem is that your experiment results have very marginal improvements, which is not sufficient to support your claims, e.g., no-memory RL baseline, Avg. ADE only improves from 1.883 to 1.864. In addition, no close-loop results provided.

  3. The visualizations are difficult to follow, and the differences between consecutive frames are not clearly illustrated.

  4. Does this memory mechanism accumulate errors? Not like latent memory that can directly be trained in a end-to-end manner with the planning task, the language memory might accumulate errors (e.g., the vehicle position, the traffic light color, etc.). Does this happen? How to handle it? More analysis should be provided.

Questions for authors

Please refer to the weakness part for raised questions.

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer t4ke (1/3)

We thank the reviewer for their time and feedback. We appreciate the reviewer highlighting the clear presentation and interesting RL training strategy. Here are our responses:

W1. Benefits of using language memory.

  1. We agree that language memory works well because it comes from well-pretrained LLMs. But “LLMs are too powerful” is not a limitation. In contrast, our very motivation is to best utilize the reasoning capability of an LLM which we feel are not sufficiently unleashed in current VLA agents. We believe leveraging this ability is promising. Indeed, the recent success of GPT-6 Astra proved that LLMs with stronger reasoning ability yield better control policies.

  2. To extend to larger models such as Alpamayo 2 Super and real-life visual input, the memory module is not necessarily generalizable with a small amount of data. Orion works well on the CARLA simulator, but real-life data is out-of-distribution; this is stated in their paper (Appendix C.1), where their method is suboptimal to other VLM-based methods on real-life scenarios (nuScene) due to out-of-distribution trajectories.

  3. As we state in line 77-78, our motivation is to build an explainable and portable memory, which we emphasize in our teaser and show in Sec. 4.3. Compared to language-based memory, vector-based memory is much harder, if at all possible, to comprehend by humans or plug into other models in a zero-shot manner.

  4. We agree that language memory, in its current form, cannot fully recover information such as grounding, spatial, geometric and motion; future augmentation may be needed to fully capture this information. However, there is also no guarantee that vectors can capture all this information. Compared to vectors, language merits include:

Concentrated: By recording in language, the model already conducts reasoning on whether an object is important to current driving, and which kind of spatial relation is the most important to notice, which helps simplify the decision-making process.

Extendable: Language can easily extend to minutes or even hours of information that is indeed important (e.g., traffic regulations such as speed signs) without introducing an additional architecture design; related discussion is listed in Appendix F. Also, data for language memory can be shared across all language-based models; latent vectors, on the contrary, might be attached to specific architectures. Thus, applying language memory leads to minimal training infrastructure and no model architecture change, which is more convenient for iterating production-size models.

  1. While both works use language to describe the scene, there is one key difference between our language in memory and GPT-Driver: GPT-Driver’s input contains coordinate numbers, while we intentionally remove such numbers (“NEVER output a numeric ego speed or units” in Appendix C.3). The reason is that LLMs are pretrained strong enough for reasoning, but not pretrained to deal with these numbers quickly and properly. By only keeping the high-level relation for the agent to consider, we force the agent to focus on reasoning and prevent hallucination. We fully leverage the reasoning ability of LLM, but also avoid leveraging them where there is no such ability.

  2. Thanks for the suggestions. We already use decision graphs during data curation (Appendix C.2.1) and it helps produce better memory. As for utilizing the LLM’s power for better latent memory representation, we agree that it is an interesting direction to explore especially with the current progress on embedding models (https://arxiv.org/abs/2412.12591v2) and research agent (https://arxiv.org/abs/2505.19955). However, we did not include it in our scope as we value the explainability and portability of the memory, which latent memory cannot provide.

AuthorsOpenReview ↗

Rebuttal to Reviewer t4ke (2/3)

W2 (1). Improvements.

As both the language memory and RL are our contributions, the performance should be studied for (1) models before and after RL for each memory variant (no memory / CoT-as-memory / ours), and also for (2) memory variants that go through the same stage of training (SFT / SFT+RL).

  1. The performance gain from RL training is significant for all variants of memory, as acknowledged by the reviewer ZNwV (“which also substantially improves the no memory baseline”). While some metrics did not improve, such as minADE for general driving, we explained in the paper (Sec. 4.2; now line 405-407) that the minimum metrics (minADE, minFDE, and corner distance) do not improve because RL sharpens policy distribution towards prediction in a single direction; thus, RL models suffer more when the driver’s intention cannot be determined at a crossing. This does not necessarily mean worse driving quality. On scenarios with clear intention such as all-way stop, our experimental result in Sec. 4.1 shows that such a phenomenon does not happen. We also do achieve a significantly lower most-likely driving metric (ML. ADE/FDE) in the original Tabs. 1 and 2 (now Tabs. 1 and 3).

  2. Our memory also achieves significant performance gains within the SFT models on the current Tabs. 1 and 3, which serves as an ablation showing the validity of language memory. On the general driving dataset, the best performance on every metric belongs to either our SFT model or the RL model.

  3. Furthermore, Besides control advantage, our proposed method also shows significantly better scene understanding ability as it achieves much higher MCQ accuracy on both all-way stop and general driving datasets. It also shows versatile 0-shot portability to other models in Sec. 4.3 that prior vector-based methods cannot achieve. This is especially important with the recent success of GPT-6 Astra which translates language reasoning to robot policy, but it is largely underexplored in prior works. Thus, we also kindly disagree with the point “not sufficient to support your claims”.

W2 (2). Closed-loop evaluation results.

To address the reviewer’s question, we test 396 unseen all-way stop scenes using Alpasim (https://github.com/NVlabs/alpasim) with our model trained on the all-way stop data, each with 6 rollouts. Controls are applied every 0.1s with memory generated every 0.4s; the length is on average 14 seconds. Memories generated between each keyframe separated by 0.4s are discarded, but the performance is evaluated on each control frame.

We list the closed-loop evaluation results below, where higher success rates are better, distance traveled is an auxiliary sanity check metric, and lower is better for the remaining metrics:

ModelsStop success rate ↑Go success rate ↑Roll through rate ↓Δpos\Delta_{\text{pos}} ↓Offroad ↓Collision at fault ↓Wrong lane ↓Plan deviation ↓Distance traveled
Base Model44.4%35.1%4.7%4.780.040.030.510.2474.48
Alpamayo 2 Super57.3%38.1%3.2%3.440.040.020.450.2871.24
No mem. (SFT only)48.8%40.5%1%4.220.040.030.540.2478.46
CoT mem. (SFT only)47.3%39.2%0.9%4.390.040.050.550.2477.08
AD-Memo (SFT only)48.8%38.3%1%4.130.040.040.510.2374.19
No mem. (SFT+Da Capo)57.0%42.3%1.3%3.530.040.030.560.2082.57
CoT mem. (SFT+Da Capo)53.0%39.4%1.7%3.850.040.040.570.1979.02
AD-Memo (Ours)59.2%39.5%1.2%3.310.040.040.580.1879.35

The result shows that AD-Memo achieves higher stop success rate and lower Δpos\Delta_{\text{pos}} with better or similar safety metrics (offroad, collision at fault, wrong lane, plan deviation). We hypothesize that such safety metrics can be further improved if we integrate these terms into the RL reward (for now we only use distance to ground truth for driving reward).

AuthorsOpenReview ↗

Rebuttal to Reviewer t4ke (3/3)

W3. Visualizations.

Thanks for pointing this out, and sorry for the confusion. We assume this point is about our video, since we use red boxes to emphasize objects of interest in the paper such as Fig. 1, 12, 13. We modified the video to clarify.

W4. Analysis on the accumulation of errors in language memory.

  1. Great question. In fact, preventing propagation of incorrect memory is the reason why we develop our RL algorithm to recover from inaccurate memory generated by the agent itself, because RL optimizes the agent’s driving quality conditioning on the agent’s own, imperfect output instead of ground truth perfect memory. We mentioned this in line 231, and now modified the wording to further emphasize this.

  2. We do have an ablation on propagation of incorrect memory, which uses the ground truth memory (SFT+ref. mem.) in Tab. 1 and Tab. 2 (now Tab. 3). The result shows that there is indeed a large gap due to incorrect memory propagation between SFT and “SFT+ref. mem.” models. The result also shows that RL can level or even go beyond this gap by outperforming the “SFT+ref. mem” results.

  3. We also updated our paper to include Figs. 20 and 21 as ablations on how the maximum number of memory entries affects performance on the same trained model. The result shows that some metrics degrade due to compound error in memory, while others keep improving (especially MCQ accuracy). This being said, our method consistently outperforms CoT-with-memory, which shows the validity of our memory.

  4. To further study how incorrect memory affects driving predictions, we conduct an ablation where we substitute self-generated memory with memory generated from a random wrong episode on the all-way stop dataset. The result is listed below and also added in Appendix D.13. The result shows that the driving quality notably degrades, but does not collapse; meanwhile, MCQ accuracy largely decreased. The result indicates that 1) our method is reliably robust with inaccurate memory, and 2) the scene-understanding gain entirely relies on the memory content received at test time, while the driving performance gain from memory is less reliant on test-time input.

MethodminADEAvg. ADEML ADEstop succ.go succ.roll-throughMCQ
AD-Memo0.9511.8661.91589.3%45.0%5.3%51.7%
AD-Memo w./ wrong memory input1.0111.9091.96689.2%43.8%5.4%34.9%
no-memory (SFT + Da Capo)0.9441.9522.00588.9%43.8%5.2%35.3%
Reviewer_ZNwVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AD-Memo, a VLA-based autonomous driving framework that maintains language-based memory across sequential driving observations. The model generates memory as part of its Chain-of-Thought output and reuses it in subsequent steps for trajectory prediction and scene understanding. The authors also introduce Da Capo, a semi-closed-loop RL method that replays ground-truth observations while propagating self-generated memory across steps. It uses step-level advantages for driving actions and future-dependent returns for memory generation. Experiments on all-way-stop and general driving datasets demonstrate improvements in trajectory prediction, VQA, and cross-model memory portability.

Critical strengths
  1. The language-memory design addresses a practical limitation of VLA models, and the semi-closed-loop training formulation is well motivated. Keeping a compact textual record of objects and their temporal behavior is useful when a short visual history cannot capture information such as vehicle arrival order or temporarily occluded pedestrians. The design is also interpretable and transferable to other VLMs. More importantly, Da Capo distinguishes the causal dependencies of memory and driving outputs: predicted trajectories do not influence subsequent observations under replay, while generated memory affects later decisions. This distinction provides a reasonable basis for assigning different reward components to different output tokens.

  2. The experimental evaluation covers several aspects of the proposed framework. On the all-way-stop dataset, AD-Memo with Da Capo achieves an average ADE of 1.866 m, compared with 1.952 m for the no-memory counterpart. On general driving, it achieves 1.864 m versus 1.883 m. Its general-driving VQA accuracy also increases from 57.43% to 65.51%. Table 5 provides useful evidence for the RL component, where Da Capo improves over trajectory-level GRPO. I also find the memory portability results on WaymoQA and LingoQA interesting, since they indicate that the generated memory can help a separate model without retraining it.

Critical weaknesses
  1. The contribution of language memory to actual driving performance is still relatively limited, especially in general driving scenarios. Most of the improvement over SFT comes from RL training, which also substantially improves the no-memory baseline. After Da Capo, the general-driving average ADE improvement from adding memory is only 0.019 m, and minADE actually becomes slightly worse, from 1.090 m to 1.091 m. On all-way-stop, the roll-through rate is also slightly worse than the no-memory RL baseline, at 5.26% versus 5.21%. These results support improved average trajectory matching, but they do not yet establish a clear safety or control advantage. In addition, although the paper motivates language memory against latent-vector memory, there is no direct comparison with a learned latent-memory or compressed-visual-history baseline under comparable resource budgets. Appendix F explains the difficulties of adapting existing systems, but this still leaves an important experimental gap.

  2. The evaluation remains open-loop with respect to vehicle control, which limits the practical conclusions. Da Capo closes the loop for memory generation, but future observations and ego trajectories are still replayed from ground truth. Consequently, errors in predicted actions cannot influence the subsequent scene, and the model is not evaluated on whether its memory improves closed-loop safety, collision avoidance, or recovery after incorrect decisions. This limitation is particularly relevant because language memory may contain inaccurate or outdated object information. Furthermore, the main experiments use the Alpamayo 2 VLM backbone without its action expert, so the behavior of the proposed memory system in a complete driving VLA remains unclear. The additional memory-generation latency and runtime cost are also not quantified in sufficient detail.

Questions for authors
  1. Could the authors include a lightweight latent-memory or compressed-frame-history baseline with a similar input-token and parameter budget? This would clarify whether the improvement comes specifically from structured language memory, rather than simply providing the model with additional temporal information.

  2. Can the authors provide more direct evidence that the generated memory improves driving decisions beyond open-loop ADE, particularly in all-way-stop or occlusion scenarios? A small controlled closed-loop evaluation, or an analysis of how incorrect memory propagates into later driving predictions, would be informative. Reporting the additional inference latency per keyframe would also help establish the practicality of the approach.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer ZNwV (1/3)

We thank the reviewer for their time and feedback. We appreciate the reviewer highlighting our idea as practical, well-motivated, reasonable and useful, and our memory as interpretable and transferable. We also appreciate the reviewer highlighting the breadth of our evaluation and regarding our experiments as interesting. Here are our responses:

W1 (1). Limited performance increase especially in the general driving scenario.

As both the language memory and RL are our contributions, the performance should be studied for (1) models before and after RL for each memory variant (no memory / CoT-as-memory / ours), and also for (2) memory variants that go through the same stage of training (SFT / SFT+RL).

  1. The performance gain from RL training is significant for all variants of memory, as acknowledged by the reviewer (“which also substantially improves the no memory baseline”). While the reviewer points out some metrics that did not improve, such as minADE for general driving, we explained in the paper (Sec. 4.2; now line 405-407) that the minimum metrics (minADE, minFDE, and corner distance) do not improve because RL sharpens policy distribution towards prediction in a single direction; thus, RL models suffer more when the driver’s intention cannot be determined at a crossing. This does not necessarily mean worse driving quality. On scenarios with clear intention such as all-way stop, our experimental result in Sec. 4.1 shows that such a phenomenon does not happen. We do achieve a significantly lower mean and most-likely driving metric (Avg. ADE/FDE, ML. ADE/FDE) in the original Tabs. 1 and 2 (now Tabs. 1 and 3).

  2. Our memory also achieves significant performance gains within the SFT models on the current Tabs. 1 and 3, which serves as an ablation showing the validity of language memory. On the general driving dataset, the best performance on every metric belongs to either our SFT model or the RL model.

  3. Within the RL models, our memory achieves significantly better performance on the average and most-likely metrics on both the all-way stop and general driving datasets, as shown in the current Tabs. 1 and 3.

With the results listed above, we believe our method (both the language memory and RL component) establishes a clear control advantage. As for safety measures, please see our response to the closed-loop evaluation (W2 (1) / Q2 (1)).

Furthermore, Besides control advantage, our proposed method also shows significantly better scene understanding ability as it achieves much higher MCQ accuracy on both all-way stop and general driving datasets. It also shows versatile 0-shot portability to other models in Sec. 4.3 that prior vector-based methods cannot achieve. This is especially important with the recent success of GPT-6 Astra which translates language reasoning to robot policy, but it is largely underexplored in prior works.

W1 (2) / Q1. Direct comparison to other types of baselines.

  1. As discussed in Appendix F, comparing with vector-based baselines in a fair manner is difficult: they either do not provide code, or use a different model architecture that requires a significant amount of work to adapt. Due to this difficulty, prior works such as COMPACT-VA (https://arxiv.org/abs/2606.07464) also do not compare with other memory-based driving methods.

  2. However, to answer the reviewer’s question, we run three different latent memory planners with available checkpoints - MoMAD (https://arxiv.org/abs/2503.03125), MoMADv2 (https://arxiv.org/abs/2608.23405), and ORION (https://arxiv.org/html/2503.19755v1) on our all-way stop data. Note, the baselines need recorded history before a keyframe while our AD-Memo does not. Thus, we can only score the keyframes which appear after 4 s of recorded video (a total of 49,365 out of 73,075 frames for all-way stop), so every scored prediction has ≥ 10 frames of memory. AD-Memo and its no-memory / CoT-as-memory variants are re-scored on the same keyframes from the paper's rollouts, with the ground truth truncated to each baseline's native horizon (6 s for MoMAD / MoMADv2, 3 s for ORION). The results are listed in the next part.

AuthorsOpenReview ↗

Rebuttal to Reviewer ZNwV (2/3)

(continuning from the last part)

MethodAvg. ADE@3s (m) ↓Avg. FDE@3s (m) ↓minADE@3s (m) ↓Stop success@3s ↑Go success@3s ↑Roll-through@3s ↓Avg. ADE@6s (m) ↓Avg. FDE@6s (m) ↓minADE@6s (m) ↓Stop success@6s ↑Go success@6s ↑Roll-through@6s ↓
MoMAD3.466.261.4516.4%6.8%82.7%7.0713.943.2513.5%3.3%85.0%
MoMADv23.275.881.6019.6%7.7%78.6%6.7013.343.5216.2%4.7%81.3%
ORION2.896.602.89†53.4%17.1%46.4%N/AN/AN/AN/AN/AN/A
No memory (SFT + Da Capo)0.381.240.1993.3%64.2%5.7%1.755.410.8490.6%45.4%4.7%
CoT-as-memory (SFT + Da Capo)0.381.220.1993.4%65.1%5.7%1.715.250.8690.8%46.6%4.7%
AD-Memo (SFT + Da Capo)0.371.190.1893.5%65.5%5.6%1.685.160.8490.8%47.0%4.8%

ORION's minADE is the minimum ADE across its six navigation-command branches. ORION predicts only 3 s, so its 6 s results are N/A. For MoMAD and MoMADv2, the 3 s results use the first 3 s of each 6 s prediction. The result shows that all three baselines are unable to transfer to our test case. Retraining these systems on our data would require the 3D / map supervision on which their perception stacks are built; our datasets do not contain such information.

W2 (1) / Q2 (1). Closed-loop evaluation.

To address the reviewer’s question, we test 396 unseen all-way stop scenes using Alpasim (https://github.com/NVlabs/alpasim) with our model trained on the all-way stop data, each with 6 rollouts. Controls are applied every 0.1s with memory generated every 0.4s; the length is on average 14 seconds. Memories generated between each keyframe separated by 0.4s are discarded, but the performance is evaluated on each control frame.

We list the closed-loop evaluation results below, where higher success rates are better, distance traveled is an auxiliary sanity check metric, and lower is better for the remaining metrics:

ModelsStop success rate ↑Go success rate ↑Roll through rate ↓Δpos\Delta_{\text{pos}} ↓Offroad ↓Collision at fault ↓Wrong lane ↓Plan deviation ↓Distance traveled
Base Model44.4%35.1%4.7%4.780.040.030.510.2474.48
Alpamayo 2 Super57.3%38.1%3.2%3.440.040.020.450.2871.24
No mem. (SFT only)48.8%40.5%1%4.220.040.030.540.2478.46
CoT mem. (SFT only)47.3%39.2%0.9%4.390.040.050.550.2477.08
AD-Memo (SFT only)48.8%38.3%1%4.130.040.040.510.2374.19
No mem. (SFT+Da Capo)57.0%42.3%1.3%3.530.040.030.560.2082.57
CoT mem. (SFT+Da Capo)53.0%39.4%1.7%3.850.040.040.570.1979.02
AD-Memo (Ours)59.2%39.5%1.2%3.310.040.040.580.1879.35

The result shows that AD-Memo achieves higher stop success rate and lower Δpos\Delta_{\text{pos}} with better or similar safety metrics (offroad, collision at fault, wrong lane, plan deviation). We hypothesize that such safety metrics can be further improved if we integrate these terms into the RL reward (for now we only use distance to ground truth for driving reward).

W2 (2). Performance in a complete VLA driving system.

Thanks for pointing this out. The reason for not using an action expert is because the training process of Alpamayo is to first finetune its VLA backbone, then conduct RL on the VLA backbone, and then training the action expert (https://www.nvidia.com/en-us/on-demand/session/alpamayo26-alpa03/). During early exploration, we find that models after RL are too resistant to our new tasks. Thus, to test the performance of the action expert, we not only need to adjust production-scale RL training accordingly, but we also need to then train the action expert from scratch, which is too expensive.

We discussed this as a limitation in the original submission of Appendix F. To address the reviewer’s concern, we add discussion on the proposed memory system in a complete driving VLA. From a dual-process perspective, we hypothesize that language memory improves the backbone’s deliberative reasoning (“System 2”), while the action expert translates that reasoning and the current scene into a continuous trajectory (“System 1”). The VLA agent hence will possess greater temporal grounding-based reasoning ability and make better driving decisions without sacrificing the inference speed advantage of the action expert.

AuthorsOpenReview ↗

Rebuttal to Reviewer ZNwV (3/3)

W2 (3) / Q2 (2). Memory-generation latency and runtime cost.

To verify the actual inference cost, we test the inference speed difference between AD-Memo and the baselines on our all-way stop data. We use the three RL checkpoints of Tab. 1 (no memory, CoT-as-memory, AD-Memo) and the same driver stack as our closed-loop evaluation (vLLM 0.11, bf16) on one H100 80GB GPU at batch size 1, i.e., the worst case of a single vehicle without batching. No speculative decoding is performed. We replay 60 unseen all-way-stop test episodes (1,580 keyframes per model, the same episodes for every model); the memory grows to up to 47 entries within an episode. We report vLLM in the eager mode used by our evaluation pipeline and with CUDA graphs, which is closer to a deployment setting. Decoding succeeded on 99.8% of keyframes for every model. Below are the results, which we added in Appendix E.1:

Memory entries readKeyframesLatency, eagerLatency, CUDA graphsText prompt tokensOutput tokens
1–51883.61 s0.81 s252160
6–102923.59 s0.80 s339160
11–204513.59 s0.80 s495160
21–302663.58 s0.80 s721158
31–502113.53 s0.79 s940157
ModelvLLM eager: latency mean (p50 / p90)rel.CUDA graphs: latency mean (p50 / p90)rel.Text prompt tokens mean (max)Output tokens mean (max)
No memory3.18 s (3.17 / 3.30)1.00×0.72 s (0.71 / 0.76)1.00×124 (124)141 (168)
CoT-as-memory3.23 s (3.21 / 3.34)1.01×0.72 s (0.71 / 0.76)1.00×492 (1,217)142 (171)
AD-Memo3.58 s (3.56 / 3.74)1.12×0.80 s (0.79 / 0.86)1.11×540 (1,390)159 (199)

The result shows that the inference overhead is only 11–12% per keyframe, which is modest. It is almost proportional to output length, as the current inference is decoding-bound.

Q2 (3). Propagation of incorrect memory into later driving predictions.

  1. Great question. In fact, preventing propagation of incorrect memory is the reason why we develop our RL algorithm to recover from inaccurate memory generated by the agent itself, because RL optimizes the agent’s driving quality conditioning on the agent’s own, imperfect output instead of ground truth perfect memory. We mentioned this in line 231, and now modified the wording to further emphasize this.

  2. We do have an ablation on propagation of incorrect memory, which uses the ground truth memory (SFT+ref. mem.) in Tab. 1 and Tab. 2 (now Tab. 3). There is indeed a large gap due to incorrect memory propagation between SFT and “SFT+ref. mem.” models. Also, RL can level or even go beyond this gap by outperforming the “SFT+ref. mem” results.

  3. We also updated our paper to include Figs. 20 and 21 as ablations on how the max number of memory entries affects performance on the same model. The result shows that some metrics degrade due to compound error in memory, while others keep improving (especially MCQ accuracy). Also, our method consistently outperforms CoT-with-memory, which shows the validity of our memory.

  4. To further study how incorrect memory affects driving predictions, we conduct an ablation where we substitute self-generated memory with memory generated from a random wrong episode on the all-way stop dataset. The result is listed below and also added in Appendix D.13. The driving quality notably degrades, but does not collapse; meanwhile, MCQ accuracy largely decreased. The result indicates that 1) our method is reliably robust with inaccurate memory, and 2) the scene-understanding gain entirely relies on the memory content received at test time, while the driving performance gain from memory is less reliant on test-time input.

MethodminADEAvg. ADEML ADEstop succ.go succ.roll-throughMCQ
AD-Memo0.9511.8661.91589.3%45.0%5.3%51.7%
AD-Memo w./ wrong memory input1.0111.9091.96689.2%43.8%5.4%34.9%
no-memory (SFT + Da Capo)0.9441.9522.00588.9%43.8%5.2%35.3%
Reviewer_eGXWOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AD-Memo, a VLA-based autonomous driving agent with in-episode language memory. The model records important road objects and events as short text, then uses this memory together with recent images to make driving decisions. The authors build memory-annotated datasets for all-way-stop and general driving scenes, and train the model with supervised fine-tuning followed by Da Capo, a semi-closed-loop RL method. During RL, the driving inputs are replayed from recorded data, while the model uses its own generated memory. Driving outputs receive step-level feedback, while memory receives feedback from current and future driving performance and final scene question answering. The experiments show improvements in average open-loop trajectory error and scene question answering compared with no-memory and CoT-as-memory baselines. The generated text memory can also help another model answer questions on external driving benchmarks, showing its potential as a portable memory representation.

Critical strengths
  • Nice writing, very fluent and easy to follow!
  • The procedure is intuitive: image → key-event memory (text), then image + memory → driving decision. This offers a promising approach to in-episode memory, and the text memory is easy to interpret.
  • The semi-closed-loop RL design also makes sense. The model learns to use its own generated memory while the driving inputs are replayed from real data. Assigning different reward signals to memory and driving outputs based on their effects on future steps is a reasonable design.
Critical weaknesses
  • How about the time cost of introducing extra in-episode memory? The model needs to read past memory and generate new memory before outputting the driving trajectory. Does this slow down inference?
  • The experiments mainly compare variants of the same base model with different memory settings and training methods. Although Alpamayo 2 Super 34B is also included, comparisons with other memory-based driving methods, such as latent-vector memory or ICL approaches, are missing. It is therefore hard to tell how this language-memory design compares with existing memory solutions.
  • The memory captions for the general-driving SFT data are generated by GPT-5.6 Luna. How do the authors ensure their accuracy?
  • The driving evaluation is mainly open-loop. Lower trajectory error is encouraging, but it would be good to see whether the gains remain in closed-loop driving, where the model’s actions affect future observations.
Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer eGXW (1/2)

We thank the reviewer for their time and feedback. We also thank the reviewer for appreciating our writing, intuitive procedure, high informativeness and reasonable RL algorithm. Below are our answers:

W1. Time cost of introducing extra in-episode memory.

To verify the actual inference cost, we test the inference speed difference between AD-Memo and the baselines on our all-way stop data. We use the three RL checkpoints of Tab. 1 (no memory, CoT-as-memory, AD-Memo) and the same driver stack as our closed-loop evaluation (vLLM 0.11, bf16) on one H100 80GB GPU at batch size 1, i.e., the worst case of a single vehicle without batching. No speculative decoding is performed. We replay 60 unseen all-way-stop test episodes (1,580 keyframes per model, the same episodes for every model); the memory grows to up to 47 entries within an episode. We report vLLM in the eager mode used by our evaluation pipeline and with CUDA graphs, which is closer to a deployment setting. Decoding succeeded on $\geq$99.8% of keyframes for every model. Below are the results, which we added in Appendix E.1:

Memory entries readKeyframesLatency, eagerLatency, CUDA graphsText prompt tokensOutput tokens
1–51883.61 s0.81 s252160
6–102923.59 s0.80 s339160
11–204513.59 s0.80 s495160
21–302663.58 s0.80 s721158
31–502113.53 s0.79 s940157
ModelvLLM eager: latency mean (p50 / p90)rel.CUDA graphs: latency mean (p50 / p90)rel.Text prompt tokens mean (max)Output tokens mean (max)
No memory3.18 s (3.17 / 3.30)1.00×0.72 s (0.71 / 0.76)1.00×124 (124)141 (168)
CoT-as-memory3.23 s (3.21 / 3.34)1.01×0.72 s (0.71 / 0.76)1.00×492 (1,217)142 (171)
AD-Memo3.58 s (3.56 / 3.74)1.12×0.80 s (0.79 / 0.86)1.11×540 (1,390)159 (199)

The result shows that the inference overhead is only 11–12% per keyframe, which is modest. It is almost proportional to output length, as the current inference is decoding-bound.

W2. Comparison with other memory-based driving methods.

  1. As discussed in Appendix F, comparing with vector-based baselines in a fair manner is difficult: they either do not provide code, or use a different model architecture that requires a significant amount of work to adapt. Due to this difficulty, prior works such as COMPACT-VA (https://arxiv.org/abs/2606.07464) also do not compare with other memory-based driving methods.

  2. However, to answer the reviewer’s question, we run three different latent memory planners with available checkpoints - MoMAD (https://arxiv.org/abs/2503.03125), MoMADv2 (https://arxiv.org/abs/2608.23405), and ORION (https://arxiv.org/html/2503.19755v1) on our all-way stop data. Note, the baselines need recorded history before a keyframe while our AD-Memo does not. Thus, we can only score the keyframes which appear after 4 s of recorded video (a total of 49,365 out of 73,075 frames for all-way stop), so every scored prediction has ≥ 10 frames of memory. AD-Memo and its no-memory / CoT-as-memory variants are re-scored on the same keyframes from the paper's rollouts, with the ground truth truncated to each baseline's native horizon (6 s for MoMAD / MoMADv2, 3 s for ORION). The results are shown in the next part.

AuthorsOpenReview ↗

Rebuttal to Reviewer eGXW (2/2)

(continuning from the last part)

MethodAvg. ADE@3s (m) ↓Avg. FDE@3s (m) ↓minADE@3s (m) ↓Stop success@3s ↑Go success@3s ↑Roll-through@3s ↓Avg. ADE@6s (m) ↓Avg. FDE@6s (m) ↓minADE@6s (m) ↓Stop success@6s ↑Go success@6s ↑Roll-through@6s ↓
MoMAD3.466.261.4516.4%6.8%82.7%7.0713.943.2513.5%3.3%85.0%
MoMADv23.275.881.6019.6%7.7%78.6%6.7013.343.5216.2%4.7%81.3%
ORION2.896.602.89†53.4%17.1%46.4%N/AN/AN/AN/AN/AN/A
No memory (SFT+Da Capo)0.381.240.1993.3%64.2%5.7%1.755.410.8490.6%45.4%4.7%
CoT-as-memory (SFT+Da Capo)0.381.220.1993.4%65.1%5.7%1.715.250.8690.8%46.6%4.7%
AD-Memo (SFT+Da Capo)0.371.190.1893.5%65.5%5.6%1.685.160.8490.8%47.0%4.8%

ORION's minADE is the minimum ADE across its six navigation-command branches. ORION predicts only 3 s, so its 6 s results are N/A. For MoMAD and MoMADv2, the 3 s results use the first 3 s of each 6 s prediction. The result shows that all three baselines are unable to transfer to our test case. Retraining these systems on our data would require the 3D / map supervision on which their perception stacks are built; our datasets do not contain such information.

W3. Accuracy of memory.

  1. The motion and object information in memory are grounded by product-grade tools. For example, the ego speed is provided by the metadata of the clips, and the existence and motion of other objects on the road are retrieved via invoking tried-and-true task-specific models. In our human review, delegating that information to non-LLM models significantly improves both the coverage and accuracy of the memories.

  2. We design multiple review processes during data curation as mentioned in Appendix C.2.1, including the “evidence gathering” stage, the “description-based linting” stage, and memory merging where the model conducts another review with access to the full video. We also conduct user studies and manually examine a subset of memory to ensure that they are correct.

W4. Closed-loop evaluation.

To address the reviewer’s question, we test 396 unseen all-way stop scenes using Alpasim (https://github.com/NVlabs/alpasim) with our model trained on the all-way stop data, each with 6 rollouts. Controls are applied every 0.1s with memory generated every 0.4s; the length is on average 14 seconds. Memories generated between each keyframe separated by 0.4s are discarded, but the performance is evaluated on each control frame.

We list the closed-loop evaluation results below, where higher success rates are better, distance traveled is an auxiliary sanity check metric, and lower is better for the remaining metrics:

ModelsStop success rate ↑Go success rate ↑Roll through rate ↓Δpos\Delta_{\text{pos}} ↓Offroad ↓Collision at fault ↓Wrong lane ↓Plan deviation ↓Distance traveled
Base Model44.4%35.1%4.7%4.780.040.030.510.2474.48
Alpamayo 2 Super57.3%38.1%3.2%3.440.040.020.450.2871.24
No mem. (SFT only)48.8%40.5%1%4.220.040.030.540.2478.46
CoT mem. (SFT only)47.3%39.2%0.9%4.390.040.050.550.2477.08
AD-Memo (SFT only)48.8%38.3%1%4.130.040.040.510.2374.19
No mem. (SFT+Da Capo)57.0%42.3%1.3%3.530.040.030.560.2082.57
CoT mem. (SFT+Da Capo)53.0%39.4%1.7%3.850.040.040.570.1979.02
AD-Memo (Ours)59.2%39.5%1.2%3.310.040.040.580.1879.35

The result shows that AD-Memo achieves higher stop success rate and lower Δpos\Delta_{\text{pos}} with better or similar safety metrics (offroad, collision at fault, wrong lane, plan deviation). We hypothesize that such safety metrics can be further improved if we integrate these terms into the RL reward (for now we only use distance to ground truth for driving reward).

Thank you for the detailed responses. The authors have addressed most of my concerns through additional experiments and clarifications. I will maintain my positive score.

We thank the reviewer for the appreciation of our rebuttal and for maintaining a positive evaluation towards our paper. We have added the new results and addressed suggestions from you and the other reviewers in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the AC and the reviewers for their time and feedback. We also thank the reviewers for highlighting the strengths of our paper, including the clear presentation (all reviewers), the high informativeness (reviewer eGXW, ZNwV), the intuitive, reasonable and promising approach (reviewer eGXW, ZNwV, and t4ke on RL training), and the interesting and diverse empirical results (reviewer ZNwV).

As the discussion period is closing, we briefly summarize our response below:

We updated our paper and supplementary materials to address the reviewers’ questions:

  1. We added closed-loop evaluation results with Alpasim to Sec. 4.1, showing that our method, even with out-of-distribution input, achieves competitive performance while attaining comparable safety metrics (all reviewers);

  2. We added discussion and empirical results on inference cost to Appendix E.1, which shows only 11-12% overhead (reviewer eGXW, t4ke);

  3. We added a comparison to vector-based memory baselines in Appendix D.12, which shows that such baselines do not work well in our scenario (reviewer eGXW, t4ke);

  4. We analyzed the propagation of incorrect memory in Appendix D.13 and Fig. 20/21, which shows our method is reasonably robust (reviewer ZNwV, t4ke);

  5. We updated the supplementary material with clearer visuals (reviewer t4ke).

We also want to emphasize that:

  1. We propose both the RL algorithm and the language memory. Performance gains due to either component are our contributions (reviewer ZNwV, t4ke on performance);

  2. Language memory is more explainable and portable than vector-based memory, easier to use in production-level foundation models without architecture change, better suitable for longer-term memory (e.g., traffic regulations) that lasts minutes to hours, and enables the model to focus on high-level reasoning where it is good at (reviewer t4ke on benefits of language memory);

  3. Our contribution in this work also includes an agentic data pipeline based on decision graphs and a tool-use harness (Appendix C.2.1) to curate challenging clips with memory and VQA questions, which is less noticed by the reviewers.

We hope these updates address all reviewers’ questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.