Memorize to Accelerate: BEV Action Engram for Efficient Diffusion-based VLA Planning in Autonomous Driving

Diffusion-based vision-language-action (VLA) planners generate driving trajectories through iterative denoising, but aggressive reduction of the denoising budget degrades planning quality. We observe that each step re-derives two kinds of reusable structure: short-horizon motion primitives that recur across scenes, and refinement dynamics that recur across sampling steps; the Transformer stores only these two in its weights. This redundancy motivates a process-level memory, BEV Action Engram, that supplies scene-conditioned action embeddings during decoding. Its two branches address the two redundancies in turn: a Spatial branch maps local motions to BEV displacement n-grams and addresses a learnable embedding table via cross-modal condition hashing for geometric disambiguation, while a CrossStep branch captures the shared refinement dynamics across sampling steps; the resulting features are injected into intermediate layers through gated residuals. In addition, we adapt flow matching group relative policy optimization to this planner and study the memory under reinforcement fine-tuning. Experiments on the NAVSIM dataset show that BEV Action Engram can be learned jointly with the decoder under imitation learning and reused as an action prior during reinforcement fine-tuning. At matched planning quality, the planner uses 40% fewer sampling steps, about $38\%$ lower planner FLOPs, and about 14% lower planner wall-clock, improving the step–compute–accuracy trade-off of the generative decoder. When the decoder is fine-tuned under the same EPDMS reward, removing the memory raises ego progress while the collision-related factors drop and the aggregate score falls below its imitation-learning starting point, whereas retaining the memory restores the safety factors and reaches 88.9 EPDMS on NAVSIMv2, and 89.6 PDMS on NAVSIMv1.