acceptodds
Under review as a conference paper at ICLR 2027

Memorize to Accelerate: BEV Action Engram for Efficient Diffusion-based VLA Planning in Autonomous Driving

Abstract

Diffusion-based vision-language-action (VLA) planners generate driving trajectories through iterative denoising, but aggressive reduction of the denoising budget degrades planning quality. We observe that each step re-derives two kinds of reusable structure: short-horizon motion primitives that recur across scenes, and refinement dynamics that recur across sampling steps; the Transformer stores only these two in its weights. This redundancy motivates a process-level memory, BEV Action Engram, that supplies scene-conditioned action embeddings during decoding. Its two branches address the two redundancies in turn: a Spatial branch maps local motions to BEV displacement n-grams and addresses a learnable embedding table via cross-modal condition hashing for geometric disambiguation, while a CrossStep branch captures the shared refinement dynamics across sampling steps; the resulting features are injected into intermediate layers through gated residuals. In addition, we adapt flow matching group relative policy optimization to this planner and study the memory under reinforcement fine-tuning. Experiments on the NAVSIM dataset show that BEV Action Engram can be learned jointly with the decoder under imitation learning and reused as an action prior during reinforcement fine-tuning. At matched planning quality, the planner uses 40% fewer sampling steps, about lower planner FLOPs, and about 14% lower planner wall-clock, improving the step–compute–accuracy trade-off of the generative decoder. When the decoder is fine-tuned under the same EPDMS reward, removing the memory raises ego progress while the collision-related factors drop and the aggregate score falls below its imitation-learning starting point, whereas retaining the memory restores the safety factors and reaches 88.9 EPDMS on NAVSIMv2, and 89.6 PDMS on NAVSIMv1.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, flow-matching, trajectory planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WUiXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper proposes BEV Action Engram, a conditional lookup memory for flow-based autonomous driving planners. Spatial displacement n-grams and prior-step velocity predictions index into embedding tables whose outputs modulate the action decoder, aiming to preserve planning quality with fewer denoising steps under imitation learning and Flow-GRPO fine-tuning.

Critical strengths
  1. The memory architecture is concrete and well-ablated in the imitation learning setting: Table 3 compares against a parameter-matched FFN and systematically ablates branch selection, conditioning signals, and injection depth.

  2. At reduced denoising steps (K=3K=3), Table 8 shows that the model reaches 86.64 PDMS compared to 86.19 for the 5-step baseline while cutting planner FLOPs by ~38% and planner latency by ~14% (yielding a ~4.8% end-to-end latency reduction once the visual backbone is included).

Critical weaknesses
  1. The rollout-based Flow Matching training objective in Section 3.7 and Algorithm 2 is mathematically underspecified and inconsistent with the appendix. Algorithm 2 queries the velocity field on self-generated rollout states x^ts\hat{x}_{t_s} while supervising every step with the constant vector x∗−εx^* - \varepsilon. Under standard conditional Flow Matching, x∗−εx^* - \varepsilon is the conditional vector field along the linear interpolation path xt=(1−t)ε+tx∗x_t = (1-t)\varepsilon + t x^*, not along an arbitrary off-path rollout trajectory. Moreover, Appendix Eq. 27 (p. 16) states that tst_s is sampled continuously during IL, without explaining how continuous time sampling defines sequential integration steps in a KK-step rollout.

  2. Both the efficiency claim and the RL safety gains are confounded by step count and benchmark noise. At any fixed step budget in Table 8, adding Engram substantially increases planner latency (54.3→72.354.3 \to 72.3 ms at K=3K=3; 84.4→122.084.4 \to 122.0 ms at K=5K=5), so any speedup relies entirely on dropping from 5 steps to 3. However, the IL metric differences are small and lack multi-seed variance—which the appendix itself notes is close to NAVSIM evaluation noise. Meanwhile, Table 1 evaluates the Flow-GRPO baseline at K=5K=5 against Engram at K=3K=3, making it impossible to tell whether the reported RL gains come from the memory module or simply from changing the inference step count.

Questions for authors
  1. Please clarify the exact training procedure in Algorithm 2 versus Eq. 27: how are time steps tst_s sampled and integrated during the KK-step training rollout, are gradients backpropagated through the rollout states, and why is x∗−εx^* - \varepsilon a valid supervision target off the linear interpolation path? How does a standard interpolation-trained Flow Matching baseline perform at K=3K=3 and K=5K=5?

  2. What are the Flow-GRPO results when the baseline and Engram are evaluated at the exact same step count (K=3K=3 vs. K=3K=3, and K=5K=5 vs. K=5K=5), and what is the multi-seed variance for the main IL and RL comparisons?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer WUiX: Comment 1, training procedure (W1 and Q1)

We thank the reviewer for recognizing the concrete memory architecture and the systematic IL ablations of capacity, branch selection, conditioning, and injection depth. Your question concerns whether the constant velocity target remains valid on the training states. The ambiguity comes from the term “rollout” in Sec. 3.7 and Alg. 2: during IL, it denotes sequential memory queries, not numerical integration of predicted velocities.

For each shared ($ ε, x^* $) pair, we draw K=3 independent time points from Beta(1.5, 1.0), map them through the noise schedule, and sort them in ascending order. At each tst_s, the query state is xts=(1−ts)ε+ts⋅x∗x_{t_s} = (1-t_s)ε + t_s\cdot x^*. All IL states therefore lie on the linear interpolation path; none is a self-integrated, off-path state. Continuous time sampling specifies these query times, while sorting establishes the noisy-to-clean order. The uniform integration step Δt=1/K\Delta t = 1/K is used only in inference and RL.

At every query, v∗=x∗−εv^* = x^* - ε is the valid conditional FM target for this path. We compute the velocity MSE at each state and average the K losses with equal weight (sum/K). This is an average of conditional FM regression terms sharing a noise–target pair, with the history-dependent memory context supplied by CrossStep.

The states do not depend on predicted velocities. Predictions are detached before being written to the CrossStep buffer, so gradients do not propagate through earlier predictions or a state-integration chain. Each current prediction and the retrieved table entries receive gradients from the corresponding planning loss; the discrete hash indices do not. Thus, the procedure does not require full or truncated BPTT across query steps.

Sequential queries supply CrossStep with model-predicted velocity prefixes, the input type it uses at inference. This avoids constructing training keys from ground-truth velocities while using predicted velocities at test time. Training still queries interpolation states, whereas inference queries self-integrated states; this distinction does not invalidate the on-path IL supervision. We will make the state construction, time sampling, and detach operations explicit in Alg. 2 and App. 1.2.

To distinguish the memory contribution from this protocol, the memory-free baseline, Equal-FFN control, and branch variants all use the same training procedure. A standard random-time interpolation-trained baseline without sequential queries achieves 85.78 PDMS at K=3 and 86.01 at K=5 on navtest-v1, compared with 86.01 and 86.19 for the sequential-query baseline. The protocol comparison shows close single-run scores; the shared-protocol controls provide the direct comparison of memory architectures.

The addressing operations are fixed at initialization, while the embedding contents are learned through the planning loss. Engram therefore combines fixed symbolic addressing with learned contents. We hope these details resolve the mathematical and implementation ambiguity.

AuthorsOpenReview ↗

Reply to Reviewer WUiX: Comment 2, controlled performance and efficiency (W2 and Q2)

Thank you for recognizing the reduced-step planning result and the associated FLOP and latency measurements. Your concern is whether these comparisons isolate the memory contribution and establish a practical latency benefit. We distinguish matched-budget quality gains from the measured latency comparison below.

  1. Matched-budget RL (Q2). Both models were trained with K=5 rollouts. Evaluating the trained checkpoints without retraining gives the following navtest-v2 EPDMS scores. Engram-RL improves EPDMS by 3.94 points at K=3 and 4.04 at K=5. At K=3, NC improves from 94.08 to 98.37 and TTC from 92.99 to 97.79, despite lower EP (87.38 versus 93.04). Thus, the aggregate gain accompanies better safety-related metrics, not greater ego progress, and persists after controlling the step count. Both models are close to their five-step scores at K=3. The low-budget distinction is clearer at K=2: baseline-RL falls from 84.96 to 81.57, whereas Engram-RL changes from 88.90 to 88.78. This supports stronger quality retention at low step counts.
ModelKNCDACDDCTLCEPTTCLKHCECEPDMS
baseline-RL194.7292.4898.01100.0090.0795.6496.3297.0840.1377.17
Engram-RL198.4695.0799.60100.0084.9197.6596.5998.0874.7385.64
baseline-RL291.6895.0598.64100.0094.5490.6094.8197.9580.1681.57
Engram-RL298.3296.7099.53100.0087.6497.7396.9698.3184.7988.78
baseline-RL394.0896.0698.87100.0093.0492.9995.2098.3084.7384.96
Engram-RL398.3796.9099.50100.0087.3897.7997.0798.3184.3388.90
baseline-RL594.4296.0398.86100.0092.8093.2794.9898.3384.3785.02
Engram-RL598.4997.0199.49100.0086.9697.9897.3098.3184.2189.06
  1. Planner quality and measured efficiency (W2). Structured action memory supplies reusable motion and refinement information, and the matched-budget RL results above show its planning-quality contribution. Separately, the latency measurements of IL provide a practical latency result: three-step Engram achieves 86.64 PDMS versus 86.19 for the five-step baseline, with 14.3% lower planner latency (72.3 versus 84.4 ms), including memory overhead. At K=3, the baseline scores 86.01 PDMS and is faster at 54.3 ms, so this saving applies to the reported configuration pair, not acceleration at equal K. The end-to-end reduction is more modest, approximately 4.8%, because the VLM dominates runtime. We will make the planner-level scope and comparison settings clearer, distinguishing the matched-budget quality gains from the measured latency saving.

  2. Memory overhead (W2). The absolute planner timings above were measured on an idle GPU with CUDA synchronization and 50 measurements. Separate profiling at K=3 under concurrent GPU load estimates in table. CPU–GPU synchronization and small-kernel launches are the main identified bottlenecks. These percentages describe the separate profiling run, not a split of the idle-GPU timings. We will document both setups in App. 2.5.

ComponentShare of memory overhead
Hashing and lookup46%
Gated injection54%
  1. Statistical reliability (W2/Q2). The reported scores and 95% scenario-bootstrap CIs are shown in table. The IL point estimates differ by 0.45 PDMS and have overlapping intervals; we describe this result as comparable observed quality, rather than a statistically established improvement. For RL, the 3.88-point gap between three-step Engram and the five-step baseline is accompanied by clearly separated intervals, supporting the reported difference with respect to scenario-sampling uncertainty on 12,146 scenarios. The matched-budget evaluations above address the step-count confound separately: gains remain at 3.94 and 4.04 points at K=3 and K=5. Thus, the RL quality benefit persists after controlling K, while the CIs here describe the original cross-budget comparison.
SettingModelKScore95% CI
IL, navtest-v1 (PDMS)Engram386.64[86.23, 87.05]
IL, navtest-v1 (PDMS)Baseline586.19[85.77, 86.63]
RL, navtest-v2 (EPDMS)Engram-RL388.90[88.51, 89.27]
RL, navtest-v2 (EPDMS)baseline-RL585.02[84.51, 85.56]

We hope the matched-budget results, overhead analysis, and clarified statistical scope help assess the structured-memory contribution and would appreciate reconsideration of the assessment.

Reviewer_WUiXOpenReview ↗

Official Comment by Reviewer

Thank you for your response. My concerns also remain unresolved.

AuthorsOpenReview ↗

Thanks for your response

Thank you very much for taking the time to review our responses and additional experiments. We understand that your concerns remain. We would appreciate any further clarification on which specific points our response has not sufficiently addressed, as this would help us provide a more focused follow-up.

Thanks again!

Reviewer_SpNLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper has a clear motivation, i.e. recurring motion patterns and denoising dynamics should not have to be reconstructed from scratch at every diffusion step. It proposed a memory method, which is promising. It presents thorough experiments to support the proposed memory methods.

Critical strengths
  1. The paper presents a clear and interesting idea: using a conditional action memory to reuse local motion structure and cross-step refinement patterns in diffusion-based driving planners. The BEV displacement representation and the separation between Spatial and CrossStep memory are reasonably motivated.

  2. The method is implemented carefully. The paper includes ablations on the two memory branches, condition hashing, injection layers, and parameter-matched controls, which provide useful evidence that the gain is not solely due to adding parameters.

  3. The step-quality trade-off is promising. The Engram model can match or slightly exceed a five-step baseline with three sampling steps, and the transfer result on a masked-diffusion VLA suggests that the spatial memory is not tied to one particular decoder.

Critical weaknesses
  1. The paper does not sufficiently establish that the memory retrieval is reliable rather than a dataset-specific shortcut. Although condition hashing is intended to disambiguate similar motions in different scenes, there is no analysis of hash collisions, slot utilization, or cases where an incorrect memory prior is retrieved and harms planning.

  2. The reported gains are relatively small and their statistical reliability is unclear. For example, the three-step Engram model improves PDMS only modestly over the corresponding baseline, but the paper does not report repeated runs, standard deviations, or confidence intervals.

  3. The evaluation remains narrow. Most experiments are conducted within the NA VSIM setting, which is non-reactive, and the transfer experiment only considers one additional masked-diffusion VLA. It remains unclear whether the memory is robust under closed-loop interaction, distribution shift, or different driving datasets and backbones.

  4. The practical acceleration benefit is overstated somewhat. The memory reduces the required number of decoder steps, but it adds per-step overhead and the full VLM forward dominates end-to-end latency; the reported end-to-end reduction is therefore much smaller than the planner-level FLOP reduction.

Questions for authors

All questions are present in above weaknesses.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer SpNL: Comment 1, retrieval reliability (W1)

We thank the reviewer for recognizing the clear conditional-memory idea, the motivated Spatial–CrossStep separation, and the useful evidence from the implementation and ablations. Your question concerns whether the retrieved contents help distinguish scene-dependent motion patterns and whether an inappropriate prior can harm planning. We examine the use of learned contents alongside the requested collision and utilization diagnostics.

  1. Conditional memory and learned contents.
  • In the retrained K=3 IL ablations on navtest-v1 (Table 3), the full model scores 86.6 PDMS, versus 86.1 without condition hashing, 85.5 with Spatial alone, 85.1 with CrossStep alone, and 85.9 for the parameter-matched Equal-FFN control. These results support scene conditioning and the combined memory structure rather than added capacity alone.
  • To examine the learned contents at inference, we also intervene on the same Engram-RL checkpoint at K=3 on navtest-v2, without retraining. Row permutation is applied to every embedding table, with null and padding rows fixed. It preserves the addressing scheme and table size but breaks the address–content correspondence. The score decrease supports a contribution from the learned correspondence during inference. These fixed-checkpoint interventions complement the retrained IL ablations. | Inference intervention | EPDMS | Change from full | |---|---:|---:| | Full retrieval and injection | 88.90 | — | | Embedding-row permutation | 87.93 | −0.97 | | Injection disabled | 87.79 | −1.11 |
  1. Collisions and utilization. The 2-gram tables occupy approximately 9.6k of 10,007 slots (96%), and the 3-gram tables reach full occupancy. Among the 100 most frequent n-grams of each order, all have distinct addresses in the corresponding four-head joint space, although individual heads have 0–1 colliding pairs. Thus, multi-head addressing distinguishes the tested frequent patterns despite single-head slot sharing; this is a result for that subset, not the full query space. Replaying navtest ground-truth displacement n-grams against tables constructed from 103,288 training samples gives lookup-weighted null-hit rates of 0.09–0.12% for 2-grams and 0.00% for 3-grams (0.06% overall; 0.40% at slot level). This measures occupied-slot coverage for those queries. Low null-hit rates, especially in fully occupied tables, are not evidence of semantic retrieval correctness.

  2. Planning failures. In the reported scenario analysis, 823 cases move from a near-zero baseline-RL score to an Engram-RL score above 0.5, compared with 225 reverse transitions. The worst negative cases often involve drivable-area compliance or collision-avoidance factors falling from 1 to 0. These outcomes characterize where planning can deteriorate, but do not identify incorrect retrieval as the cause. We will add the diagnostics and qualitative examples to App. 2.3 and App. 2.6.

The results support functional, condition-dependent memory use on NAVSIM. Reliability under distribution shift and closed-loop interaction remains untested, as clarified in the second reply.

AuthorsOpenReview ↗

Reply to Reviewer SpNL: Comment 2, reliability and scope (W2–W4)

Thank you for recognizing the promising quality–step trade-off and the transfer result suggesting that Spatial memory is not tied to one decoder. We clarify the statistical and deployment scope of these findings below.

  1. Statistical reliability (W2). The reported scores and 95% scenario-bootstrap CIs are in the table. The IL point estimates differ by 0.45 PDMS and have overlapping intervals; we describe this result as comparable observed quality, rather than a statistically established improvement. For RL, the 3.88-point gap between three-step Engram and the five-step baseline is accompanied by clearly separated intervals, supporting the reported difference with respect to scenario-sampling uncertainty on 12,146 scenarios. The matched-budget evaluations in the first reply address the step-count confound separately: gains remain at 3.94 and 4.04 points at K=3 and K=5. Thus, the RL quality benefit persists after controlling K, while the CIs here describe the original cross-budget comparison.
SettingModelKScore95% CI
IL, navtest-v1 (PDMS)Engram386.64[86.23, 87.05]
IL, navtest-v1 (PDMS)Baseline586.19[85.77, 86.63]
RL, navtest-v2 (EPDMS)Engram-RL388.90[88.51, 89.27]
RL, navtest-v2 (EPDMS)baseline-RL585.02[84.51, 85.56]
  1. Evaluation scope (W3). NAVSIM is non-reactive, so the current results do not establish robustness to closed-loop interaction or distribution shift. The masked-diffusion transfer (85.9 to 86.3, Sec. 4.7) addresses a narrower question: the same Spatial branch can improve another decoder without changing its memory construction. This supports portability of the design, while validation across driving datasets, additional backbones, and reactive settings remains future work. We will make this distinction clear in the Limitations section.

  2. Efficiency scope (W4). We agree that the end-to-end reduction is modest: 252.1 to 240.0 ms, or 12.1 ms per frame (4.8%), with VLM forward dominating runtime. The paper reports planner-level acceleration, but this distinction should be more prominent. In the IL planner-only rerun, three-step Engram takes 72.3 ms versus 84.4 ms for the five-step baseline, saving 14.3% with close observed scores (86.64 versus 86.19 PDMS). This is a net saving for those operating points, including memory overhead, rather than a general speed advantage: the three-step baseline is faster at 54.3 ms and scores 86.01 PDMS. The matched-budget RL gains of 3.94 and 4.04 EPDMS points separately establish the planning-quality contribution. We will distinguish these RL quality results from the IL latency comparison and state the planner and end-to-end scopes in the abstract, conclusion, and App. 2.5.

We hope the retrieval diagnostics and the clarified reliability and scope address the concerns and would appreciate reconsideration of the assessment.

Reviewer_zzEpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes BEV Action Engram, which augments an autonomous-driving VLA planner with spatial-action and cross-step memories, injecting their features into a DiT decoder through gated residuals. On NAVSIM, the authors report that a 3-step memory-augmented planner matches or exceeds a 5-step baseline, and that retaining the memory during reinforcement-learning fine-tuning improves safety-related metrics.

Critical strengths
  1. Practical motivation. Reducing the sampling steps of a generative planner while preserving planning quality is an important engineering goal.
  2. Domain-specific design. The method represents reusable motion patterns as BEV displacement n-grams and uses a CrossStep branch to capture refinement information across sampling steps. The delayed-injection interface is designed to remain consistent between training and inference.
  3. Useful ablations. The paper reports a parameter-matched FFN control, single- versus dual-branch variants, and ablations of condition hashing and injection layers. These results provide preliminary evidence that the gains arise from the memory structure, rather than simply from adding parameters.
Critical weaknesses
  1. The RL comparison is confounded by different inference budgets. The memory-free RL model is evaluated with 5 sampling steps, while the Engram RL model uses 3. Consequently, the reported EPDMS and safety-metric differences cannot be attributed to the memory alone. Matched-budget RL results are needed to support the claim that Engram preserves safety during reward optimization.

  2. The practical end-to-end speedup is small. The reported 14% latency reduction concerns the planner alone. In the paper’s NAVSIMv1 measurements, end-to-end latency falls from 252.1 ms for the 5-step baseline to 240.0 ms for the 3-step Engram model—approximately 4.8%—because the memory adds per-step overhead and the VLM dominates runtime. The paper should better establish whether this system-level gain is meaningful for real-time deployment.

  3. The comparison with alternative acceleration methods is not controlled. Although the paper reports results for MeanFuser, this is a cross-paper benchmark comparison rather than an evaluation under matched models, training setups, and compute budgets. There is also no experiment combining Engram with MeanFlow or Shortcut-style methods, leaving unclear whether Engram offers a distinct advantage or is complementary to these approaches.

  4. The robustness of small gains is unclear. The paper does not report variation across random seeds or confidence intervals. This is important because some ablation differences are small, and the authors note that differences of this scale may be close to benchmark noise.

  5. Evidence for deployment generalization is limited. Evaluation is confined to NAVSIM; closed-loop driving, multi-view settings, and on-vehicle hardware are not tested. Thus, the results demonstrate gains on the reported benchmark, but do not yet establish that the method improves safety or latency in a deployed driving system.

Questions for authors

Can you report NAVSIMv2 submetrics for memory-free and Engram RL at the same sampling budgets? Does Engram retain its benefits when integrated with a MeanFlow- or Shortcut-based planner?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer zzEp: Comment 1, matched-budget RL submetrics (W1 and Question, first part)

We thank the reviewer for recognizing the practical motivation, the BEV displacement and CrossStep design, and the preliminary evidence from the ablations for a memory-structure contribution beyond added parameters. The requested matched-budget comparison tests whether the RL safety gain persists when step count is held fixed. We report that comparison below.

Both RL models were trained with K=5 rollouts (App. 1.1). The following navtest-v2 results evaluate the trained checkpoints without retraining; submetric names follow Table 1.

ModelKNCDACDDCTLCEPTTCLKHCECEPDMS
baseline-RL194.7292.4898.01100.0090.0795.6496.3297.0840.1377.17
Engram-RL198.4695.0799.60100.0084.9197.6596.5998.0874.7385.64
baseline-RL291.6895.0598.64100.0094.5490.6094.8197.9580.1681.57
Engram-RL298.3296.7099.53100.0087.6497.7396.9698.3184.7988.78
baseline-RL394.0896.0698.87100.0093.0492.9995.2098.3084.7384.96
Engram-RL398.3796.9099.50100.0087.3897.7997.0798.3184.3388.90
baseline-RL594.4296.0398.86100.0092.8093.2794.9898.3384.3785.02
Engram-RL598.4997.0199.49100.0086.9697.9897.3098.3184.2189.06

The gain remains at equal budgets: 3.94 EPDMS points at K=3 and 4.04 at K=5. At K=3, NC improves from 94.08 to 98.37, TTC from 92.99 to 97.79, DAC from 96.06 to 96.90, and LK from 95.20 to 97.07. EP is lower (87.38 versus 93.04), so the higher EPDMS reflects a better safety–progress balance rather than greater progress. The K=5 submetrics show the same pattern. This answers the attribution concern: the safety-related benefit is retained when the inference step count is controlled.

The sweep shows that Engram-RL is better at every tested budget, with gains of 8.47, 7.21, 3.94, and 4.04 points at K=1, 2, 3, and 5. Both models are close to their five-step scores at K=3: baseline-RL scores 84.96 versus 85.02, and Engram-RL scores 88.90 versus 89.06. The distinction is clearer at K=2, where Engram retains 88.78 while the baseline falls to 81.57. These results support higher quality at matched budgets and stronger quality retention at low step counts. We will include this table in the revision.

We hope these results resolve the step-count ambiguity; the second reply addresses the combination question and the remaining concerns.

AuthorsOpenReview ↗

Reply to Reviewer zzEp: Comment 2, efficiency, related methods, and robustness (W2–W5 and Question, second part)

Thank you for recognizing the importance of reducing planner steps while preserving quality and the useful evidence from the controlled ablations. Your remaining questions concern practical benefit, distinction from other acceleration methods, and reliability beyond a single comparison. We address these separately.

  1. End-to-end efficiency (W2). We agree that the end-to-end reduction is modest: 252.1 to 240.0 ms, or 12.1 ms per frame (4.8%), with VLM forward dominating runtime. The paper reports planner-level acceleration, but this distinction should be more prominent. In the IL planner-only rerun, three-step Engram takes 72.3 ms versus 84.4 ms for the five-step baseline, saving 14.3% with close observed scores (86.64 versus 86.19 PDMS). This is a net saving for those operating points, including memory overhead, rather than a general speed advantage: the three-step baseline is faster at 54.3 ms and scores 86.01 PDMS. The matched-budget RL gains of 3.94 and 4.04 EPDMS points separately establish the planning-quality contribution. We will distinguish these RL quality results from the IL latency comparison and state the planner and end-to-end scopes in the abstract, conclusion, and App. 2.5. Whether the measured savings meet a particular real-time deadline requires deployment-specific validation.

  2. Distinct contribution and combinations (W3 and Question). We appreciate this valuable question, which concerns both composability with MeanFlow/Shortcut and their competitiveness as alternatives. Engram improves the planning quality–efficiency trade-off through structured action memory: under our evaluated flow formulation, three-step Engram achieves higher PDMS than the five-step baseline (86.64 vs. 86.19) with lower measured planner latency (72.3 vs. 84.4 ms). Same-step comparisons, the parameter-matched Equal-FFN control, and matched-budget RL results further support the contribution of memory structure. Mechanistically, spatial memory remains available within a single decoder evaluation, whereas CrossStep’s history-based contribution requires previous velocity predictions. Our existing (K=1) result (84.81→85.29 PDMS) is a limited observation under the current flow formulation. We will consider integration with MeanFlow/Shortcut in future work.

  3. Statistical reliability (W4).Statistical reliability (W4). The reported scores and 95% scenario-bootstrap CIs are in the table. The IL point estimates differ by 0.45 PDMS and have overlapping intervals; we describe this result as comparable observed quality, rather than a statistically established improvement. For RL, the 3.88-point gap between three-step Engram and the five-step baseline is accompanied by clearly separated intervals, supporting the reported difference with respect to scenario-sampling uncertainty on 12,146 scenarios. The matched-budget evaluations in the first reply address the step-count confound separately: gains remain at 3.94 and 4.04 points at K=3 and K=5. Thus, the RL quality benefit persists after controlling K, while the CIs here describe the original cross-budget comparison.

SettingModelKScore95% CI
IL, navtest-v1 (PDMS)Engram386.64[86.23, 87.05]
IL, navtest-v1 (PDMS)Baseline586.19[85.77, 86.63]
RL, navtest-v2 (EPDMS)Engram-RL388.90[88.51, 89.27]
RL, navtest-v2 (EPDMS)baseline-RL585.02[84.51, 85.56]
  1. Deployment scope (W5). The results establish matched-budget RL quality gains and latency savings at the reported IL operating points on NAVSIM. Decoder transfer tests architectural portability, and the approximately 11 MB FP16 memory footprint (about 0.3% of the backbone footprint, App. 2.7) quantifies storage cost. Neither establishes on-vehicle safety or real-time performance. Closed-loop, multi-view, and on-vehicle validation remain future work, and we will distinguish these directions in the Limitations section.

We hope the controlled results and clarified scope address the concerns and would appreciate reconsideration of the assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Response

We sincerely thank the reviewers and the Area Chair for their time and constructive feedback. Below, we summarize the key clarifications and additional experiments provided in our responses.

The core contribution of BEV Action Engram is a structured action-memory mechanism that enables the planner to reuse spatial motion patterns and cross-step refinement information internally.

Our responses clarify the training procedure and examine whether the observed gains persist at matched sampling steps and extend beyond increased parameter count.

  1. Training procedure (WUiX). In IL, the queried states follow the prescribed linear interpolation path; they are not obtained by integrating predicted velocities. The constant conditional flow-matching target therefore remains valid. Predicted velocity prefixes are detached before entering CrossStep.

  2. Quality at matched sampling steps (WUiX, zzEp). Both RL checkpoints are trained at K=5. On navtest-v2, Engram-RL improves EPDMS over baseline-RL by 3.94 points at inference K=3 and 4.04 points at K=5. At K=2, Engram-RL scores 88.78, compared with 81.57 for baseline-RL. These results show that the gains persist at equal sampling-step counts, including at low-step inference settings.

  3. Evidence for the memory contribution (WUiX, SpNL). Under the same IL protocol and evaluation budget, full Engram achieves 86.6 PDMS, compared with 85.9 for the parameter-matched Equal-FFN control. Separately, permuting the embedding-table rows of the same RL checkpoint reduces EPDMS from 88.90 to 87.93. These experiments provide complementary evidence for a benefit beyond parameter count and for the role of the learned address–content mapping.

Our individual responses also report measured efficiency and retrieval diagnostics, clarify the scope of the statistical evidence, and discuss decoder transfer. We have incorporated these clarifications and additional results into the revised manuscript (highlighted in blue).

We sincerely hope that our responses and supplementary experiments help demonstrate the effectiveness of BEV Action Engram and address the concerns raised by the reviewers. We respectfully ask the reviewers and the Area Chair to consider this additional evidence alongside the original reviews and our detailed responses when assessing the contribution of our work. We thank all reviewers and the Area Chair for their careful consideration.

Best regards,

Authors of Submission 28581

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.