acceptodds
Under review as a conference paper at ICLR 2027

SNaP: Self-Narrating Policy with Fast-Weight Context for VLA Control

Abstract

Long-horizon robot manipulation requires recalling past observations and tracking task progress that may not be apparent from the current observation. We introduce SNaP, a self-narrating vision-language-action (VLA) policy that builds context from its own past narrations to guide future decisions. The policy generates short descriptions of its current subtask and relevant task state, using them to condition the current action and accumulating them as context for future narrations and actions. Rather than retaining a growing text history, SNaP encodes past narrations into token-level associations stored in a fixed-size fast-weight narration bank that is updated online during deployment. A separate sensory bank retains proprioceptive context. Readouts from both banks condition subsequent narration and action prediction. The policy thus builds and uses its context online from its own descriptions and observed states. Across three real-robot manipulation tasks requiring object-location recall, cue-based counting, and multi-stage progress tracking, SNaP achieves 90.0% average success, outperforming the strongest evaluated baseline by 23.3 percentage points. Targeted context interventions further show that omitting specific past events changes subsequent subtask predictions, supporting the policy's use of accumulated context.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot memory, test-time training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TL1VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SNaP gives a narration-conditioned VLA two fast-weight memory banks for each episode. One stores token associations from the policy's own subtask descriptions, and the other stores proprioceptive history. Learned queries read a fixed number of context tokens for both the next narration and action prediction. The banks use normalized delta updates; the backbone and address networks stay fixed during deployment. Evaluation covers three real-robot tasks: hidden object locations, visual cue counting, and a two-drink recipe. Average success is 90.0%, compared with 66.7% for MemER. The authors also remove earlier events from the context to test their effect on later subtask predictions.

Critical strengths

The large gap on Cued Bean Scooping is convincing: 28/30 successes for SNaP against 40.0% for MemER and 50.0% for past-sentence conditioning. Here the policy has to remember the earlier cue and count completed transfers, so recalling an object location alone is not enough. On Two-Drink Recipe, 23/30 successes gives some evidence that the benefit extends to a longer sequence with repeated operations. All methods use the same demonstrations, making these comparisons easier to interpret.

I found the interventions in Figure 5 and Appendix D more informative about context use than success rates alone. Removing a blink or a completed scoop changes the predicted count in the expected direction, with the current observation and policy kept fixed. These are small experiments. The authors report subtask-prediction accuracy here and clearly distinguish it from closed-loop task success.

Critical weaknesses

I am less convinced that the experiments identify the benefit of self-narration itself. Every Table 1 variant has the narration bank; there is no comparison with a similarly sized fast-weight memory over sensory or visual features alone. The nonlinear address network gives very large gains, but it also adds capacity, as the authors acknowledge. Past-sentence conditioning is a relevant baseline, though I could not judge the comparison without more detail on its retained history length and training configuration. Since the claimed contribution is how history is represented, I would want a matched control that already gives the policy access to history.

Appendix B.1 leaves me with a further concern about training. Second-stage writes use argmax predictions under teacher forcing, and the action branch still receives annotated current narration. B.3 also reconstructs preceding narration context from annotations. At deployment, autoregressive narrations both build the memory and condition actions. This is a substantial mismatch, with no measurement of its practical effect. The 0.9 confidence gate may help, but the paper has no stage-2 or gate ablation and no analysis of errors committed to narration memory. Even with the encouraging robot results, it is unclear how much the recursive memory depends on the constrained task vocabulary and histories. I recommend weak reject because this issue and the missing memory control leave the main contribution insufficiently established.

Questions for authors
  1. How does a capacity-matched fast-weight policy without historical narration perform on the counting tasks? Could you also compare retaining only the latest narration with retaining the accumulated narration bank?

  2. Please specify what history the past-sentence baseline retains and how it is truncated. Does it receive comparable narration supervision and sequence training?

  3. How much does the second training stage improve performance? During actual rollouts, how often does the confidence gate reject a correct narration or commit an incorrect one?

Reviewer_xPaxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds two episode-level fast-weight memories to a narration-conditioned π0.5 policy. A narration bank stores the policy's own subtask sentences at the token level: the key is the causal hidden state preceding each token, the value is the token embedding, and writes use a normalized delta rule. A sensory bank stores one proprioceptive key-value pair per tick. Each bank is read with fixed learned queries. The resulting 16 readout tokens are inserted after block 9 of the VLM and condition both the next narration and the action chunk. Only the two fast-weight matrices change at test time. On three real-robot tasks, SNaP reaches 90.0% average success, compared with 66.7% for MemER and 54.4% for π0.5 conditioned on past predicted sentences. Context interventions also show that removing a single earlier event changes later subtask predictions.

Critical strengths
  1. The idea of reusing the subtask narration that π0.5 already produces as the content of memory is simple, and it fits naturally into the existing architecture.
  2. The tasks design is clear and the experiment gain is large especially on Cued Bean Scooping.
  3. The intervention study in Q3 is good. Removing one green blink shifts the predicted scoop count from 3 to 2, and removing the banana placement drops target-bin accuracy to roughly chance. This is direct evidence that later narrations depend on the stored context.
  4. The method is described in enough detail to reimplement the memory itself.
Critical weaknesses
  1. The central claim is not tested against the obvious alternative. The paper motivates fast weights by saying a text history "must be truncated or summarized." In these tasks, accepted narrations are deduplicated, so a likely long episode yields at most a few short sentences, which fit easily in the VLM context. The decisive comparison is the same model, trained the same way, with the full list of accepted narrations in the prompt. The "π0.5 + past sentences" baseline may be this comparison, but the paper gives no details: how many sentences are kept, whether training uses annotated or predicted sentences, where they are placed, and how many training steps are used. Without this, a large success rate gap could reflect an under-trained baseline rather than the value of fast-weight storage.
  2. Unequal supervision across baselines. The narration annotations directly encode the hidden variable for each task ("banana in bin 3", "yellow go: 3 scoops", "done; close boba bin"). SNaP and the past-sentence baseline are trained with these labels. It is unclear whether MemoryVLA and MemER receive them. The large gaps to π0.5 and MemoryVLA therefore mix the effect of the memory mechanism with the effect of supervising hidden state in language.
  3. Shortcut through the last narration. The read queries are conditioned on the last accepted narration (Eq. 7). If narrations carry the progress state themselves (like "scoop 2 of 3"), the state can propagate as a chain of sentences without the bank holding anything from earlier steps. Every variant in Table 1 includes the narration bank, so there is no ablation that separates bank content from the last sentence. Please add (a) the last accepted narrationonly, with no banks, and (b) sensory bank only.
  4. Retention vs. task length. The paper defines a tick as one policy step, and execution replans every 5 control steps at 30 Hz, which is about 6 Hz. If I understand correctly, with ρ = 0.99 per tick, the degrading scale index is nearly zero after 1 minute execution. For a long manipulation task like Two-Drink Recipe, whether old associations survive depends on the value scale, which is not reported. Please state the actual tick rate, report the effective weight of early writes at retrieval time, and include a sweep analysis over ρ. As written, it is not clear what mechanism carries information across long-lasting wall-clock time.
  5. Train/test mismatch in what gets written. In stage 2, the narration bank stores argmax predictions under teacher forcing, and the action expert is always conditioned on annotated narrations. At deployment, narrations are decoded autoregressively and fed to both the bank and the action expert. There is no report of narration error rates during deployment, how often the 0.9 confidence gate rejects a write, or how a wrong write ay therefore come from visual writes interfering with sensory content in a shared matrix, not from visual history being unhelpful. The conclusion "storing more history is not always better" does not follow from this setup.
  6. Evaluation scope. The evaluation uses three custom tasks with separately trained single-task policies on one platform, and no standard benchmark, although RoboMME and RMBench are cited. The Two-Drink recipe is fixed (3 boba, 1 red bean), so counts are memorized from training rather than inferred. Inference latency with per-tick autoregressive decoding, narration re-encoding and two bank reads is not reported.
Questions for authors
  1. How is the past-sentence baseline built and trained? How were MemoryVLA and MemER adapted, and do they receive narration labels?
  2. What is the tick rate in practice, and what is the effective weight of a write from the start of a Two-Drink episode at the time it is needed?
  3. What happens if the bank is disabled but the query still sees the last narrow? What about the sensory bank alone?
  4. In Q3, is the context rebuilt from annotations or from the policy's own narrations?
  5. How often are narrations wrong or rejected at deployment, and how often does a wrong write lead to task failure?
1 comment

As the discussion period is closing soon and no response from the author, I will keep my rating.

Reviewer_YWGbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SNaP augments a narration-conditioned VLA policy with two fixed-size fast-weight banks that store token-level narration associations and proprioceptive history. Retrieved context conditions subsequent narration and action prediction, while the banks are updated online with a normalized delta rule. Across three real-robot tasks involving location recall, counting, and progress tracking, SNaP achieves 90.0% average success. Historical-event interventions additionally demonstrate that subsequent subtask predictions depend on accumulated context.

Critical strengths
  1. Strong empirical gains on real robots. SNaP achieves 90.0% average task success, compared with 66.7% for MemER and 54.4% for historical-subtask conditioning. The substantial improvement on cue-based counting provides useful evidence that the proposed system supports progress-dependent control.

  2. Concrete memory design with complementary validation. The method integrates narration and proprioceptive history into a fixed-capacity read–write mechanism. The address-network and context-modality ablations, together with historical-event interventions, provide evidence beyond aggregate task success.

Critical weaknesses
  1. Limited evaluation coverage. Experiments are restricted to three author-designed real-world tasks, without standardized evaluation on benchmarks such as RMBench or RoboMME. This limits reproducibility and evidence that the gains generalize beyond the tested settings. The paper also does not assess whether adding memory degrades performance on tasks with limited historical dependence, such as selected LIBERO tasks.

  2. Insufficient training details and memory analysis. The paper omits key settings, including sequence-window and prefill lengths, the TBPTT truncation interval, and total and per-stage training steps, limiting reproducibility. The computational cost of prefill and treatment of repeated annotated narrations are also unclear when training. Furthermore, it lacks analysis of how memory readouts evolve within and across task stages, and whether these changes reflect meaningful memory updates rather than changes in the queries.

Questions for authors
  1. Additional evaluation: Can the authors provide results on a standardized memory benchmark, such as RMBench or RoboMME, and assess whether adding memory affects performance on tasks with limited historical dependence, such as LIBERO?

  2. Clarification: What are the sequence-window length, history-prefill length, TBPTT truncation interval, and training steps for each stage? How are repeated annotated narrations handled during training, and what is the measured effect of prefill on training time and memory usage?

  3. Address network: The analysis of the address network is insufficient, but it will significantly affect the results in Table 1. (1) Although it is shared between reading and writing, read-side gradients alone do not necessarily ensure adequate supervision of the write mapping. Could the authors clarify the training-time write gates and how subsequent prediction losses backpropagate through memory updates to supervise write addresses? If model-generated narrations are written only when they change, could the backpropagate of writing become too sparse? (2) Furthermore, is the substantial improvement of nonlinear over linear addressing with narration-only memory in Table 1 primarily due to additional model capacity? If the gain persists after controlling for capacity, why cannot linear variation addressing it?

  4. Memory readout analysis: Could the authors analyze how the retrieved representations �0��0� vary within and across task stages, and whether these changes reflect task progress? Controls that hold queries fixed or remove relevant writes would help determine whether the differences arise from meaningful memory updates and support subsequent predictions.

I would raise my score if additional benchmark evaluations, controlled experiments, or ablations provide compelling evidence of the memory mechanism’s effectiveness and explain the key factors underlying the address network’s gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.