acceptodds
Under review as a conference paper at ICLR 2027

Key-Gram: Extensible World Knowledge for Embodied Manipulation

Abstract

Language-conditioned manipulation requires both reusable task knowledge and state-dependent visual reasoning. Most vision-language-action policies represent both within a shared backbone or conditioning path, making targeted knowledge extension difficult. We introduce Key-Gram, a conditional-memory framework that extracts short, task-specific key-grams, maps them to learnable memory entries by deterministic multi-head hashing, and injects the retrieved entries into selected Transformer layers through token-wise gating and lightweight convolutional fusion. The memory is organized as independently addressable sub-tables, allowing capacity to grow without changing the backbone architecture and supporting constant-time lookup with respect to total table size. Across RoboTwin2.0, LIBERO/LIBERO-Plus, and real-world dual-arm manipulation, Key-Gram improves both and . Relative gains are on the hard RoboTwin2.0 split, on LIBERO-Plus transfer without target-domain fine-tuning, and on real-world long-horizon tasks. In four-stage continual compositional assembly, Stage 4 terminal-node success rises from to . These results support external linguistic memory as a useful architectural bias for compositional grounding and transfer under task expansion.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Learning, World-Action Model, Language-conditioned Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_fCAYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Key-Gram, an external conditional-memory module for language-conditioned robotic manipulation. An offline language-model parser decomposes instructions into short semantic phrases, which index learnable embeddings through deterministic multi-head hashing. Retrieved embeddings are injected into selected Transformer layers through context-dependent gating and convolutional fusion. The framework also incorporates future visual-latent prediction. Its slot/head table organization is intended to support memory expansion without changing the backbone architecture. Experiments cover RoboTwin2.0, LIBERO/LIBERO-Plus, and real-world dual-arm manipulation, including compositional and continual task expansion. The reported results show substantial improvements in several challenging settings.

Critical strengths
  1. Substantial empirical gains across multiple challenging settings and two backbones. The reported improvements extend across simulation and real-world manipulation. On RoboTwin2.0 Hard, success increases from 58.4% to 75.6% for pi 0 and from 76.8% to 84.4% for pi 0.5 (Table 1). Transfer from LIBERO to LIBERO-Plus without target-domain fine-tuning improves from 53.6% to 72.8% and from 83.9% to 87.7%, respectively (Table 2). The real-world continual assembly results also show appreciable improvements in final-stage completion (Table 5). These results support the practical usefulness of the proposed augmentation in difficult manipulation settings.

  2. The ablations support the usefulness of correctly matched key-gram conditioning. Table 6 includes a trained, parameter-matched +VAE control, shuffled trained memories, CRF-based phrases, and KG without the visual-foresight branch. Correct KG retrieval improves the four-task weighted score from 61.950 to 77.175 when visual foresight is present; without that branch, KG achieves 68.925 compared with 51.000 for the vanilla model. Shuffled memories and CRF phrases achieve lower scores of 58.825 and 68.075. Together with the shared language augmentation described in Appendix B, these controls support the usefulness of the correctly matched KG pathway and the importance of semantic phrase construction and instruction correspondence in this ablation setting.

Critical weaknesses
  1. The experiments do not isolate the storage-specific contribution of the proposed memory architecture from the benefit of semantic key-gram conditioning. The trained +VAE, noVAE, CRF, and shuffled-memory controls in Table 6 provide substantive evidence for the usefulness of the complete KG pathway and correct instruction–memory correspondence. However, none holds the cached semantic key-grams fixed while replacing hashed retrieval with a conventional phrase encoder under comparable fusion and training conditions. Because the lookup-and-fusion mechanism substantially follows Engram (Cheng et al., 2026), this missing comparison leaves the added value of the proposed store unresolved. A focused comparison within the existing four-task ablation setting is needed to assess that storage-specific contribution.

  2. The visual-foresight branch lacks essential training definitions, limiting reproducibility and assessment of the training conditions. Section 3.4 describes future-latent prediction, but the manuscript and appendices do not define the joint training objective, future-target sampling horizon, VAE model, latent-loss formulation, or its weight relative to action learning. The main optimization settings are also incomplete. This is a consequential component in the reported ablation: Table 6 gives a weighted score of 77.175 for the full configuration versus 68.925 without visual foresight. Without these definitions, the reader cannot reconstruct how visual supervision is coupled to action learning. The paper needs the complete objective and training configuration.

Questions for authors

Q1 : Can you provide one control on the existing four-task ablation that uses the same cached semantic key-grams, but replaces hashed retrieval with a conventional phrase encoder while keeping the insertion layers, gating/fusion, and visual-foresight configuration comparable? Please report parameter counts and the scope of parameter matching. Q2 : What are the joint training objective, future-target sampling horizon, VAE model, and relative loss weights for the visual-foresight and action pathways? Please also provide the main optimization settings and training schedule. Additional clarifications Q3: Does memory allocation change across the four stages in Table 5? Please report the per-stage allocation. If tables or slots are added, how are keys routed while preserving existing mappings, and how are the key/value projections adapted? Q4: Which Assembly task definition and success criteria produced Table 3? Does the plate-transfer task in Figure 3 use the same evaluation protocol as the tray-pulling and bread-pouring task in Appendix D? Q5: The vanilla row in Table 6 matches all eight corresponding π₀.₅ Easy/Hard values in Table 7. Was vanilla separately trained and evaluated for the ablation, and does the disjoint-data/checkpoint statement in Section 4.4 and Appendix C apply to this row?

Reviewer_sJ2ROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper augments π0\pi_0 and π0.5\pi_{0.5} with an external conditional memory. An LLM decomposes each instruction into short key-grams, deterministic multi-head hashing maps them to learned memory entries, and gated modules inject the retrieved entries into selected Transformer layers. The method reports gains on RoboTwin 2.0, LIBERO-Plus transfer, and several real-world compositional and continual-manipulation settings.

Critical strengths
  • The idea of representing reusable world knowledge in an external, independently addressable memory is interesting.

  • The proposed memory path is modular and can be attached to two strong VLA backbones without redesigning the action expert. The evaluation is broad, covering two backbones, multiple simulation benchmarks, distribution transfer, and real-world dual-arm tasks.

Critical weaknesses

None

Questions for authors

The paper suggests that Key-Gram can support knowledge extension with the backbone frozen, but all reported experiments use full-parameter training. Does the method remain effective when only the new memory entries are updated? This result would materially affect my assessment of the claimed extensibility.

Reviewer_JRgCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Key-Gram, a memory module for VLA models. It turns instructions into short semantic phrases, retrieves learned embeddings through hashing, and feeds them into selected Transformer layers. Results on RoboTwin2.0, LIBERO/LIBERO-Plus, and a real dual-arm robot show improvements in manipulation, transfer, and sequential task learning.

Critical strengths
  1. Strong results across settings. Key-Gram improves both π0 and π0.5 on several simulation benchmarks and on real hardware. The gains on harder tasks and unseen object–target combinations are especially encouraging.

  2. Useful retrieval ablations. In Table 6, shuffling the learned instruction–memory matches hurts performance substantially. This is good evidence that the model benefits from the retrieved information, rather than just having extra parameters.

Critical weaknesses
  1. Limited methodological novelty. Much of the design follows Engram (Cheng et al., 2026): multi-head hashed lookup, learned memory embeddings, context-dependent gating, and convolutional fusion. The main change is using LLM-extracted task phrases instead of token n-grams. That may be useful for robotics, but without a direct Engram-style baseline on the same VLA backbones, it is hard to see how much is genuinely new.

  2. The extensibility claim is not directly tested. The paper emphasizes adding new knowledge without disturbing what is already stored, yet the continual-learning experiments update the entire model at every stage. They show better sequential fine-tuning, not that new memory can be added while preserving the old backbone and memory. This leaves the paper's central claim only partly supported.

Questions for authors
  1. How does Key-Gram compare with a direct Engram-style token n-gram memory added to the same VLA backbone, using a matched parameter and training budget?

  2. Can you show an actual memory-expansion experiment where the backbone and existing memory are frozen, only new entries are trained, and both old-task retention and new-task performance are measured? Also, how are existing hash addresses preserved if the table size in Eq. (1) is increased?

  3. Could you report the parameter-matched +VAE control from Table 6 on the main RoboTwin2.0 and LIBERO-Plus evaluations? That would make it easier to separate the memory gain from the auxiliary prediction pathway.

Reviewer_NRNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds a memory module, Key-Gram, to VLA policies (π0\pi_0 and π0.5\pi_{0.5}). An external LLM splits each instruction into short phrases (key-grams). Each phrase is hashed to rows of learnable embedding tables, and the retrieved vectors are injected into the visual tokens at three backbone layers through a gate and a convolution. The model also adds a branch that predicts future visual latents. Experiments on RoboTwin2.0, LIBERO, LIBERO-Plus and a real dual-arm robot show clear gains for π0\pi_0 and smaller gains for π0.5\pi_{0.5}, including on unseen object-target pairs and a four-stage continual task.

Critical strengths
  1. The module is a simple add-on and works with two widely used VLA backbones.
  2. The evaluation is broad and includes real-robot experiments with clearly described protocols.
  3. Some controls are useful, such as CRF phrases and the no-VAE variant, and the paper reports its data generation and limitations openly.
Critical weaknesses
  1. The module behaves like stronger text conditioning, not like a reusable memory.

All key-gram embeddings are merged into one vector per sample (Vm∈RB×1×dV_m \in \mathbb{R}^{B\times 1\times d}), and every visual token receives this same vector, scaled by its gate. In effect, this is an instruction-dependent bias that pushes the model to attend more to the instruction. Since the key-grams are already merged at this point, the convolution in Eq. (4) cannot model relations between them as claimed in lines 187 to 190. Simpler ways to strengthen the instruction signal are not compared: appending the key-grams to the prompt, re-weighting attention toward instruction tokens, or FiLM. The random and shuffled controls do not settle this either, because any conditioning method would fail with untrained vectors or a mismatched instruction. Without these baselines, it is unclear why hashing and a separate memory are needed.

  1. The extensibility claim is not supported.

Eq. (1) computes addresses modulo the table size. Enlarging a table therefore sends most existing key-grams to new, untrained rows, which contradicts Section 3.5. Table 5 does not show whether memory actually grows across stages, and it trains all parameters with no fixed-size or frozen-backbone comparison. The better retention may therefore come from stronger conditioning rather than from the memory. Localized writes also do not prevent forgetting. A collision in one head already changes part of an old entry, and the layers that read the memory are retrained at every stage.

  1. The gains are not clearly due to Key-Gram.

Key-Gram adds both the memory and a future-latent (VAE) branch. In the ablation, the VAE branch alone raises the score from 51.0 to 61.95, about 42% of the total gain. However, the paper does not say whether the main baselines include this branch, and it does not report the branch on the main benchmarks. Its loss, horizon and weighting are also not described.

  1. Some results are difficult to rely on.

The Vanilla row of Table 6 is identical to the π0.5\pi_{0.5} results for the same tasks in Table 7, although the ablation is said to use separate checkpoints. All results come from single runs, and the final layer choice rests on a gain of only 0.375 points. With 50 real-robot trials per result, most per-task differences in Tables 3 and 4 are not statistically significant.

Questions for authors
  1. How does Key-Gram compare with simpler ways of strengthening the instruction, such as appending the same key-grams to the text prompt or scaling attention toward instruction tokens, under the same training setup?
  2. Does the memory actually grow across the four stages? If so, how are old addresses preserved, and how does this compare with a fixed-size Key-Gram?
  3. Do the baselines in Tables 1 to 5 include the VAE branch? Please report +VAE-only results on at least one main benchmark and describe the loss used for this branch.
  4. Was the Vanilla row in Table 6 reused from the main run in Table 7? Could the authors also report repeated runs or confidence intervals for a representative subset?

Comments on the paper

1 comment

Dear Reviewers,

Thank you for the review. The authors have responded to the points you raised, and the response period closes on 18 November (AoE). I would be grateful if you could post a brief reply on whether your concerns are resolved, partly resolved, or still open, so that the authors have a fair chance to address them. If anything still needs clarifying, please ask now while they can still respond.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.