Key-Gram: Extensible World Knowledge for Embodied Manipulation

Language-conditioned manipulation requires both reusable task knowledge and state-dependent visual reasoning. Most vision-language-action policies represent both within a shared backbone or conditioning path, making targeted knowledge extension difficult. We introduce Key-Gram, a conditional-memory framework that extracts short, task-specific key-grams, maps them to learnable memory entries by deterministic multi-head hashing, and injects the retrieved entries into selected Transformer layers through token-wise gating and lightweight convolutional fusion. The memory is organized as independently addressable sub-tables, allowing capacity to grow without changing the backbone architecture and supporting constant-time lookup with respect to total table size. Across RoboTwin2.0, LIBERO/LIBERO-Plus, and real-world dual-arm manipulation, Key-Gram improves both ${\pi }_0$ and ${\pi }_{0.5}$. Relative gains are $29.5\%/9.9\%$ on the hard RoboTwin2.0 split, $35.8\%/4.5\%$ on LIBERO-Plus transfer without target-domain fine-tuning, and $15.4\%/8.1\%$ on real-world long-horizon tasks. In four-stage continual compositional assembly, Stage 4 terminal-node success rises from $20\%/52\%$ to $58\%/74\%$. These results support external linguistic memory as a useful architectural bias for compositional grounding and transfer under task expansion.