T²Mem: Learning Test-Time Memory for Robotics

Memory-dependent robotic manipulation requires policies to use information that is no longer available in the current observation. Retaining history alone is insufficient: memory must preserve information that supports future actions. We introduce T²MEM, a framework that develops this capability within a pretrained vision-language-action policy, without external reasoning models or memory-specific annotations. T²MEM uses test-time training to encode observation history into compact fast weights through online self-supervised updates, avoiding repeated processing of the full history. An observation-grounded interface extracts vision-language information for memory formation and supplies retrieved context to the action expert. Action supervision shapes what the memory learns to retain and use, while alternating memory–policy learning gives each component a fixed counterpart during optimization. Across 16 RoboMME tasks, T²MEM improves average success from 17.93% to 56.83% over the memory-free base policy and outperforms the recurrent-memory methods reported in the benchmark, while controlled profiling indicates at least 3x inference speedup over explicit methods.