AdaMem: Action-Utility Guided Memory Management for Budgeted Long-Horizon Vision-Language-Action Policies

Long-horizon robotic manipulation requires preserving task-relevant information beyond the current observation, yet VLA policies operate under limited temporal context and deployment resources. Existing event-memory methods mainly focus on memory admission, while post-admission maintenance is often handled by simple replacement heuristics. We introduce ADAMEM, an action-utility guided memory manager that learns to maintain a bounded episodic memory with a frozen VLA backbone. For each incoming event, ADAMEM evaluates feasible retain, merge, and evict operations according to their predicted effect on future action quality, using counterfactual future-action supervision. ADAMEM-SELECT learns budget-aware raw-frame selection, while ADAMEM-FULL further supports compact representations, temporally ordered merging, and gated latent-memory integration. We also provide a finite-candidate regret analysis and evaluate both offline action-proxy quality and closed-loop performance. Across eight RoboTwin-MeM tasks, ADAMEM consistently improves performance under matched memory budgets. At K = 3, ADAMEM-FULL achieves 90.1% closed-loop success versus 80.9% for Raw-FIFO. These results show that learned memory maintenance can improve long-horizon VLA execution without modifying the action backbone.