SegMem: Segment-wise Procedural Memory with Future-aware Gating for Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation requires conditioning on procedural history. However, naively accumulating all past observations causes inference latency to grow unboundedly, which is intolerable in the real world. This necessitates organizing manipulation memory under a strictly controlled context budget. Prior work typically addresses this by either retaining sparse visual frames, risking collapse when a critical evidence is omitted, or repeatedly compressing the full interaction into a global latent state, where high-frequency read or write cycles induce information interference. These trade-offs highlight $\text{temporal granularity}$ as a central design problem. Our pilot study reveals that segmented recurrence mitigates global state interference, yet its effectiveness remains sensitive to the chosen granularity. To this end, we propose SegMem, a segment-wise memory architecture that recurrently compresses variable-length observation subsequences into fixed-size segment slots. Crucially, to overcome the sensitivity of manual granularity, SegMem employs a future-aware gate that learns when to close an active segment based on its anticipated utility for subsequent action prediction. Under a fixed budget, SegMem significantly outperforms prior methods across both simulation benchmarks and real-world experiments. Further analyses demonstrate that the learned gate surpasses alternative segmentation strategies and remains robust under temporal shifts, establishing adaptive temporal organization as a key principle for memory-augmented robotic manipulation.