Persistent Memory Bridges Occlusions in Video Object Segmentation

Memory-based video object segmentation models such as SAM2 condition each frame on a fixed-size window of recent memory frames. We show that this fixed memory horizon is a substantial, previously unaddressed cause of a costly failure: once an object is occluded for longer than the window, every pre-occlusion memory frame has been evicted and the model falls back on a stale initialization it rarely recovers from. We introduce Persistent Memory Reinjection (PMR), a training-free mechanism that keeps a small buffer of pre-occlusion memory alive past the window and reinjects it into the model's own cross-attention when the recency window is incomplete, an exact and free occlusion signal whenever tracking is interrupted; for unconstrained video, where memory is written on every frame, we evaluate always-on and confidence-triggered variants of the same mechanism. On a controlled occlusion benchmark over 1500 MOSEv2 sequences, PMR improves post-occlusion region accuracy by up to 17 $\mathcal{J}$ points (up to 6 on DAVIS-2017), bootstrap-significant at every occlusion length beyond the window, while leaving continuous-tracking accuracy exactly unchanged. Using only the four freshest pre-occlusion frames, PMR matches an infinite-horizon oracle that attends to all past memory, showing that of the re-acquisition accuracy recoverable without new supervision, the bounded horizon rather than memory content or capacity is the main limiting factor. PMR composes with fine-tuning, which improves continuous tracking but not re-acquisition, and extends to naturally occurring failures of several types.