Learning What to Remember: Reinforcement-Guided Memory for SAM2 in 3D Medical Segmentation

Recent medical foundation models such as Medical SAM2 have demonstrated strong performance for 3D multi-modal segmentation by propagating information across 2D slices via a memory bank. SAM2 uses a naive first-in-first-out memory, and discards informative long-range context. This leads to a problems, coined context mis-propagation: SAM2 is susceptible to mistakenly propagating the most recent patterns instead of mining past relevant contexts. To this end, we introduce Agentic Frame Selection (AFS), a novel reinforcement learning-based policy that dynamically selects important frames based on their expected contribution to segmentation performance. In both Interactive Segmentation and In-Context Learning settings that match various real-world clinical context, AFS demonstrates state-of-the-arts performance, improving upon recent works by 4.6% in Interactive Segmentation setting and 1.8% in In-Context Learning setting.