MeMix: Selective State Preservation for Streaming 3D Reconstruction

Streaming 3D reconstruction must absorb new observations while retaining historical context in bounded memory. This requires deciding not only how strongly to update a recurrent state, but which locations should temporarily avoid a write and when to reselect them. We present MeMix, a training-free module that allocates a fixed preservation budget using current-frame compatibility. At each frame, it scores candidate state locations against the observation, preserves the previous values at the highest-scoring locations, and leaves the remaining locations on the original update path. MeMix adds no learnable parameters and keeps recurrent memory constant in stream length. Matched interventions show that compatibility-guided selection and current-frame reselection improve long-horizon performance beyond gate binarization or a fixed protected subset. We evaluate MeMix across update rules sharing a pretrained CUT3R checkpoint. On 16 1,000-view 7-Scenes streams, MeMix reduces TTSA3R's accuracy and completeness errors by 29.9% and 30.7%. With the FILT3R Kalman update, it lowers 1,000-view ScanNet pose ATE by 17.8% across 65 scenes.