SWIM: Sparse Weight-Injected Memories

Persistent visual memory enables vision-language models to answer multiple queries about an image without repeatedly processing visual tokens. Existing approaches encode such memories as low-rank weight updates. We show that low-rank memory updates are insufficient, particularly for dense-text and OCR-intensive tasks. We introduce a training-free diagnostic that measures the minimum activation-subspace rank required to retain a target level of image-conditioned performance. Guided by this analysis, we propose sparse weight-injected memories (SWIM) which replace low-rank updates with sparse, high-rank updates. Under the same hypernetwork output budget, SWIM provides substantially greater rank than LoRA, improving exactly the rank-hungry benchmarks the diagnostic flagged, by $5.3\%$ and $10.5\%$ normalized dense-text accuracy across two backbones, while preserving general-perception accuracy. These results expose the low-rank bottleneck in persistent visual memory and show that SWIM overcomes it without generating more parameters.