- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Sparse Weight-Injected Memories explores how to store visual information directly in LLM weights, allowing models to answer future questions without repeatedly processing image tokens.
Existing approaches use hypernetworks to generate LoRA updates, but their low-rank structure may limit the amount of visual information retained, especially for dense-text and OCR tasks.
The paper first introduce a training-free diagnostic to estimate the rank required to preserve image-conditioned performance by analyzing changes in hidden activations. They then propose SWIM, which replaces low-rank updates with sparse, high-rank weight updates under the same parameter-generation budget. Fisher information is used to select important weight locations.
Experiments on two model backbones and 11 vision-language benchmarks show that SWIM improves normalized dense-text accuracy by 5.3% and 10.5% over LoRA, while largely maintaining general visual perception performance.
Critical strengths
A useful diagnostic for rank requirements: The paper introduces a training-free diagnostic that estimates the rank needed to preserve image-conditioned performance by analyzing changes in hidden activations. The results show that document understanding and OCR tasks require more independent directions than general visual perception tasks, providing experimental motivation for high-rank weight updates.
Critical weaknesses
Limited direct evaluation of persistent visual memory: the paper mainly evaluates single-query visual question answering. It does not systematically test whether one weight update generated from an image can support multiple diverse, unseen queries, or whether memories from multiple images can be combined reliably. I suggest evaluating one-time memory writing followed by multiple unseen queries, with comparisons against visual-token input and KV-cache reuse.
Questions for authors
Can a single weight update reliably support multiple diverse and unseen queries about the same image? The main experiments evaluate visual question-answering accuracy, but do not explicitly measure how much information a single persistent weight update retains across multiple unseen queries. Could the authors conduct a one-image, one-write, multiple-query evaluation, where each image is encoded into weights only once and then queried with diverse questions without accessing the original image? Comparing SWIM, LoRA, and visual-token baselines would provide more direct evidence for the claimed persistent visual memory capability.