acceptodds
Withdrawn from ICLR 2027

SWIM: Sparse Weight-Injected Memories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Persistent visual memory enables vision-language models to answer multiple queries about an image without repeatedly processing visual tokens. Existing approaches encode such memories as low-rank weight updates. We show that low-rank memory updates are insufficient, particularly for dense-text and OCR-intensive tasks. We introduce a training-free diagnostic that measures the minimum activation-subspace rank required to retain a target level of image-conditioned performance. Guided by this analysis, we propose sparse weight-injected memories (SWIM) which replace low-rank updates with sparse, high-rank updates. Under the same hypernetwork output budget, SWIM provides substantially greater rank than LoRA, improving exactly the rank-hungry benchmarks the diagnostic flagged, by and normalized dense-text accuracy across two backbones, while preserving general-perception accuracy. These results expose the low-rank bottleneck in persistent visual memory and show that SWIM overcomes it without generating more parameters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hypernetworks, LoRA, Low-rank adaptation, Vision-language models, Weight-space memory, Model adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dnCiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Sparse Weight-Injected Memories explores how to store visual information directly in LLM weights, allowing models to answer future questions without repeatedly processing image tokens.

Existing approaches use hypernetworks to generate LoRA updates, but their low-rank structure may limit the amount of visual information retained, especially for dense-text and OCR tasks.

The paper first introduce a training-free diagnostic to estimate the rank required to preserve image-conditioned performance by analyzing changes in hidden activations. They then propose SWIM, which replaces low-rank updates with sparse, high-rank weight updates under the same parameter-generation budget. Fisher information is used to select important weight locations.

Experiments on two model backbones and 11 vision-language benchmarks show that SWIM improves normalized dense-text accuracy by 5.3% and 10.5% over LoRA, while largely maintaining general visual perception performance.

Critical strengths

A useful diagnostic for rank requirements: The paper introduces a training-free diagnostic that estimates the rank needed to preserve image-conditioned performance by analyzing changes in hidden activations. The results show that document understanding and OCR tasks require more independent directions than general visual perception tasks, providing experimental motivation for high-rank weight updates.

Critical weaknesses

Limited direct evaluation of persistent visual memory: the paper mainly evaluates single-query visual question answering. It does not systematically test whether one weight update generated from an image can support multiple diverse, unseen queries, or whether memories from multiple images can be combined reliably. I suggest evaluating one-time memory writing followed by multiple unseen queries, with comparisons against visual-token input and KV-cache reuse.

Questions for authors

Can a single weight update reliably support multiple diverse and unseen queries about the same image? The main experiments evaluate visual question-answering accuracy, but do not explicitly measure how much information a single persistent weight update retains across multiple unseen queries. Could the authors conduct a one-image, one-write, multiple-query evaluation, where each image is encoded into weights only once and then queried with diverse questions without accessing the original image? Comparing SWIM, LoRA, and visual-token baselines would provide more direct evidence for the claimed persistent visual memory capability.

Reviewer_W2ZUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies persistent visual memory for vision-language models, where an image is encoded into model weights so that subsequent questions can be answered without repeatedly inserting visual tokens. The authors argue that hypernetwork-generated LoRA updates impose a rank limitation that is particularly harmful for dense-text and OCR tasks.

The paper proposes a training-free activation-rank diagnostic and Sparse Weight-Injected Memories (SWIM), which generates values at fixed sparse locations in the language model weights. Under the same generated-output budget, SWIM can produce higher-rank updates than LoRA. Experiments on two backbone configurations and eleven benchmarks show improvements on dense-text/OCR tasks while keeping general-perception performance close to the LoRA baseline. The paper also studies a hybrid SWIM + LoRA design, mask selection, different output budgets, and inference efficiency.

Critical strengths

S1. The paper compares LoRA, SWIM, and SWIM + LoRA under the same generated-output budget while keeping the image encoder, language model, active sites, training data, and training schedule fixed within each backbone. This gives a relatively clean comparison of update geometry.

S2. The evaluation covers two backbone configurations and eleven vision-language benchmarks, including both general-perception and dense-text/OCR tasks. The results show a meaningful pattern: SWIM improves dense-text performance while remaining close to LoRA on general perception. Table 1 reports dense-text normalized scores of 105.3% and 110.5% for SWIM on the two backbones, while general-perception scores remain 97.9% and 99.2%.

Critical weaknesses

Weakness 1

In Sections 3.2–3.3 and Figure 3, the diagnostic constructs a separate activation basis for each image using that image’s native benchmark questions together with 256 additional questions from other images in the same benchmark. The native questions are then used for evaluation, while the additional questions are used to fit the basis.

This protocol is reasonable for measuring an optimistic, benchmark-conditioned activation capacity, and the paper explicitly describes the diagnostic as optimistic. However, it does not directly measure the rank required by a deployable image-only memory for arbitrary future queries. The basis is constructed from question-dependent activation corrections, whereas SWIM must generate its update from the image before the downstream query is known.

Moreover, the SVD projection minimizes activation reconstruction error in Frobenius norm. It is not itself an optimization over rank-(k) weight updates or task accuracy, and the projected activation correction is not necessarily realizable by one shared rank-(k) weight update for all future queries. Therefore, the diagnostic supports an optimistic upper-bound-style analysis, but the stronger interpretation that it directly measures the rank required by a trained hypernetwork is not fully demonstrated.

The diagnostic is also performed on a frozen Qwen2.5-VL-7B model, whereas the trained SWIM comparisons use SigLIP-2 + Llama-3 and CLIP + Vicuna. Figure 1 shows a qualitative relationship between diagnostic rank demand and SWIM gains, but the paper does not quantify this relationship or establish that it transfers across the trained backbones.

Weakness 2

The main matched-budget results in Table 1 and Table 4 cover two backbone configurations. However, the Fisher-versus-random mask ablation in Table 6 and the wider capacity sweep in Table 7 are reported only for CLIP + Vicuna. The diagnostic and the lossless image-token-removal check in Appendix A are based on Qwen2.5-VL-7B.

Thus, the paper provides evidence that SWIM works on two backbones, but the separate claims about Fisher mask selection, capacity trends, and diagnostic-guided design are each supported by narrower settings. These results are useful, but the conclusions should be scoped accordingly.

Questions for authors

Question 1

Could the authors clarify whether the rank diagnostic is intended as an optimistic benchmark-conditioned probe rather than a direct estimate of the rank required for unseen future queries? A query-disjoint evaluation, or a quantitative correlation analysis between diagnostic rank demand and SWIM gains across the trained backbones, would help establish the strength and scope of the predictive claim.

Question 2

Could the authors clarify whether the Fisher-mask and capacity-sweep conclusions are expected to transfer to the SigLIP-2 + Llama-3 backbone? If these analyses are specific to CLIP + Vicuna, the corresponding claims should be stated more narrowly.

Reviewer_tWe7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the rank requirements of persistent visual memories generated by hypernetworks. The authors propose a training-free projected-rollout diagnostic that compresses image-induced activation corrections to estimate the rank required by different visual tasks. The analysis suggests that dense-text and OCR tasks require higher rank than general-perception tasks.

Based on this observation, the paper proposes SWIM, which replaces LoRA’s low-rank update with independent values written to a fixed sparse support. Under matched hypernetwork-output budgets, SWIM improves dense-text/OCR performance across two backbones while largely preserving general-perception performance. The paper also reports latency and context-length advantages over visual-token prefixing.

Critical strengths
  1. Novel and well-motivated perspective. The paper identifies update rank as a potentially important bottleneck for query-agnostic visual memories and introduces an informative diagnostic.

  2. Carefully matched empirical comparison. LoRA, SWIM, and SWIM+LoRA use the same output budget, training recipe, active sites, and backbones. The consistent dense-text improvements across two backbones and multiple budgets support the usefulness of sparse injection.

Critical weaknesses
  1. The diagnostic does not directly establish a LoRA-rank lower bound. It constrains the complete image-induced activation correction at every layer, whereas LoRA applies multiple low-rank updates across many layers and modules. Their composition through attention, residual connections, and nonlinearities may produce activation changes with rank higher than the rank of each individual update. Moreover, the SVD basis minimizes Frobenius reconstruction error, not downstream task error, so the resulting accuracy curve is not necessarily a strict upper bound on all rank-k modules.

  2. Important empirical and systems evidence is incomplete. The diagnostic basis is fitted using each image’s native evaluation questions, making the claimed query-agnostic interpretation partly transductive. The paper also reports only one training seed and does not measure the numerical rank of the learned SWIM updates. Finally, the efficiency comparison reports context and KV-cache savings but does not include the storage cost of approximately 8.9M generated values per image or the cost of switching between stored memories.

Questions for authors
  1. How does the per-layer activation-rank constraint imply a rank requirement for a system containing 40 low-rank updates across multiple nonlinear layers?

  2. Why is the projected-rollout accuracy a strict upper bound when truncated SVD optimizes Frobenius error rather than benchmark accuracy?

  3. What happens when the diagnostic basis is fitted without using the native evaluation questions of each image?

  4. What are the measured numerical ranks or singular-value spectra of the learned SWIM updates, rather than their maximum structural rank?

  5. What is the per-image storage and switching cost of the generated weight updates, and is it included in the efficiency comparison?

  6. How sensitive are the reported improvements to training randomness, particularly the smaller general-perception differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.