acceptodds
Under review as a conference paper at ICLR 2027

PixelMem: Persistent Images for conteXt Efficient Language-model Memory

Abstract

For language models to support long-term user interactions, they need to retain and reuse user information, past events, and procedural knowledge. These memories require storage space, and reading them back into the model occupies its limited input capacity. Rendering text as images can shorten model inputs, but may enlarge the storage cost. This motivates a memory format that supports both compact storage and short model inputs. We introduce PixelMem, which encodes information from text memories into the pixel values of images. A trained visual reader maps each image to a small number of continuous memory slots, which the language model uses together with the question to generate an answer. The image size remains fixed, while the reader determines how many input positions the memory occupies. We evaluate storage, input length, and answer accuracy on four types of tasks, ranging from simple facts to detailed procedures. Each question is answered from a single memory. With 16 memory input positions, PixelMem uses 74.25% fewer input cost than full text and 93.98% less storage than rendered-text images under the same input budget. Its accuracy is 89.4%, only 3.1 percentage points below full text. In a separate comparison with memories stored as vectors, it reaches 76.2% accuracy with four memory input positions, exceeding the vector baseline's 48.5% by 27.7 percentage points. These results suggest that small pixel images can preserve information useful for answering questions while requiring few model input positions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, persistent memory, context compression, pixel-based memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_9eFWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PixelMem stores a frozen BGE embedding as a projected, quantized 16×16 RGB image. A trained reader maps its global and local features into a small number of continuous memory slots for a LoRA-adapted language model. Evaluation uses four categories of synthetic single-memory QA to compare answer accuracy, persistent storage, and memory input length across text, rendered-image, vector, and pixel representations. Additional analyses examine slot budgets, reader-branch interventions, and reuse of stored images across two separately adapted language models.

Critical strengths
  1. Useful evaluation. Separating persistent storage from reading-time context occupancy exposes trade-offs that token compression alone misses.
  2. Informative memory interventions. Donor-memory substitutions show that both reader branches transmit memory-specific information, strengthening the evidence beyond ordinary branch removal.
Critical weaknesses
  1. The advantage over direct vectors remains unclear. Matched reader and training conditions are not established, so the contribution of the image representation is not isolated.
  2. The advantage of two-dimensional organization is unverified. Branch interventions demonstrate use of local features, but not superiority over alternative arrangements of the same numerical code.
  3. Broader benchmark evaluation is missing. Evaluation is limited to template-generated QA, leaving generalization to established conversational and long-term memory benchmarks untested.
Questions for authors
  1. Direct-vector comparison: Does PixelMem retain its advantage over direct vector baselines under matched reader and training conditions?
  2. Spatial organization: Could you compare the same numerical code under two-dimensional, one-dimensional, and alternative arrangements to assess the benefit of spatial organization?
  3. Benchmark coverage: Could you evaluate on LoCoMo and LongMemEval, optionally supplemented by VTCBench’s memory track, to assess generalization beyond template-generated QA?
Reviewer_AretOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors present an alternative method to store the output of an existing, frozen textual encoder, transforming it into small png images of 16x16 rgb pixels. The transformation consists in a frozen random projection and a clipping mask.

After training a reader that transform the pixel images back into K "soft-token" vectors to be processed by the backbone of existing LLMs, they claim that this storage method is more efficient than other competing paths, such as storing the full original text, the raw vector encodings or transforming it into rendered images.

Critical strengths

None.

Critical weaknesses

I have deep concerns about the core elements of the paper.

First of all, the idea of splitting text into chunks and encoding them into vectorial form before inserting them back into LLM backbones, is clearly not new. The encoder itself, among other things, is used with frozen parameters, and is just one of the many pre-trained encoders available off-the-shelf.

The "original" contribution of the paper is the idea to store the resulting encoded vectors into pixel images. My criticisms will focus on this specific aspect. The authors' central claim is that storing the processed pixel images is strictly better than storing the raw vector.

This idea, to me, is deeply flawed for two main reasons: i) the processed images are the same size of the encoded vector ii) they have a strictly inferior informational content with respect to the original raw vectors.

In the following I will go in deeper detail.

First, the textual input to be compressed and stored is the output of a text encoder, specifically a vector of dimension 384. This vector, stored as FP16, occupies 768bytes. The proposed pixel image, of 16x16 8bytes rgb pixels, occupies 768bytes. This is acknowledged by the authors themselves in the paper. So, from the point of view of storage, there is no gain.

Furthermore, the projection from the vectors to the pixel images is a frozen, randomly selected orthogonal matrix. Projection is followed by a clipping mask. This means that the information stored in the image is strictly lower than the original vector. Being a 2D image, moreover, should not introduce any additional useful structure: the original vector is not 2D and the projection is random. Why is there a "Conv2d with kernel size 2 and stride 2" to process an image that cannot have any local 2D structure?

Despite of this, the paper shows that the performance on downstream task of a trained reader on the pixel image is better than that of the original vector.

However, nowhere in the text it is clarified whether the computational capabilities of the readers are equivalent for the pixel image and the original vector. The authors also acknowledge that "The comparison therefore evaluates each stored representation together with its adapted reader and language model; it does not isolate the effect of pixel encoding alone."

In my opinion the discrepancy between the reader capacity, or training pipeline, is the most likely explaination of the seemingly contradictory results of the paper that using a randomly processed, informationally inferior version of a semantically encoded vector performs better than the vector itself.

Questions for authors

The questions I have all circle around the comparison between the downstream efficiencies of the pixel image and the raw encoded vector.

  1. What is the shape/architecture of the reader for the original FP16 vector? Is it compatible to that of the pixel image? This fundamental information is missing from the paper.

  2. The pixel image is constructed first normalizing the raw vectors on the unit sphere. Is the reader for the raw vector also trained on the normalized FP16 vectors? The presence of highly variable norms in the FP16 vectors could by itself impair the training of the decoder.

  3. Why do you expect that the (encoded vector + random frozen projection), once reshaped into a 16x16 rgb image have a local structure that calls for the use of CNNs?

Reviewer_5TygOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors identify the storage and input overhead of textual memory systems as a challenge in language models. The authors present PixelMem, a method that encodes information in the form of an image for storage and decodes the image into LLM readable memory slots for inference. The method uses an existing text encoder and a trained image reader. The authors experiment on a custom QA dataset and show improvements on storage and number of tokens while maintaining comparable or even higher performance in some settings.

Critical strengths
  1. The writing is clear and easy to follow. The authors stated the limitations clearly, and did not overclaim the contributions in the main paper.
  2. The experiment setup, analysis, and ablations are coherent and thorough.
  3. The idea of storing textual memory in image form is natively multimodal, and genuinely novel.
Critical weaknesses
  1. At the current scale of the QA tasks, the absolute reduction in storage and memory cost can be viewed as trivial, given that LLMs consume much more in general, especially in long context settings.
  2. The tasks are synthetic, and do not control for complexity of the textual memory, which is directly related to its compressibility.
Questions for authors
  1. Can the authors show that the performance is sustained (information is preserved in the image) when the task becomes more complex, dense, or long?

  2. How sensitive is the method to the type of text encoder?

  3. How does text-based and image-based memory scale given information parity (which can be demonstrated by sustained performance)? For example, when text grows linearly, does the image also have to grow linearly?

Reviewer_VNbeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to store a language model memory compactly and read it back with few input positions. The proposed PixelMem encodes each memory text with the frozen bge-small-en-v1.5 encoder, maps the 384 dimensional embedding to 768 values with a fixed orthonormal projection, quantizes them to 8 bits and saves them as a 16 by 16 RGB PNG of 852 bytes. A trained visual reader with a global branch over the flattened pixels and a local branch over 64 patches of 2 by 2 pixels produces K continuous slots, which replace placeholder embeddings in Qwen3-VL-8B-Instruct. The reader and a LoRA adapter are trained with answer cross entropy. The evaluation uses a synthetic English template corpus of 9,600 memories in four types, and each question is answered from one given memory. With 16 slots, PixelMem reaches 89.4 percent against 92.5 percent for full text, uses 16 positions instead of 62.13, and uses 93.98 percent less storage than rendered text images at the same budget. Against a direct FP16 vector baseline it reaches 76.2, 85.2 and 89.4 percent at 4, 8 and 16 slots, compared with 48.5, 70.8 and 77.2 percent. Branch ablation, memory mismatch and a Qwen2.5-VL comparison are also reported. The paper is clearly written and the appendix is very detailed. However, I think the pixel image is only a reshaped quantized embedding, the main gains cannot be attributed to the representation, and the evaluation is narrow and favors the trained method. Therefore I lean to reject.

Critical strengths
  1. The paper reports storage bytes, memory input positions and QA accuracy together for text, rendered images, vectors and pixel codes in Table 1, Table 12 and Table 15. This joint accounting is useful for memory systems, where storage and context cost are often discussed separately. It is the main reason I see value in the work.
  2. The paper is unusually transparent. Appendix A to F give seeds, splits, reader size, training stages, checkpoint rules and sample counts, and Section 5.1 and Appendix B.4 state openly that the vector comparison does not isolate the pixel encoding. This makes the claims easy to check and is the reason my rating is not lower.
Critical weaknesses
  1. The pixel image adds no new information or structure beyond the BGE vector. By Eq. 1, Eq. 2 and Appendix B.1, the image is the 384 dimensional embedding after a fixed projection and 8 bit quantization, placed in a fixed order. P has orthonormal rows, so the 768 values still carry only 384 degrees of freedom, and the spatial layout has no meaning, so a 2 by 2 patch is just a group of 12 arbitrary coordinates. The global branch in Table 6 is an MLP on the flattened 768 values, so the claim in Section 5.2 that the reader does not treat the image as a flat collection of values is not convincing. The 27.7 point gain over the FP16 vector in Section 4.3 then must come from the reader and training, not from pixels. Appendix C documents four training stages only for the main sixteen slot pixel system, including initialization from an earlier Qwen2.5-VL reader and a final stage that samples complex facts and procedures at 35 percent each. The vector reader architecture, size and schedule are not reported in Appendix B.4. The largest pixel gains are exactly on complex facts and procedures, 28.8 and 44.0 points at four slots. The authors admit in Section 5.1 that the difference should not be attributed to the representation, but the abstract, Figure 1 and contribution 1 still present the pixel image as the key idea. Without a matched reader on the same vector, the central claim is not supported. The storage claim has the same problem. PixelMem is 2.61 times larger than full text and 84 bytes larger than the FP16 vector, and an 8 bit version of the 384 dimensional vector would need only 384 bytes. The 93.98 percent saving is only against rendered text images, which is a weak reference.
  2. The evaluation is in distribution and the baselines are not trained. Appendix A.1 states that the test split uses the same templates, vocabulary and tasks as training, and each question is answered from one given memory with no retrieval. PixelMem is trained on 160,000 such examples, while Section 4.1 states that the text and rendered image baselines disable LoRA. Appendix A.3 also states that the text baselines receive memories with exact values replaced by a placeholder, while pixel memories use the original text. The held out wording is used for checkpoint selection in Appendix C.4, so it is not fully held out. Learned compression methods that the paper cites in Section 2, such as xRAG, ICAE and 500xCompressor, are not compared, although xRAG also maps one retrieval embedding to the language model input. The results use 100 test memories per batch and 20 memories for the branch analysis, with no confidence intervals. Under these conditions, the 3.1 point gap to full text and the gains over budgeted text in Table 14 and Table 15 say little about real long term memory, where memories are free form and must be retrieved and combined.
Questions for authors
  1. Please report the vector reader architecture, parameter count, initialization and full training schedule in Appendix B.4, and compare its total training updates with the four stages in Table 9. If the vector reader received much less training, the Section 4.3 result changes my view of the gain.
  2. Please run the same reader in Table 6 on the BGE vector itself, for example by reshaping z or the projected u without quantization into the same 64 groups, with the same initialization and schedule. If this matches PixelMem, the pixel format is not the source of the gain. This experiment could directly change my soundness score.
  3. Please report full text and selected text with a LoRA adapter trained on the same QA data, and at least one learned compression baseline such as xRAG or ICAE from Section 2. I need this to judge whether PixelMem is competitive with trained text methods.
  4. Please explain why exact values are masked only for the text and rendered image baselines in Appendix A.3, and report Table 13 without masking.
  5. Please clarify the gap between the full sixteen slot reader at 76.5 percent in Table 2 and 89.4 percent in Table 1. Please also clarify the phrase 8 byte identical pixel memories in Section 5.3 and how the Qwen2.5-VL reader was trained, since Appendix C.2 says the Qwen3 reader was initialized from it.
  6. Please report confidence intervals or results over several seeds for Table 1, Table 2 and Table 15.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.