acceptodds
Withdrawn from ICLR 2027

VCRM: Empowering Efficient Memory for Vision-Language Models with Visual Context Record Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Although large language models excel at complex reasoning tasks, effectively leveraging continuously growing long-term historical information remains a critical challenge. Existing memory-augmented methods primarily rely on storing and retrieving historical information in textual form. However, as interaction history continues to grow, memory content faces a dual bottleneck of ever-increasing context overhead and key evidence dispersed across voluminous retrieval results. To address these issues, we propose Visual Context Record Memory (VCRM), which achieves significant token compression with stable performance gains. VCRM consists of two synergistic components: (1) Hierarchical Evidence Prospecting (HEP), which filters redundant information through tiered confidence routing and adaptive cliff pruning; (2) Discriminative Semantic Rendering (DSR), which renders critical evidence into two-dimensional visual records with non-uniform information density, enabling key evidence to be preserved more effectively under constrained token budgets. We evaluate our method on three widely adopted benchmarks spanning long-term conversational memory and multi-hop question answering, i.e., LoCoMo, LongMemEval-S, and HotpotQA. Results show that VCRM improves downstream task performance while substantially reducing token consumption, validating the effectiveness of the method itself. Under constrained budgets, VCRM maintains near-full performance through visual memory, demonstrating strong robustness under compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, long-term memory, memory-augmented agents, context compression, long-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QcvnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes VCRM, a training free framework for long-context question-answering. VCRM retrieves relevant information from the context, and renders the retrieved context into an image, and the model answers the question from this rendered image. VCRM has three main modules: Hierarchical Evidence Prospecting (HEP), Semantic Fact Extraction (SFE), and Density-Differentiated rendering (DDR).

The paper experiments on LoCoMo, LongMemEval-S and HotpotQA using two small Qwen2.5-VL and Qwen3-VL models, and it reports improved accuracy with much fewer memory tokens than the baselines. The authors did ablation experiments to examine the individual components of VCRM. The experiments also demonstrate strong performance of VCRM even at only 64 visual tokens.

Critical strengths
  • The observation that converting textual input into visual input can sometimes be more effective and efficient in textual QA tasks is a very interesting, surprising, and counter-intuitive result, which could have high significance if thoroughly studied and investigated.
  • The experiments reports strong performance under very tight memory budgets, as few as 64 tokens on Qwen3-VL-4B, suggesting potential of VCRM in low-budget settings.
Critical weaknesses
  • The paper did not provide sufficient evidence to show the benefit of visual representation over raw text. This is a very surprising observation so it also required substantial evidence to fully investigate when we shall expect such benefit. Table 4 provides a useful text-versus-image comparison, but does not report the corresponding token costs or repeat this comparison across memory budgets. Most of the experiments also bundles image conversion with visual emphasis, without comparing uniform images or text given equivalent emphasis. It therefore remains unclear when images help, particularly when the extracted facts are already short enough to fit as text (such as the example in Figure 8).
  • The hyperparameter selection protocol is unclear. HEP introduces fusion weights, confidence thresholds, retrieval quotas, and pruning parameters; rendering adds further configuration choices. Their values and selection procedures are incompletely documented.
Questions for authors
  • How are the hyperparameters selected? VCRM involves many hyperparameters and it is not clear whether any of the hyperparameters are tuned on the test set. Many hyperparameter selection details are missing. The authors should carefully list all the hyperparameters required for their method and explain how these hyperparameters are set. Is it set by standard/default evaluation practice, a separate validation dataset, or what else?
  • How much of the reported token reduction comes from retrieval and extraction (HEP, SFE) versus rendering (DDR)? What are the token consumption and end-to-end latency of the HEP+SFE-only variant?
1 comment

Dear Reviewer Qcvn,

Thank you again for your careful review and constructive feedback. We have made additional clarifications and provided supplementary analyses in response to the concerns raised in your review.

With the discussion period nearing its end, we would greatly appreciate your taking the time to review our responses and share any remaining questions or concerns. If you find that our responses have adequately addressed your concerns, we would sincerely appreciate your consideration of an updated rating.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_tKFEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper questions on the optimal format of memory in modern agent systems. Inspired by the recent advances on visual token compression, it proposes a hierarchical evidence prospecting and discriminative semantic rendering algorithm to down-sample the existing conversations, extracting the query-specific facts and render as a visual image. In the experiments, both experiments against baselines and own ablations yield substantial improvement.

Critical strengths
  1. The idea of using visual tokens are memory is insightful and answers many of the practitioner's questions including me. The proposed method seems to be strikingly simple and effective in terms of ranking and cliff pruning.

  2. The full method's performance is a lot better than baselines, although some of the improvements might not solely from the modality itself.

Critical weaknesses
  1. The comparison against the baselines may not be fair, given the VCRM uses query-specific fact extraction and runtime rendering of the memory. The current results (especially HSP only ablation) didn't demonstrate the convincing performance/technical soundness that is superior to its baselines. In SFE, the fact extraction in conditioned on query, which in fact utilizes more information than some of the baselines such as NaiveRAG/Mem0 and True Memory if I understand it correctly.

  2. The efficiency study focues only on tokens, it's a widely known fact that visual tokens can be fewer than (DeepSeek OCR etc.) textual tokens, but the prefilling time of visual tokens are longer than text, it is not revealed in the paper and appendix on what's the real-time wallclock comparison of these models.

Questions for authors
  1. Can you elaborate on inference time what's the additional cost of VRCM? how do we measure the addtional rendering cost?

  2. Is it possible to reduce the computations like KV-cache re-use if using this visual memory. For example, if turn t and t+1, following current method, retrieves different memory context, they will be rendered as two different image although most of the content might overlap.

2 comments

Dear Reviewer tKFE,

Thank you again for your insightful comments and valuable suggestions. We have provided detailed responses to the questions and concerns raised in your review.

As the discussion period is drawing to a close, we would appreciate it if you could review our responses and let us know if you have any remaining questions. If our responses have addressed your concerns, we would be grateful if you would consider updating your rating accordingly.

Thank you for your time and consideration.

Best regards,

The Authors

I read the author's rebuttal. While I appreciate the author's great effort on adding text-only baselines and ablation study, I think the motivation of using visual memory for textual conversations is limited still. Given the fact that visual memory's biggest strength comes from the high compression ratio, real use cases should be justified. I believe other reviewers also hold the similar opinion by reading their initial reviews.

Reviewer_Rz4sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VCRM is a training-free pipeline for memory-based question answering with VLMs. Hierarchical Evidence Prospecting combines embedding similarity, BM25, and reranking, then uses confidence-based candidate quotas and score-cliff pruning. Discriminative Semantic Rendering performs query-conditioned fact extraction and renders the extracted facts into an image with differentiated visual salience. The final VLM answers from the image and query. Experiments on LoCoMo, LongMemEval-S, and a distractor-extended HotpotQA subset report improved accuracy and reduced answer-stage memory tokens, including substantial retention under a 64-token visual-memory budget.

Critical strengths
  1. Filtering before extraction and rendering addresses an important weakness of indiscriminate visual compression: irrelevant content consumes limited readable image area. Preserving temporal context and explicit entities is sensible for conversational-memory questions.

  2. With Qwen3-VL-4B, Table 1 reports J-scores of 82.86 on LoCoMo and 75.80 on LongMemEval-S, compared with 77.40 and 71.40 for True Memory. Table 2 reports HotpotQA F1 of 79.16 versus 68.85 for Mem0. A second backbone and lexical metrics provide useful corroboration beyond a single judge score.

Critical weaknesses
  1. Evaluation uncertainty and tuning separation are insufficiently established. Three judge votes reduce verdict variability; they do not quantify uncertainty in method differences.

  2. The primary judge may accept incomplete or partially incorrect answers. Figures 10–11 instruct it to mark an answer correct if it mentions the same core topic/fact. This is weaker than the claimed assessment of comprehensiveness and logical coherence, especially for multi-session aggregation and conflicting updates. F1/BLEU and HotpotQA EM help, so this does not invalidate all results. However, a blinded human audit or stricter correctness check on a representative subset would establish whether the several-point J-score advantage survives.

Questions for authors
  1. Can you report retrieval/reranking cost, extraction input/output tokens, vision-encoding cost, and total latency against True Memory and the matched HEP-plus-text-extraction variant, with repeated queries over the same history?
  2. With exactly the same extracted facts, how do uniform visual rendering, differentiated rendering, and budget-matched text compression compare at 1024, 256, and 64 memory tokens?
  3. What exact processor settings enforce the visual budgets on both backbones? Can you provide actual token-count distributions and the text-baseline budget enforcement code?
  4. Which data selected all retrieval and rendering hyperparameters? Can you supply the relative-drop definition and paired uncertainty estimates, clustering LoCoMo questions by conversation?
1 comment

Dear Reviewer Rz4s,

Thank you again for your thoughtful feedback and constructive suggestions. We have carefully addressed the concerns raised in your review in our author response.

As the discussion period is drawing to a close, we would greatly appreciate it if you could take a moment to review our responses and let us know if any concerns remain. If our clarifications have adequately addressed your concerns, we would sincerely appreciate your consideration of an updated rating.

Thank you for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Additional Clarifications

Dear Area Chairs and Reviewers,

We sincerely thank the Area Chairs and all reviewers for their time, effort, and constructive feedback on our submission. We greatly appreciate the opportunity to clarify our work and further strengthen the paper through this discussion.

In response to the reviewers' concerns, we have provided detailed responses and additional analyses addressing the main questions raised:

  • Hyperparameter Configuration and Sensitivity: We disclosed the specific hyperparameter values used in VCRM and conducted sensitivity analyses on key parameters. The results show that VCRM maintains competitive performance across the tested configurations, rather than relying on a narrow set of specific values.

  • Evaluation Reliability: We followed established evaluation criteria in prior studies to ensure fair comparisons. To address concerns regarding LLM-as-a-Judge evaluation, we further assessed the reliability of our results using an alternative judging criterion, stronger LLM judges, and blinded human evaluation.

  • Token Consumption and Runtime: We reported detailed token consumption and end-to-end runtime measurements. The results demonstrate that VCRM substantially reduces both token consumption and runtime, with negligible rendering overhead.

  • Visual Representation Benefits: We compared text extraction, uniform visual rendering, and differentiated visual rendering across multiple target token budgets on both LongMemEval-S and LoCoMo, further demonstrating the effectiveness of our visual representation.

  • Baseline Fairness: We clarified the fairness constraints of our overall process and conducted additional controlled comparisons using query-conditioned fact extraction on existing baselines. The results show that VCRM's performance gains stem from the complementary contributions of multiple components, rather than any single component.

Overall, these additional analyses provide further evidence for VCRM's effectiveness, robustness under constrained token budgets, and computational efficiency. We have also clarified several implementation details to improve the reproducibility of our approach.

As the discussion period has only a few days left, we would greatly appreciate it if the Area Chairs and reviewers could take a moment to review our responses and share any further questions, concerns, or feedback. This would allow us sufficient time to address remaining issues and clarify points that may still be unclear.

Thank you again for your time, consideration, and valuable feedback. We sincerely appreciate your engagement in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.