- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
This paper proposes MemEye to evaluate a model's ability to answer questions based on historical text-and-image dialogues. The benchmark contains 1,200 questions, focusing on visual detail, cross-turn association, and visual state updates.
-
The paper divides the task into two dimensions: visual granularity and memory operations, and compares various memory methods.
Critical strengths
- The question has practical implications. The loss of details in historical pictures and the interference of old information with current answers are all problems that multi-modal assistants will encounter.
- The division of tasks is relatively clear. Separating the need for visual detail from memory operations helps understand the specific types of problems where the model fails.
- Diagnostic tests are relatively adequate. Comparing the original image with the caption, replacing evidence, and retrieval analysis are more valuable than simply reporting the total score.
Critical weaknesses
- The difference from existing work is not significant enough. Work such as MMRC[1] has studied question answering based on historical graphic dialogues. The new value of this article is mainly in fine-grained visual evidence and status updates, but how much new evaluation capabilities these designs bring needs to be explained more directly.
- The span of “long-term memory” is underexplained. Lacking question-by-question history length, evidence distance, and cross-session span distribution, it is difficult to judge its actual difference from general long-context multi-graph question answering.
- State updates are not sufficiently differentiated from historical reasoning. On the visual state update subset, the scores were 0.712 for manually providing the latest key evidence and 0.727 for providing the complete chain of evidence. This result illustrates the need to further distinguish between "finding the latest valid information" and "synthesizing multiple historical states for reasoning."
- Comparison of strong base models is lacking. The memory method itself is not universally outdated, but the bases are dominated by mini, nano, Flash-Lite and 8B, and only GPT-5.4-mini covers the full test set. How much of the observed failures resulted from base capacity limitations is unclear.
[1] Xue, H., Tang, F., Hu, M., Liu, Y., Huang, Q., Li, Y., Liu, C., Xu, Z., Zhang, C., Feng, C., Xie, Y., Razzak, I., Ge, Z., Su, J., He, J., & Qiao, Y. (2025). MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation. Annual Meeting of the Association for Computational Linguistics.
Questions for authors
See weakness, and, with a unified storage budget, how much difference remains between a more detailed caption or structured visual description and the original image?
8 comments
W2: The span of “long-term memory” is underexplained. Lacking question-by-question history length, evidence distance, and cross-session span distribution, it is difficult to judge its actual difference from general long-context multi-graph question answering.
Thank you for pointing out the need to separate caption quality from captioner choice and representation length. Table 11 shows that richer captions recover substantial visual information: for FC, the High-X image advantage decreases from 0.134 to 0.014, while the X4 gap remains 0.100. The original comparison changes both the captioner and the prompt, so it measures the combined effect of these changes.
To address this, we conducted a new evaluation on 240 stratified questions, with 20 questions per X/Y cell. We generated generic and exhaustive captions using the same captioner, then evaluated both representations with GPT-6.1 Sol as the answering model and GPT-5.2 as the judge. FC used full histories, and SRAG used standard retrieval from full histories. All conditions used the same question IDs.
Statistical reliability. We calculated native-image minus caption differences within each question and report 95% paired percentile bootstrap confidence intervals using 10,000 resamples:
| Method | Group | n | Native − Generic [95% CI] | Native − Exhaustive [95% CI] |
|---|---|---|---|---|
| FC | Low-X | 120 | +0.025 [−0.017, 0.067] | +0.033 [0.002, 0.069] |
| FC | High-X | 120 | +0.194 [0.129, 0.258] | +0.046 [0.002, 0.096] |
| FC | X4 | 60 | +0.125 [0.025, 0.208] | +0.025 [−0.050, 0.083] |
| SRAG | Low-X | 120 | +0.142 [0.108, 0.175] | +0.129 [0.088, 0.175] |
| SRAG | High-X | 120 | +0.208 [0.135, 0.279] | +0.198 [0.121, 0.254] |
| SRAG | X4 | 60 | +0.163 [0.071, 0.258] | +0.133 [0.046, 0.225] |
Low-X comprises X1–X2; High-X comprises X3–X4. X4 is a subset of High-X. Differences are calculated before rounding.
The residual High-X gaps remain statistically supported for both methods in this new evaluation, including with exhaustive captions. At X4, SRAG retains a positive gap of 0.133 [0.046, 0.225]. For FC, exhaustive captions reduce the X4 gap to 0.025 [−0.050, 0.083], which does not establish a residual advantage. These results show that the effect depends on both caption quality and the memory pipeline.
Representation budgets. We did not impose a dedicated per-image caption token cap or match generic and exhaustive captions to the same length. On average, the generated task-aware captions are typically 2–3 times longer. Thus, this comparison primarily focuses how informative caption representations perform in the memory pipeline, rather than performance under equal storage budgets. We have made this explicit and report the measured caption-token distributions.
The results support MemEye's role in evaluating which visual evidence survives textual representation and remains accessible at answer time. Richer captions can substantially improve preservation, while native-image access still provides measurable gains in several settings. We have added this new, broader evaluation results to the paper.
W3: State updates are not sufficiently differentiated from historical reasoning. On the visual state update subset, the scores were 0.712 for manually providing the latest key evidence and 0.727 for providing the complete chain of evidence. This result illustrates the need to further distinguish between "finding the latest valid information" and "synthesizing multiple historical states for reasoning."
Thank you for highlighting this distinction. We agree that finding the latest valid information and reasoning across historical states are different memory capabilities. Table 15 reports aggregate results on the evolving visual-state subset: latest-only evidence scores 0.712, compared with 0.727 for the full chain. This comparison motivates separating the two capabilities within Y3.
We reviewed all 400 Y3 questions and identified two subtypes:
- Final-valid-state selection: 222 questions (55.5%). Once the latest decisive observation is supplied, earlier states are unnecessary for answering. The memory challenge is identifying the valid observation within a history containing updates or outdated evidence.
- History-dependent reasoning: 178 questions (44.5%). Earlier states remain necessary even after the latest decisive observation is supplied, for example to determine what changed or compare states over time.
Both types of Y3 questions have histories that contain updates, conflicts, or overrides, but the questions differ in whether earlier states remain necessary once the valid observation is identified. Therefore, a memory system must track evolving evidence to support both types of Y3 questions.
To test this distinction, we conducted a new evaluation on all 400 Y3 questions, using GPT-5.4-mini for answering and GPT-5.2 for scoring. We supplied either the latest decisive observation or the full annotated evidence chain, with the same necessary entity-reference context.
| Y3 group | n | Latest-only | Full-chain | Full − Latest, paired 95% CI |
|---|---|---|---|---|
| Final-valid-state selection | 222 | 0.761 | 0.773 | +0.011 [−0.027, 0.050] |
| History-dependent reasoning | 178 | 0.331 | 0.702 | +0.371 [0.295, 0.447] |
| All Y3 | 400 | 0.570 | 0.741 | +0.171 [0.128, 0.216] |
Intervals use 10,000 paired bootstrap resamples. Differences are calculated before rounding.
The results support the distinction: full-chain evidence provides a substantial benefit for history-dependent questions, while its observed benefit is small for final-valid-state questions once the valid observation is supplied. The difference between subtype effects is 0.360 [0.276, 0.447], and remains positive under shared-history cluster resampling.
Both capabilities remain relevant to agent memory. In the new evaluation, FC(V) and SRAG(V) score 0.381 and 0.423 on final-valid-state questions, compared with 0.761 under latest-only oracle evidence. On history-dependent questions, they score 0.225 and 0.258, compared with 0.702 under full-chain oracle evidence. These results show the difficulty of answering from evolving histories in both settings.
We have revised the Y3 definition and Appendix A.4 to distinguish these subtypes explicitly and report their results separately. Y3 covers selecting the valid state amid updates and reasoning across changing states and only the history-dependent subtype requires earlier states after the latest decisive observation is supplied. This clarification makes the taxonomy more precise while preserving its focus on how memory systems process evolving evidence.
Thank you for asking us to make the added evaluation capability clearer. We agree that MMRC already studies question answering over multimodal dialogue histories. MemEye’s contribution is to examine what visual evidence survives in agent memory and how that evidence is used when states change over time.
Figure 2 provides a direct comparison with existing benchmarks, including MMRC, LoCoMo, and Mem-Gallery. Using the same answering model, we compare three input settings: no visual information, caption substitutes, and original images. We provide answer-relevant clue context to reduce the effect of memory search. Under the evaluated captioning setup, MemEye shows a larger caption-to-image gain, indicating that it exposes more questions whose decisive visual information is not preserved by those captions. This gives a concrete measure of the additional visual-memory demands introduced by our design.
MemEye also separates two evaluation capabilities:
Together, Figure 2 establishes the additional visual dependence under the tested representations, while the taxonomy and evidence controls identify where memory systems fail: preserving fine visual details, selecting the valid observation, or reasoning across changing states. These capabilities make MemEye useful beyond an aggregate dialogue-QA score. We have made this distinction explicit in the introduction and the comparison with MMRC.