acceptodds
Under review as a conference paper at ICLR 2027

MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory

16 views

Abstract

Long-term agent memory is becoming increasingly multimodal, but existing evaluations rarely test whether agents can preserve visual details across long interactions. In many prior benchmarks, questions involving images can still be answered from captions, dialogue, or other textual cues, without relying on the original visual information. Existing evaluations also provide limited coverage of cases where visual states change over time and later evidence updates or replaces earlier observations. We introduce MemEye, a benchmark built around two challenges in multimodal memory: preserving visual information at the right level of detail and using that information correctly as memory evolves over time. We capture these challenges with two axes, ranging from scene-level to pixel-level visual evidence, and from single-evidence retrieval to reasoning over changing states. Under this framework, we construct a new benchmark of 1,200 questions across eight life-scenario tasks, with 100 questions per taxonomy cell and ablation-driven validation gates for assessing answerability, shortcut resistance, visual necessity, and reasoning structure. We evaluate 13 memory methods across four vision-language model (VLM) backbones and find that current systems . Further analysis shows that failures often come from lost visual details, stale or incomplete retrieval, and difficulty tracking the current visual state.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language and multimodal language models, Agent memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4rTMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper proposes MemEye to evaluate a model's ability to answer questions based on historical text-and-image dialogues. The benchmark contains 1,200 questions, focusing on visual detail, cross-turn association, and visual state updates.

  • The paper divides the task into two dimensions: visual granularity and memory operations, and compares various memory methods.

Critical strengths
  • The question has practical implications. The loss of details in historical pictures and the interference of old information with current answers are all problems that multi-modal assistants will encounter.
  • The division of tasks is relatively clear. Separating the need for visual detail from memory operations helps understand the specific types of problems where the model fails.
  • Diagnostic tests are relatively adequate. Comparing the original image with the caption, replacing evidence, and retrieval analysis are more valuable than simply reporting the total score.
Critical weaknesses
  • The difference from existing work is not significant enough. Work such as MMRC[1] has studied question answering based on historical graphic dialogues. The new value of this article is mainly in fine-grained visual evidence and status updates, but how much new evaluation capabilities these designs bring needs to be explained more directly.
  • The span of “long-term memory” is underexplained. Lacking question-by-question history length, evidence distance, and cross-session span distribution, it is difficult to judge its actual difference from general long-context multi-graph question answering.
  • State updates are not sufficiently differentiated from historical reasoning. On the visual state update subset, the scores were 0.712 for manually providing the latest key evidence and 0.727 for providing the complete chain of evidence. This result illustrates the need to further distinguish between "finding the latest valid information" and "synthesizing multiple historical states for reasoning."
  • Comparison of strong base models is lacking. The memory method itself is not universally outdated, but the bases are dominated by mini, nano, Flash-Lite and 8B, and only GPT-5.4-mini covers the full test set. How much of the observed failures resulted from base capacity limitations is unclear.

[1] Xue, H., Tang, F., Hu, M., Liu, Y., Huang, Q., Li, Y., Liu, C., Xu, Z., Zhang, C., Feng, C., Xie, Y., Razzak, I., Ge, Z., Su, J., He, J., & Qiao, Y. (2025). MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation. Annual Meeting of the Association for Computational Linguistics.

Questions for authors

See weakness, and, with a unified storage budget, how much difference remains between a more detailed caption or structured visual description and the original image?

8 comments

W1: The difference from existing work is not significant enough. Work such as MMRC[1] has studied question answering based on historical graphic dialogues. The new value of this article is mainly in fine-grained visual evidence and status updates, but how much new evaluation capabilities these designs bring needs to be explained more directly.

Thank you for asking us to make the added evaluation capability clearer. We agree that MMRC already studies question answering over multimodal dialogue histories. MemEye’s contribution is to examine what visual evidence survives in agent memory and how that evidence is used when states change over time.

Figure 2 provides a direct comparison with existing benchmarks, including MMRC, LoCoMo, and Mem-Gallery. Using the same answering model, we compare three input settings: no visual information, caption substitutes, and original images. We provide answer-relevant clue context to reduce the effect of memory search. Under the evaluated captioning setup, MemEye shows a larger caption-to-image gain, indicating that it exposes more questions whose decisive visual information is not preserved by those captions. This gives a concrete measure of the additional visual-memory demands introduced by our design.

MemEye also separates two evaluation capabilities:

  • Fine-grained visual preservation. The X-axis distinguishes scene, region, instance, and pixel-level evidence. This helps identify whether a memory representation preserves the overall scene but loses details needed for object identity, exact attributes, counting, or small-text recognition.
  • Memory use under changing states. The Y-axis distinguishes single-evidence retrieval, relational association, and evolving-state memory. Within Y3, our new analysis separates valid-state selection (55.5%) from history-dependent reasoning (44.5%). Full-chain evidence provides a substantially larger benefit for history-dependent questions: +0.371, compared with +0.011 for valid-state selection. The difference between these effects is 0.360 [0.276, 0.447].

Together, Figure 2 establishes the additional visual dependence under the tested representations, while the taxonomy and evidence controls identify where memory systems fail: preserving fine visual details, selecting the valid observation, or reasoning across changing states. These capabilities make MemEye useful beyond an aggregate dialogue-QA score. We have made this distinction explicit in the introduction and the comparison with MMRC.

W2: The span of “long-term memory” is underexplained. Lacking question-by-question history length, evidence distance, and cross-session span distribution, it is difficult to judge its actual difference from general long-context multi-graph question answering.

Thank you for pointing out the need to separate caption quality from captioner choice and representation length. Table 11 shows that richer captions recover substantial visual information: for FC, the High-X image advantage decreases from 0.134 to 0.014, while the X4 gap remains 0.100. The original comparison changes both the captioner and the prompt, so it measures the combined effect of these changes.

To address this, we conducted a new evaluation on 240 stratified questions, with 20 questions per X/Y cell. We generated generic and exhaustive captions using the same captioner, then evaluated both representations with GPT-6.1 Sol as the answering model and GPT-5.2 as the judge. FC used full histories, and SRAG used standard retrieval from full histories. All conditions used the same question IDs.

Statistical reliability. We calculated native-image minus caption differences within each question and report 95% paired percentile bootstrap confidence intervals using 10,000 resamples:

MethodGroupnNative − Generic [95% CI]Native − Exhaustive [95% CI]
FCLow-X120+0.025 [−0.017, 0.067]+0.033 [0.002, 0.069]
FCHigh-X120+0.194 [0.129, 0.258]+0.046 [0.002, 0.096]
FCX460+0.125 [0.025, 0.208]+0.025 [−0.050, 0.083]
SRAGLow-X120+0.142 [0.108, 0.175]+0.129 [0.088, 0.175]
SRAGHigh-X120+0.208 [0.135, 0.279]+0.198 [0.121, 0.254]
SRAGX460+0.163 [0.071, 0.258]+0.133 [0.046, 0.225]

Low-X comprises X1–X2; High-X comprises X3–X4. X4 is a subset of High-X. Differences are calculated before rounding.

The residual High-X gaps remain statistically supported for both methods in this new evaluation, including with exhaustive captions. At X4, SRAG retains a positive gap of 0.133 [0.046, 0.225]. For FC, exhaustive captions reduce the X4 gap to 0.025 [−0.050, 0.083], which does not establish a residual advantage. These results show that the effect depends on both caption quality and the memory pipeline.

Representation budgets. We did not impose a dedicated per-image caption token cap or match generic and exhaustive captions to the same length. On average, the generated task-aware captions are typically 2–3 times longer. Thus, this comparison primarily focuses how informative caption representations perform in the memory pipeline, rather than performance under equal storage budgets. We have made this explicit and report the measured caption-token distributions.

The results support MemEye's role in evaluating which visual evidence survives textual representation and remains accessible at answer time. Richer captions can substantially improve preservation, while native-image access still provides measurable gains in several settings. We have added this new, broader evaluation results to the paper.

W3: State updates are not sufficiently differentiated from historical reasoning. On the visual state update subset, the scores were 0.712 for manually providing the latest key evidence and 0.727 for providing the complete chain of evidence. This result illustrates the need to further distinguish between "finding the latest valid information" and "synthesizing multiple historical states for reasoning."

Thank you for highlighting this distinction. We agree that finding the latest valid information and reasoning across historical states are different memory capabilities. Table 15 reports aggregate results on the evolving visual-state subset: latest-only evidence scores 0.712, compared with 0.727 for the full chain. This comparison motivates separating the two capabilities within Y3.

We reviewed all 400 Y3 questions and identified two subtypes:

  • Final-valid-state selection: 222 questions (55.5%). Once the latest decisive observation is supplied, earlier states are unnecessary for answering. The memory challenge is identifying the valid observation within a history containing updates or outdated evidence.
  • History-dependent reasoning: 178 questions (44.5%). Earlier states remain necessary even after the latest decisive observation is supplied, for example to determine what changed or compare states over time.

Both types of Y3 questions have histories that contain updates, conflicts, or overrides, but the questions differ in whether earlier states remain necessary once the valid observation is identified. Therefore, a memory system must track evolving evidence to support both types of Y3 questions.

To test this distinction, we conducted a new evaluation on all 400 Y3 questions, using GPT-5.4-mini for answering and GPT-5.2 for scoring. We supplied either the latest decisive observation or the full annotated evidence chain, with the same necessary entity-reference context.

Y3 groupnLatest-onlyFull-chainFull − Latest, paired 95% CI
Final-valid-state selection2220.7610.773+0.011 [−0.027, 0.050]
History-dependent reasoning1780.3310.702+0.371 [0.295, 0.447]
All Y34000.5700.741+0.171 [0.128, 0.216]

Intervals use 10,000 paired bootstrap resamples. Differences are calculated before rounding.

The results support the distinction: full-chain evidence provides a substantial benefit for history-dependent questions, while its observed benefit is small for final-valid-state questions once the valid observation is supplied. The difference between subtype effects is 0.360 [0.276, 0.447], and remains positive under shared-history cluster resampling.

Both capabilities remain relevant to agent memory. In the new evaluation, FC(V) and SRAG(V) score 0.381 and 0.423 on final-valid-state questions, compared with 0.761 under latest-only oracle evidence. On history-dependent questions, they score 0.225 and 0.258, compared with 0.702 under full-chain oracle evidence. These results show the difficulty of answering from evolving histories in both settings.

We have revised the Y3 definition and Appendix A.4 to distinguish these subtypes explicitly and report their results separately. Y3 covers selecting the valid state amid updates and reasoning across changing states and only the history-dependent subtype requires earlier states after the latest decisive observation is supplied. This clarification makes the taxonomy more precise while preserving its focus on how memory systems process evolving evidence.

Reviewer_s1NAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

MemEye introduces 1,200 multimodal memory questions organized by visual granularity and memory reasoning depth. Evaluations across 13 memory methods and 4 backbones use caption substitution, oracle evidence, and retrieval diagnostics to study visual information loss and outdated-state retrieval.

Critical strengths

The benchmark explicitly checks whether visual evidence is needed, addressing textual shortcuts in multimodal evaluation. Its oracle and retrieval diagnostics help distinguish missing evidence from failures to use available evidence.

Critical weaknesses

Y3 appears to mix current-version retrieval with history-dependent reasoning. Latest-only evidence nearly matches the full evidence chain (0.712 versus 0.727, Table 15). In the fossil-tag example, once the latest image is selected, its label directly answers the question; the earlier label’s content is unnecessary for identifying the current number.

Visual irreplaceability depends strongly on caption quality. Stronger captions reduce the high-X image advantage from 0.134 to 0.014 for Full Context, while X4 retains a gap of approximately 0.10 (Table 11). The ablation simultaneously changes the captioner and prompt and increases caption length by 2–3 times.

The X/Y comparisons vary task content alongside the intended ability. Even within X4, the examples involve different perceptual operations: a Y2 question compares cartoon characters’ eye colors, while a Y3 question requires locating an UNO hand-size transition and counting another player’s red cards at that moment.

Questions for authors

What fraction of Y3 questions requires earlier states after the latest relevant observation is supplied?

Are the residual caption–image gaps statistically reliable, and what representation budgets were used?

Do the reported X/Y trends persist within scenarios after accounting for evidence count?

6 comments

W1: Y3 appears to mix current-version retrieval with history-dependent reasoning. Latest-only evidence nearly matches the full evidence chain (0.712 versus 0.727, Table 15). In the fossil-tag example, once the latest image is selected, its label directly answers the question; the earlier label’s content is unnecessary for identifying the current number.

Q1: What fraction of Y3 questions requires earlier states after the latest relevant observation is supplied?

Thank you for pointing out this distinction. 178 of the 400 Y3 questions (44.5%) require earlier states even after the latest decisive observation is supplied. The remaining 222 questions (55.5%) require selecting the final valid state from an evolving history. All Y3 questions have histories that contain updates, conflicts, or overrides, but the questions differ in whether earlier states remain necessary once the valid observation is identified. Therefore, a memory system must track evolving evidence to support both types of Y3 questions: selecting the valid state and reasoning across changing states.

The fossil-tag example belongs to the second group. Once the latest valid image is supplied, its label directly answers the question. Its memory challenge is selecting that image from a history containing outdated labels. We agree that this differs from reasoning that still requires earlier states after evidence selection.

To make this distinction explicit, we conducted a new oracle evaluation on all 400 Y3 questions, extending beyond the evolving visual-state subset in Table 15. We used GPT-5.4-mini for answering and GPT-5.2 for scoring. Both conditions received the same necessary entity-reference context.

Y3 subtypenLatest-onlyFull-chainFull − Latest, paired 95% CI
Final-valid-state selection2220.7610.773+0.011 [−0.027, 0.050]
History-dependent reasoning1780.3310.702+0.371 [0.295, 0.447]
All Y34000.5700.741+0.171 [0.128, 0.216]

Intervals use 10,000 paired bootstrap resamples. Differences are calculated before rounding.

The results support the two distinct evidence requirements. Earlier states provide a substantial benefit for history-dependent questions, while their observed benefit is small for final-valid-state questions once the valid observation is supplied. The difference between subtype effects is 0.360 [0.276, 0.447] and remains positive under shared-history cluster resampling.

Final-valid-state selection also remains challenging under normal memory access. In the new evaluation, FC(V) and SRAG(V) score 0.381 and 0.423 on this subtype, compared with 0.761 when the latest decisive observation is supplied directly.

We have reported these subtypes separately and revise the taxonomy description to clarify that Y3 covers selecting the valid state amid updates and reasoning across changing states. This distinguishes the fossil-tag case from questions that require earlier states, while both question types require the memory system to process evolving evidence.

W2: Visual irreplaceability depends strongly on caption quality. Stronger captions reduce the high-X image advantage from 0.134 to 0.014 for Full Context, while X4 retains a gap of approximately 0.10 (Table 11). The ablation simultaneously changes the captioner and prompt and increases caption length by 2–3 times.

Q2: Are the residual caption–image gaps statistically reliable, and what representation budgets were used?

Thank you for pointing out the need to separate caption quality from captioner choice and representation length. Table 11 shows that richer captions recover substantial visual information: for FC, the High-X image advantage decreases from 0.134 to 0.014, while the X4 gap remains 0.100. The original comparison changes both the captioner and the prompt, so it measures the combined effect of these changes.

To address this, we conducted a new evaluation on 240 stratified questions, with 20 questions per X/Y cell. We generated generic and exhaustive captions using the same captioner, then evaluated both representations with GPT-6.1 Sol as the answering model and GPT-5.2 as the judge. FC used full histories, and SRAG used standard retrieval from full histories. All conditions used the same question IDs.

Statistical reliability. We calculated native-image minus caption differences within each question and report 95% paired percentile bootstrap confidence intervals using 10,000 resamples:

MethodGroupnNative − Generic [95% CI]Native − Exhaustive [95% CI]
FCLow-X120+0.025 [−0.017, 0.067]+0.033 [0.002, 0.069]
FCHigh-X120+0.194 [0.129, 0.258]+0.046 [0.002, 0.096]
FCX460+0.125 [0.025, 0.208]+0.025 [−0.050, 0.083]
SRAGLow-X120+0.142 [0.108, 0.175]+0.129 [0.088, 0.175]
SRAGHigh-X120+0.208 [0.135, 0.279]+0.198 [0.121, 0.254]
SRAGX460+0.163 [0.071, 0.258]+0.133 [0.046, 0.225]

Low-X comprises X1–X2; High-X comprises X3–X4. X4 is a subset of High-X. Differences are calculated before rounding.

The residual High-X gaps remain statistically supported for both methods in this new evaluation, including with exhaustive captions. At X4, SRAG retains a positive gap of 0.133 [0.046, 0.225]. These results show that the effect depends on both caption quality and the memory pipeline.

Representation budgets. We did not impose a per-image caption token limit or match generic and exhaustive captions to the same length. Caption length was determined by the generation prompt. In the original Table 11 experiment, task-aware captions were typically 2–3 times longer. Thus, this comparison tests how informative caption representations perform in the memory pipeline, rather than performance under equal storage budgets. Using the same captioner removes the captioner-change confound, but prompt and resulting length still vary together. We have made this explicit and report the measured caption-token distributions.

The results support MemEye’s role in evaluating which visual evidence survives textual representation and remains accessible at answer time. Richer captions can substantially improve preservation, while native-image access still provides measurable gains. We have revised the paper to reflect this.

W3: The X/Y comparisons vary task content alongside the intended ability. Even within X4, the examples involve different perceptual operations: a Y2 question compares cartoon characters’ eye colors, while a Y3 question requires locating an UNO hand-size transition and counting another player’s red cards at that moment.

Q3: Do the reported X/Y trends persist within scenarios after accounting for evidence count?

Thank you for raising this concern. X4 identifies the granularity of the required visual evidence, rather than a single perceptual operation or a fixed difficulty level. Both eye-color recognition and exact card counting require fine-grained visual evidence. The Y label describes how that evidence must be used: comparing attributes across observations involves relational association, while locating a hand-size transition requires reasoning across changing states. These examples illustrate the categories and they do not establish a performance difference caused only by Y.

To test whether differences persist under comparable conditions, we conducted a new evaluation of 240 questions organized into 120 matched Y2/Y3 pairs across seven scenarios. Each pair matches exactly on scenario, X level, and gold clue-round count. We separately evaluate final-valid-state selection and history-dependent reasoning, with 60 pairs per subtype. We use GPT-5.4-mini for answering and GPT-5.2 for judging.

History-dependent Y3 shows a larger benefit from gold-evidence provision than matched Y2:

Matched categoryFC(V)Gold-evidence oracleGain
Y2: Relational association0.26250.5250+0.2625
Y3: History-dependent reasoning0.18750.6833+0.4958

The difference in gains is +0.2333 [95% CI: 0.0993, 0.4062], using 10,000 bootstrap resamples over shared source histories. This supports a differential response to evidence access after matching scenario, visual granularity, and evidence count.

The matched results do not preserve a universal Y3-below-Y2 ordering: history-dependent Y3 performs better under oracle evidence. The supported distinction is therefore how memory performance responds to evidence access, rather than a fixed difficulty hierarchy.

We have added this analysis and clarify the interpretation.

Reviewer_zzyGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MemEye, a benchmark for evaluating multimodal agent memory along two axes: visual evidence granularity, from scene-level to pixel-level, and memory reasoning depth, from atomic retrieval to reasoning over evolving states. The benchmark contains 1,200 questions balanced across 12 taxonomy cells, with mirrored multiple-choice and open-ended formats. Experiments compare 13 memory methods across four VLM backbones, supplemented by caption substitution, oracle-evidence controls, retrieval diagnostics, and memory-scaling analyses.

The paper addresses a relevant evaluation gap and provides useful diagnostic tools. However, several central interpretations require stronger validation, particularly the claims of visual irreplaceability and a distinct hierarchy of memory reasoning.

Critical strengths
  1. A useful evaluation focus. The benchmark explicitly targets fine-grained visual retention and visually conveyed state updates. These are meaningful challenges that aggregate memory accuracy can obscure.

  2. Informative experimental diagnostics. Gold-clue annotations, latest-only and stale-only controls, and retrieval provenance help distinguish evidence-access failures from downstream answer-generation failures. The paired textual and visual streams also provide useful comparisons within memory families.

  3. Broad evaluation and transparent caveats. The paper evaluates diverse memory systems and reports both multiple-choice and open-ended results. It appropriately acknowledges implementation differences, limited human validation, and the lack of statistically reliable answer-quality improvements from recency reranking.

Critical weaknesses
  1. “Caption-proof” is stronger than the evidence establishes. The construction filter uses minimal captions that retain only the image type, which provides a weak test of whether informative text can substitute for images. The stronger caption ablation is valuable, but it substantially reduces the claimed gap: in Table 11, task-aware captions reduce the high-X visual advantage to 0.014 for Full Context and 0.041 for Semantic RAG. The remaining X4 gaps are more substantial, but uncertainty estimates are not provided. These results support sensitivity to visual compression more directly than general visual irreplaceability. The claims should be narrowed accordingly.

  2. The distinction between Y3 synthesis and retrieving the latest state needs clarification. Appendix A.4 states that Y3 questions should require resolving temporally ordered evidence rather than retrieving either clue alone. Yet Table 15 shows that latest-only evidence nearly matches the complete oracle chain: 0.712 versus 0.727. This is a useful finding, but it suggests that many items primarily test selecting the current state. Selecting that state from a full history remains challenging; nevertheless, it does not necessarily require synthesizing an update chain once retrieved. Reporting separate results for latest-state selection and genuinely multi-state reasoning would strengthen the taxonomy.

  3. Independent validation of the taxonomy remains limited. Although all 1,200 questions were human-reviewed with image access, the quantitative annotation-agreement study uses two LLMs without original images. Independent image-based human agreement would better establish the taxonomy boundaries. The human oracle study reports Y3 majority accuracy of 0.81 but omits sample counts and error analysis. Moreover, declining oracle Judge scores alone cannot separate reasoning depth from differences in task composition or perceptual difficulty.

  4. Some diagnostic protocols need clarification. Several supporting analyses use a 371-question subset whose selection and representativeness are insufficiently explained. The recency experiment also describes reranking a fixed candidate pool, yet Table 13 reports changing Latest@10 values. This requires clarifying whether a larger pool is reranked before selecting ten. Additionally, vanilla SRAG(V)’s latest-clue recall differs between Tables 12 and 13 (0.533 vs. 0.600).

Questions for authors
  1. Do the task-aware caption gaps remain significant with paired confidence intervals and broader evaluation?
  2. What proportion of Y3 questions requires multiple states after the latest decisive clue is supplied?
  3. Can you report independent human taxonomy agreement, human-oracle sample counts, and an analysis of errors? Note that I do not expect full-scale human evaluation in the rebuttal; results on a representative subset would suffice, or the authors could provide evidence explaining why this additional validation would offer little value for the present work.
  4. How was the 371-question subset selected, and how can the recency protocol and the latest-clue recall values in Tables 12–13 be reconciled?

Overall Assessment

MemEye is a promising diagnostic benchmark with a relevant focus and several thoughtful analyses. My main reservations concern whether the current validation supports the strongest conceptual claims. Clearer taxonomy boundaries, stronger independent annotation checks, and reconciled experimental protocols would materially strengthen the paper. I would be willing to revise my assessment upward if the authors convincingly address these concerns.

6 comments

W1: “Caption-proof” is stronger than the evidence establishes. The construction filter uses minimal captions that retain only the image type, which provides a weak test of whether informative text can substitute for images. The stronger caption ablation is valuable, but it substantially reduces the claimed gap: in Table 11, task-aware captions reduce the high-X visual advantage to 0.014 for Full Context and 0.041 for Semantic RAG. The remaining X4 gaps are more substantial, but uncertainty estimates are not provided. These results support sensitivity to visual compression more directly than general visual irreplaceability. The claims should be narrowed accordingly.

Q1: Do the task-aware caption gaps remain significant with paired confidence intervals and broader evaluation?

Thank you for raising this point. Our intended claim is that caption-based memory can lose answer-relevant visual evidence, and MemEye provides a way to measure that loss. The minimal-caption filter serves as an initial shortcut check during construction. The stronger caption ablations test whether informative descriptions preserve the evidence needed for answering.

We agree that caption quality is important. In Table 11, task-aware captions reduce the High-X native-image advantage to 0.014 for FC and 0.041 for SRAG. To examine residual gaps with broader coverage and paired uncertainty estimates, we conducted a new evaluation on 240 stratified questions, with 20 questions per X/Y cell, using GPT-6.1 Sol for answering and GPT-5.2 for scoring. We generated generic and exhaustive captions using the same captioner, controlling the captioner change in the original comparison. FC used full histories, and SRAG used standard retrieval from full histories.

We report native-image minus caption differences with 95% paired percentile bootstrap confidence intervals from 10,000 resamples:

MethodGroupnNative − Generic [95% CI]Native − Exhaustive [95% CI]
FCLow-X120+0.025 [−0.017, 0.067]+0.033 [0.002, 0.069]
FCHigh-X120+0.194 [0.129, 0.258]+0.046 [0.002, 0.096]
FCX460+0.125 [0.025, 0.208]+0.025 [−0.050, 0.083]
SRAGLow-X120+0.142 [0.108, 0.175]+0.129 [0.088, 0.175]
SRAGHigh-X120+0.208 [0.135, 0.279]+0.198 [0.121, 0.254]
SRAGX460+0.163 [0.071, 0.258]+0.133 [0.046, 0.225]

Low-X comprises X1–X2; High-X comprises X3–X4. X4 is a subset of High-X. Differences are calculated before rounding.

In this broader evaluation, the High-X native-image advantage remains positive with confidence intervals excluding zero for both methods, even with exhaustive captions. SRAG also retains an X4 advantage of 0.133 [0.046, 0.225].

These findings strengthen the benchmark’s diagnostic value. MemEye reveals both where detailed captions preserve sufficient visual information and where native-image access still improves memory performance. The residual SRAG gaps reflect the complete memory pipeline, including retrieval and answering, rather than caption generation alone.

We have clarified that “caption-proof” describes resistance to the evaluated caption substitutions, rather than a guarantee against every possible textual representation. We also add the new paired CIs and distinguish this broader evaluation from the original Table 11 runs. The results support the paper’s central motivation: evaluating whether memory representations preserve and recover the visual evidence that later questions require.

W2: The distinction between Y3 synthesis and retrieving the latest state needs clarification. Appendix A.4 states that Y3 questions should require resolving temporally ordered evidence rather than retrieving either clue alone. Yet Table 15 shows that latest-only evidence nearly matches the complete oracle chain: 0.712 versus 0.727. This is a useful finding, but it suggests that many items primarily test selecting the current state. Selecting that state from a full history remains challenging; nevertheless, it does not necessarily require synthesizing an update chain once retrieved. Reporting separate results for latest-state selection and genuinely multi-state reasoning would strengthen the taxonomy.

Q2: What proportion of Y3 questions requires multiple states after the latest decisive clue is supplied?

Thank you for highlighting this distinction. 178 of the 400 Y3 questions (44.5%) require earlier states even after the latest decisive observation is supplied. The remaining 222 questions (55.5%) require selecting the final valid state from an evolving history. All Y3 questions have histories that contain updates, conflicts, or overrides, but the questions differ in whether earlier states remain necessary once the valid observation is identified. Therefore, a memory system must track evolving evidence to support both types of Y3 questions: selecting the valid state and reasoning across changing states.

We agree that selecting the valid state and reasoning across multiple states should be reported separately. To test this distinction, we conducted a new evaluation on all 400 Y3 questions, extending beyond the evolving visual-state subset in Table 15. We generated fresh answers using GPT-5.4-mini and scored them using GPT-5.2. The two oracle conditions provide either the latest decisive observation or the full annotated evidence chain, with the same necessary entity-reference context.

Y3 groupnLatest-onlyFull-chainFull − Latest, paired 95% CI
All Y34000.5700.741+0.171 [0.128, 0.216]
Final-valid-state selection2220.7610.773+0.011 [−0.027, 0.050]
History-dependent reasoning1780.3310.702+0.371 [0.295, 0.447]

Intervals use 10,000 paired bootstrap resamples. Differences are calculated before rounding.

The full-chain benefit is concentrated in the history-dependent subtype. The difference between subtype effects is 0.360, with a 95% CI of [0.276, 0.447]. This positive contrast also remains under shared-history cluster resampling. For final-valid-state questions, the observed benefit of earlier evidence is small once the decisive valid observation is supplied.

The new memory-system results also show why final-valid-state selection remains useful to evaluate. On these 222 questions, FC(V) and SRAG(V) score 0.381 and 0.423, compared with 0.761 when the latest decisive observation is supplied directly. Thus, answering from the evolving history remains challenging even when the answer can be read from a single correctly selected observation.

We have revised Appendix A.4 to make the boundary explicit: Y3 tests selecting the valid state amid updates or reasoning across changing states and only the history-dependent subtype requires multiple states after oracle selection. We also report the two subtypes separately. This preserves Y3's focus on evolving memory while making clear which questions test valid-state selection and which test multi-state reasoning.

W3: Independent validation of the taxonomy remains limited. Although all 1,200 questions were human-reviewed with image access, the quantitative annotation-agreement study uses two LLMs without original images. Independent image-based human agreement would better establish the taxonomy boundaries. The human oracle study reports Y3 majority accuracy of 0.81 but omits sample counts and error analysis. Moreover, declining oracle Judge scores alone cannot separate reasoning depth from differences in task composition or perceptual difficulty.

Q3: Can you report independent human taxonomy agreement, human-oracle sample counts, and an analysis of errors? Note that I do not expect full-scale human evaluation in the rebuttal; results on a representative subset would suffice, or the authors could provide evidence explaining why this additional validation would offer little value for the present work.

Thank you for raising this point. Appendix A.5 describes our human review process, but its quantitative agreement analysis uses LLM annotators. Since submission, we have completed additional image-based human validation covering all 1,200 questions.

Independent human taxonomy agreement. Eight human annotators independently labeled all 1,200 questions. Each question was assigned to two annotators, and each annotator labeled 300 questions, resulting in 2,400 item-level annotations. Annotators received the original images, dialogue context, clue rounds, and question–answer pairs, together with the predefined taxonomy rules and boundary examples. The original X/Y labels were hidden. For X, annotators identified the finest level of visual evidence needed to answer the question. For Y, they identified the required memory operation and the minimal sufficient evidence structure.

Treating each (X,Y) pair as one joint category, the average pairwise Cohen’s κ was 0.92. We calculated κ for each annotator pair with shared questions and averaged these values with equal weight across pairs. This provides direct evidence that humans inspecting the original visual evidence can consistently apply the combined taxonomy.

Human-oracle sample counts and results. Separately, two annotators independently answered all 1,200 questions using only the gold evidence rounds presented in chronological order, including the original images. A third reviewer examined disagreements and errors and confirmed the reference answers. Annotators revised their responses following review. The following results therefore report post-adjudication accuracy.

Memory categoryUnique questionsPost-adjudication accuracy
Y1: Atomic retrieval40098.1%
Y2: Relational association40096.0%
Y3: Evolutionary synthesis40090.7%
Overall1,20094.9%

This full-benchmark study is separate from the smaller human-oracle sanity check reported in Table 9. It provides broader evidence of answerability under gold evidence and review.

Qualitative error analysis and Y3 clarification. Our review of the original human responses found relatively few errors in Y1 and Y2. These mainly involved misreading answer options, overlooking relevant details, or missing the temporal order of events.

Another question also highlighted the importance of distinguishing two operations within Y3. All Y3 evidence sequences involve updates, conflicts, or overrides. Answering requires selecting the valid state amid updates or reasoning across changing states. Final-valid-state questions test whether the agent identifies the authoritative observation among competing states, while history-dependent questions require information from earlier and later states.

Both operations belong to Y3 because the agent must account for changes in the evidence rather than treat all observations as equally valid. Providing the latest decisive observation directly removes the state-selection challenge. Strong latest-only oracle performance is therefore consistent with questions that remain challenging when the agent must find that observation within the full history. We report the subtype breakdown in our response to W2/Q2.

Together, the human labeling study supports the reproducibility of the taxonomy, while the oracle study assesses answerability. We agree that declining oracle scores alone cannot isolate reasoning depth from perceptual difficulty or task composition. We have distinguished these roles clearly and add the human study protocols, sample counts, results, and qualitative analysis to the revised appendix.

Comments on the paper

4 comments

We have uploaded a revised PDF incorporating the updates discussed in our responses. The revised passages and additions are marked in purple for ease of reference. We thank the reviewers and Area Chair for their time and valuable feedback!

Dear Reviewers and Area Chair,

Thank you for your reviews. We posted our responses on Nov 11 and uploaded a revised manuscript on Nov 13, with the changes marked in purple. We would be glad to address any follow-up questions before the discussion period closes on Nov 18.

Best regards, The Authors

Area_Chair_G2JVOpenReview ↗

Please review the rebuttal and provide further comments as appropriate

Dear Reviewers,

The authors have submitted their rebuttal. Please review it and provide further comments as appropriate.

Best, Your AC

AuthorsOpenReview ↗

Summary of Revisions and Thanks

Dear Reviewers and Area Chair,

Thank you for reading our paper and for the discussion. Your comments guide us to conduct new experiments that significantly strengthen the paper's core claims. We have uploaded the revised manuscript, and here are the main changes.

  1. Caption comparison.

We reran the comparison on 240 questions using GPT-5.2 as the captioner for all models and added paired confidence intervals. We narrowed our claim about the FC High-X gaps and updated the discussion to reflect the gap that remains for SRAG. Appendix F.1 describes the setup, and Table 24 in Appendix F.3 reports results for both matched groups.

  1. What Y3 measures.

We separated questions that ask for the final valid state (222) from questions that require reasoning over the history (178). We revised the main claim to reflect this distinction and added an oracle evaluation for the two groups in Appendix A.4.

  1. Benchmark validation.

We added an independent annotation check with eight annotators (κ = 0.92) and ran the oracle evaluation on all 1,200 questions (94.9%). We also clarified that the 371-question subset was the full first version of the benchmark and corrected the value in Table 13 from 0.600 to 0.533.

  1. How far back the evidence goes.

Appendix A.1 now reports history length, evidence distance, and cross-session span for all 1,200 questions. The median available history was 230 rounds, 34 sessions, and 177 images. At least one clue appeared before the final session in 98.5% of questions, and the evidence spanned multiple sessions in 55.83%.

Thank you again for your time and feedback!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.