- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces M³Eval, a benchmark for evaluating multimodal memory through four video-based paradigms inspired by cognitive psychology: divided attention, memory interference, interleaved events, and N-back. The authors evaluate several multimodal models and video agents alongside human participants, examining source identification, temporal organization, content retention, and cross-clip attribute matching. The experiments reveal substantial difficulties under these settings and interesting differences in how humans and models respond to interference and increasing sequence length. The main contribution is the structured evaluation design and its diagnostic findings.
Critical strengths
- The four cognitively motivated paradigms provide a clear evaluation framework, with useful comparisons between different interference conditions.
- The analyses reveal interesting patterns, including repetition benefits and different effects of temporal lag and sequence length.
- The evaluation covers multiple models, agents, and human references, supported by single-video baselines, manual verification, and human agreement checks.
Critical weaknesses
- The benchmark does not fully disentangle memory failures from perception and evidence retrieval. Although the single-video controls are helpful, reduced target-frame coverage or changes in effective resolution may also contribute to the performance drops.
- Some mechanistic interpretations are stronger than the evidence supports. The observed human–model differences are interesting, but differing viewing conditions and limited causal controls warrant more cautious claims about the underlying memory mechanisms.
Questions for authors
- Can the authors better disentangle memory interference from reduced visual evidence? For most models, the multi-video setting still uses a fixed total of 96 sampled frames, which may reduce the number of frames representing the target video compared with the single-video baseline.
- For the False Memory Discrimination task, the construction makes “Neither video” always the correct semantic answer. Could the authors include balanced positive cases where the queried event appears in Video 1, Video 2, or both, to rule out this shortcut?
3 comments
Response to Reviewer sT1d (1/2): Matched Target Evidence and Mechanistic Interpretations
W1 & Q1: Matched target evidence
Our single-video baselines already match target frames and per-stream resolution, and the main effects hold under this matched setting.
Setting.
- Divided Attention (split-screen). We place the two videos side by side at their original resolution, without resizing. Each stream thus has the same number of visual tokens per frame as the single-video input. The two streams play in sync. The target stream receives 96 frames in both the single-video and split-screen inputs.
- Memory Interference (concatenation) and Interleaved Events (interleaving). The total number of input frames scales with the number of videos. We use 48 frames for one video, 96 frames for two videos, and 144 frames for the three-video setting in Fig. 8. The target video thus receives 48 frames in both the single-video and multi-video inputs.
Results. These matched comparisons use the same questions as the main evaluation, for seven models (Tables 6–8). Single-video accuracy exceeds multi-video accuracy by 19–29 pp for Divided Attention, 11–27 pp for Memory Interference, and 16–28 pp for Interleaved Events (per-model averages). We have added these input settings to Appendices B and H.
W2: Strength of mechanistic interpretations
Our human–model comparison follows standard practice, and the human interference asymmetry persists under input matched to the models. Our mechanism-level statements are empirical interpretations meant to inform future model design. Establishing causal mechanisms lies beyond the scope of this benchmark.
Fair comparison. Humans, models, and agents access video in different ways, such as which frames they see and whether they can revisit the video. This access is part of each method. It is not an external variable that can be controlled separately. Video-MME, MLVU, and InfiniBench also evaluate each model under its native input setting instead of a uniform frame count (Appendix H). Within this practice, we align the conditions where possible. Humans and models receive the same constructed videos. Humans may pause and rewind. This gives them the same capability as agents such as VideoLucy, which re-access the video during inference (Appendix C.2). Further, our key human–model comparisons are contrasts within each method, such as proactive versus retroactive interference (Table 2) and the effects of N and K on N-Back accuracy (Fig. 11). Each method keeps the same input access across all conditions of such a contrast. Differences in input access across methods thus do not directly enter these comparisons.
Matched human control. We also test 4 new participants under input matched to the models (a supplementary experiment requested by Reviewer vCcL; details are in our response to that reviewer). They see the same 96 uniformly sampled frames as the default model input, once and without pausing or rewinding, with counterbalanced conditions. On the 55 paired Memory Interference questions, human retroactive accuracy is still 16.36 pp lower than proactive accuracy, close to the 20.00 pp under the original protocol. Viewing conditions thus do not explain why humans show this asymmetry while models do not.
Scope of our interpretations. Our contribution is a benchmark that reveals where model memory differs from human memory under controlled paradigms. We present the mechanism-level statements as empirical interpretations of these differences, with wording such as "suggests" and "we hypothesize" (e.g., Sections 4.1 and 4.2). They aim to offer insights for future model design (Section 5). Establishing causal mechanisms would require interventions on model internals. We have stated this scope explicitly in Appendix C.3, together with the matched control.
Response to Reviewer sT1d (2/2): False Memory Discrimination
Q2: Answer shortcut in False Memory Discrimination
The Neither video shortcut does not affect our reported False Memory Discrimination (FMD) results. On the balanced set requested by the reviewer, models still attribute most absent events to a video.
Why the reported results are not affected. We query each question in a separate call. The model cannot see other questions, so it cannot learn that the FMD answer is always Neither video. In Table 3, all models and agents score at most 40.00% on FMD, far below the 100% of the shortcut.
Balanced experiment. Although the shortcut does not affect our reported results, we add a balanced experiment at the reviewer's request. For each FMD video, we add three new questions. In these questions, the queried event appears only in Video 1, only in Video 2, or in both videos. The new questions use the same template and the same four options as the original question. We report two error rates:
- False positive rate (FPR): the fraction of absent-event questions (the original ones) answered with Video 1, Video 2, or Both videos.
- False negative rate (FNR): the fraction of present-event questions (the new ones) answered with Neither video.
We have completed 16 of the 95 videos (64 questions) so far and will cover all 95 videos in the revision.
| Method | Acc ↑ | FPR ↓ | FNR ↓ |
|---|---|---|---|
| Random | 25.00 | 75.00 | 25.00 |
| Always Neither video | 25.00 | 0.00 | 100.00 |
| Qwen3.5-4B | 32.81 | 81.25 | 0.00 |
| Qwen3-VL-8B-Instruct | 40.63 | 87.50 | 2.08 |
| InternVL3.5-8B | 34.38 | 100.00 | 0.00 |
| Qwen3.5-9B | 43.75 | 81.25 | 0.00 |
Results. Always choosing Neither video now reaches only 25.00% accuracy, so the shortcut no longer works. All four models attribute 81.25%–100.00% of absent events to a video (FPR), above the random level of 75.00%. In contrast, they answer Neither video for at most 2.08% of present events (FNR). This indicates that the low FMD accuracy comes from false memory, not from a bias toward Neither video. Even with 64 questions, the gap between FPR and FNR is significant for each model (disjoint 95% Wilson intervals).
We have added this experiment to Appendix J.2 and the one-call protocol to Appendix H.
If our responses have addressed your concerns, we would appreciate it if you could consider raising your score.
Overall Response to Reviewer sT1d
We sincerely thank the reviewer for the positive assessment and constructive questions. Below we briefly summarize our key responses; details for each point are in the follow-up comment replies.
We have incorporated all these clarifications and supplementary results into the revised paper, highlighted in blue. All section, figure, and appendix numbers below refer to the revised paper.