acceptodds
Under review as a conference paper at ICLR 2027

Towards Multimodal Lifelong Understanding: A Multi-Scale Benchmark

Abstract

Multimodal systems can access increasingly long video histories, but their ability to connect observations of the same subject across days and months remains insufficiently evaluated. We formalize the Lifelong Horizon, distinguishing recorded duration from the physical temporal span of a persistent subject's history. To evaluate this setting, we introduce MM-Lifelong, a benchmark with 181.1 hours of video and 1289 questions across Day, Week, and Month scales. Its Month subset contains 105.6 hours of footage spanning 51 days. Every question is paired with manually annotated clue intervals. We evaluate answer quality and temporal grounding separately, using question-specific rubrics to accommodate valid alternative answers and partial correctness. Experiments with multimodal large language models (MLLMs) and agents reveal uneven gains from larger input budgets, with the best MLLM scoring 26.53 out of 100. Among agents, answer quality and grounding rankings diverge. Error analysis identifies lost answer-relevant details, confusion between event instances, and unresolved conflicting observations. These findings motivate memory systems that preserve event-specific evidence and verification that checks its support for the final answer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Lifelong Understanding, Long-form Video Benchmarking, Autonomous Memory Agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_fix8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MM-Lifelong, a benchmark for multimodal understanding over a persistent subject’s history at Day, Week, and Month scales. It distinguishes recorded duration from physical temporal span, and includes 181.1 hours of video and 1,289 clue-grounded questions; the Month subset contains 105.6 hours of livestream footage spanning 51 days. The paper evaluates a broad range of MLLMs and video agents using both answer quality and temporal grounding, finding that current systems remain far from human performance.

Critical strengths

A valuable new evaluation setting

The distinction between recorded duration and physical temporal span is useful. The Month subset extends prior day-scale longitudinal benchmarks such as EgoLife and TeleEgo to a substantially longer, sparse history, where recurring events and changing contexts can matter.

Strong benchmark construction and evaluation

The paper provides manually annotated clue intervals and evaluates answer quality separately from temporal grounding. The comparison covers both end-to-end MLLMs and agentic systems, making the empirical picture more informative than answer accuracy alone.

Critical weaknesses

The central claim about unobserved intervals is not directly evaluated

The paper motivates the Lifelong Horizon by arguing that models must reason about events occurring during unobserved periods. However, the QA pairs are grounded in recorded clue intervals. The authors explicitly acknowledge in Appendix D(iii) that the benchmark contains no questions or annotations directly probing the content of an unobserved interval. It therefore evaluates connecting observed events across time, rather than the stronger gap-inference ability used to motivate the setting.

The distinct value of the Month scale is not established

Extending prior day-scale longitudinal benchmarks such as EgoLife and TeleEgo to the Month scale is a main contribution. However, temporal scale is fully coupled with domain and subject: Day uses gameplay, Week uses egocentric daily life, and Month uses livestreams. The paper intentionally introduces this content variation, but provides no matched comparison that separates temporal horizon from domain difficulty. Indeed, the paper notes that most MLLMs perform worse on the shorter Day subset than on Week. The results establish several challenging long-history settings, but do not show what unique value the Month horizon adds beyond broader data coverage.

Questions for authors
  1. Could the authors add a small set of questions that directly probe reasoning about the content of unobserved intervals? This would provide direct evidence for the central capability used to motivate the Lifelong Horizon.

  2. Since the Month-scale horizon is a central contribution beyond EgoLife and TeleEgo, could the authors provide an analysis of what substantive challenges or failure modes become more prominent at this scale? This would help establish the added value of Month-scale evaluation beyond expanding the benchmark’s coverage.

Reviewer_JvaKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formalizes multimodal lifelong understanding through the Lifelong Horizon, which separates observational duration (T_dur, total playback) from physical temporal span (T_span, the real-world time covered), and defines the setting via three constraints: daily-scale observation of at least 12 hours, a cross-day span of at least 24 hours containing an unobserved interval, and continuous tracking of a single persistent subject. The authors release MM-Lifelong, a benchmark of 181.1 hours of video and 1289 questions at Day (gameplay), Week (egocentric life), and Month (livestream) scales; the Month subset spans 51 days with 105.6 hours of footage. Every question is paired with manually annotated clue intervals and question-specific rubrics. Answer quality (0–5 rubric scoring) and temporal grounding (Ref@N bin IoU) are evaluated separately. Across 18 MLLMs and 8 agents in a training-free setting, the best MLLM scores 26.53/100; larger input budgets yield uneven gains, answer and grounding rankings diverge among agents, and error analysis identifies lost evidence, event-instance confusion, and unresolved conflicting observations as recurring failures that motivate memory designs preserving event-specific evidence and verifying its support for answers.

Critical strengths
  • The Lifelong Horizon formalization separates T_dur from T_span via three operational constraints, and the month-scale subset (105.6h over 51 days) exceeds prior corpora in physical span by 2–3 orders of magnitude.
  • All 1289 questions come with 1810 manually annotated clue intervals and per-question rubrics, backed by common-sense filtering and independent human re-annotation (>80% agreement).
  • Results reveal uneven gains from larger input budgets, diverging answer/grounding rankings among agents, and three recurring failure modes (lost evidence, instance confusion, unresolved conflicts) that give concrete targets for memory-system design.
Critical weaknesses
  • The rubric verifier and the best-performing model come from the same family: GPT-5.6-luna both achieves the top answer score and grades every prediction (§4.5), creating a risk of self-grading bias; verifier–human agreement (κ=0.63, MAE 0.69) is below human–human agreement (0.67, 0.58), and the alignment study covers only 400 responses.
  • The benchmark is statistically thin: one subject per scale, 1289 questions in total, and no held-out test partition for the month scale (Table 2 shows zero Month test questions), so the headline result is reported on validation and subject-specific quirks cannot be separated from general capability, as acknowledged in §D.
  • Grounding evaluation is coarse and heterogeneous: the default Ref@300 uses 5-minute bins while 57.4% of clue intervals are shorter than 90 seconds (Table 2); answer and grounding runs are separate (Table 4 note), and agent intervals come from heterogeneous sources (reported intervals, selected frames, tool traces), limiting comparability across agent families.
Questions for authors
  • Remove the verifier confound by re-scoring at least the top systems with a verifier from a different model family (or a substantially larger human-adjudicated sample), reporting verifier sensitivity per model, and releasing verifier outputs alongside the rubrics for reproducibility and audit.
  • Strengthen the statistical footing by adding a held-out test partition at the month scale (e.g., the temporally last ~15% of the stream) or reporting dev–val gaps, and by reporting results broken down by the 11 question categories, clue-duration buckets, and the two task families so the Type I/II design in §4.2 is actually validated.
  • Unify and refine the grounding evaluation by reporting Ref@N at finer default bins (e.g., 60–120 seconds) or full bin-width curves in the main text, defining a uniform protocol for converting agent tool traces into time intervals, and evaluating grounding on the same runs as answer scoring or explicitly justifying the separate-run design.
Reviewer_BpG2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces MM-Lifelong, a long video-QA benchmark spanning ~181 hours and 3 scales (Day, Week and Month) that distinguishes recorded duration from physical temporal span (real time covered) of a single subject's history, a concept termed as Lifelong Horizon. The authors further evaluate SOTA Multimodal LLMs as well as agentic methods without benchmark specific finetuning and evaluate the answer quality (on a scale of 0-5 with question-specific rubrics and LLM as a Judge) and temporal grounding (Ref@N). The best performing model achieves 26.53 / 100 as compared to the human performance of 88.45. Experiments and other analyses show that more input context helps unevenly, answer quality and grounding assign different ranks to the models and three major types of errors occur : missed details, confusion between similar events and unresolved conflicting evidence.

Critical strengths

S1. Separating the amount of recorded footage from the physical temporal span is simple and practically relevant. This distinction matches how real assistants work as they encounter footage with gaps for sleep, privacy etc and yet have to answer queries like "when did this happen ?" which requires temporal depth over breadth. This framing also helps in designing systems. Computational costs grow as the duration of videos increase, while aspects related to memory and retrieval must deal with the gaps in footage. Decoupling them makes it clearer what to optimize.

S2. Valuable effort to manually annotate month-level longitudnal data, which the community needs for building efficient real-world assistants, so that they can process long streams of a single subject and align it's responses with subject specific requirements.

S3. Rubric-based partial credit is a simple and nice way for evaluating free form questions rather than exact matches. The system prompt used for grading and a working example further make things clear.

S4. Data contamination is satisfactorily addressed. For instance, the authors' dropped a year-scale split because models could answer it from web knowledge (Appendix 4.2). Other experiments show that - more tokens help unevenly (Qwen3.5-9B gets a gain of 4.36 points by increasing input context from 32K to 1M, whereas Eagle-2.5 only 0.49), content matters and not length as many methods score low on the day split as compared to the week split, Appendix C highlights failure cases of the strongest agentic baseline (GPT5.6 luna).

Critical weaknesses

W1. It is not yet fully shown that the benchmark tests lifelong understanding. The paper does not report how many questions require evidence from multiple sessions or across days. Moreover, Table 2 shows that a significant number of questions (~69%) have a temporal certificate of under one hour, so many questions might be long-video retrieval questions. The splits are not clear as well - Table 9 (Week Split) shows the cumulative duration of the videos to be around 80 hours, whereas Table 1 and Table 3 report 51.9 hours.

W2.Though simple and intuitve,the partial credit score makes the scores hard to read, and scores in the range 20-30 might be interpreted as guessing and prior knowledge. The jump in performance of models on using web search (text-only Gemini in Appendix A.4.1), show that there are Month-scale questions that can still be answered using information online.

W3. The paper asserts that "missing evidence" causes approximately 50 % of the errors for the three MLLMs analyzed in Fig.4 and interprets this as a memory problem. Input token budget is given but frame sampling and FPS and frame resolution information is not reported. Please report the sampling settings and add an additional oracle experiment, where only the annotated evidence intervals are provided. If the scores jump, the bottleneck becomes finding the evidence, else the bottleneck is understanding the evidence itself.

W4. Answer quality and grounding are scored in separate runs, which shows that the diversion in two results is a built-in design choice. In Example.1 in Appendix C, Codex answers the question correctly yet inspects evidence outside the manually annotated reference interval. Evaluation using a metric that gives credit only when the answer is correct and temporal window overlaps the annotated interval would give better interpretation.

The central claim of the paper is that it probes Lifelong Understanding, however the current experiments and analyses do not sufficiently solidify this. Since scores are average partial credit, the numbers are hard to interpret as lifelong understanding. The reviewer, however wants to acknowledge that the problem is well-motivated and the manual annotation required substantial effort.

Questions for authors

Questions

  1. What fraction of questions need evidence from two or more sessions or days? How does accuracy change with certificate length and with single vs cross-session questions?
  2. What are random-guess, text-only and strict-accuracy results for all systems?
  3. Can you report agreement with a judge from a different model family ?
  4. Please show a joint metric that gives credit only when the answer is correct and the predicted interval overlaps the marked interval (maybe above a certain threshold) for a subset of the benchmark ? How did annotators confirm no other valid interval exists, especially for counting and “entire trip” questions?
  5. Is 51.9 hours in Table 1 a subset of the ~80 hours (week-split) shown by Table 9? Additionally, please report the temporal information received by the models, if concatenated clips alone are the input to the model, the task essentially becomes long video QA over a large concatenated video.

Minor Concerns

  1. (iii) in Appendix D states that unobserved-gap reasoning is not tested as it requires new task formulation and ways of validating ground-truth without access to the unrecorded footage. While Sec.2. describes probing subject's state across weeks of unobserved gap.
  2. The data comes from livestreams but evaluation appears offline. Please explain this clearly.
Reviewer_Vy5LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalizes the Lifelong Horizon, distinguishing recorded duration from the physical temporal span of a persistent subject’s history. To evaluate this setting, we introduce MM-Lifelong, a benchmark with 181.1 hours of video and 1289 questions across Day, Week, and Month scales.

Critical strengths

1.The paper establishs evaluation Benchmarks and analyzes the challenges of long-form video understanding. It covers very long video understanding 2. Beyond simply asking "is the answer correct?", the evaluation also requires identifying the supporting evidence span, assessed separately using the Ref@300 metric. This makes it possible to distinguish between cases where the model "finds relevant segments but provides an incorrect answer" and cases where it "arrives at the correct answer but fails to pinpoint the supporting evidence accurately."

Critical weaknesses

1.Compared to existing long-video benchmarks, the value of this new evaluation framework still requires further substantiation. 2. A long timespan does not in itself prove that answering these questions requires a long-term historical perspective. Conducting controlled evaluations, such as shortening the historical time window, shuffling the chronological order of events, or removing early evidence, can help distinguish between long-term reasoning and localized information retrieval.

Questions for authors
  1. It is hard to tell whether the model truly utilizes long-term history. The video coverage spans 51 days, but this doesn't mean every question requires understanding the entire 51-day history. This can be achieved by limiting the history window, shuffling the order of segments, or removing comparisons from earlier events.
  2. Beyond what is covered by existing long-video benchmarks, what additional capabilities does MM-Lifelong evaluate? And what evidence is there to suggest that it offers value beyond simply addressing longer time spans?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.