acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies

Abstract

Answering questions about long videos often requires connecting events involving the same objects across hours or days. Chronological descriptions and text-derived entities can leave physical identity unresolved: different objects may share a description, while observations of the same object remain disconnected across events. Retrieving relevant events therefore does not necessarily recover the “biography” of the particular entity a question concerns. To address this, we introduce Grounded Entity Biographies (GEB), a long-video memory framework that groups visually grounded observations of the same physical instance across clips into retrievable biographies while preserving the context of each moment. During question answering, the biography is retrieved alongside episodic evidence, allowing the model to follow an entity through events using identity links established during memory construction. Evaluations across four benchmarks, including day-long and week-long recordings, demonstrate improvements over prior memory frameworks in both multiple-choice and open-ended question answering. On EgoLifeQA, GEB achieves 72.0% accuracy, 4.4 percentage points above the best published result. Ablations show that grounded identity association and biography reading both contribute to the gains, which additional descriptions alone do not fully recover.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding, Video Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nNViOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Grounded Entity Biographies (GEB), a memory framework for long-video understanding. Its core idea is to associate observations belonging to the same physical instance across different video clips, building entity biographies that carry persistent identity information, and to connect entity identity with episodic information to support cross-time event retrieval and reasoning. Experiments on multiple long-video question-answering benchmarks show that GEB outperforms existing memory-based methods; ablation studies and evidence-retrieval analysis further validate the role of each major component.

Critical strengths
  1. The paper addresses entity identity ambiguity in long-video memory systems: text descriptions alone often cannot distinguish different physical instances or associate observations of the same entity across time. The proposed entity-biography representation offers a sound approach to organizing and retrieving information across time.
  2. The method outperforms recent memory-based approaches on multiple benchmarks, and the roles of the main components are validated through modular ablations and evidence-retrieval analysis.
  3. The paper is well organized, the method is clearly described, and the experimental tables and qualitative examples are presented effectively, facilitating understanding of both the method design and its outcomes.
Critical weaknesses
  1. The memory structure used by GEB is substantially larger than that of several compared methods. Although the paper controls for the reasoning model and retrieval setup, it does not adequately separate the effect of increased memory capacity on performance, making it difficult to fully attribute the performance gains to the identity-based organization of the memory.

  2. On MultiHop-EgoQA, removing identity association drops complete evidence coverage from 52.1% to 48.4%, yet the answer score remains at 3.11. The paper does not adequately explain why the improvement in evidence-retrieval performance fails to translate into a sustained gain in answer quality.

  3. GEB introduces additional processing for object tracking, visual description generation, and cross-clip identity association. However, the paper does not adequately quantify the computational cost of constructing the memory, making it difficult to assess the overall computational overhead of the proposed framework.

Questions for authors
  1. Could the authors provide visualizations of representative entity-identity association failure cases, particularly incorrect identity merges (where distinct entities are wrongly merged) and fragmented biographies (where observations of the same entity are incorrectly split across multiple entity biographies)?

  2. Could the authors report QA performance separately on different question types, depending on whether a question requires cross-event identification of the same physical instance, so as to further analyze under which conditions entity biographies yield more significant gains?

  3. On MultiHop-EgoQA, removing identity association leads to a drop in complete evidence coverage while the answer score stays unchanged. Could the authors explain this phenomenon?

  4. Could the authors report GEB's offline memory construction time, and further break down the time cost of each major computational module?

4 comments
AuthorsOpenReview ↗

Response to Reviewer nNVi (1/2)

We thank the reviewer for their time and feedback. We appreciate that the reviewer found the approach "sound" and the paper "well organized". We answer questions next.

A reminder of the terms we use. GEB (our method) tracks every object inside each 30-second clip of a long video; one such track is an observation. GEB then decides which observations from different clips show the same physical object, and links them. This step is the association, and all linked observations of one object form an entity.

W1: Separating the gain from the effect of a larger memory

We ran exactly this experiment. A GEB memory that is smaller than MAGIC-Video's still outperforms it, so GEB's lead does not depend on a larger memory.

We rebuilt GEB's memory from only 15% of its observations, selected without looking at any question. This memory takes 11 MB per hour of video; the memory of MAGIC-Video, the strongest published memory framework, takes 13 MB.

MemorySize per hour of videoEgoLifeQAEgo-R1-BenchTest@Week
MAGIC-Video13 MB67.664.730.92
GEB, 15% of the observations11 MB70.065.335.75
GEB, as in the paper44 MB72.071.336.83

Two facts from the paper point the same way. First, GEB and MAGIC-Video run under the same limits on what reaches the model that writes the answer: the same number of retrieved items, search rounds and video frames. Second, when we keep all of GEB's descriptions of the observations but attach them to the clips instead of organizing them by object, accuracy drops by 3.8 points on EgoLifeQA (Table 4).

The matched-size comparison is now in Appendix H.2.

W2 / Q3: Coverage drops but the answer score stays the same

The answers do get worse without association: they point to less of the right evidence. The 1–10 answer score is simply too coarse to show a difference of this size.

What the reviewer observed. MultiHop-EgoQA is a benchmark whose questions need evidence from several moments of a video. Evidence coverage is the share of questions for which all the annotated evidence reaches the model that writes the answer. Without association it falls from 52.1% to 48.4%, while the answer score stays at 3.11 (Table 3).

The answers do get worse. On MultiHop-EgoQA, the model must also cite the time intervals that support its answer. Without association, these cited intervals match the annotated evidence less well: their overlap (mIoG) falls from 36.7 to 34.1 (Table 11). The 95% confidence interval of this drop, 1.7 to 3.6 points, excludes zero.

Why the score does not move. The score is a grade from 1 to 10 that a judge model gives to the written answer, and it reacts slowly to how much evidence is retrieved. To measure how slowly, we ran a new experiment on GEB itself: we limited the number of search rounds per question, which lowers the coverage in a controlled way. The paper allows 5 rounds.

Search rounds allowedEvidence coverageAnswer score (1–10)
129.5%2.60
239.1%2.95
344.5%3.00
5 (the paper)52.1%3.11

The score follows the coverage, but slowly. Going from 5 rounds to 3 lowers the coverage by about 7.5 points, twice the loss from removing association, and the score changes by only 0.11. A coverage loss half as large, the 3.7 points lost without association, would move the score by about 0.05 on the 1–10 scale, which is too small to observe.

This explanation and the new experiment are now in Appendix H.4.

W3 / Q4: Construction cost, per module

Here is the breakdown by module. We report the cost per 30-second clip, the unit in which GEB processes a video, and in numbers of model calls, so that it depends neither on the length of the video nor on the hardware. The numbers are averages over the 52-hour EgoLife video.

ModuleModelPer 30-second clip
Detect and track objectsYOLOE-26x-seg + BoT-SORT300 frames, giving 49 observations
Describe each observationQwen3.5-35B (VLM)49 calls
Embed each observation for the associationQwen3-VL-Embedding-8B49 embeddings
Associate observations across clipssimilarity comparison, no model call12 entities
Embed the descriptions and build the graphQwen3-Embedding-4B49 embeddings

The description module dominates the cost. It is the only module that calls a large model: one vision-language-model call per observation, about 49 per 30-second clip. Tracking and embedding use small models, and the association itself needs no model call. The memory is built once and then answers every question about the video. A fraction of the calls already suffices: with only 15% of the observations described (7 calls per clip), GEB is still ahead of MAGIC-Video on all three benchmarks, as the table in our answer to W1 shows.

This table is now in Appendix H.2.

AuthorsOpenReview ↗

Response to Reviewer nNVi (2/2)

Q1: Failure cases of the association

We have added Figures 11 and 12 to the revised paper (Appendix H.1). They show a wrong merge and a fragmented biography.

How often do these failures happen? We measured the association against human judgment on three datasets. Wrong merges are rare: 87% to 92% of GEB's links are correct. Fragmented biographies are more common: GEB finds 32% to 46% of the true same-object pairs, because it links two observations only when the visual evidence is strong. We chose this side on purpose: a missed link can still be recovered when a question is asked, while a wrong link would mix the histories of two objects.

In each figure, a row is one observation: video frames with the object marked by a box, the time of each frame, and GEB's description of the object.

  • A wrong merge (Figure 11). Two pillows of the same kind lie side by side. Nothing in their appearance tells them apart, and GEB merges them. Objects whose detection boxes are apart in the same frame are never merged, so this happens only across clips.
  • A fragmented biography (Figure 12). The same plush toy appears in two consecutive clips, and GEB keeps the two observations in two entities, because a new observation must agree with every recent observation of an entity. Both observations carry the same name, "lion plush toy". When they are retrieved together, GEB presents them with a note that they may be the same object.

Q2: Performance by question type

GEB is ahead of both memory frameworks on every question type, and its gain is large on the questions about specific objects and people, where recognizing the same instance across events matters.

We answer on MultiHop-EgoQA, because it is the benchmark that labels every question with a category and annotates all the moments that contain its evidence. The table gives the answer score (1–10) per category, from Table 12 of the paper.

Category (number of questions)MAGIC-VideoWorldMMGEBGain over the better baseline
Repeated activities (73)2.912.973.24+0.27
Multiple actions (241)2.532.752.91+0.16
Multiple objects (193)2.402.432.90+0.47
Locations / people (72)2.512.593.25+0.66
Event composition (111)2.322.572.90+0.33
Event comparison (34)4.234.685.90+1.22

GEB is ahead of both in every category. Among the five large categories, the two about specific objects and people gain the most (+0.47 and +0.66); the small category that compares two events gains even more (+1.22).

Questions whose evidence is spread over three or more moments (285 questions) show the same advantage: GEB's answer score is 2.97, against 2.80 for WorldMM and 2.63 for MAGIC-Video.

Reviewer_nNViOpenReview ↗

Additional Comment of Reviewer nNVi

Thank you for the detailed response and additional experiments. Most of my concerns have been addressed, but two points remain:

Memory Construction Cost. The breakdown of model calls is helpful, but the actual construction time remains unclear. Could the authors report the wall-clock time and hardware configuration?

Contribution of Identity Association. The question-type analysis is useful, but it does not directly distinguish questions that require cross-event identification of the same entity. Could the authors compare performance on questions with and without this requirement?

We sincerely thank the reviewer for the follow-up. We are very glad that most of the concerns are now addressed. The two remaining questions are both valuable, and answering them has made the paper stronger. We answer them next.

Q4: Wall-clock time and hardware of memory construction

Using one node with 8 H100 GPUs which also hosts the LLM to describe observations, GEB builds its memory in 19.8 minutes per hour of video on average. If the LLM is called through the OpenRouter API, the time falls to 7.5 minutes per hour of video. With our two nodes and a self-hosted LLM, memory for the 52-hour EgoLife video is built in about 8.8 hours.

The table gives the average wall-clock time of each module per hour of video. The description module and the total have two values. The first value is the time needed when hosting Qwen3.5-35B on our node. The second value is the time needed when the same model is called through the OpenRouter API.

ModuleHardwareTime per hour of video
Detect and track objects (YOLOE-26x-seg + BoT-SORT)1 H100 node1.4 min
Describe each observation (Qwen3.5-35B)1 H100 node / OpenRouter API17.1 / 4.8 min
Embed each observation for the association (Qwen3-VL-Embedding-8B)1 H100 node0.7 min
Associate observations across clipsCPU only0.5 min
Embed the descriptions and build the graph (Qwen3-Embedding-4B)1 H100 node and CPU0.1 min
Total19.8 / 7.5 min

Describing the observations is the most time-consuming module. We choose to host the Qwen3.5-35B model ourselves. Each copy of the model runs on two GPUs (tensor parallelism of 2), so one node serves four copies at the same time, and the module takes 17.1 minutes per hour of video. If one prefers not to host the model, an external API can be used instead: through OpenRouter, the module takes 4.8 minutes, which is 3.6 times as fast. The time of this module therefore depends on which vision-language model is used and how it is hosted, and it is proportional to the number of observations described. Describing only 15% of the observations takes about 2.6 minutes per hour of video (self-hosted). Despite the reduced memory, GEB is still ahead of MAGIC-Video on all three benchmarks (70.0 against 67.6 on EgoLifeQA, for example). The memory is built once offline and then answers every question about the video online. These timings are now in Appendix H.2.

Q2: Questions with and without cross-event identification of the same entity

On the questions that require recognizing the same object or person across events, association raises evidence coverage by 5.9 points. On the other questions, it adds 1.3 points.

Concretely, we labeled each of the 1,080 questions of MultiHop-EgoQA, from the question and its reference answer only. A question requires cross-event identification when one object or person is the subject of two or more of its evidence moments. Either the question asks what happened to this object, as in "What did I do with the box before putting it on the wall?", or it asks about an action repeated on it, as in "When did I open the tap?". This holds for 569 questions. Of the other 511, 254 have several evidence moments that each concern a different object, as in "What items did I place on the kitchen counter?", and 257 have all their evidence in one moment. The table compares GEB with association, as in the paper, and without it, on evidence coverage and answer score.

Questions (number)GEBEvidence coverageAnswer score
With the requirement (569)Without association35.4%3.01
With association41.3% (+5.9)3.06
Without the requirement (511)Without association62.8%3.14
With association64.1% (+1.3)3.17

The coverage gain from association is in parentheses. It is 4.6 points larger on the questions with the requirement than on the others.

Table 3 of the paper reports that removing association lowers evidence coverage by 3.7 points on the whole benchmark. 83% of this loss falls on the questions with the requirement, which make up 53% of the benchmark.

The answer score is also higher with association, by 0.05 on the questions with the requirement and by 0.03 on the others.

This table is now in Appendix H.5.

Reviewer_p5iUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Grounded Entity Biographies (GEB), a long-video memory framework that organizes visually grounded observations of the same physical instance into persistent entity biographies while preserving their episodic context and visual evidence. By linking observations across time through identity-aware associations, GEB enables retrieval to follow an entity across events and supports long-horizon video question answering. Experiments on four benchmarks demonstrate improvements over existing memory frameworks, with ablations examining the contributions of entity association, episodic connections, and biography reading.

Critical strengths

1、 The paper identifies an important limitation of existing long-video memory systems: retrieving semantically relevant events does not guarantee that they involve the same physical entity. Organizing memory around persistent physical instances provides a compelling solution to this problem.

2、 Comprehensive memory design and empirical validation. GEB integrates visually grounded observations, cross-time entity association, episodic context, and identity-aware retrieval into a coherent framework. The evaluation spans multiple long-video benchmarks, and the ablation studies provide evidence that physical-instance organization and biography reading contribute beyond simply adding more textual descriptions.

Critical weaknesses
  1. Entity identity association is central to GEB, as incorrect associations can connect unrelated events and introduce misleading evidence into downstream reasoning. However, the evaluation primarily measures downstream question-answering performance rather than directly quantifying cross-clip identity association accuracy. Reporting identity-level precision and recall against manually verified correspondences would provide stronger evidence for the reliability of the proposed memory representation.

  2. The association thresholds are calibrated through visual inspection and differ across recording domains, raising questions about whether GEB can generalize to unseen environments without manual adjustment. Further sensitivity analyses and evaluations across diverse recording conditions would help establish the robustness of the proposed association mechanism. In addition, the improvement on MM-Lifelong Test@Day over the strongest reported baseline is relatively small, and the reported 95% confidence interval includes zero, leaving the statistical significance of this gain uncertain.

  3. The reported retrieval latency averages 13.6 seconds per question, excluding controller and answer-model inference. Given the substantial size of the memory graph, further analysis of end-to-end latency, memory construction overhead, and computational cost as recording duration and entity count increase would strengthen the case for practical deployment.

Questions for authors

see Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer p5iU (1/2)

We thank the reviewer for their time and feedback. We appreciate the reviewer recognizing that "organizing memory around persistent physical instances provides a compelling solution" to a real limitation of current systems. We answer questions next.

A reminder of the terms we use. GEB (our method) tracks every object inside each 30-second clip of a long video; one such track is an observation. GEB then decides which observations from different clips show the same physical object, and links them. This step is the association, and all linked observations of one object form an entity.

W1: Precision and recall of the association

We agree, and we have now measured both. When GEB links two observations, the link is correct about 9 times out of 10.

Concretely, we compared GEB's links with human judgment on three datasets:

  • EgoLife, the week-long home video of our main experiments, and the gameplay video, a day-long game stream used by the MM-Lifelong Test@Day benchmark. We sampled pairs of observations at random, looked at their video frames, and decided whether the two show the same object.
  • EgoTracks, a public dataset in which human annotators marked each object with a box throughout a video, also after the object leaves the view and comes back. We ran GEB on 60 EgoTracks videos (7 hours, 172 annotated objects) with the same settings as on EgoLife.

Precision is the share of GEB's links that are correct. Recall is the share of true same-object pairs that GEB links.

DataPrecisionRecall
EgoLife (day 1)87% of 200 links34% of 300 same-object pairs
Gameplay video89% of 100 links32% of 100 same-object pairs
EgoTracks (public annotations)92% of 912 links46% of 2,658 same-object pairs

Precision is high on all three, including EgoTracks, where nothing was tuned. Recall is lower because GEB links two observations only when the visual evidence is strong. We made this choice on purpose. A missed link can still be recovered when a question is asked, because a search returns several entities, and the model that writes the answer reads them together. A wrong link would mix the histories of two different objects.

This evaluation is now in Appendix H.1 of the paper.

W2: Generalization to unseen videos, and sensitivity to the thresholds

Yes, GEB generalizes without manual adjustment, and its results do not depend on a finely tuned threshold.

A different dataset. On EgoTracks, the EgoLife thresholds give 92% precision without any change (table above). With EgoLife (daily life in a house), EgoTracks (many short videos by different people) and the gameplay video, the association is now evaluated under three very different recording conditions.

Sensitivity. We rebuilt the EgoLife memory with looser and stricter thresholds, changing nothing else, and re-ran the three benchmarks on this video.

ThresholdsEgoLifeQAEgo-R1-BenchTest@Week
0.55 / 0.70 (looser)69.671.335.75
0.60 / 0.75 (used in the paper)72.071.336.83
0.70 / 0.85 (stricter)71.869.335.00
0.75 / 0.85 (strictest)71.467.334.75
MAGIC-Video, for reference67.664.730.92

Every setting stays ahead of MAGIC-Video, the strongest published memory framework, on all three benchmarks. The gameplay video behaves similarly: across four threshold settings its MM-Lifelong Test@Day score stays between 16.25 and 17.58, against 10.08 for MAGIC-Video.

No question, answer or QA result is used when the thresholds are chosen. Appendix A now describes how they are chosen, and Appendix H.1 has the tables.

W2: The gain on MM-Lifelong Test@Day over the strongest reported baseline

That baseline, ReMA, runs on GPT-5, while GEB runs on the much smaller open model Qwen3.5-35B. With the same model for both, GEB is 10 points ahead.

To make the comparison fair, we ran ReMA with Qwen3.5-35B in every role, with the clip length that its paper found best.

SystemLanguage modelTest@Day
ReMA (published)GPT-516.75
ReMAQwen3.5-35B7.33
WorldMMQwen3.5-35B8.50
MAGIC-VideoQwen3.5-35B10.08
GEB (ours)Qwen3.5-35B17.58

With the same language model, GEB is 10.25 points above ReMA. Put differently, GEB's memory lets a 35B open model match a GPT-5 agent. Table 1 now includes the new ReMA row.

AuthorsOpenReview ↗

Response to Reviewer p5iU (2/2)

W3: End-to-end latency, and how it grows with the video

End to end, answering a question of EgoLifeQA takes about 40 s, most of it in language-model calls. The cost of a memory search grows at most linearly with the video, and slower in practice.

End to end. On average, a question of EgoLifeQA takes 3.5 calls to decide what to search, 2.7 memory searches, and one call to write the answer. A memory search itself calls no language model.

Growth with the video. A search compares the question with every item in the memory and then runs Personalized PageRank over the links between the items. Each observation adds one item and at most three links, so the work grows linearly with the length of the video. We measured it on memories built from the first 1, 3, 5 and 7 days of EgoLife.

Memory built fromItems in the memoryTime per search
day 154K (1.0×)1.0×
days 1–3156K (2.9×)1.5×
days 1–5263K (4.9×)2.2×
days 1–7, the full week350K (6.5×)2.0×

A memory 6.5 times larger takes only twice as long to search, because a part of every search, such as encoding the question, has a fixed cost. This analysis is now in Appendix H.3.

W3: Memory construction overhead

Building the memory takes one vision-language-model (VLM) call per observation, about 49 per 30-second clip. This cost grows linearly with the video and is paid only once.

We report the cost per 30-second clip, the unit in which GEB processes a video, so that it depends neither on the length of the video nor on the hardware. The numbers are averages over the 52-hour EgoLife video.

StepModelPer 30-second clip
Detect and track objectsYOLOE-26x-seg + BoT-SORT300 frames, giving 49 observations
Describe each observationQwen3.5-35B (VLM)49 calls
Embed each observation for the associationQwen3-VL-Embedding-8B49 embeddings
Associate observations across clipssimilarity comparison, no model call12 entities
Embed the descriptions and build the graphQwen3-Embedding-4B49 embeddings

The description step dominates: it is the only step that calls a large model. Detection, tracking and embedding use small models, and the association itself needs no model call. Once built, the memory answers every question about the video. A fraction of the calls already suffices: describing only 15% of the observations (7 calls per clip) still gives 70.0% on EgoLifeQA, against 72.0% with all of them and 67.6% for MAGIC-Video. The table is now in Appendix H.2.

Dear Reviewer p5iU,

As the discussion period draws to a close, we wanted to briefly highlight the updates addressing your concerns.

  1. Association reliability (W1): Direct evaluation on EgoLife, gameplay, and EgoTracks shows 87–92% precision and 32–46% recall, reflecting our conservative approach to avoiding incorrect merges.
  2. Generalization and robustness (W2): EgoLife thresholds achieve 92% precision on 60 EgoTracks videos without adjustment. All four tested threshold settings outperform MAGIC-Video on the three EgoLife-based benchmarks, and calibration uses no QA supervision.
  3. Controlled ReMA comparison (W2): With Qwen3.5-35B used by both methods, GEB scores 17.58 versus ReMA’s 7.33 on Test@Day, and matches the published GPT-5-based ReMA (16.75).
  4. Latency and scaling (W3): End-to-end QA takes approximately 40 seconds per question on EgoLifeQA. From one day to seven days, memory size grows 6.5× while measured search time grows approximately 2×.
  5. Construction cost (W3): Construction takes 19.8 minutes per video hour on an eight-H100 node. Describing only 15% of observations reduces description calls by approximately 85% while preserving gains over MAGIC-Video on all three EgoLife-based benchmarks.

Thank you again for your time and feedback. We would be happy to discuss any remaining concerns.

Best regards,

The Authors of GEB

Reviewer_vMuYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GEB which is a long-video memory framework that links visual observations of the same instance across clips into persistent entity biographies while retaining episodic context and source evidence. At query time, GEB retrieves through same-instance and context links so that a matched observation can lead to other appearances of the same entity and their surrounding events, enabling reasoning over distributed evidence across long videos. Experiments on long video qa benchmarks show improvements over prior memory frameworks under matched controller and answer-model settings, with ablations supporting the contributions of instance association, identity-aware retrieval, and biography reading.

Critical strengths

1. The paper identifies a clear and practically important limitation of existing long-video memory frameworks.

In particular, it highlights that retrieving semantically relevant events is insufficient when the system cannot determine whether observations correspond to the same physical instance across time. This motivates an entity-centric representation that is well aligned with the challenges of long-horizon video understanding.

2. The empirical evaluation is carefully controlled and includes targeted ablations.

GEB is compared with major memory baselines under matched controller and answer-model settings, and the ablations separately examine identity association, same-instance connections, episodic links, and biography reading. This makes the reported gains more informative by showing which parts of the framework contribute to downstream performance.

Critical weaknesses

W1. The core physical-instance association mechanism is not directly validated and appears to rely on recording-specific manual calibration.

GEB’s main contribution depends on correctly linking observations of the same physical instance across clips, yet the paper primarily evaluates this component indirectly through downstream QA and retrieval performance rather than reporting direct association quality such as precision, recall, or false-merge and false-split rates. This is particularly important because the association thresholds are calibrated separately for each recording through visual inspection of sampled merges, with different thresholds used for EgoLife and the gameplay stream. Moreover, the paper explicitly notes that its evidence-coverage metric measures access to temporally relevant context rather than the correctness of identity assignments. Since an incorrect association can create spurious links between otherwise unrelated events, the lack of direct identity-association evaluation makes it difficult to assess the robustness and generalization of the central mechanism beyond the calibrated recordings.

W2. The methodological novelty is somewhat limited relative to the additional system complexity and offline construction cost.

GEB combines several established components, including open-vocabulary detection and tracking, VLM-based observation descriptions, embedding-based cross-clip association, graph-based memory, and Personalized PageRank retrieval, with the main novelty lying in organizing these components around persistent entity biographies. At the same time, this design introduces substantial preprocessing and memory overhead: each observation is described with a Qwen3.5-35B call, and the 52-hour EgoLife recording produces more than 300K tracked observations. The resulting memory contains roughly 350K nodes and over 900K edges, substantially larger than the compared graph-based memories. Although the paper reports query-time retrieval latency, it does not provide a comparable analysis of total memory-construction time, compute, or cost relative to the baselines. Given that the conceptual contribution is largely an integration of existing techniques, a clearer cost-benefit analysis is important for establishing whether the added complexity is justified by the reported gains.

Questions for authors
  • Could the authors clarify how the association thresholds are selected for a new unseen recording and whether any evaluation data or downstream QA results are used during their recording-specific calibration?

  • Do the authors have direct measurements of physical-instance association quality, such as precision, recall, or false-merge and false-split rates, to validate the central mechanism independently of downstream QA performance?

  • Could the authors quantify the total offline memory-construction cost, including VLM calls, compute time, and storage, and compare it with the corresponding costs existing models?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vMuY (1/2)

We thank the reviewer for their time and feedback. We appreciate the reviewer noting that the paper "identifies a clear and practically important limitation of existing long-video memory frameworks" and that its evaluation is "carefully controlled". We answer questions next.

A reminder of the terms we use. GEB (our method) tracks every object inside each 30-second clip of a long video; one such track is an observation. GEB then decides which observations from different clips show the same physical object, and links them. This step is the association, and all linked observations of one object form an entity.

Q2: Do the authors have direct measurements of the association quality?

Yes, we now do. When GEB links two observations, the link is correct about 9 times out of 10.

Concretely, we compared GEB's links with human judgment on three datasets:

  • EgoLife, the week-long home video of our main experiments, and the gameplay video, a day-long game stream used by the MM-Lifelong Test@Day benchmark. We sampled pairs of observations at random, looked at their video frames, and decided whether the two show the same object.
  • EgoTracks, a public dataset in which human annotators marked each object with a box throughout a video, also after the object leaves the view and comes back. We ran GEB on 60 EgoTracks videos (7 hours, 172 annotated objects) with the same settings as on EgoLife.

Precision is the share of GEB's links that are correct. Recall is the share of true same-object pairs that GEB links.

DataPrecisionRecall
EgoLife (day 1)87% of 200 links34% of 300 same-object pairs
Gameplay video89% of 100 links32% of 100 same-object pairs
EgoTracks (public annotations)92% of 912 links46% of 2,658 same-object pairs

Precision is high on all three, including EgoTracks, where nothing was tuned. Recall is lower because GEB links two observations only when the visual evidence is strong. We made this choice on purpose. A missed link can still be recovered when a question is asked, because a search returns several entities, and the model that writes the answer reads them together. A wrong link would mix the histories of two different objects.

This evaluation is now in Appendix H.1 of the paper.

Q1: How the thresholds are chosen

No evaluation data or QA result is used. For a new video, the thresholds can be set automatically.

How we chose them. GEB links a new observation to an entity when the two are similar enough. Two similarity thresholds define "enough", as discussed in Section 3.3 of the paper (Equation 3): a consistency threshold, which the new observation must reach with every recent observation of the entity, and a matching threshold, which it must reach with at least one of them. On EgoLife, we used the first hour of the video and chose thresholds such that about 90% of the links were correct. The thresholds of the gameplay video were chosen in the same way.

A new video. The EgoLife thresholds are reused as is. On EgoTracks, a public dataset of videos with human tracking annotations, they yield 92% precision without any change. The gameplay video is the one case where we raised the thresholds, because a video game shows many identical copies of the same object. Even there, keeping the EgoLife thresholds would change the MM-Lifelong Test@Day score only from 17.58 to 17.00.

Setting the thresholds automatically. The inspection can be avoided. Two objects whose detection boxes are apart in the same frame are certainly different objects, so linking them is an error. We choose the thresholds such that this error is about as rare as on EgoLife. On the gameplay video, this rule leads to nearly the same thresholds as our inspection.

Appendix A now describes this procedure.

W1: Robustness beyond the calibrated recordings

GEB's results do not depend on a finely tuned threshold. Beyond the EgoTracks result above, we rebuilt the EgoLife memory with looser and stricter thresholds, changing nothing else, and re-ran the three benchmarks on this video.

ThresholdsEgoLifeQAEgo-R1-BenchTest@Week
0.55 / 0.70 (looser)69.671.335.75
0.60 / 0.75 (used in the paper)72.071.336.83
0.70 / 0.85 (stricter)71.869.335.00
0.75 / 0.85 (strictest)71.467.334.75
MAGIC-Video, for reference67.664.730.92

Every setting stays ahead of MAGIC-Video, the strongest published memory framework, on all three benchmarks. The table is now in Appendix H.1.

AuthorsOpenReview ↗

Response to Reviewer vMuY (2/2)

W2: Limited methodological novelty

We respectfully disagree. The contribution of GEB is a new way to organize the memory of a long video, and the ablations show that the gains depend on this organization.

The memory frameworks we compare with, MAGIC-Video and WorldMM, make a long video searchable by storing what happened in each clip. They identify objects by the names extracted from their captions. A name, however, does not identify a physical object: two different mugs can both be "a red mug", and one mug can be named differently in different clips. On EgoLife, 77% of the objects that are seen more than once receive two or more names (Appendix B). The goal of GEB: keep one record per physical object, linked by its appearance. The record lists, in chronological order, everything that happened to that object, and points back to the video frames. A question about an object can then follow that object through the week instead of searching for its name.

Three experiments in the paper keep the same descriptions and change only how they are organized. Each one lowers GEB's accuracy on EgoLifeQA, which is 72.0% (Table 4):

  • observations grouped by name instead of by appearance: −2.8 points;
  • descriptions attached to the clips instead of to the objects: −3.8 points;
  • no linking across clips: −3.4 points.

Q3: Cost of building the memory

Building the memory takes one vision-language-model (VLM) call per observation, about 49 per 30-second clip, and the memory occupies 44 MB per hour of video. Describing only 15% of the observations is already enough to stay ahead of MAGIC-Video.

We report the cost per 30-second clip, the unit in which GEB processes a video, so that it depends neither on the length of the video nor on the hardware. The numbers are averages over the 52-hour EgoLife video.

StepModelPer 30-second clip
Detect and track objectsYOLOE-26x-seg + BoT-SORT300 frames, giving 49 observations
Describe each observationQwen3.5-35B (VLM)49 calls
Embed each observation for the associationQwen3-VL-Embedding-8B49 embeddings
Associate observations across clipssimilarity comparison, no model call12 entities
Embed the descriptions and build the graphQwen3-Embedding-4B49 embeddings

Comparison with existing models. MAGIC-Video and WorldMM build their memories from clip captions. GEB uses the same captions, so the table above is exactly the extra work that GEB does. In storage, GEB needs 44 MB per hour of video, against 13 MB for MAGIC-Video and 15 MB for WorldMM. For scale, the video itself takes 2.0 GB per hour.

The table and the storage numbers are now in Appendix H.2.

W2: Is the cost justified by the gains?

The cost is paid once per video. With a small fraction of it, GEB is still ahead of MAGIC-Video, and its memory is then even smaller than MAGIC-Video's.

Once the memory is built, it answers every question about the video: the same EgoLife memory serves EgoLifeQA, Ego-R1-Bench and MM-Lifelong Test@Week. To see how much of the cost is really needed, we rebuilt this memory while describing only a part of the observations. The table gives the number of VLM calls, the size of the resulting memory, and the accuracy.

Observations describedVLM calls per 30-second clipMemory per hour of videoEgoLifeQAEgo-R1-BenchTest@Week
MAGIC-Video, for reference—13 MB67.664.730.92
5%27 MB68.062.734.50
15%711 MB70.065.335.75
30%1517 MB70.466.736.50
70%3432 MB71.070.036.75
100% (the paper)4944 MB72.071.336.83

With only 5% of the calls, GEB's memory is about half the size of MAGIC-Video's (7 MB against 13 MB per hour of video), and GEB is already ahead on EgoLifeQA and MM-Lifelong Test@Week. With 15% of the calls, the memory is still smaller than MAGIC-Video's (11 MB), and GEB is ahead on all three benchmarks. The full memory is the most accurate. The cost is therefore a dial: a user can trade VLM calls and storage for accuracy (Appendix H.2).

Dear Reviewer vMuY,

As the discussion period draws to a close, we wanted to briefly highlight the updates addressing your concerns.

  1. Association quality (W1, Q2): Direct evaluation on EgoLife, gameplay, and EgoTracks shows 87–92% precision and 32–46% recall, reflecting our conservative approach to avoiding incorrect merges.
  2. Threshold robustness (W1, Q1): EgoLife thresholds achieve 92% precision on 60 EgoTracks videos without adjustment. All four tested threshold settings outperform MAGIC-Video on the three EgoLife-based benchmarks. We also clarified that calibration uses no QA supervision and added an automatic procedure.
  3. Contribution of entity organization (W2): Keeping the same descriptions but attaching them to clips reduces EgoLifeQA accuracy by 3.8 points. Our new analysis also shows that association improves evidence coverage by 5.9 points on questions requiring cross-event identification, versus 1.3 points on other questions.
  4. Construction cost and tradeoffs (W2, Q3): Construction takes 19.8 minutes per video hour on an eight-H100 node. Describing only 15% of observations yields a smaller memory than MAGIC-Video’s (11 versus 13 MB per video hour), while outperforming it on all three benchmarks, including 70.0% versus 67.6% on EgoLifeQA.

Thank you again for your time and feedback. We would be happy to discuss any remaining concerns.

Best regards,

The Authors of GEB

Comments on the paper

1 comment
Area_Chair_bLaMOpenReview ↗

Friendly reminder from AC

Dear Reviewers,

Thank you for your reviews. The initial ratings are 2 / 3 / 3. The authors have posted detailed responses to all three reviews. Several reviewers raised overlapping concerns, and the response addresses each of them.

Please read the authors' response and let us know in the discussion whether these points have been resolved and whether your rating has changed. If concerns remain, please state them specifically so the authors still have a chance to answer. Because the paper is borderline, we would especially value input from reviewers on both sides about whether the new association and cost analyses change their view.

Thank you for your efforts.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.