- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Grounded Entity Biographies (GEB), a memory framework for long-video understanding. Its core idea is to associate observations belonging to the same physical instance across different video clips, building entity biographies that carry persistent identity information, and to connect entity identity with episodic information to support cross-time event retrieval and reasoning. Experiments on multiple long-video question-answering benchmarks show that GEB outperforms existing memory-based methods; ablation studies and evidence-retrieval analysis further validate the role of each major component.
Critical strengths
- The paper addresses entity identity ambiguity in long-video memory systems: text descriptions alone often cannot distinguish different physical instances or associate observations of the same entity across time. The proposed entity-biography representation offers a sound approach to organizing and retrieving information across time.
- The method outperforms recent memory-based approaches on multiple benchmarks, and the roles of the main components are validated through modular ablations and evidence-retrieval analysis.
- The paper is well organized, the method is clearly described, and the experimental tables and qualitative examples are presented effectively, facilitating understanding of both the method design and its outcomes.
Critical weaknesses
-
The memory structure used by GEB is substantially larger than that of several compared methods. Although the paper controls for the reasoning model and retrieval setup, it does not adequately separate the effect of increased memory capacity on performance, making it difficult to fully attribute the performance gains to the identity-based organization of the memory.
-
On MultiHop-EgoQA, removing identity association drops complete evidence coverage from 52.1% to 48.4%, yet the answer score remains at 3.11. The paper does not adequately explain why the improvement in evidence-retrieval performance fails to translate into a sustained gain in answer quality.
-
GEB introduces additional processing for object tracking, visual description generation, and cross-clip identity association. However, the paper does not adequately quantify the computational cost of constructing the memory, making it difficult to assess the overall computational overhead of the proposed framework.
Questions for authors
-
Could the authors provide visualizations of representative entity-identity association failure cases, particularly incorrect identity merges (where distinct entities are wrongly merged) and fragmented biographies (where observations of the same entity are incorrectly split across multiple entity biographies)?
-
Could the authors report QA performance separately on different question types, depending on whether a question requires cross-event identification of the same physical instance, so as to further analyze under which conditions entity biographies yield more significant gains?
-
On MultiHop-EgoQA, removing identity association leads to a drop in complete evidence coverage while the answer score stays unchanged. Could the authors explain this phenomenon?
-
Could the authors report GEB's offline memory construction time, and further break down the time cost of each major computational module?
4 comments
Response to Reviewer nNVi (2/2)
Q1: Failure cases of the association
We have added Figures 11 and 12 to the revised paper (Appendix H.1). They show a wrong merge and a fragmented biography.
How often do these failures happen? We measured the association against human judgment on three datasets. Wrong merges are rare: 87% to 92% of GEB's links are correct. Fragmented biographies are more common: GEB finds 32% to 46% of the true same-object pairs, because it links two observations only when the visual evidence is strong. We chose this side on purpose: a missed link can still be recovered when a question is asked, while a wrong link would mix the histories of two objects.
In each figure, a row is one observation: video frames with the object marked by a box, the time of each frame, and GEB's description of the object.
- A wrong merge (Figure 11). Two pillows of the same kind lie side by side. Nothing in their appearance tells them apart, and GEB merges them. Objects whose detection boxes are apart in the same frame are never merged, so this happens only across clips.
- A fragmented biography (Figure 12). The same plush toy appears in two consecutive clips, and GEB keeps the two observations in two entities, because a new observation must agree with every recent observation of an entity. Both observations carry the same name, "lion plush toy". When they are retrieved together, GEB presents them with a note that they may be the same object.
Q2: Performance by question type
GEB is ahead of both memory frameworks on every question type, and its gain is large on the questions about specific objects and people, where recognizing the same instance across events matters.
We answer on MultiHop-EgoQA, because it is the benchmark that labels every question with a category and annotates all the moments that contain its evidence. The table gives the answer score (1–10) per category, from Table 12 of the paper.
| Category (number of questions) | MAGIC-Video | WorldMM | GEB | Gain over the better baseline |
|---|---|---|---|---|
| Repeated activities (73) | 2.91 | 2.97 | 3.24 | +0.27 |
| Multiple actions (241) | 2.53 | 2.75 | 2.91 | +0.16 |
| Multiple objects (193) | 2.40 | 2.43 | 2.90 | +0.47 |
| Locations / people (72) | 2.51 | 2.59 | 3.25 | +0.66 |
| Event composition (111) | 2.32 | 2.57 | 2.90 | +0.33 |
| Event comparison (34) | 4.23 | 4.68 | 5.90 | +1.22 |
GEB is ahead of both in every category. Among the five large categories, the two about specific objects and people gain the most (+0.47 and +0.66); the small category that compares two events gains even more (+1.22).
Questions whose evidence is spread over three or more moments (285 questions) show the same advantage: GEB's answer score is 2.97, against 2.80 for WorldMM and 2.63 for MAGIC-Video.
Additional Comment of Reviewer nNVi
Thank you for the detailed response and additional experiments. Most of my concerns have been addressed, but two points remain:
Memory Construction Cost. The breakdown of model calls is helpful, but the actual construction time remains unclear. Could the authors report the wall-clock time and hardware configuration?
Contribution of Identity Association. The question-type analysis is useful, but it does not directly distinguish questions that require cross-event identification of the same entity. Could the authors compare performance on questions with and without this requirement?
We sincerely thank the reviewer for the follow-up. We are very glad that most of the concerns are now addressed. The two remaining questions are both valuable, and answering them has made the paper stronger. We answer them next.
Q4: Wall-clock time and hardware of memory construction
Using one node with 8 H100 GPUs which also hosts the LLM to describe observations, GEB builds its memory in 19.8 minutes per hour of video on average. If the LLM is called through the OpenRouter API, the time falls to 7.5 minutes per hour of video. With our two nodes and a self-hosted LLM, memory for the 52-hour EgoLife video is built in about 8.8 hours.
The table gives the average wall-clock time of each module per hour of video. The description module and the total have two values. The first value is the time needed when hosting Qwen3.5-35B on our node. The second value is the time needed when the same model is called through the OpenRouter API.
| Module | Hardware | Time per hour of video |
|---|---|---|
| Detect and track objects (YOLOE-26x-seg + BoT-SORT) | 1 H100 node | 1.4 min |
| Describe each observation (Qwen3.5-35B) | 1 H100 node / OpenRouter API | 17.1 / 4.8 min |
| Embed each observation for the association (Qwen3-VL-Embedding-8B) | 1 H100 node | 0.7 min |
| Associate observations across clips | CPU only | 0.5 min |
| Embed the descriptions and build the graph (Qwen3-Embedding-4B) | 1 H100 node and CPU | 0.1 min |
| Total | 19.8 / 7.5 min |
Describing the observations is the most time-consuming module. We choose to host the Qwen3.5-35B model ourselves. Each copy of the model runs on two GPUs (tensor parallelism of 2), so one node serves four copies at the same time, and the module takes 17.1 minutes per hour of video. If one prefers not to host the model, an external API can be used instead: through OpenRouter, the module takes 4.8 minutes, which is 3.6 times as fast. The time of this module therefore depends on which vision-language model is used and how it is hosted, and it is proportional to the number of observations described. Describing only 15% of the observations takes about 2.6 minutes per hour of video (self-hosted). Despite the reduced memory, GEB is still ahead of MAGIC-Video on all three benchmarks (70.0 against 67.6 on EgoLifeQA, for example). The memory is built once offline and then answers every question about the video online. These timings are now in Appendix H.2.
Q2: Questions with and without cross-event identification of the same entity
On the questions that require recognizing the same object or person across events, association raises evidence coverage by 5.9 points. On the other questions, it adds 1.3 points.
Concretely, we labeled each of the 1,080 questions of MultiHop-EgoQA, from the question and its reference answer only. A question requires cross-event identification when one object or person is the subject of two or more of its evidence moments. Either the question asks what happened to this object, as in "What did I do with the box before putting it on the wall?", or it asks about an action repeated on it, as in "When did I open the tap?". This holds for 569 questions. Of the other 511, 254 have several evidence moments that each concern a different object, as in "What items did I place on the kitchen counter?", and 257 have all their evidence in one moment. The table compares GEB with association, as in the paper, and without it, on evidence coverage and answer score.
| Questions (number) | GEB | Evidence coverage | Answer score |
|---|---|---|---|
| With the requirement (569) | Without association | 35.4% | 3.01 |
| With association | 41.3% (+5.9) | 3.06 | |
| Without the requirement (511) | Without association | 62.8% | 3.14 |
| With association | 64.1% (+1.3) | 3.17 |
The coverage gain from association is in parentheses. It is 4.6 points larger on the questions with the requirement than on the others.
Table 3 of the paper reports that removing association lowers evidence coverage by 3.7 points on the whole benchmark. 83% of this loss falls on the questions with the requirement, which make up 53% of the benchmark.
The answer score is also higher with association, by 0.05 on the questions with the requirement and by 0.03 on the others.
This table is now in Appendix H.5.
Response to Reviewer nNVi (1/2)
We thank the reviewer for their time and feedback. We appreciate that the reviewer found the approach "sound" and the paper "well organized". We answer questions next.
A reminder of the terms we use. GEB (our method) tracks every object inside each 30-second clip of a long video; one such track is an observation. GEB then decides which observations from different clips show the same physical object, and links them. This step is the association, and all linked observations of one object form an entity.
W1: Separating the gain from the effect of a larger memory
We ran exactly this experiment. A GEB memory that is smaller than MAGIC-Video's still outperforms it, so GEB's lead does not depend on a larger memory.
We rebuilt GEB's memory from only 15% of its observations, selected without looking at any question. This memory takes 11 MB per hour of video; the memory of MAGIC-Video, the strongest published memory framework, takes 13 MB.
Two facts from the paper point the same way. First, GEB and MAGIC-Video run under the same limits on what reaches the model that writes the answer: the same number of retrieved items, search rounds and video frames. Second, when we keep all of GEB's descriptions of the observations but attach them to the clips instead of organizing them by object, accuracy drops by 3.8 points on EgoLifeQA (Table 4).
The matched-size comparison is now in Appendix H.2.
W2 / Q3: Coverage drops but the answer score stays the same
The answers do get worse without association: they point to less of the right evidence. The 1–10 answer score is simply too coarse to show a difference of this size.
What the reviewer observed. MultiHop-EgoQA is a benchmark whose questions need evidence from several moments of a video. Evidence coverage is the share of questions for which all the annotated evidence reaches the model that writes the answer. Without association it falls from 52.1% to 48.4%, while the answer score stays at 3.11 (Table 3).
The answers do get worse. On MultiHop-EgoQA, the model must also cite the time intervals that support its answer. Without association, these cited intervals match the annotated evidence less well: their overlap (mIoG) falls from 36.7 to 34.1 (Table 11). The 95% confidence interval of this drop, 1.7 to 3.6 points, excludes zero.
Why the score does not move. The score is a grade from 1 to 10 that a judge model gives to the written answer, and it reacts slowly to how much evidence is retrieved. To measure how slowly, we ran a new experiment on GEB itself: we limited the number of search rounds per question, which lowers the coverage in a controlled way. The paper allows 5 rounds.
The score follows the coverage, but slowly. Going from 5 rounds to 3 lowers the coverage by about 7.5 points, twice the loss from removing association, and the score changes by only 0.11. A coverage loss half as large, the 3.7 points lost without association, would move the score by about 0.05 on the 1–10 scale, which is too small to observe.
This explanation and the new experiment are now in Appendix H.4.
W3 / Q4: Construction cost, per module
Here is the breakdown by module. We report the cost per 30-second clip, the unit in which GEB processes a video, and in numbers of model calls, so that it depends neither on the length of the video nor on the hardware. The numbers are averages over the 52-hour EgoLife video.
The description module dominates the cost. It is the only module that calls a large model: one vision-language-model call per observation, about 49 per 30-second clip. Tracking and embedding use small models, and the association itself needs no model call. The memory is built once and then answers every question about the video. A fraction of the calls already suffices: with only 15% of the observations described (7 calls per clip), GEB is still ahead of MAGIC-Video on all three benchmarks, as the table in our answer to W1 shows.
This table is now in Appendix H.2.