acceptodds
Under review as a conference paper at ICLR 2027

SENRI: Understanding the Space-Event Nexus in Real-Life Videos

Abstract

Human memories are organized in a way that connects events with their corresponding spatial contexts. An AI assistant should also be able to memorize and recall the nexus of events and spaces. Thus we introduce **SENRI**, a task that studies this space-event connection in long, continuous real-life egocentric videos. It contains six task types in two directions: *EventSpace*, which recovers spatial knowledge from event cues, and *SpaceEvent*, which retrieves events from regions and trajectories. To evaluate this, we build **SENRI-Bench**, containing 500 _manually annotated_ multiple-choice questions over 10 uninterrupted _self-recorded_ videos with around 25.3 hours of content. Also, we evaluate three typical long video understanding paradigms: direct perception, evidence seeking, and persistent memory. The task remains challenging across all three paradigms, highlighting substantial room for improvement. Then we show two representative failure modes of existing memory methods and address these evidence gaps with **SENRI-Agent**, a controller that uses _Target-Guided Visual Check (TGVC)_ to propose local visual check tasks for parallel verifiers and integrate the reports into a working memory to help with reasoning. Built upon existing memory systems, SENRI-Agent improves the performance of existing baselines without modifying their stored memories. We hope SENRI will provide valuable insights into how real-life egocentric understanding should be evaluated, reveal the gaps in existing approaches, and offer a possible direction for improvement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding, Spatial Understanding, Agentic Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rc4LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SENRI-Bench to study event–space associations in long egocentric videos through six tasks and 500 manually annotated questions drawn from 10 continuous recordings totaling 25.3 hours. It evaluates 28 configurations across direct perception, evidence seeking, and persistent memory, with the strongest baseline reaching 59.0% accuracy. The proposed SENRI-Agent adds target-guided visual checks and working memory to existing memory systems, improving accuracy by 2.6–3.8 percentage points across four evaluated configurations.

Critical strengths
  1. The paper offers a useful perspective on spatial intelligence by jointly considering events and space, providing a new basis for studying VLMs’ spatial understanding of videos.
Critical weaknesses
  1. The benchmark includes trajectory retrieval and navigation planning, but it does not separately assess reasoning across multiple places and events, leaving its coverage of spatial topology unclear.

  2. SENRI-Agent improves existing memory systems through targeted visual checks, but it does not explicitly represent 3D structure or connections between places, making it difficult to attribute the gains to spatial modeling.

Questions for authors
  1. Could the authors include questions that connect multiple events at the same place or trace events across places and times? This would more fully test the proposed space–event nexus.
  2. The paper emphasizes spatial understanding, while SENRI-Agent verifies evidence from video frames. How does this approach compare with a representative 3D spatial-memory baseline, and what are its relative strengths?
  3. How do accuracy and SENRI-Agent’s gains vary with the number of places covered and the video duration? An analysis beyond the per-video results would help establish how the method scales.
2 comments
AuthorsOpenReview ↗

Response to Reviewer rc4L

We appreciate Reviewer rc4L for their constructive feedback. We’re glad that they find our task a novel perspective of spatial intelligence which connect events and spaces in everyday life. For the concerns we address them as follows:

(note that all new experiments are based on the revised benchmark as asked by Reviewer 9sRK in W3 and Q3):

W1 and Q1:multiple places and events.

We thank the reviewer for the valuable question. We have to clarify that we already include the reasoning across multiple places and events in our benchmark.

  • Navigation Planning (NP), “From when I exited the underground mall to when I threw away my drink bottle, how did I walk?” This connects two events through a route spanning multiple intermediate places, requiring recovery of their spatial and temporal progression.
  • Trajectory Event Retrieval (TER), “During a roughly L-shaped walking route inside a certain store, which two events can be included within this L-shaped route?” This explicitly asks the model to associate multiple events with a shared spatial trajectory.
  • Region-Conditioned Event Retrieval (RER), “What did I see when I was at the entrance of the subway for the second time?” This requires distinguishing repeated visits to the same place and retrieving the observation associated with the specified visit.

They are included in the above three types, though they’re not listed as a separate task type by our definition.

W2 and Q2: whether spatial memories help

We thank very much for noticing that. Actually in our early exploration for the task we tried constructing retrievable 3d memories by getting spatial coordinates and relations using 3d models. However we found too much noise in the real-world which make it less effective (e.g. you don’t know which object to propose) and constructing spatial memories is quite heavy so we didn’t include it or claim it in our method.

However, we found a related work DirectMe [1] which does it by constructing 3d scene graphs but requires the time of the question (which is not natively acquirable in our task because lots of our questions require reasoning over a period of time). We tried to adapt it by using time priors by WorldMM and MagicVideo for Qwen to select query time for the geometry information. Results show that they still fall behind our SENRI-Agent:

RetrievalDirectMe adaptationSENRI-Agent
WorldMM45.3%49.4%
MAGIC-Video37.8%47.0%

Details of the experiment are in revised version of our pdf at Appendix C.4.

Q3: scale with number of places and video duration

We thank the reviewer for the suggestion. We did this experiment by dividing to 5 fewer / more places videos and 5 shorter / longer videos. Results are as follows:

Memory systemInference modelShorter videosLonger videosFewer placesMore places
WorldMMQwen3.8-27B+5.3+2.5+6.1+1.7
WorldMMGemini 3.7 Flash+3.9+6.3+3.9+6.3
MAGIC-VideoQwen3.8-27B+2.2+6.7+3.9+5.1
MAGIC-VideoGemini 3.7 Flash+3.5+5.0−0.9+9.3

3 of 4 settings of SENRI-Agent scales with that though the variance of our videos and number of places are relatively small, the discussion has been added in Appendix C.5.

[1] Wang et al. Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams. ICML 2026.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer rc4L,

Thank you again for your thoughtful feedback on the spatial part of our project, which we've tried to address in our rebuttal and our revision. As the discussion period draws to a close, we would appreciate hearing whether our response has addressed your concerns. If any points remain unclear, we would be happy to clarify them.

Thank you for your time and for helping us improve our work.

Best regards,

The Authors

Reviewer_Bth3OpenReview ↗
Rating
4
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

I am not familiar with this area

Critical strengths

I am not familiar with this area

Critical weaknesses

I am not familiar with this area

Questions for authors

I am not familiar with this area

1 comment
AuthorsOpenReview ↗

Response to Reviewer Bth3

We thank Reviewer Bth3 for their sincere and transparent feedback.

Reviewer_9sRKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether models can connect what happened with where it happened in long real-life videos. The authors recorded 10 videos themselves, each a single uncut take from a chest camera averaging about 2.5 hours (25.3 hours total). The person who recorded each video wrote 50 four-choice questions about it, so 500 in total. Some questions start from an event and ask about the place ("where was I when..."), and some start from a place or a route and ask what happened there. The authors test 28 setups, including models that watch the frames directly, search agents, and memory systems, and the best one gets 59%. They also propose SENRI-Agent, which adds a "go back and look" step to existing memory systems. It sends small visual checks to helper models that can't see the answer options, and keeps what they find in a short working memory. This adds 2.6 to 3.8 points to WorldMM and MAGIC-Video.

Critical strengths

1.The task is a good one. Asking where I was when I tasted the fruit flower tea, or what I saw walking back along the other side of the canal, is exactly what a personal assistant should handle, and current benchmarks don't really test it.

  1. Long uncut recordings that move through real places (malls, subway, parks, museums) are rare, and new recordings help avoid data leakage.
Critical weaknesses
  1. The benchmark is relatively small. 500 questions means only 69 to 90 per task type, so each task score has a margin of error of roughly ±10 points. Several per-task gaps in Sec. 3.2 (about 8 to 9 points) may not be significant. and the paper also doesn't say how many people, cities or places are covered.

  2. The person who filmed each video wrote its questions, and "an independent verifier" checked them. There's no agreement score, no count of removed questions, and no human accuracy. Some questions depend on which moment or view you look at, so accuracy from people who didn't film the video is needed.

  3. Qwen3.8-27B gets 34.6% with no video vs 25% chance, and 512 frames only add 6.4 points (Table 11). Three tested systems score below that no-video number. Please run this test on the strongest models and drop questions that can be answered without watching to eliminate language bias.

  4. license or release plan?

  5. Which model is the verifier? The same one as the planner?

Questions for authors

see weakness and

  1. How many people recorded videos and wrote questions, and how many cities and places are covered?

  2. Can you report agreement between annotators, and accuracy from people who didn't film the video?

  3. What's the no-video accuracy for Gemini 3.7 Flash and GPT-5.5?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9sRK

We thank reviewer 9sRK for their constructive feedback. We’re glad that they find our task is a good and novel one for everyday AI assistant and recognize the value of our self-recorded videos and human-annotated benchmark questions which are rare in existing benchmarks and help avoid data leakage. We address the concerns as follows

(note that all new experiments are based on the revised benchmark as asked in W3 and Q3):

W1 and Q1: significance of the per-task difference finding and more statistics of the benchmark

We appreciate the reviewer for pointing this out.

For the per-task difference finding in Sec 3.2, we agree that small per-task gaps should be interpreted cautiously. On the revised benchmark, paired 95% bootstrap CIs for the object-counting gap (+8.33 points, CI [−4.76, 21.43]) and RER gap (+8.97 points, CI [−3.85, 21.79]). Though they include zero, we have to clarify that our finding describes observed task-dependent performance patterns, instead of superiority of particular methods on individual tasks.

For more benchmark statistics, five camera wearers took the videos in two cities. We also categorize the places covered by the videos into at least 12 types (given that it’s hard to define distinct places), including shopping malls, retail stores, supermarkets and convenience stores, restaurants, cafés, museums, parks and gardens, amusement areas, university campuses, metro stations, temples, and riverside promenades. We add the information in Sec 2.

W2 and Q2: verification process and human tests

We appreciate the reviewer for this valuable feedback. Around 60 questions are removed due to its correctness or ambiguity in the verification process (resulting in 500). We chose 180 questions (30 each task) and did a human evaluation test by allowing human to freely scroll or inspect the video. Results are listed below,

TaskAccuracy
L80.0%
NP96.7%
OC93.3%
RR90.0%
RER100.0%
TER83.3%
Overall90.6%

We add the human evaluation in Appendix C.6.

W3 and Q3: strongest model no-video accuracies and potential language bias

We thank the reviewer for pointing this out. We evaluated no video accuracy of GPT 5.5 and gemini 3.7 flash on the original 500 questions and list the comparison of video and no-video below. They have significant improvement.

ModelNo-video accuracy (%)Video accuracy (%)
GPT-5.533.648.2
Gemini 3.7 Flash37.052.2

Regarding potential language bias, we found that most of the improvement over 25% come from that there are fixed patterns in options though they cannot be confirmed from the text alone. Due to the difficulty of defining and finding language bias or shortcut, we tried our best to locate them by investigating the 67 commonly right questions by the 3 models (GPT, Gemini and Qwen). We find some potiential language bias e.g. (option structure and wording asymmetry, textually eliminable distractors ,commonsense and contextual priors) and locate the 34 that have manifest patterns and dropped them to help reduce the potential language bias. And the video and no-video accuracies of Qwen, GPT and Gemini are shown below. Qwen has 9% improvement with video.

ModelNo-video accuracy (%)Video accuracy (%)
Qwen3.8-27B29.838.8
GPT-5.528.845.7
Gemini 3.7 Flash32.450.4

After we drop the 34 questions, all the main findings and effectiveness of SENRI-Agent stay consistent. We’ve updated the manuscript to this new benchmark version and add the analysis of language bias in Appendix C.3. Note that all our findings stay consistent, despite few changes in benchmark rankings.

W4: license or release plan

We thank the reviewer for asking about release plan. SENRI-bench would be released upon acceptance under CC BY-NC-SA 4.0 license.

W5: planner backbone

We thank the reviewer for the suggestion on this clarification. Yes, the backbone is shared between planner (controller as we state in the paper) and verifier.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 9sRK:

Thank you again for your thoughtful feedback. As the discussion period approaching its very end, we'd be glad to hear from you whether our clarification and experiments listed above have addressed your concern. We've also uploaded a revision pdf based on your suggestions.

We would be grateful for any further thoughts you may have and thank you once again for your suggestions to help us improve our work.

Best Regards,

Authors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their insightful and constructive reviews. We’re glad that they find our task SENRI is a useful and novel perspective in spatial intelligence (9sRK, rc4L) which existing benchmarks do not cover (9sRK), and our self-recorded, manual-annotated benchmark is valuable and avoids data leakage (9sRK), which provides a new basis for studying VLMs’ spatial understanding of videos (rc4L).

We have provided individual responses to each reviewer. Prompted by reviewer 9sRK’s suggestion in W3 and Q3, we dropped 34 questions in our bench which have potential risks of language bias which might make some options more or less likely to be chosen. We revised figures and experimental results in the pdf revision corresponding to the new version of benchmark. All our findings (3 main findings in benchmark results, SENRI-Agent improvement and ablations) stay consistent. Other experiments and clarification asked by reviewers are based on that new version and included in the pdf in blue font as a further revision listed below,

  1. Reviewer 9sRK W1 and Q1: more statistics of the benchmark (Sec 2)
  2. Reviewer 9sRK W2 and Q2, W3 and Q3: human results, no-video accuracy of stronger models and our process of locating and dropping potential language bias questions (Sec 2, Appendix C.3, C.6)
  3. Reviewer rc4L W1 and Q1, Q3: how SENRI-Agent performs varying number of places and video duration (Appendix C.5).
  4. Reviewer rc4L W2 and Q2: discussion of whether explicit spatial representation helps and comparison on a spatial baseline (Appendix C.4).

We also address reviewers concerns on significance of the per-task difference finding (Reviewer 9sRK W1), license and release plan (Reviewer 9sRK W4), backbone of the controller (Reviewer 9sRK W5) and clarification on our coverage of multiple places and events (Reviewer rc4L W1 and Q1) in the corresponding responses.

Again we thank all reviewers and the AC for their work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.