acceptodds
Under review as a conference paper at ICLR 2027

SuperMemory-VQA: An Egocentric Visual Question Answering Dataset and Benchmark for Long-Horizon Memory

Abstract

AI glasses present a compelling platform for AI agents to serve as personalized memory assistants. To be genuinely useful, such systems must move beyond short-term video comprehension and address memory gaps that humans experience for practical, personal, or social purposes over longitudinal egocentric video streams. However, existing egocentric datasets predominantly focus on action recognition or generic QAs from short clips, measuring perceptual capabilities rather than realistic human memory needs. We introduce SuperMemory-VQA, an egocentric visual question answering (VQA) dataset for evaluating AI assistants on practical, long-horizon memory tasks. It contains 52.9 hours of everyday activities recorded with AI glasses, including synchronized RGB video, audio transcription, eye gaze, IMU, and SLAM trajectories. Through a human-verified annotation pipeline, we construct grounded 4,853 question-answer pairs that span object and location memory, intent recall, visual scene recall, timeline reconstruction, conversational memory, and in-context retrieval. Each question is posed as multiple-choice with an explicit "unanswerable" option to test hallucination robustness. Benchmarking leading agentic frameworks and LLM backbones reveals that existing systems remain far from reliable on real-world memory tasks, highlighting the need for new architectures for grounded AI memory that can answer only when evidence is sufficient. A participant survey further supports that our questions are realistic, useful, and aligned with everyday memory needs.

Primary area:
datasets and benchmarks
Keywords:
Egocentric Vision, VQA, Long-Horizon Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qgL3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SuperMemory-VQA introduces 4,853 evidence-grounded questions from 52.9 hours of egocentric recordings to evaluate practical memory assistance and abstention. Its evaluation shows that current systems often answer incorrectly or abstain despite available evidence.

Critical strengths
  • Practical memory tasks with evidence timestamps and an explicit abstention option.
  • Human-reviewed annotations and useful diagnostic evaluations.
Critical weaknesses
  • The novelty positioning overlooks EgoMemReason, a closely related benchmark for long-horizon, multi-evidence egocentric memory. This makes it difficult to assess what new evaluation capability SuperMemory-VQA provides. A direct comparison of task design, evidence requirements, and temporal coverage is needed to substantiate the claimed contribution.
  • The evidence does not fully support the long-horizon claim: the median evidence-to-question gap is only 1.5 hours. The baseline comparisons also do not isolate whether errors arise from temporal distance, evidence composition, or differences in retrieval procedures. Consequently, the results establish difficulty on this benchmark but provide limited evidence for the proposed explanation of that difficulty.
Questions for authors
  • How many questions require cross-day evidence, and how do systems perform on them? This would show whether the reported difficulty persists on genuinely long-horizon cases.
  • What is the benchmark’s distinct contribution relative to EgoMemReason and EgoLifeQA? A comparison using consistent definitions of temporal gap and multi-evidence reasoning would clarify the novelty claim.
  • Do the findings hold against simple retrieval and direct VLM baselines under matched budgets? These controls would help separate task difficulty from differences in retrieval procedures and computational resources.
  • What is human accuracy on these questions? A human reference would help establish how much of the observed error reflects model limitations rather than ambiguous questions or answer choices.
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer qgL3

W1a. "The novelty positioning overlooks EgoMemReason, a closely related benchmark for long-horizon, multi-evidence egocentric memory."

Unlike EgoMemReason [1], which does not have unanswerable QAs [1, App. A.4] and uses standard MCQs without ordered vague distractors, SuperMemory-VQA evaluates epistemic abstention (30.2% unanswerable/false-premise QAs) and ordered specificity grading (Correct > Vague > Wrong > N/A). When retrieval surfaces an object under a false premise (Fig. 21), we test whether reasoners hallucinate compliance or abstain.

W1b. "A direct comparison of task design, evidence requirements, and temporal coverage is needed to substantiate the claimed contribution."

In the revised manuscript, we add EgoMemReason [1] and a comparison to §2 and Table 1 across: (i) Task design: 6 memory tasks with ordered specificity choices + 30.2% unanswerable QAs (vs. answerable-only standard MCQs); (ii) Evidence: 30.3% disjoint multi-interval QAs (10.2m median, 7.70h mean span; 35.0% cross-session) and 68.1% multi-modal QAs across 6 Aria Gen 1 streams; and (iii) Coverage: 52.9h across 87 sessions (10 wearers), up to 55.9d (W2a).

W2a. "The evidence does not fully support the long-horizon claim: the median evidence-to-question gap is only 1.5 hours."

First, a 1.5h median is ~2× the ~45m native VLM context limit: all QAs are causal, 95.7% have >45m preceding video (80.6% >2h), and 66.0% of answerable QAs have a >45m gap, forcing retrieval and reasoning outside native context. Second, Fig. 11a's 1.5h median measured only answerable QAs, omitting the 30.2% unanswerable QAs that must verify the full preceding context; combining both lifts the median to 1.79h, mean to 101.2h, and multi-day (>24h) rate from 4.2% to 13.7% (see Q1). In the revised manuscript, we update Abs., §1, and Fig. 11a with both horizons.

W2b. "The baseline comparisons also do not isolate whether errors arise from temporal distance, evidence composition, or differences in retrieval procedures."

Table 8 (App. 16.4), Figs. 15–16, and logs isolate all three factors: (1) Retrieval (Table 8): holding Flash fixed, Gold Evidence lifts QA-Acc +8.7 pts over Video-RAG (61.0% to 69.7%) and +15.6 pts over EgoButler (54.1% to 69.7%), separating retrieval/summary loss from the 30.3-pt reasoning/calibration gap; (2) Temporal distance (Fig. 15): gaps >45m drop Video-RAG by 3.5–7.6 pts (-11.0 to -15.6 pts across sessions), while crossing a day drops EgoButler by 8.4 pts on Flash (43.2% to 34.8%); and (3) Evidence composition (Fig. 16): moving from single- to multi-interval (>15m) QAs on EgoButler drops accuracy by 6.3–9.5 pts across closed models (37.1% to 30.8% on GPT-6-sol). In the revised manuscript, we add this breakdown to §5.2 and App. 16.4.

References

Dear Reviewer qgL3,

As the discussion period comes to a close, we wanted to politely check in on whether our response, revised manuscript, and additional experiments in the thread have addressed your concerns. Please let us know if you have any remaining questions we can clarify; if your concerns have been addressed, we would really appreciate it if you would consider updating your score. Thank you again for your helpful review!

Reviewer_38tqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark called SuperMemory-VQA, an egocentric visual question answering dataset for evaluating AI assistants on long-horizon memory tasks. The benchmark includes multiple modalities, such as RGB video, audio transcriptions, eye gaze, IMU, and SLAM trajectories. It also covers different memory tasks and provides an abstention option for questions that cannot be answered based on the available evidence.

Critical strengths
  1. The benchmark includes rich multimodal data, such as RGB video, audio transcriptions, eye gaze, IMU, and SLAM trajectories, which could be valuable for studying long-term memory in real-world settings.

  2. The benchmark focuses on practical memory questions that users may actually encounter in daily life, making it relevant to the development of personal AI assistants.

Critical weaknesses
  1. I am not fully convinced by the definition of long-horizon memory in this paper. Having long recordings does not necessarily mean that the questions require long-term memory reasoning. Many questions may still be answered by retrieving a single moment from the video. I think the authors should better clarify what makes these questions challenging from a memory perspective, rather than simply having a long video context.

  2. I am also not sure how much multi-evidence reasoning is actually required. The authors report that 34% of the questions involve multiple evidence clips, but how do they verify that all these clips are necessary to answer the question? Could the model answer correctly by retrieving just one relevant moment? Although the paper analyzes temporal gaps and evidence counts, it would be helpful to further analyze the temporal span across the required evidence and distinguish retrieval from actual memory reasoning.

  3. The comparison with existing long-term memory benchmarks is quite limited. There are several closely related works, such as EgoMemReason, MA-EgoQA, and EgoMem, that are not discussed. These benchmarks have already explored long-horizon memory and reasoning over multiple pieces of evidence. I think a more thorough comparison is needed to clarify what new challenges SuperMemory-VQA introduces.

  4. I feel the experimental analysis is somewhat limited. Most of the findings are expected, such as performance dropping with longer temporal gaps or more evidence, and models struggling with answerability. What new insights do we actually learn from this benchmark? For example, is the main bottleneck retrieval, memory construction, or reasoning over retrieved evidence? A deeper analysis would make the benchmark more useful for understanding how future memory systems should be improved.

Questions for authors

See questions in weakness.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer 38tq

W1a. "I think the authors should better clarify what makes these questions challenging from a memory perspective, rather than simply having a long video context."

Beyond raw video length, recurring routines over 3–12h create three memory challenges:

  • (1) Exceeding native VLM context: A 1.5h answerable median is ~2× the ~45m native video context limit of frontier VLMs—95.7% of QAs have >45m of preceding video and 66.0% of answerable QAs have a >45m gap, forcing retrieval and reasoning outside native context;
  • (2) Full-history verification for abstention: Fig. 11a measured only answerable QAs, omitting the 30.2% unanswerable QAs that must verify the entire preceding context, lifting the combined median to 1.79h, mean to 101.2h, and multi-day (>24h) rate to 13.7%;
  • (3) Spatio-temporal aliasing: Wearers revisit the same rooms and objects across 5–16 sessions, creating near-duplicate episodes that reduce Video-RAG accuracy by 3.5–7.6 pts for gaps >45m and 11.0–15.6 pts across sessions (Fig. 15). In the revised manuscript, we add the >45m context analysis and combined horizon to §1 and Fig. 11a.

W1b. "Many questions may still be answered by retrieving a single moment from the video."

While 62.7% of answerable QAs test pinpoint retrieval amidst routines, 37.3% annotate ≥2 spans (Fig. 11b) and 30.3% require ≥2 disjoint intervals (>5s apart)—including >80% of In-Context Retrieval and Timeline Reconstruction. Combined with the 30.2% unanswerable QAs requiring full-history verification, 51.4% of the benchmark cannot be answered from one moment. In the revised manuscript, we add this breakdown to §3.5.

W2a. "The authors report that 34% of the questions involve multiple evidence clips, but how do they verify that all these clips are necessary to answer the question?"

In Phase 2 (Fig. 9b, App. 12.2 & 12.4), the closed-loop Verifier checks multi-clip necessity against the retrieved video segments and prunes redundant spans, followed by a human review of evidence (Fig. 14, App. 13.1).

W2b. "Although the paper analyzes temporal gaps and evidence counts, it would be helpful to further analyze the temporal span across the required evidence and distinguish retrieval from actual memory reasoning."

Across the 1,026 disjoint multi-evidence QAs, the span between the first and last required clip has a 10.2m median and 7.70h mean, with 43.5% spanning >15m, 35.0% cross-session, and 68.9% multi-modal (extending Figs. 11b & 12). In the revised manuscript, we add this span distribution to §3.5 and App. 13.

On EgoButler, moving from single- to multi-interval (>15m) QAs reduces accuracy by 6.3–9.5 pts across closed models (37.1% to 30.8% on GPT-6-sol; 23.3% to 16.7% on 3.1-Pro; cf. Fig. 16). When Gold Evidence removes retrieval error (Table 8, App. 16.4), Flash reaches 69.7%, isolating a 30.3-pt gap due purely to memory reasoning and calibration. In the revised manuscript, we add this breakdown to §5.2 and App. 16.4.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.

The new statistics help clarify the long-horizon setting, especially the distinction between single- retrieval and multi-evidence questions. The expanded comparison with related benchmarks is also helpful.

However, I still have some concerns. While the verification process checks multi-evidence questions, it is not entirely clear whether all annotated clips are actually necessary to answer them. Also, the gold evidence experiments are useful, but I am not fully convinced that they can clearly separate retrieval, memory construction, and reasoning errors.

Overall, I think the revisions improve the paper, but my main concerns remain, so I will keep my original rating. Thanks again for the effort!

Thank you for following up and clarifying your two remaining concerns. We ran two controlled experiments and pulled concrete examples from the dataset to address both points directly.

1. Are all annotated clips necessary for multi-evidence questions?

We tested clip necessity on 239 multi-evidence questions from three random subjects, feeding the annotated clips directly with no retrieval:

Setting (P8–P10, 239 QAs)Qwen3.5-9B (%)Chance (%)
All clips given47.325.0
One clip at a time, averaged26.625.0
One clip at a time, best41.843.8

(Note on "best" chance: 231/239 QAs have 2 clips; in 4-way MCQ with 25.0% single-run chance, counting a question correct if either single-clip run guesses right raises random chance to 1 - 0.75^2 = 43.8%.)

When given one clip instead of all clips, Qwen3.5-9B drops by 20.7 points to 26.6%, right near the 25.0% chance level. Even taking the best result across separate single-clip runs (41.8%) sits below the 43.8% two-try chance baseline and below giving all clips together (47.3%). This rules out both duplicate clips and cases where one clip already has the answer while the extra clip is unnecessary.

Across the dataset, 94.4% of lookup questions annotate only a single interval, whereas multi-clip questions are multi-evidence by design because they compare or link distinct moments. Two examples show why a single clip cannot answer them:

  • Example 1 (P5, Q177; two clips, 40 min apart): "Did we put whole cloves in the pulao rice too, or just the butter chicken?"
    • Clip 1 (Sat 16:22, Video + Audio): Whole spices including cloves are plated and added to the butter chicken ("Can I use whole of this?").
    • Clip 2 (Sat 17:02, Video, no relevant speech): The pulao rice is seasoned in the pot with no cloves added.
    • Answer: Cloves went into the butter chicken only, not the pulao rice.
    • Why 1 clip is not enough: With Clip 1 alone, you see cloves added to the butter chicken, which cannot rule out putting cloves in the pulao 40 minutes later. With Clip 2 alone, you see no cloves in the pulao, which cannot confirm cloves were ever used in the butter chicken. Both clips are needed to conclude "just the butter chicken."
  • Example 2 (P6, Q414; two clips, 7 min apart): "I'm washing up the cutlery now. Did I use the same knife to slice the raw chicken as I did to open the minced garlic bottle?"
    • Clip 1 (Sun 13:21, Video, no speech): Raw chicken is sliced on the cutting board with a black-handled chef's knife.
    • Clip 2 (Sun 13:28, Video, no speech): The blue minced-garlic jar is pried open with a small silver knife.
    • Answer: No, the chef's knife was used for the chicken and a small silver knife for the garlic jar.
    • Why 1 clip is not enough: The question compares two tools used 7 minutes apart. Clip 1 only shows the knife used on the chicken (nothing about the garlic jar), and Clip 2 only shows the knife used on the garlic jar (nothing about the chicken). Nobody mentions either knife out loud, so only watching both clips answers whether it was the same knife.

2. Separating retrieval, memory construction, and reasoning errors

To avoid mixing frameworks in the gold-evidence comparison, we ran a step-by-step ablation inside EgoButler on P9 (310 QAs), changing only one component at each step:

Setting (P9, 310 QAs)Qwen3.5-9B (%)Gemini 3.8 Flash (%)
(A) EgoButler memory + own retrieval40.357.8
(B) EgoButler memory on GT segment50.571.4
(C) 1 FPS frames + full transcript in GT segment59.278.2
(D) Upper bound100.0100.0
Retrieval loss (B - A)10.213.6
Memory construction loss (C - B)8.76.8
Reasoning loss (D - C)40.821.8

Holding EgoButler's memory fixed, replacing retrieved segments with GT segments isolates retrieval loss (10.2–13.6 points). On those same GT segments, replacing EgoButler's summaries with 1 FPS frames and full transcripts isolates memory construction loss (6.8–8.7 points), showing that pre-query summarization drops unremarked visual details needed later (such as which knife was used in Example 2). The remaining gap to 100% reflects reasoning and calibration error: for the 9B model, reasoning is the main bottleneck (40.8 points), whereas with Gemini 3.8 Flash, reasoning error drops nearly in half (21.8 points) so retrieval and memory construction together make up almost half of the remaining errors (20.4 of 42.2 points).

We will add both controlled experiments on the full dataset and these examples to Appendix. Thanks for the discussion and we hope these address your concerns.

Reviewer_PSWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SuperMemory-VQA, a egocentric VQA benchmark designed for long-term personal memory. Ten participants wear Meta Aria Gen 1 smart glasses for a total of 52.9 hours (3–12 hours per person). A two-stage agent-based processing workflow generates 4,853 QA pairs covering six tasks: object and location memory, conversational memory, visual scene recall, in-context retrieval, timeline reconstruction and intent recall. Each question has four ordered options, approximately 30% of questions remain unanswered, and each answer is associated with timestamped evidence spans.

Critical strengths
  • This work presents 52.9 hours of IRB-approved, publicly available multi-sensor Aria recordings (RGB, gaze, IMU, SLAM, de-identified transcripts), accompanied by 4,853 evidence-level QA examples labelled with time intervals, rooms and modalities, along with the annotation pipeline, review tools and evaluation code.
  • The setting response format distinguishes between "answerable" and "incorrect". The work shows that for seven out of the eight models, the primary source of error was excessive omission. Existing egocentric MCQ benchmarks are unable to achieve this, yet for a memory assistant, this design makes it a measurable metric.
Critical weaknesses
  • The claims do not align with the data presented. The paper refers to "everyday activities", "real-world memory tasks" and "spanning days and weeks", but the tasks involved scripted cooking, board games and jigsaw puzzles within a single Airbnb property, with a median time interval of 1.5 hours. Tasks spanning multiple days appear to account for only 4–5%. This directly undermines the paper's justification for differentiation from EgoLife (300 hours, over a continuous week).
  • The metric for measuring the ability to refrain from answering shows almost no discrimination. The paper does not specify the positive class for Ans-F1, refer to Fig.7, I assume the positive is "answerable", while approximately 30% of questions are unanswerable. Consequently, the strategy of "answerable questions" achieves an F1 score of 0.82. The best result in Table 2, 83.9, is on a par with this.
  • Dense captions, the Super Ledger, QA generation, verification, retrieval and enhancement all generated by Gemini-3.1-Pro / Gemini-3-Flash. Three models evaluated belong to the Gemini series, and they lead in seven out of the nine columns in Tab. 2. Because answerability and naturalness were filtered by a Gemini verifier and distractor style comes from Gemini, this introduces a risk of family-level bias that the paper neither analyses nor acknowledges.
Questions for authors
  • Sec. 3.5 and Fig. 4 contradict Tab. 5, Sec. 3.5 and Fig. 4 state that Object Location (~19%) and Conversational Memory (~18%) accounted for the largest proportions, while Intent Recall accounted for the smallest. However, tasks in Tab. 5 have different statistic numbers.
  • No baseline prompt is provided, there is no explanation of how the model ranks the options (QA-MRR relies on ranking scores).
  • Sec. 3.4 states that "the Captioning, Planner, and Verifier agents use Gemini-3.1-Pro-preview, but in Sec. 12.2 and 12.3 state that Gemini-3-flash-preview is used for captioning. Please clarify which model produced the released captions.
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer PSWY

W1a. "The paper refers to 'everyday activities' and 'spanning days and weeks', but the tasks involved scripted cooking, board games and jigsaw puzzles within a single Airbnb property, with a median time interval of 1.5 hours."

While high-level activities followed loose scripts (recipes, board games, puzzles; App. 8.2) to create multi-step interactions, participants planned and conversed naturally without scripted dialogue or fixed object placements across indoor rooms (living room, kitchen, bedroom, hallway, entryway) and outdoor segments (street walks, car/parking), and 100% of wearers agreed the QAs reflect genuine memory lapses (Fig. 8). In the revised manuscript, we avoid the term "everyday activities" and specifically say "scripted naturalistic daily activities" in Abs. and §1.

W1b. "Tasks spanning multiple days appear to account for only 4–5%."

First, a 1.5h median is ~2× the ~45m native video context limit of most frontier VLMs: all QAs are strictly causal, 95.7% have >45m of preceding video (80.6% >2h), and 66.0% of answerable QAs have a >45m gap, forcing retrieval and reasoning outside native context. Second, Fig. 11a's 1.5h median (23.3h mean) and 4.2% multi-day rate measured only answerable QAs, omitting the 30.2% unanswerable QAs that must verify the entire preceding history. Combining answerable and unanswerable QAs lifts the median to 1.79h, mean to 101.2h, and multi-day (>24h) rate to 13.7%. In the revised manuscript, we update Abs., §1, and Fig. 11a to report both the answerable-only gap and the combined horizon.

W1c. "This directly undermines the paper's justification for differentiation from EgoLife (300 hours, over a continuous week)."

While EgoLifeQA also contains natural, memory-oriented questions, it is not specifically designed for benchmarking memory. We collected our own data to design loosely guided activities around practical, personal, and social information a wearer might later want to recall, instead of only mining existing footage for QAs. Cooking, games, and conversations elicited plans, object movements, and commitments; dialogue and object placements remained unscripted (App. 8.2). This allows targeted memory-use-case coverage, not greater naturalness than EgoLife. Our QA design also adds 30.2% unanswerable/false-premise QAs and ordered Correct/Vague/Wrong/N/A choices, alongside six sensor streams (§3.2, Fig. 2).

W2a. "The paper does not specify the positive class for Ans-F1, refer to Fig.7, I assume the positive is 'answerable', while approximately 30% of questions are unanswerable."

Ans-F1 treats answerable as positive (A/B/C vs. D). In the revised manuscript, we specifically add this definition to §4.

W2b. "The metric for measuring the ability to refrain from answering shows almost no discrimination. Consequently, the strategy of 'answerable questions' achieves an F1 score of 0.82. The best result in Table 2, 83.9, is on a par with this."

Because 69.8% of QAs are answerable, a trivial baseline that never abstains gets 82.2% Ans-F1 simply by predicting "answerable" 100% of the time—while catching 0.0% of unanswerable QAs (0.0% Abstention-F1, 41.1% Macro-F1). In contrast, evaluated models strongly discriminate both classes: under Video-RAG, Gemini-3-Flash abstains on 68.3% of unanswerable QAs while answering 82.2% of answerable QAs, reaching 65.3% Abstention-F1 and 74.6% Macro-F1 (+33.5 pts); 3.8-Flash abstains on 94.6% (66.2% Abstention-F1, 71.1% Macro-F1), and GPT-6-sol abstains on 65.2% (Video-RAG) and 75.9% (EgoButler) (71.7% and 70.4% Macro-F1). In the revised manuscript, we add Abstention-F1 and Macro-F1 to §4 and Table 2.

W3a. "Three models evaluated belong to the Gemini series, and they lead in seven out of the nine columns in Tab. 2."

(1) The model used for Generator/verifier, Gemini 3.1-Pro loses to GPT-6-sol in 9/9 columns of Table 2a (and Table 2b) as well as in Tables 6–7; (2) GPT-6-sol beats Flash (the captioner) in 6/9 columns and hits the highest QA-Acc (70.6%, WorldMM); (3) 3/4 closed models in Table 2a are Gemini variants vs. 1 non-Gemini, 3.8-Flash (winning 5/7) was never in the pipeline, and Gemma-4-E4B IT is the weakest open model (35.3%/36.4%).

W3b. "Because answerability and naturalness were filtered by a Gemini verifier and distractor style comes from Gemini, this introduces a risk of family-level bias that the paper neither analyses nor acknowledges."

Blind text-only runs score near 25% chance on 3.5-flash (31.0%, Table 5) and Qwen3-8B (23.8%), reducing chances of shortcuts, and Fig. 7 & Figs. 22–26 (App. 16.5) show 3.1-Pro's lower MCQ score reflects conservative option-D abstention (8.3% wrong answers vs. Flash's 14.7%, but 47.6% false abstention vs. 18.2%); 2-stage human review (Figs. 13–14) also rejected 23.5% of candidates and audited all 4,853 items. In the revised manuscript, we add this cross-family bias analysis to §5.2 and App. 16.2.

References

  • [4] Yang et al., EgoLife and EgoLifeQA (CVPR 2025)

Dear Reviewer PSWY,

As the discussion period wraps up, we wanted to politely check in to see if our response and revised manuscript have addressed your concerns and questions. Please let us know if anything remains unclear or if we can provide any further details. If your concerns have been resolved, we would be grateful if you would consider updating your score. Thank you again for your time and feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all reviewers (PSWY, 38tq, qgL3) for their constructive feedback. SuperMemory-VQA jointly evaluates strictly causal long-horizon retrieval, ordered specificity grading (Correct > Vague > Wrong > N/A), and epistemic abstention (30.2% unanswerable and false-premise queries) over 6-sensor Aria Gen 1 streams (52.9h, 87 sessions, 10 wearers). In the revised manuscript, we update the Abstract, §1–§5, Fig. 11a, and Tables 1, 2, and 5 with four key additions:

(1) Abstention Calibration (PSWY W2): Because 69.8% of the dataset is answerable, a trivial baseline that never abstains gets 82.2% Ans-F1 while catching 0.0% of unanswerable questions; in contrast, evaluated Video-RAG models correctly abstain on 65.2%–94.6% of unanswerable QAs and reach 61.7%–66.2% Abstention-F1 and 71.1%–74.6% Macro-F1. In the revised manuscript, we add the positive-class definition, Abstention-F1, and Macro-F1 to §4 and Table 2.

(2) Gemini Bias (PSWY W3): The generator/verifier model, Gemini 3.1-Pro, loses to non-Gemini GPT-6-sol in 9/9 columns of Table 2a (and Table 2b) as well as in Tables 6–7, and GPT-6-sol beats Flash in 6/9 columns (reaching the paper's highest QA-Acc, 70.6%, under WorldMM); blind text-only runs score near 25% chance on both 3.5-flash (31.0%) and Qwen3-8B (23.8%). In the revised manuscript, we add this cross-family bias analysis to §5.2 and App. 16.2.

(3) Long-Horizon & Multi-Day Performance (PSWY W1, 38tq W1, qgL3 W2/Q1): Even a 1.5h answerable median is ~2× the ~45m native VLM context limit (95.7% of QAs have >45m preceding video; 66.0% of answerable QAs have a >45m gap). Moreover, Fig. 11a measured only answerable QAs; including the 30.2% unanswerable QAs that must verify the entire preceding history lifts the combined median to 1.79h, mean to 101.2h, and multi-day (>24h) rate from 4.2% to 13.7% (140 answerable + 522 unanswerable), where EgoButler drops across all models (43.2% to 34.8% on Flash, trailing Video-RAG's 66.7% by 31.9 pts). In the revised manuscript, we update Abs., §1, Fig. 11a, and §5.2 with these horizon and cross-day statistics.

(4) Multi-Evidence Span, Related Work & Bottlenecks (38tq W2–W4, qgL3 W1–Q3): Across 1,026 disjoint multi-evidence QAs, the span between the first and last required clip has a 10.2m median and 7.70h mean (35.0% cross-session), and Table 8 cleanly separates retrieval loss (8.7 pts), hierarchical summary loss (6.9 pts overall, 31.9 pts cross-day), and reasoning/calibration limits (30.3 pts). In the revised manuscript, we add EgoMemReason [1], MA-EgoQA [2], and EgoMem [3] to §2 and Table 1, and add the multi-evidence span and bottleneck analyses to §3.5, §5.2, and App. 16.4.

References

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.