- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SuperMemory-VQA introduces 4,853 evidence-grounded questions from 52.9 hours of egocentric recordings to evaluate practical memory assistance and abstention. Its evaluation shows that current systems often answer incorrectly or abstain despite available evidence.
Critical strengths
- Practical memory tasks with evidence timestamps and an explicit abstention option.
- Human-reviewed annotations and useful diagnostic evaluations.
Critical weaknesses
- The novelty positioning overlooks EgoMemReason, a closely related benchmark for long-horizon, multi-evidence egocentric memory. This makes it difficult to assess what new evaluation capability SuperMemory-VQA provides. A direct comparison of task design, evidence requirements, and temporal coverage is needed to substantiate the claimed contribution.
- The evidence does not fully support the long-horizon claim: the median evidence-to-question gap is only 1.5 hours. The baseline comparisons also do not isolate whether errors arise from temporal distance, evidence composition, or differences in retrieval procedures. Consequently, the results establish difficulty on this benchmark but provide limited evidence for the proposed explanation of that difficulty.
Questions for authors
- How many questions require cross-day evidence, and how do systems perform on them? This would show whether the reported difficulty persists on genuinely long-horizon cases.
- What is the benchmark’s distinct contribution relative to EgoMemReason and EgoLifeQA? A comparison using consistent definitions of temporal gap and multi-evidence reasoning would clarify the novelty claim.
- Do the findings hold against simple retrieval and direct VLM baselines under matched budgets? These controls would help separate task difficulty from differences in retrieval procedures and computational resources.
- What is human accuracy on these questions? A human reference would help establish how much of the observed error reflects model limitations rather than ambiguous questions or answer choices.
2 comments
Dear Reviewer qgL3,
As the discussion period comes to a close, we wanted to politely check in on whether our response, revised manuscript, and additional experiments in the thread have addressed your concerns. Please let us know if you have any remaining questions we can clarify; if your concerns have been addressed, we would really appreciate it if you would consider updating your score. Thank you again for your helpful review!
Author Response to Reviewer qgL3
Unlike EgoMemReason [1], which does not have unanswerable QAs [1, App. A.4] and uses standard MCQs without ordered vague distractors, SuperMemory-VQA evaluates epistemic abstention (30.2% unanswerable/false-premise QAs) and ordered specificity grading (Correct > Vague > Wrong > N/A). When retrieval surfaces an object under a false premise (Fig. 21), we test whether reasoners hallucinate compliance or abstain.
In the revised manuscript, we add EgoMemReason [1] and a comparison to §2 and Table 1 across: (i) Task design: 6 memory tasks with ordered specificity choices + 30.2% unanswerable QAs (vs. answerable-only standard MCQs); (ii) Evidence: 30.3% disjoint multi-interval QAs (10.2m median, 7.70h mean span; 35.0% cross-session) and 68.1% multi-modal QAs across 6 Aria Gen 1 streams; and (iii) Coverage: 52.9h across 87 sessions (10 wearers), up to 55.9d (W2a).
First, a 1.5h median is ~2× the ~45m native VLM context limit: all QAs are causal, 95.7% have >45m preceding video (80.6% >2h), and 66.0% of answerable QAs have a >45m gap, forcing retrieval and reasoning outside native context. Second, Fig. 11a's 1.5h median measured only answerable QAs, omitting the 30.2% unanswerable QAs that must verify the full preceding context; combining both lifts the median to 1.79h, mean to 101.2h, and multi-day (>24h) rate from 4.2% to 13.7% (see Q1). In the revised manuscript, we update Abs., §1, and Fig. 11a with both horizons.
Table 8 (App. 16.4), Figs. 15–16, and logs isolate all three factors: (1) Retrieval (Table 8): holding Flash fixed, Gold Evidence lifts QA-Acc +8.7 pts over Video-RAG (61.0% to 69.7%) and +15.6 pts over EgoButler (54.1% to 69.7%), separating retrieval/summary loss from the 30.3-pt reasoning/calibration gap; (2) Temporal distance (Fig. 15): gaps >45m drop Video-RAG by 3.5–7.6 pts (-11.0 to -15.6 pts across sessions), while crossing a day drops EgoButler by 8.4 pts on Flash (43.2% to 34.8%); and (3) Evidence composition (Fig. 16): moving from single- to multi-interval (>15m) QAs on EgoButler drops accuracy by 6.3–9.5 pts across closed models (37.1% to 30.8% on GPT-6-sol). In the revised manuscript, we add this breakdown to §5.2 and App. 16.4.
References