acceptodds
Under review as a conference paper at ICLR 2027

EgoCITE: Situation-Aware Context-Augmented Indexing and Retrieval for Long-Horizon Egocentric Memory

Abstract

Long-horizon egocentric memory requires agents to recover relevant experiences from days of continuous first-person video and audio. Existing systems typically store isolated video captions or speech transcripts and retrieve them using raw user queries. We find that this pipeline loses the situational context needed for reliable memory search: stored fragments are not self-contained, complementary descriptions of the same experience remain disconnected, and queries often express temporal intent only implicitly. We introduce EgoCITE, a situation-aware framework that contextualizes both memory storage and retrieval for long-horizon egocentric question answering. EgoScheme uses local multimodal context to transform fragmentary captions and transcripts into self-contained atomic memories. EgoIndex organizes complementary action, activity, utterance, and conversation representations into multi-view, multi-granularity indices that capture each situation from different perspectives. EgoRetrv interprets the situation implied by a question through question-conditioned temporal relevance scoring, then iteratively retrieves and curates matching evidence. We evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench using answer accuracy and target-event retrieval alignment. Across the three benchmarks, EgoCITE improves average answer accuracy over the strongest agentic memory baselines by 4.4–14.2% while achieving 36 lower normalized cost than long-context LLM agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric vision, long-horizon video understanding, agentic memory, temporal reasoning, retrieval-augmented generation, multimodal question answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RQ6eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce EgoCITE, a situation-aware framework that contextualizes both memory storage and retrieval for long-horizon egocentric question answering and evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench using answer accuracy and target-event retrieval alignment. Across the three benchmarks, EgoCITE improves average answer accuracy over the strongest agentic memory baselines by 4.4–14.2% while achieving 36×lower normalized cost than long-context LLM agents.

Critical strengths

S1: The framework is clear and useful. The four-stage pipeline is well-designed with clean module decoupling, EgoScheme handles context resolution, EgoIndex handles multi-view organization, and EgoRetrv handles temporal ranking. S2: Evaluation is solid, three benchmarks and multiple baselines are evaluated. EgoCITE-GPT achieves hit rate improvements of 15.1/17.0/22.7% on the three benchmarks. The efficiency is considered, The 36× cost reduction vs. long-context agents, combined with higher accuracy, provides clear evidence of a Pareto-optimal trade-off across accuracy, latency, and cost.

Critical weaknesses

W1: The method seems to have limited algorithmic novelty. All core components are combinations of known techniques. Eq. (1) is essentially simple exponential decay, λ=0.99 is close to "equals 1," and the contribution of temporal scoring may be overstated. W2: The method depends heavily on prompt engineering. EgoScheme, EgoIndex, and EgoRetrv all rely heavily on the prompt execution quality of GPT-5.4. And the prompt is complicated, contains rules, definitions, examples, etc., which seems more like a few-shot + rule-based prompting. This may means this is a victory of prompt design rather than methodological contribution. W3: The cost comparison is unfair. The GPT-5.4 agent baseline processes raw JSON documents directly, while EgoCITE uses preprocessed atomic indices + curated pool. The two systems' costs are not computed on the same baseline, since the preprocessing cost of EgoCITE is ignored.

Questions for authors
  1. Whether the cost comparison is accounting for preprocessing. The 36× cost advantage compares the "GPT-5.4 agent baseline" processing raw JSON against EgoCITE-GPT consuming pre-generated atomic indice. If the offline indexing cost is amortized into EgoCITE's per-query cost, what is the actual cost ratio?
  2. W2
3 comments

We thank the reviewer for the insightful comments. Below, we address each concern in turn.

W1: Novelty and temporal scoring

We agree the building blocks are established and do not claim them individually. Our contribution is the finding that semantically similar evidence is not necessarily situationally relevant (Appx. A), and a design in which each component addresses one resulting failure. Tab. 10 shows the progression on EgoR1-Bench: accuracy 61.3% → 69.7% (+EgoScheme) → 69.7% (+EgoIndex) → 75.3% (+EgoRetrv), and hit rate 40.7% → 55.7% → 59.3% → 62.7%.

Eq. (1) is deliberately simple. What is new is that the time query Q_t is estimated from the question's temporal intent and used as a ranking signal rather than a hard filter. Unlike recency scoring, which decays with the time elapsed since the present, Eq. (1) decays with the distance to Q_t. λ = 0.99 is applied per hour over a 56-hour timeline, so the weight spans 1 to ≈ 0.57. Because top-candidate similarities are tightly clustered (ranks 1 and 24 differ by 0.16 in Fig. 14), this moves the target-overlapping index from rank 24 to 4, 21 to 4, and 26 to 8 (Figs. 14–16). Tab. 10 and Tab. 11 quantify its contribution on EgoR1-Bench: adding EgoRetrv raises accuracy from 69.7% to 75.3% and hit rate from 59.3% to 62.7%, and removing only the decay (λ = 1) lowers accuracy from 75.3% to 74.3% and hit rate from 62.7% to 61.0%.

W2 / Q2: Prompt engineering

We do not propose a sophisticated memory architecture. Instead, we reveal that as long as a human-centered memory is situation-aware, it gains 14.0% in accuracy over caption RAG (61.3% → 75.3% on EgoR1-Bench, Tab. 10), despite using the simplest vector database. Its methodological contribution lies in two findings revealed by our results:

  1. What to index. For a situation-aware, human-centered memory system built from video and dialogue, what matters is the content of each atomic memory index, beyond what traditional memory systems store. Atomic memory indices must be self-contained (resolving coreferences and ellipsis), anchored to a specific person, and organized into four memory views: actions, utterances, activities, and conversations. Existing systems miss this: 10.1% of WorldMM triples contain unresolved references and 17.4% of LoCoMo observations contain verbatim quotes (Appx. A.1). Indexing this content raises accuracy from 61.3% to 69.7% and hit rate from 40.7% to 59.3% over caption RAG (Tab. 10).
  2. How to retrieve. We show how off-the-shelf LLM agents can extract the temporal cues in a question (e.g., "last time", "usually", "this morning") and turn them into view and time queries for EgoRetrv that retrieve the relevant atomic memory indices. 76% of EgoLifeQA questions carry such cues, yet existing systems lose 9.8–13.3% accuracy on them (Appx. A.2). On these questions, EgoCITE-GPT reaches 62.6% vs. 47.1% for WorldMM-GPT (Tab. 5).

Both findings hold across models: the GPT-based memory baselines use the same GPT-5.4 retrieval and answering backbone as EgoCITE-GPT, and a fully open-weight pipeline without GPT-5.4 still reaches 57.1% on EgoLifeQA (Tab. 15), vs. 49.6% for WorldMM-GPT.

W3 / Q1: Cost comparison

The 36× figure compares per-question cost only (3.979 dollars vs. 0.110 dollars, Tab. 13). Both systems start from the same multimodal captions; EgoCITE additionally builds the index. The indexing cost is 0.018 dollars per minute of recording (new Tab. 14). Indexing one participant's 56-hour recording therefore costs ≈ 60.5 dollars once (3,360 min × 0.018 dollars), i.e., ≈ 0.125 dollars per question on EgoLifeQA (≈ 484 questions per participant). Including it, EgoCITE-GPT costs 0.235 dollars per question, ≈ 17× cheaper than the GPT-5.4 agent (≈ 25× with one round). The indexing cost of all methods is reported in the revision (Tab. 14).

EgoCITE-GPTGPT-5.4 agentVideoRAGWorldMM-GPTA-MemEgoRAG
Index cost (dollars / min input)0.01800.0110.0200.0040.003
Cost per question (dollars)0.1103.9790.1000.2140.0160.048

The review comments were originally written by the reviewer and polished by an AI tool.

Thanks for the response and additional experiments, but I still intend to maintain my original score. I expect a explanation more theoretically and explainable.

Reviewer_gq3POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets question answering over multi-day egocentric recordings. EgoCITE rewrites captions and transcripts into self-contained atomic memories, organizes them into action, utterance, activity, and conversation views, and retrieves by multiplying embedding similarity with a question-conditioned time-decay score before an LLM curates evidence. Three EgoLife-based benchmarks show higher accuracy than memory baselines and long-context agents at lower cost.

Critical strengths

The strengths of the paper are listed as follows:

  • Well written and easy to follow. The exposition moves from two failure cases (Fig. 2) to two insights (Sec. 3), four designs that each answer one insight (Sec. 4), and experiments ordered by claim (Sec. 5); Fig. 3 and Table 1 give the pipeline and index specification, so the method is reconstructible from the main text.
  • Transparent, model-agnostic implementation. Prompts and the tool schema are printed, configurations listed, anonymized code linked, and the pipeline runs with proprietary GPT-5.4 and open-weight Qwen3.6-27B and Gemma-4-31B. This eases reproduction.
  • Quantified failure diagnosis. The motivation rests on measurements: 10.1% of 92,131 WorldMM triples contain unresolved references, 17.4% of 254,997 LoCoMo observations keep verbatim quotes, and WorldMM and VideoRAG lose 9.8 to 13.3 points on time-aware questions. Each design choice thus answers a measured gap.
Critical weaknesses

The weaknesses of the paper are listed as follows:

  • Baseline re-implementation gap. Published WorldMM-GPT reports 65.6% on EgoLifeQA (500 A1_JAKE queries) and 65.3% on Ego-R1 Bench [1]; the re-implementation scores 49.6% and 64.0% (Table 2) with a newer backbone and the same dataset captions, and EgoRAG and VideoRAG fall similarly (52.0 to 34.3; 55.4 to 46.5). EgoCITE-GPT's 63.8% on the full 2,905-question set does not even reach WorldMM's own published figure, so the +14.2% gain (Sec. 5) cannot be verified until both systems are evaluated on a common question set.
  • Temporal mechanism never isolated. Insight 2 rests on question-conditioned temporal scoring (Eq. 1), yet no ablation sets λ = 1, drops time queries, removes the sampling agent, or tests the hard window criticized in Sec. 4.2 [2]. Table 10 adds all of EgoRetrv at once, and its “+EgoIndex” and “+EgoScheme” configurations are undefined.
  • Single corpus and dataset-specific engineering. All three benchmarks are carved from the same week-long EgoLife recording of six housemates [3], the extraction prompts hardcode those six names (Figs. 18 and 19), and identities for VLM captioning are manually annotated per scene. No second corpus is tested, so generalization beyond EgoLife is unsupported.
  • Weak statistical support. EgoR1-Bench has 300 questions [4], a size the paper omits, yet hosts all component and granularity ablations. λ is tuned on a 100-question subset of this test set (Table 11). No seeds, intervals, or significance tests appear, although EgoMem margins are 0.9 points (Table 2).

[1] Yeo, Woongyeong, et al. "Worldmm: Dynamic multimodal memory agent for long video reasoning (2025)."

[2] Sun, Shitong, et al. "EgoGraph: Temporal Knowledge Graph for Egocentric Video Understanding." arXiv preprint arXiv:2602.23709 (2026).

[3] Yang, Jingkang, et al. "Egolife: Towards egocentric life assistant." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[4] Tian, Shulin, et al. "Ego-r1: Chain-of-tool-thought for ultra-long egocentric video reasoning." arXiv preprint arXiv:2506.13654 (2025).

Questions for authors

Answers to the following questions could materially change my assessment.

  • WorldMM-GPT scores 49.6% on EgoLifeQA in Table 2, whereas WorldMM reports 65.6% for itself on its 500-query A1_JAKE subset, and your Ego-R1 Bench re-implementation (64.0%) matches the published 65.3%. Which step of your EgoLifeQA pipeline (Qwen3.6 translation, 30-second re-alignment, the 15-index cap, question normalization) explains the EgoLifeQA-specific drop? Please report EgoCITE-GPT and the official WorldMM release (prebuilt memory, gpt-5-mini/gpt-5 defaults) on the same 500 A1_JAKE questions.
  • What are the accuracy and hit rate of EgoCITE-GPT with λ = 1, with time queries disabled, and without the sampling agent, each under the full 5-round drafting loop, on EgoR1-Bench and the time-aware EgoLifeQA subset? What exactly do "Caption RAG" (61.3% on EgoR1, versus 49.5% for EgoRAG in Table 2), "+EgoIndex" without EgoScheme, and "+EgoScheme" consist of in Table 10?
  • How many questions does EgoR1-Bench contain in your evaluation, and are the 100 questions used to tune λ (Table 11) excluded from the EgoR1 numbers in Tables 2, 3 and 10? Can you report run-to-run variation (three runs, or bootstrap intervals with paired tests against the best baseline) for Tables 2, 3, 10 and 11?
  • Do you have any result on recordings other than EgoLife, e.g., HippoVlog, where WorldMM reports 78.3%? How does EgoCITE behave when the six housemate names hardcoded in Figs. 18 and 19 are replaced by an identity list passed as input, and how many person-hours did the per-scene identity annotation of Appx. B.1 require?
3 comments

We thank the reviewer for the insightful comments. Below, we address each concern in turn.

Q1: EgoCITE-GPT vs. the official WorldMM on the same 500 A1_JAKE questions

  • EgoCITE-GPT. We report EgoCITE-GPT on the 475 A1_JAKE questions of EgoLifeQA, as released in the HuggingFace repository (A1_JAKE.json), where it achieves 72.0% accuracy (342/475). This result is also recorded in our submitted supplementary files (log).

  • WorldMM-GPT. We implement WorldMM-GPT with our GPT-5.4 setup rather than the GPT-5 and GPT-5-mini used in the WorldMM paper. At the time of our evaluation (Aug 2026), GPT-5 and GPT-5-mini were legacy OpenAI models and were slow to be served: a single batch of requests (we treat each 30-second window as one request and each 5-minute window as one batch) takes ten minutes or longer. This does not scale to the roughly 300 hours of recordings over six participants: processing would take at least 25 days if run sequentially. We therefore chose GPT-5.4, the up-to-date and fast model at the time of evaluation, for all baselines.

  • Comparison. On the A1_JAKE questions, the accuracies are: WorldMM-GPT as reported in the WorldMM paper (65.6%), our WorldMM-GPT implementation (61.5%, 292/475), and EgoCITE-GPT (72.0%). EgoCITE-GPT therefore improves over the number reported in the WorldMM paper by 6.4%. We were unable to reproduce the number reported in the WorldMM paper and could not identify the cause of the 4.1% gap between it and our implementation. This is also why we release all our code, together with the retrieval and QA traces, in our GitHub repository, to ease evaluation and comparison in future work.

Q2: Ablations on λ = 1, time queries, and the sampling agent; composition of the Table 10 columns

All numbers below are for EgoCITE-GPT under the full 5-round drafting loop on EgoR1-Bench (300 questions).

SettingAccuracy (%)Hit rate (%)
Full EgoCITE-GPT (λ = 0.99)75.362.7
λ = 1 (no decay)74.361.0
Time decay and time-aware prompts both removed69.759.3
  • λ = 1. We added λ = 1 to Tab. 11. Removing the decay alone lowers accuracy by 1.0% and hit rate by 1.7%.
  • Time queries disabled. Removing both the time decay and the time-aware prompts lowers accuracy by 5.6% and hit rate by 3.4%. This setting is the "+EgoIndex" column of Tab. 10.
  • Sampling agent. We do not have a run without the sampling agent under the full 5-round loop, and we will add it.
  • Time-aware EgoLifeQA subset. We do not have the 5-round ablations on this subset. As a first-round analysis on EgoLifeQA, we replayed the first search call of each question with and without the decay: recall@10 rises from 43.6% to 49.3%, and from 42.7% to 49.7% on the 2,226 questions for which the drafting agent issues a time query.
  • Table 10 columns. All columns use the same GPT-5.4 agent loop (5-round drafting, sampling, and response) and differ only in the memory and retrieval they operate on. "Caption RAG" retrieves over the raw 30-second multimodal captions, without EgoScheme extraction or multiple views. It differs from the EgoRAG row of Tab. 2 (49.5%), which runs the original EgoRAG pipeline with its own multi-granularity summaries and retrieval. "+EgoScheme" replaces the captions with atomic action and utterance indices extracted with EgoScheme. "+EgoIndex" adds the activity and conversation views. "+EgoRetrv" adds the time-aware retrieval. In the revision we reordered the columns as Caption RAG → +EgoScheme → +EgoIndex → +EgoRetrv, because the views of EgoIndex are built from EgoScheme-extracted indices: accuracy is 61.3% → 69.7% → 69.7% → 75.3% and hit rate is 40.7% → 55.7% → 59.3% → 62.7%.

Q3: Other datasets than EgoLife

  • We do not currently have results on recordings other than EgoLife. We will consider adding more benchmarks and datasets, such as HippoVlog, as time and budget permit.

Q4: How does EgoCITE behave when the six housemate names hardcoded in Figs. 18 and 19 are replaced by an identity list passed as input, and how many person-hours did the per-scene identity annotation of Appx. B.1 require?

  • We would kindly ask the reviewer to clarify this question: for example, what would such an identity list look like, and when would it be provided as input, i.e., at indexing time or at retrieval time?

  • In our understanding, each subject's name must be associated with their appearance in the video for visual identification to be possible. The difficulty with the EgoLife dataset is that all faces are redacted. We tried annotating each person's shape with YOLO bounding boxes: for DAY1's recording alone, this took us more than 10 hours of labor across all 6 persons and over 2,000 bounding boxes, which is impractical and not scalable. Moreover, based on our observations, appearance (mainly clothing and hairstyle) varies from day to day and even from scene to scene.

  • We therefore use visual IDs with hardcoded names as prompt inputs to the VLM description step. Annotating all persons in a scene takes us less than 10 minutes, as long as their appearances are static within that scene.

Thanks for the authors' response. The rebuttal addresses most of my concerns, so I increased my rating.

Reviewer_8ABJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EgoCITE, a situation-aware memory framework for long-horizon egocentric QA. The authors argue that semantic similarity alone is not enough, because stored captions are fragmentary and questions have implicit temporal intent. Thus propose EgoCITE which use an LLM to rewrite captions into self contained atomic memories. Experiments on the the proposed dataset show accuracy gain over agentic memory baselines.

Critical strengths
  1. The paper shows existing dataset contain unresolved pronouns and copy utterances verbatim. Baselines also drop on time aware questions. The motivation is thus concrete and measured.
  2. The experiment results show the benefits of agentic memory. On all three benchmarks, the gains hold with the same GPT-5.4 backbone and also with Qwen. The cost is lower than WorldMM.
Critical weaknesses
  1. Rewriting fragments into atomic units is similar to some previous works like proposition indexing and decontextualisation [1,2]. Eq. 1 is basically the scoring of genrative agents [3] which uses the same settings. In addition, some similar works were missed to discussed.
  2. The temporal aware retrieval is not isolated, and the lambda seems to be tuned on the test benchmark. In the sec 5, "We ablate the temporal decay factor λof EgoCITE-GPT on a 100-question validation subset of EgoR1-Bench in Tab", which is part of a reported test benchmark, on 100 questions the curve is within noise.
  3. The comparison with long context agents and the cost claim are not well controlled. Long-context accuracy is measure with 50 questions batched in one prompt but cost and latency are measured with single question prompts on another 100 question sample. If cost is amortized over the batch, the GPT agent costs seem not more expensive than EgoCITE.
  4. Some reported numbers are inconsistent, which make it hard to trust how results are assembled. In Table 12, EgoCITE-qwen on egomem has nearhits 75.6, but hits 86.8 in table 3, by definition of nearhits should not be lower than Hits? Appendix E's decomposition does not fit the main results. 49.6 (hit) - 11.3 (hit but wrong) + 12.6 (miss but correct) =50.9, not the 63.8 accuracy? And it seems there are some other obvious number issues.
  5. All three benchmarks are built on the same egolife recordings, they are not independent environments. Person IDs are manually annotated the main results use human dense captions, and the prompts hard code the housemates names. These make the generality and baseline fairness are limited.

[1] Dense X Retrieval: What Retrieval Granularity Should We Use? [2] Decontextualization: Making Sentences Stand-Alone [3] Generative Agents: Interactive Simulacra of Human Behavior [4] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory [5] MemoryBank: Enhancing Large Language Models with Long-Term Memory

Questions for authors

see weakeness, and here are some additional questions:

  1. what is the accuracy of the long context agents with one question per prompt, on at least a subset? and what are the cached cost and the total offline memory construction cost of EgoCITE and WorldMM
  2. Are the 100 questsions used for tuning lambda excluded from the EgoR1 results in Table 2/3/10?
  3. Why is NearHits lower thanHits for EgoCITE-Qwen on EgoMem, and how do the App. E numbers fit with 63.8 accuracy?
3 comments

We thank the reviewer for the careful reading, in particular for checking our numbers. Below, we address each concern in turn.

W1: Relation to prior work [1–5]

We agree that these works are closely related, and we added a discussion of [1–5] to Sec. 2.

  • [1, 2]. They rewrite well-formed encyclopedic text, where the missing context is inside the same document. Our input is egocentric captions and dialogue, where the missing context is situational: who acts or speaks, what an elliptical utterance refers to across turns, and which activity or conversation the moment belongs to.
  • [3]. Its recency score decays with the time since a memory was last accessed, so it always favors recent memories. Eq. (1) decays with the distance to a time interval Q_t estimated from the question, so it can favor any part of the timeline. We do not claim the exponential form itself.
  • Contribution. The finding that semantically similar evidence is not necessarily situationally relevant (Appx. A), and a design in which each component addresses one resulting failure (Tab. 10).

W2 / Q2: Isolation of time-aware retrieval and selection of λ

  • Q2. No, and we acknowledge this as a mistake in the submission. The 100 questions used to select λ were not isolated: they were part of the 300 EgoR1-Bench questions reported in Tab. 2, 3, and 10, and we did not state this. We also agree that the curve on 100 questions is within noise.
  • Revision. We now treat the full EgoR1-Bench (300 questions) as our development set and state this in Sec. 5: λ and all component ablations (Tab. 10 and 11) are run on all 300 questions, and Tab. 11 now also reports accuracy and λ = 1. The EgoR1-Bench column of Tab. 2 should therefore be read as a development-set result. EgoLifeQA and EgoMem are the held-out benchmarks.
λ0.970.980.990.9951
Accuracy (%)69.073.075.374.774.3
Hit rate (%)58.359.762.761.361.0
  • Held-out benchmarks. The same λ = 0.99 is used for all benchmarks. EgoLifeQA (2,905 questions) and EgoMem (939 questions) were not used to select λ, and the gains over the strongest agentic memory baseline on them are 14.2% and 4.4% (Tab. 2). Our main claims rest on these two benchmarks.
  • Isolating time-aware retrieval. (1) Tab. 10: adding EgoRetrv on top of EgoScheme and EgoIndex raises accuracy from 69.7% to 75.3% and hit rate from 59.3% to 62.7%. (2) Tab. 11: removing only the decay (λ = 1) lowers accuracy by 1.0% and hit rate by 1.7%. We agree that the differences between λ = 0.98 and λ = 1 are small on 300 questions. (3) To isolate the decay from the variance of the LLM agents, we replayed the first search call of each EgoLifeQA question with and without the decay, with all queries fixed: recall@10 rises from 43.6% to 49.3%, and from 42.7% to 49.7% on the 2,226 questions for which the agent issues a time query.

W3 / Q1: Long-context agents and cost

  • Why batching. One independent call costs 3.979 dollars, i.e., about 16,500 dollars per model for the 4,144 questions, which is not affordable. We therefore follow batch prompting [6] (cited in Appx. C) with 50 questions per prompt for accuracy, and measure cost and latency with one question per prompt.
  • Accuracy with one question per prompt. We will test it on a small subset in a later revision if the budget allows.
  • Cached context. We compute its cost for one participant's 56-hour recording (7 days × 8 hours per day): 6,720 thirty-second captions, 783k tokens, ≈ 117 tokens per caption.
    1. Cached tokens are not free: 0.25 dollars per 1M tokens (0.50 dollars above 272K tokens), vs. 2.50 (5.00) dollars for new tokens (OpenAI pricing). Every request sends the whole prompt again and pays the cached price for the prefix (prompt caching guide).
    2. The context accumulates and the user can ask at any time, so each new caption is appended and the prompt is sent again. The i-th request pays for 117 × (i − 1) cached tokens and 117 new tokens.
    3. New tokens: 117 × 6,720 ≈ 783k tokens, i.e., 3.2 dollars. Cached tokens: 117 × (0 + 1 + … + 6,719) ≈ 2.63 billion tokens, i.e., 1,236 dollars.
    4. Total: ≈ 1,239 dollars per participant, vs. 60.5 dollars for the EgoCITE index (3,360 min × 0.018 dollars), i.e., ≈ 20× more. The cached context grows quadratically with the recording length; the EgoCITE index grows linearly.
Cumulative cost (dollars)Day 1Day 2Day 3Day 4Day 5Day 6Day 7
GPT-5.4 agent with cached context13.754.2163.2351.7593.9889.81,239.4
EgoCITE-GPT index8.617.325.934.643.251.860.5

This favors the cached baseline: it assumes that every request hits the cache (the TTL is at most 24 hours) and ignores output tokens. Each question on the cached context then costs ≈ 0.46 dollars, vs. 0.110 dollars for EgoCITE-GPT.

W4 / Q3: Inconsistent numbers

Both are our errors, and both are corrected in the revision.

  • NearHits < Hits for EgoCITE-Qwen on EgoMem. We made a mistake: this was a bug in our evaluation script, and we have corrected it. The NearHits rate of EgoCITE-Qwen is 88.0% on EgoMem (826/939; Hits 86.8%) and 71.3% on EgoR1-Bench (214/300; Hits 59.0%). Tab. 12 is corrected. The hit rates in Tab. 3 were not affected. For reproducibility, the full retrieval and QA traces of these runs have been available in our anonymous repository since the initial submission (EgoMem logs, EgoR1-Bench logs), so the corrected numbers can be verified from them.
  • Appx. E. (1) The decomposition in Appx. E is based on the 2,828 EgoLifeQA questions (out of 2,905) that have a valid target timestamp, which is the set on which hit rates are defined, while Tab. 2 reports accuracy on all 2,905 questions. (2) We made a mistake in the expression: the original sentence mixed two criteria. 11.3% was "hit but wrong" under Hits, and 12.6% was "correct but missed" under NearHits, so the two numbers cannot be combined with the 49.6% hit rate. The revision uses one criterion (Hits or NearHits) for both percentages: "retrieved but wrong" and "correct without the annotated evidence". For EgoCITE-GPT: 69.4% (NearHits) − 19.2% (retrieved but wrong) + 12.5% (correct without the annotated evidence) = 62.7%. For EgoCITE-Qwen: 63.6% − 17.9% + 15.6% = 61.3%.

W5: Generality and baseline fairness

  • Same recordings. We agree and will state this as a limitation.
  • Fairness. (1) All baselines take the same multimodal captions from the EgoLife dataset as input, which already include the expert-annotated identities, so the person annotation does not favor EgoCITE in the baseline comparisons (Tab. 2, 3, 5, and 12). (2) We use our own identity prompts (Figs. 18 and 19) exclusively in the VLM-generated caption experiment (Appx. D.4, Tab. 15), which is not part of the baseline comparisons.
  • Hardcoded names. (1) In the memory construction prompts, the names are only an output-format instruction (write a name instead of a pronoun). They do not provide any identity information: who does what comes from the EgoLife captions, which all baselines also take. (2) The VLM identity prompts map visual IDs to names because all faces in EgoLife are redacted; they apply only to Appx. D.4.

References

[6] Zhoujun Cheng, Jungo Kasai, and Tao Yu. Batch Prompting: Efficient Inference with Large Language Model APIs. EMNLP 2023 Industry Track.

I appreciate the authors' response and additional results, but intend to maintain my original score. The authors acknowledged that hyperparameter tuning was performed on a reported benchmark and that several numerical inconsistencies were due to evaluation errors. These issues still make me concerned about the reliability of the results, and I am not sufficiently convinced to recommend acceptance at ICLR.

Comments on the paper

1 comment

Dear Reviewers,

Please kindly review the authors' rebuttal and engage in discussions if you have further questions.

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.