- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies VQA over personal camera rolls, where a conversational assistant answers user-specific questions ("What did I eat after the STS-135 launch?") by locating evidence in a long personal visual history. The motivating gap: existing photo products do face/object or keyword search, and RAG-over-captions designs discard pixels, losing the fine-grained cues (identity, relationships, event context) that matter for personal memory. Two contributions follow — camroll, a benchmark of 50 real rolls (31,476 images, 2,500 human-authored questions), and camroll-agent, a training-free reference agent.
Technically the agent (Sec. 4) is a three-level pyramid — raw pixels, profile-conditioned captions C = {c_i}, and event summaries E = {e_j} = (I_j, d_j, m_j) — linked by hashed IDs and an event_id foreign key giving O(1) bidirectional navigation (Sec. 4.1, Fig. 3). Captions condition the captioning MLLM on the user's profile photo plus a look-back window of k = 3 recent images; event segmentation is an incremental ADD / UPDATE / NO_OP decision per image given the current caption, timestamp, recent captions, and the latest event summary (prompt in A.6). Five tools factor along retrieval paradigm × access depth: search (frozen all-MiniLM-L6-v2 over FAISS), grep (BM25 via FTS5), list (metadata filters), get (full record), and view(id, prompt, ≤6 images), which re-examines raw pixels with a VLM. Interaction is a ReAct loop capped at 25 tool calls with ≤5 view calls (A.3).
Results (Table 3, Gemini-2.5-Flash answerer): no context 30.0 MCQ; all captions 63.4; all images 76.5; oracle 86.4. camroll-agent reaches 70.5 MCQ with 88.5 evidence recall and 4.11 judge at ~3.2k context and ~15k total tokens — ~50× fewer than the ~750k all-images setting while retaining ~92% of its accuracy. Baselines span 32.3 (MemOS) to 54.0 (Claude Code, at ~59k tokens), with Mem0 53.2, LightMem 52.7, HippoRAG2 48.5, Self-RAG 46.2, SimpleMem 44.6. Ablations on the in-house split (Table 5, judge only): final 5.04, generic captions 4.01, no events 4.03, no captions 2.29, no search 2.88, no view 3.22. The error analysis (Table 7) assigns 14.7% of failures to wrong evidence, 24.7% to retrieving-but-never-viewing it, 10.5% to budget exhaustion, 21.7% to premature stopping, 17.5% to answering wrongly after viewing the right images.
Critical strengths
- A new, carefully built benchmark with human-authored questions. Unlike LLM-synthesized-query datasets (Deng et al., 2026; Xu et al., 2026), annotators inspect each roll, which is what produces longitudinal items ("Have I ever been to Italy?"). The semantic/episodic split, 80% three-verifier agreement, and evidence-conditioned design (98% of episodic questions carry gold evidence, Table 8) are documented checkably (Sec. 3.1, A.5, A.6).
- Unusually candid reporting of confounds. Sec. 5.2 names three confounds in Table 3 (profile-conditioned vs. generic captions, the
viewtool, agentic vs. single-pass retrieval), states that the generic-caption ablation explains part of the gap (5.04 → 4.01), and notes Claude Code's different backbone. A.3 adds that the all-images baseline is "a strong but not strictly upper-bound reference" because images are downscaled to 768px. - Ablations are unusually complete for a systems paper. Table 5 changes one component at a time across all five tools and all three memory layers, and the ordering is interpretable: captions matter most (2.29 without them),
viewsecond (3.22),searchthird (2.88) — supporting the "retrieve through abstractions, verify in pixels" thesis rather than asserting it. - Trajectory-level error analysis and careful reproducibility. The rule-based taxonomy in A.2 is fully specified (step counts,
viewcap,ρ = 1,G ∩ V = ∅), making it reproducible, and yields an actionable finding (B and D together are 46.4% of failures). A redistributable YFCC core (20 users, 1,000 questions) ships with exact tool budgets and verbatim prompts (A.3, A.6).
Critical weaknesses
- The central "modality-preserving" claim is never tested against image-embedding retrieval. Sec. 5.2 admits that disentangling the comparison "would require running baselines with personalized captions and adding image-embedding retrieval baselines (e.g., CLIP or SigLIP top-k images)", and defers this. That baseline is precisely the control for the headline design principle: without it, the gap over Mem0 (53.2) or LightMem (52.7) could come from having pixel access at all, not caption-indexed pixel access. This is the most load-bearing missing experiment.
viewdoes not return pixels, which weakens the modality-preservation framing. Sec. 4.2 statesview"returns a VLM-generated textual analysis," so the answering model never sees an image and verification is mediated by another model call. With Gemini-2.5-Flash apparently serving both roles,viewcan confirm rather than correct, and correlated errors go unmeasured.- Half the benchmark is unobtainable, and the public half is atypical. The in-house subset (30 users, 1,500 questions) is access-controlled pending legal review; YFCC is 592×514 mean resolution versus 3188×3500 in-house (Table 10), so the released core is not the setting argued for. Table 9 also shows ~59% of released images carry non-commercial licenses.
- The evaluation instrument is unvalidated and carries the ablations. Free-form scoring is GPT-4o on 0–10 with no human study; the only check is a second judge model (Table 6), which measures judge-family sensitivity, not validity. Tables 4 and 5 rest entirely on it, so "captions are the most influential memory component" is a judge-score claim. The primary MCQ metric is only 3-way (33.3% chance), and no CIs or user-level variance are reported (A.3) despite only 50 clusters.
- Ablation token accounting is non-monotonic and unexplained. In Table 5, removing
grep,list,get, orviewincreases input tokens (4.0k, 3.7k, 4.3k, 3.5k vs. 2.4k), as do generic captions (4.2k). This is never discussed and undercuts the efficiency framing; trajectory compensation for a lost tool is not isolated. - The best configuration is not benchmarked, and the main table is not the targeted regime. Table 4 shows Gemini-3.1-Preview-Pro at 5.80 judge with the same agent, yet Table 3 reports Gemini-2.5-Flash. The motivating claim concerns rolls of "thousands" of photos (1–10M tokens), but every roll is under 1,000 images (median 558; Table 10) — the regime where full context still fits. The paper concedes this (Sec. 1) but never evaluates beyond it.
Questions for authors
- What happens on a roll that genuinely exceeds the context window (e.g., 10k–50k images)? Without a full-context reference at that scale, how should the 50× efficiency claim be read?
- Since
viewreturns VLM text rather than pixels, would giving the answering model the raw images directly (within budget) close the 70.5 → 76.5 gap? How often doesviewcontradict the caption it is meant to verify? - Table 5 shows every tool removal increasing input tokens. Is this trajectory compensation, and does the total count (including the image tokens Table 5 omits) change the tool ranking?
- Why is the headline run on Gemini-2.5-Flash (5.04) rather than Gemini-3.1-Preview-Pro (5.80), and does the Table 3 ordering hold with the stronger backbone?
- Semantic questions have 0% gold evidence (Table 8), so ~20% of the benchmark cannot separate retrieval from answering, contrary to the contribution claim. How are they scored, and would per-user CIs change the significance of 70.5 vs. 76.5?
5 comments
Direct Visual Access vs. Textual View
We thank the reviewer for this insightful question. To investigate whether textual mediation through the view tool introduces an information bottleneck, we conducted an additional ablation where the agent directly receives the retrieved raw images in its context, rather than VLM-generated textual descriptions.
Interestingly, direct image access achieves 70.2% MCQ accuracy, compared with 70.5% using our original view tool, showing nearly identical performance. This suggests that textual mediation does not introduce a substantial performance bottleneck in our setting. One possible explanation is that modern VLMs can effectively extract relevant visual information, while directly incorporating multiple images into the agent's context may introduce additional irrelevant information.
Moreover, our decoupled view design offers an efficiency advantage by keeping visual tokens outside the main agent's reasoning context. For example, assuming 4k tokens per image, directly processing 10 images requires a context containing 40k visual tokens, whereas separate visual inspection operates on much shorter sequences. This reduces the attention burden on the main reasoning model while maintaining comparable accuracy.
We will incorporate this ablation and clarify that in the revised version.
Dataset Availability and Methodological Contribution
We acknowledge the limitations regarding dataset availability and representativeness. As clarified in our previous rebuttal, the YFCC subset (20 users, 1,000 questions) has already been released under its original Creative Commons licenses. For the in-house subset (30 users, 1,500 questions), we are actively consulting with our legal team regarding broader redistribution.
Importantly, our contribution goes beyond introducing a benchmark. We also propose camroll-agent, a task-specific approach that highlights the importance of structured visual memory construction and iterative agentic retrieval for personal camera-roll VQA.
Our central finding is that conventional RAG and memory-layer approaches are insufficient for this task. As shown in Table 3, existing RAG and memory systems achieve only 32.3–53.2% MCQ accuracy, while camroll-agent achieves 70.5%. These results demonstrate the value of combining a dedicated hierarchical memory database with interactive evidence retrieval and visual agent.
Thus, our work not only introduces a new benchmark but also provides empirical evidence for an effective architectural direction for personal visual memory agents.
Image-Embedding Retrieval Baselines
We thank the reviewer for this suggestion. To disentangle the effects of retrieval representations and raw-image access, we introduce three additional SigLIP-based retrieval baselines:
All three baselines retrieve original raw images (rather than captions) and pass them, together with the question, to the same Gemini-2.5-Flash answering model. We evaluate Top-K = 1, 5, and 20.
Results (Free-form Judge Score, 0–10):
For reference, camroll-agent achieves 4.11 with iterative agentic retrieval rather than fixed Top-K retrieval.
These results show that caption-based retrieval consistently outperforms direct image-embedding retrieval, with personalized captions providing additional improvements. Furthermore, despite having access to raw images, all single-pass retrieval baselines substantially underperform camroll-agent. This supports the value of memory abstractions and agentic search.