- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies "context-dependent image-set retrieval" over a personal album A = {a_i} whose photos carry visual content plus optional time and GPS, where the query Q resolves only by chaining evidence across photos (e.g., "the day the wall with drawings of fresh ingredients was seen" anchors the day; "metal chairs" specifies the target). Following DeepImageSearch (Deng et al., 2026), the task is F: (A, Q) → R̂ ⊆ A (Eq. 1), with a ReAct-style VLM policy u_t ~ π_θ(u | Q, τ_<t) (Eq. 2) and answer R̂ = G_θ(Q, τ_1:T) (Eq. 3).
MNEME contributes four components. The Album Memory Graph M_A = (V, E) with V = V_P ∪ V_C ∪ V_S (Eq. 4) stores photo nodes (raw pixels + metadata), clue nodes (recurring faces/objects, clustered with Buffalo L, YOLO26, and Qwen3-VL-Embedding, then verified by Qwen3.5; singletons dropped), and multi-scale story nodes (captions → daily/location/activity → experience memories). Seven tools expose it (Appendix B) with deliberate guards: story synopses return no Photo IDs, metadata filtering emits a "set certificate", and temporal neighbors are yielded as "anchor proposals only". Evidence-Aware Short-Term Memory keeps a bounded state M_t = (E_t, P_t, X_t) (Eq. 5) separating active evidence, search progress, and photo-exposure provenance. Self-Evolving Search Skills grow a textual bank: success is y(Q, τ) = I[EM(Q, τ) = 1] (Eq. 6), and same-query success–failure contrasts are folded in via S^(r+1) = Revise(S^(r), L^(r)) (Eq. 7), then frozen — no backbone updates, no evaluation labels. Dual-route search runs a memory-free direct route and a memory-guided reconstructive route in parallel (Eq. 8), pools both answer sets plus recently inspected candidates (Eq. 9), then closes the pool for a reconciliation verifier (Eq. 10) seeing only raw images and metadata.
Experiments use 8×A800-80GB, 30 turns per route, 4 rollouts × 3 induction generations. On DISBench (122 queries, 109,467 photos, 57 users), MNEME reaches 14.0 EM / 30.0 F1 overall with Qwen3-VL-32B-Instruct (vs. PhotoCraft 9.0/24.7) and 23.0 EM / 41.8 F1 with Qwen3.5-27B (vs. ImageSeeker 14.8/35.6), with intra-event 24.6/46.0 and inter-event 21.5/38.2 on the latter. On a 200-question stratified CamRoll subset: 90.3/79.0/93.8, Judge 5.6 (Qwen3-VL-32B) and 95.5/82.0/95.7, Judge 6.2 (Qwen3.5-27B). Table 2 shows STM helps (+3.2 EM), the Album Memory Graph alone hurts (18.0 → 14.8 EM), skills recover and exceed it (+4.9 EM), and dual-route adds +3.3 EM. Figure 5 traces skills (DISBench EM 14.8 → 18.9 → 18.9 → 19.7); Figure 4 shows the turn-budget trade-off (EM 20.5 at 40 turns → 15.6 at 50).
Critical strengths
- Consistent wins across two backbones and two tasks. MNEME is best on every DISBench column under both backbones (e.g., inter-event EM 21.5 vs. ImageSeeker's 10.8 with Qwen3.5-27B), weakening the "one lucky backbone" objection.
- Unusually honest negative result. Table 2 reports that adding the Album Memory Graph alone degrades performance (18.0 → 14.8 EM, 37.0 → 34.8 F1), and §5.3 attributes this to unguided memory complexity. Reporting a component that hurts is valuable and rare.
- Fair prompt control. Appendix E.2 states MNEME reuses ImageSeeker's system prompt verbatim, so gains over ImageSeeker come from added structure rather than prompt engineering; PhotoCraft's reimplementation additionally received the same synthetic induction set (Appendix D).
- Well-designed interface-level error containment. Story synopses withholding Photo IDs, "set certificates", and anchor-vs-target separation in the verifier target exactly the failure modes §5.4 observes (e.g., "transferring anchor-only constraints to targets").
- Careful synthetic-data hygiene. Appendix C documents disjointness from evaluation users/questions/answers/photo IDs, double blind verification passes with shuffled orders, adjudication on disagreement, and explicit rejection criteria.
- Transparent scope of claims. The paper states plainly that the verifier "does not establish completeness over the entire album" (§4.4) and that dual-route gains "cannot be attributed to reconciliation alone" (§5.3).
Critical weaknesses
- No variance, seeds, or significance testing anywhere.** With 122 DISBench queries, 14.0 EM corresponds to roughly 17 exact matches, so a +5.0-point EM gain is ~6 queries; given tool nondeterminism, no Table 1 margin is interpretable without variance.
- The Figure 4 sweep does not match the main configuration, and this is never explained. At the default 30-turn budget Figure 4 reports EM 19.7 / F1 38.3, whereas Tables 1–2 report the full dual-route system at 30 turns per route as 23.0 / 41.8. Figure 4's numbers match Table 2's "+ Search Skills" (single-route) row exactly, so the swept configuration appears to omit dual-route — but neither the caption nor §5.4 says so.
- Compute is not matched, and construction cost is unquantified. MNEME pays for two 30-turn routes plus a verification call per pooled candidate while baselines pay for one search; cost "includes both searches" (§4.4) but no wall-clock or token figures are given, nor any equal-budget baseline. Per-user graph construction over 109,467 photos likewise reports no time or storage.
- The skill-induction set is small and arguably circular. It contains only 72 DISBench-style questions (3 per album, 24 YFCC albums) and 96 CamRoll-style questions, generated by Claude-Opus-4.5 from graph samples whose photo inventories are known complete (Appendix C). Synthetic queries therefore recapitulate the very anchor→scope→target structure the Album Memory Graph was built to expose, and no ablation tests transfer to human-authored DISBench queries.
- The learned artifact is never shown. The skill bank
S^(k)is contribution (2), yet its text never appears; only aggregate curves (Figure 5) are given, so a reader cannot judge whether the bank encodes transferable procedure or benchmark-specific heuristics. - Ablations are cumulative-only. Because "+ Album Memory Graph" is evaluated only after STM, the paper cannot separate "graph hurts" from "graph + STM without skills hurts"; there is no skills-without-graph row, no clue- vs. story-node ablation, and no verifier ablation.
- CamRoll evidence is thin. The authors subsample 200 questions themselves; recall gains over the strongest baseline are ≤0.5 points (90.3 vs. 89.8; 95.5 vs. 95.0), while headline differences sit in a GPT-4o Judge score (4.1–5.8 → 6.2) graded by a self-authored rubric (Appendix E.1) without human validation. Naïve RAG's recall is identical (61.4) across backbones, indicating recall is fixed by the top-10 retriever rather than the method.
Questions for authors
- Which configuration does Figure 4 sweep? At 30 turns it shows EM 19.7 / F1 38.3, matching Table 2's "+ Search Skills" row rather than the 23.0 / 41.8 dual-route system — please confirm and state this in the caption.
- What are the mean and standard deviation over ≥3 seeds for Table 1, and is the +5.0 / +8.2 EM margin distinguishable from run-to-run variance?
- What is MNEME's per-query wall-clock and token cost versus each baseline? Does a single-route agent given the combined turn budget close the gap?
- Can the final skill bank
S^(k)be included verbatim — are the induced skills generic procedures or benchmark-specific heuristics? - Would skills induced from human-authored auxiliary queries transfer as well, separating genuine procedural learning from queries sampled from the same graph used at inference?
4 comments
Pointwise Response (2/3)
Response to Q3: Compute-Matched Evaluation and Computational Costs
We thank the reviewer for raising the concern that Mneme's performance gains may partly result from additional inference-time computation. To isolate the contribution of memory-guided reconstruction, we conduct two additional controls on all 122 DISBench queries using Qwen3.5-27B.
We compare Mneme against (1) a single memory-free STM route with an extended 60-turn budget and (2) two independent memory-free STM routes with 30 turns each, followed by the same reconciliation procedure used by Mneme.
| Method | Search Budget | Reconciliation | EM | F1 |
|---|---|---|---|---|
| STM, extended budget | 1 × 60 turns | No | 18.03 | 36.92 |
| Two independent STM routes | 2 × 30 turns | Same verifier | 19.67 | 38.82 |
| Mneme | 2 × 30 turns | Same verifier | 22.95 | 41.84 |
Mneme outperforms the two-route STM control by 3.28 EM and 3.02 F1 percentage points, and the extended single-route control by 4.92 points on both metrics. Thus, neither doubling the number of memory-free search trajectories nor extending a single route to 60 turns closes the performance gap.
Inference-time computational costs. We additionally profile per-query model calls, token consumption, and latency. The results are summarized below.
| Configuration | Time / Query | Model Calls | Input Tokens | Output Tokens | Total Tokens |
|---|---|---|---|---|---|
| ImageSeeker | 395.07 s | 20.63 | 279,945 | 9,521 | 289,467 |
| STM, 30 turns | 440.78 s | 19.19 | 278,048 | 11,386 | 289,433 |
| STM, 60 turns | 700.26 s | 31.48 | 477,872 | 18,011 | 495,883 |
| Two STM routes + verifier | 550.84 s | 39.37 | 577,934 | 23,189 | 601,123 |
| Mneme: STM route | 440.78 s | 19.19 | 278,048 | 11,386 | 289,433 |
| Mneme: STM + LTM route | 508.21 s | 19.69 | 418,787 | 12,269 | 431,056 |
| Mneme: Fusion | 22.58 s | 0.99 | 20,311 | 420 | 20,731 |
| Mneme: dual-route + Fusion | 542.53 s | 39.87 | 717,146 | 24,075 | 741,221 |
Notably, compared with the two-route STM control, Mneme achieves higher retrieval performance with comparable model-call counts and estimated latency. Compared with the 60-turn single-route baseline, Mneme also achieves better retrieval performance with approximately 22.5% lower estimated latency, while consuming more tokens. These results suggest that Mneme makes more effective use of inference-time computation through complementary memory-guided and direct search, rather than relying solely on longer memory-free trajectories.
Offline graph construction and storage. We further quantify the offline cost of constructing the Album Memory Graph (AMG) across the 57 DISBench albums containing 109,467 photos.
| Metric | Value |
|---|---|
| Number of albums | 57 |
| Number of photos | 109,467 |
| Total graph and retrieval-index storage | 7.783 GiB |
| Shared photo retrieval index | 1.732 GiB |
| Incremental memory-specific storage | 6.051 GiB |
| Estimated full-corpus construction time | Approximately 73.85 hours |
Averaged across 57 users, the total storage footprint is approximately 140 MiB per user, including only 109 MiB of incremental memory-specific storage, representing a modest storage overhead. Moreover, AMG construction is an offline process that can run asynchronously in the background without affecting online retrieval. Routine maintenance can be performed incrementally as photos are added or updated, avoiding full reconstruction and keeping ongoing maintenance overhead low.
Overall, these additional experiments provide evidence that Mneme's performance gains are not explained solely by additional search trajectories or a larger turn budget. The compute-controlled comparison supports the contribution of memory-guided reconstruction, while the profiling makes its additional online and offline resource requirements explicit.
Pointwise Response (3/3)
Response to Q4: Analysis of Learned Search Skills
We thank the reviewer for emphasizing the importance of making the learned skill bank inspectable. We agree that examining the induced skills helps clarify their relationship with our Album Memory Graph (AMG) and whether they capture reusable procedures or benchmark-specific heuristics.
We view the learned skills as graph-grounded and task-specialized procedural knowledge. AMG provides structured evidence and navigation primitives, while the skills guide the agent on when and how to exploit these capabilities for different query structures.
Below, we reproduce three representative excerpts verbatim from the final skill bank:
-
Event Context Grounding
Prioritize retrieving high-level event memories or narrative stories to establish the precise temporal and spatial scope before searching for specific visual subjects.This skill leverages Story Nodes to establish event-level context, particularly for queries involving temporal or same-event constraints.
-
Anchor Metadata Extraction
First identify the reference photo and explicitly extract its precise temporal or location metadata to define a strict search scope for the remaining items.This skill exploits Photo Node metadata to refine the candidate scope based on a reference photo.
-
Semantic Exclusion Anchoring
Prioritize semantic evidence search using distinctive descriptive phrases to identify the exclusion target and anchor the location context using descriptive memory captions.This skill leverages caption-based semantic evidence to handle exclusion constraints during retrieval.
Together, these skills guide the agent to establish event context, refine search scope through photo-level anchors, and enforce semantic constraints. The remaining skill, Narrative Scope Verification, further exploits photo–story relationships to verify the contextual consistency of retrieved candidates.
Importantly, these skills are specialized to recurring challenges in personal photo retrieval, such as temporal grounding, reference-based search, and exclusion constraints. However, they do not encode album-specific facts, photo identifiers, or ground-truth answers. We therefore view them as reusable procedures transferable across albums and queries within this task domain, rather than universally applicable search policies.
For transparency and reproducibility, we will include the complete final skill bank verbatim in the Appendix of the revised manuscript.
Response to Q5: Generalization Beyond Synthetic Skill-Induction Queries
We thank the reviewer for raising this important concern regarding potential structural biases in our skill-induction process.
We first clarify that skill induction and evaluation are conducted on disjoint albums. Specifically, synthetic queries are generated from 24 auxiliary YFCC100M albums, which are disjoint from the evaluation data in terms of users, photos, questions, and answers. Although both stages follow the same AMG construction paradigm, they operate on separate album graphs. The induced skill bank is also frozen before evaluation, without access to evaluation labels or trajectories.
Importantly, our CamRoll results provide evidence of transfer from synthetic induction to human-authored evaluation queries. Unlike our synthetic induction queries, the original CamRoll evaluation questions are human-authored. Mneme achieves 90.3% and 95.5% accuracy with Qwen3-VL-32B and Qwen3.5-27B, respectively, demonstrating system-level transfer beyond the synthetic query generation setting.
Furthermore, the learned skills focus on reusable procedures for exploiting AMG capabilities, such as story-level contextual grounding, photo-level metadata refinement, and semantic evidence verification, rather than memorizing album-specific content or ground-truth answers.
Thank You for Your Positive Reassessment
Thank you very much for your thoughtful reconsideration and for raising your rating following our response. We are glad that our additional experiments and clarifications helped address your concerns, and we truly appreciate your time and constructive feedback, which have significantly helped us improve our work.
Pointwise Response (1/3)
We sincerely thank the reviewer for the thorough review and constructive feedback. We appreciate the recognition of Mneme's consistent improvements across backbones and tasks, as well as its error-containment design, synthetic-data construction, and transparent reporting. Our detailed responses are provided below.
Response to Q1: Clarification of the Configuration in Figure 4
Thank you for pointing out this ambiguity. We clarify that Figure 4 evaluates the single-route memory-guided configuration (with short-term memory, Album Memory Graph, and learned search skills), rather than the full dual-route Mneme system. We will explicitly clarify the evaluated configuration in the Figure 4 caption and the corresponding experimental analysis.
Response to Q2: Statistical Robustness
We thank the reviewer for highlighting the importance of statistical robustness, particularly given the relatively small size of DISBench. Following this suggestion, we conduct exact McNemar tests for EM and paired bootstrap tests for F1 using per-query predictions. We also evaluate the full Mneme system across three independent random seeds (101, 202, and 303).
Main comparisons (Table 1). The paired tests provide statistical support for Mneme's improvements over existing baselines. Overall EM improvements are significant against all four baselines (p ≤ 0.031), and all inter-event EM improvements are significant (p ≤ 0.039). For intra-event queries, significant EM improvements are observed against Naïve RAG and PhotoCraft, but not against ImageSeeker or CamRoll-Agent. Notably, all paired-bootstrap 95% confidence intervals for F1 improvements are strictly above zero.
Ablation significance (Table 2). We further conduct paired tests for each cumulative ablation step. Although none of the EM differences reaches statistical significance (all p ≥ 0.109), adding Short-Term Memory, Search Skills, and Dual-Route Search yields positive F1 improvements with 95% confidence intervals entirely above zero. In contrast, the effect of adding AMG alone is not statistically significant.
Run-to-run stability (Table 3). We evaluate the full Mneme system using three independent random seeds (101, 202, and 303), with all 122 DISBench queries evaluated in each run. Mneme achieves 23.22 ± 0.47 EM and 41.61 ± 1.49 F1 overall, demonstrating consistent performance across runs.
Overall, the paired tests support Mneme's improvements over the baselines, while the repeated-run results demonstrate that its overall performance is stable across independent runs. We will incorporate these statistical analyses and clarify the evaluation protocol in the revised manuscript.