acceptodds
Under review as a conference paper at ICLR 2027

Mneme: An Odyssey Through Reconstructive Memory for Personal Deep Image Search

Abstract

Image retrieval enables users to find relevant photos in large image collections, such as their photo albums. Common embedding-based methods rank images independently by query–image similarity, but personal album queries may identify targets through contextual clues found in other photos. An effective search must therefore recover cross-photo relations and use them to locate and verify target images. To this end, we introduce Mneme, a memory-augmented agent for reconstructive search over personal photo collections. Rather than using memory only to preserve or reuse agent search experience, Mneme reconstructs the visual history itself as a persistent relational memory. Its Album Memory Graph links grounded photo evidence, recurring visual instances, and overlapping episodic contexts, while Self-Evolving Search Skills induce and iteratively refine reusable search guidance from same-query success–failure trajectory contrasts on auxiliary tasks. At inference time, a dual-route strategy combines memory-guided reconstruction with memory-independent evidence search, and reconciles their candidates against raw images and metadata. Experiments on DISBench and CamRoll show that Mneme consistently outperforms strong retrieval and memory-augmented baselines across both tested backbones. Ablations further indicate that persistent album memory alone is insufficient; its benefits emerge when combined with reliable short-term memory, learned memory search guidance, and complementary memory-guided and direct search.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Personalized Image Retrieval, Multimodal Agents, Agent Memory, Deep Image Search

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_oderOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies "context-dependent image-set retrieval" over a personal album A = {a_i} whose photos carry visual content plus optional time and GPS, where the query Q resolves only by chaining evidence across photos (e.g., "the day the wall with drawings of fresh ingredients was seen" anchors the day; "metal chairs" specifies the target). Following DeepImageSearch (Deng et al., 2026), the task is F: (A, Q) → R̂ ⊆ A (Eq. 1), with a ReAct-style VLM policy u_t ~ π_θ(u | Q, τ_<t) (Eq. 2) and answer R̂ = G_θ(Q, τ_1:T) (Eq. 3).

MNEME contributes four components. The Album Memory Graph M_A = (V, E) with V = V_P ∪ V_C ∪ V_S (Eq. 4) stores photo nodes (raw pixels + metadata), clue nodes (recurring faces/objects, clustered with Buffalo L, YOLO26, and Qwen3-VL-Embedding, then verified by Qwen3.5; singletons dropped), and multi-scale story nodes (captions → daily/location/activity → experience memories). Seven tools expose it (Appendix B) with deliberate guards: story synopses return no Photo IDs, metadata filtering emits a "set certificate", and temporal neighbors are yielded as "anchor proposals only". Evidence-Aware Short-Term Memory keeps a bounded state M_t = (E_t, P_t, X_t) (Eq. 5) separating active evidence, search progress, and photo-exposure provenance. Self-Evolving Search Skills grow a textual bank: success is y(Q, τ) = I[EM(Q, τ) = 1] (Eq. 6), and same-query success–failure contrasts are folded in via S^(r+1) = Revise(S^(r), L^(r)) (Eq. 7), then frozen — no backbone updates, no evaluation labels. Dual-route search runs a memory-free direct route and a memory-guided reconstructive route in parallel (Eq. 8), pools both answer sets plus recently inspected candidates (Eq. 9), then closes the pool for a reconciliation verifier (Eq. 10) seeing only raw images and metadata.

Experiments use 8×A800-80GB, 30 turns per route, 4 rollouts × 3 induction generations. On DISBench (122 queries, 109,467 photos, 57 users), MNEME reaches 14.0 EM / 30.0 F1 overall with Qwen3-VL-32B-Instruct (vs. PhotoCraft 9.0/24.7) and 23.0 EM / 41.8 F1 with Qwen3.5-27B (vs. ImageSeeker 14.8/35.6), with intra-event 24.6/46.0 and inter-event 21.5/38.2 on the latter. On a 200-question stratified CamRoll subset: 90.3/79.0/93.8, Judge 5.6 (Qwen3-VL-32B) and 95.5/82.0/95.7, Judge 6.2 (Qwen3.5-27B). Table 2 shows STM helps (+3.2 EM), the Album Memory Graph alone hurts (18.0 → 14.8 EM), skills recover and exceed it (+4.9 EM), and dual-route adds +3.3 EM. Figure 5 traces skills (DISBench EM 14.8 → 18.9 → 18.9 → 19.7); Figure 4 shows the turn-budget trade-off (EM 20.5 at 40 turns → 15.6 at 50).

Critical strengths
  • Consistent wins across two backbones and two tasks. MNEME is best on every DISBench column under both backbones (e.g., inter-event EM 21.5 vs. ImageSeeker's 10.8 with Qwen3.5-27B), weakening the "one lucky backbone" objection.
  • Unusually honest negative result. Table 2 reports that adding the Album Memory Graph alone degrades performance (18.0 → 14.8 EM, 37.0 → 34.8 F1), and §5.3 attributes this to unguided memory complexity. Reporting a component that hurts is valuable and rare.
  • Fair prompt control. Appendix E.2 states MNEME reuses ImageSeeker's system prompt verbatim, so gains over ImageSeeker come from added structure rather than prompt engineering; PhotoCraft's reimplementation additionally received the same synthetic induction set (Appendix D).
  • Well-designed interface-level error containment. Story synopses withholding Photo IDs, "set certificates", and anchor-vs-target separation in the verifier target exactly the failure modes §5.4 observes (e.g., "transferring anchor-only constraints to targets").
  • Careful synthetic-data hygiene. Appendix C documents disjointness from evaluation users/questions/answers/photo IDs, double blind verification passes with shuffled orders, adjudication on disagreement, and explicit rejection criteria.
  • Transparent scope of claims. The paper states plainly that the verifier "does not establish completeness over the entire album" (§4.4) and that dual-route gains "cannot be attributed to reconciliation alone" (§5.3).
Critical weaknesses
  • No variance, seeds, or significance testing anywhere.** With 122 DISBench queries, 14.0 EM corresponds to roughly 17 exact matches, so a +5.0-point EM gain is ~6 queries; given tool nondeterminism, no Table 1 margin is interpretable without variance.
  • The Figure 4 sweep does not match the main configuration, and this is never explained. At the default 30-turn budget Figure 4 reports EM 19.7 / F1 38.3, whereas Tables 1–2 report the full dual-route system at 30 turns per route as 23.0 / 41.8. Figure 4's numbers match Table 2's "+ Search Skills" (single-route) row exactly, so the swept configuration appears to omit dual-route — but neither the caption nor §5.4 says so.
  • Compute is not matched, and construction cost is unquantified. MNEME pays for two 30-turn routes plus a verification call per pooled candidate while baselines pay for one search; cost "includes both searches" (§4.4) but no wall-clock or token figures are given, nor any equal-budget baseline. Per-user graph construction over 109,467 photos likewise reports no time or storage.
  • The skill-induction set is small and arguably circular. It contains only 72 DISBench-style questions (3 per album, 24 YFCC albums) and 96 CamRoll-style questions, generated by Claude-Opus-4.5 from graph samples whose photo inventories are known complete (Appendix C). Synthetic queries therefore recapitulate the very anchor→scope→target structure the Album Memory Graph was built to expose, and no ablation tests transfer to human-authored DISBench queries.
  • The learned artifact is never shown. The skill bank S^(k) is contribution (2), yet its text never appears; only aggregate curves (Figure 5) are given, so a reader cannot judge whether the bank encodes transferable procedure or benchmark-specific heuristics.
  • Ablations are cumulative-only. Because "+ Album Memory Graph" is evaluated only after STM, the paper cannot separate "graph hurts" from "graph + STM without skills hurts"; there is no skills-without-graph row, no clue- vs. story-node ablation, and no verifier ablation.
  • CamRoll evidence is thin. The authors subsample 200 questions themselves; recall gains over the strongest baseline are ≤0.5 points (90.3 vs. 89.8; 95.5 vs. 95.0), while headline differences sit in a GPT-4o Judge score (4.1–5.8 → 6.2) graded by a self-authored rubric (Appendix E.1) without human validation. Naïve RAG's recall is identical (61.4) across backbones, indicating recall is fixed by the top-10 retriever rather than the method.
Questions for authors
  • Which configuration does Figure 4 sweep? At 30 turns it shows EM 19.7 / F1 38.3, matching Table 2's "+ Search Skills" row rather than the 23.0 / 41.8 dual-route system — please confirm and state this in the caption.
  • What are the mean and standard deviation over ≥3 seeds for Table 1, and is the +5.0 / +8.2 EM margin distinguishable from run-to-run variance?
  • What is MNEME's per-query wall-clock and token cost versus each baseline? Does a single-route agent given the combined turn budget close the gap?
  • Can the final skill bank S^(k) be included verbatim — are the induced skills generic procedures or benchmark-specific heuristics?
  • Would skills induced from human-authored auxiliary queries transfer as well, separating genuine procedural learning from queries sampled from the same graph used at inference?
4 comments
AuthorsOpenReview ↗

Pointwise Response (1/3)

We sincerely thank the reviewer for the thorough review and constructive feedback. We appreciate the recognition of Mneme's consistent improvements across backbones and tasks, as well as its error-containment design, synthetic-data construction, and transparent reporting. Our detailed responses are provided below.

Response to Q1: Clarification of the Configuration in Figure 4

Thank you for pointing out this ambiguity. We clarify that Figure 4 evaluates the single-route memory-guided configuration (with short-term memory, Album Memory Graph, and learned search skills), rather than the full dual-route Mneme system. We will explicitly clarify the evaluated configuration in the Figure 4 caption and the corresponding experimental analysis.

Response to Q2: Statistical Robustness

We thank the reviewer for highlighting the importance of statistical robustness, particularly given the relatively small size of DISBench. Following this suggestion, we conduct exact McNemar tests for EM and paired bootstrap tests for F1 using per-query predictions. We also evaluate the full Mneme system across three independent random seeds (101, 202, and 303).

Main comparisons (Table 1). The paired tests provide statistical support for Mneme's improvements over existing baselines. Overall EM improvements are significant against all four baselines (p ≤ 0.031), and all inter-event EM improvements are significant (p ≤ 0.039). For intra-event queries, significant EM improvements are observed against Naïve RAG and PhotoCraft, but not against ImageSeeker or CamRoll-Agent. Notably, all paired-bootstrap 95% confidence intervals for F1 improvements are strictly above zero.

ComparisonIntra EM: pIntra ΔF1: 95% CIInter EM: pInter ΔF1: 95% CIOverall EM: pOverall ΔF1: 95% CI
Mneme vs. Naïve RAG0.013[+14.52, +37.84]<0.001[+12.59, +30.82]<0.001[+16.50, +31.12]
Mneme vs. ImageSeeker0.508[+1.18, +18.71]0.039[+2.51, +11.97]0.031[+0.07, +12.51]
Mneme vs. CamRoll-Agent0.375[+5.35, +17.06]<0.001[+12.24, +30.82]<0.001[+6.83, +21.44]
Mneme vs. PhotoCraft0.021[+11.96, +33.25]0.021[+2.33, +20.64]<0.001[+9.62, +23.64]

Ablation significance (Table 2). We further conduct paired tests for each cumulative ablation step. Although none of the EM differences reaches statistical significance (all p ≥ 0.109), adding Short-Term Memory, Search Skills, and Dual-Route Search yields positive F1 improvements with 95% confidence intervals entirely above zero. In contrast, the effect of adding AMG alone is not statistically significant.

Cumulative ablation stepΔEM (pp)EM: McNemar pΔF1 (pp)ΔF1: Bootstrap 95% CI
Base Agent → + Short-Term Memory+3.30.219+1.4[+1.29, +7.65]
+ Short-Term Memory → + Album Memory Graph−3.30.344−2.2[−9.05, +4.38]
+ Album Memory Graph → + Search Skills+4.90.109+3.5[+2.81, +10.26]
+ Search Skills → + Dual-Route Search+3.30.289+3.5[+2.37, +8.42]

Run-to-run stability (Table 3). We evaluate the full Mneme system using three independent random seeds (101, 202, and 303), with all 122 DISBench queries evaluated in each run. Mneme achieves 23.22 ± 0.47 EM and 41.61 ± 1.49 F1 overall, demonstrating consistent performance across runs.

Query TypeQueries per SeedEM (%) Mean ± StdF1 (%) Mean ± Std
Intra-event5723.98 ± 1.0145.98 ± 0.75
Inter-event6522.56 ± 0.8937.77 ± 2.44
Overall12223.22 ± 0.4741.61 ± 1.49

Overall, the paired tests support Mneme's improvements over the baselines, while the repeated-run results demonstrate that its overall performance is stable across independent runs. We will incorporate these statistical analyses and clarify the evaluation protocol in the revised manuscript.

AuthorsOpenReview ↗

Pointwise Response (2/3)

Response to Q3: Compute-Matched Evaluation and Computational Costs

We thank the reviewer for raising the concern that Mneme's performance gains may partly result from additional inference-time computation. To isolate the contribution of memory-guided reconstruction, we conduct two additional controls on all 122 DISBench queries using Qwen3.5-27B.

We compare Mneme against (1) a single memory-free STM route with an extended 60-turn budget and (2) two independent memory-free STM routes with 30 turns each, followed by the same reconciliation procedure used by Mneme.

MethodSearch BudgetReconciliationEMF1
STM, extended budget1 × 60 turnsNo18.0336.92
Two independent STM routes2 × 30 turnsSame verifier19.6738.82
Mneme2 × 30 turnsSame verifier22.9541.84

Mneme outperforms the two-route STM control by 3.28 EM and 3.02 F1 percentage points, and the extended single-route control by 4.92 points on both metrics. Thus, neither doubling the number of memory-free search trajectories nor extending a single route to 60 turns closes the performance gap.

Inference-time computational costs. We additionally profile per-query model calls, token consumption, and latency. The results are summarized below.

ConfigurationTime / QueryModel CallsInput TokensOutput TokensTotal Tokens
ImageSeeker395.07 s20.63279,9459,521289,467
STM, 30 turns440.78 s19.19278,04811,386289,433
STM, 60 turns700.26 s31.48477,87218,011495,883
Two STM routes + verifier550.84 s39.37577,93423,189601,123
Mneme: STM route440.78 s19.19278,04811,386289,433
Mneme: STM + LTM route508.21 s19.69418,78712,269431,056
Mneme: Fusion22.58 s0.9920,31142020,731
Mneme: dual-route + Fusion542.53 s39.87717,14624,075741,221

Notably, compared with the two-route STM control, Mneme achieves higher retrieval performance with comparable model-call counts and estimated latency. Compared with the 60-turn single-route baseline, Mneme also achieves better retrieval performance with approximately 22.5% lower estimated latency, while consuming more tokens. These results suggest that Mneme makes more effective use of inference-time computation through complementary memory-guided and direct search, rather than relying solely on longer memory-free trajectories.

Offline graph construction and storage. We further quantify the offline cost of constructing the Album Memory Graph (AMG) across the 57 DISBench albums containing 109,467 photos.

MetricValue
Number of albums57
Number of photos109,467
Total graph and retrieval-index storage7.783 GiB
Shared photo retrieval index1.732 GiB
Incremental memory-specific storage6.051 GiB
Estimated full-corpus construction timeApproximately 73.85 hours

Averaged across 57 users, the total storage footprint is approximately 140 MiB per user, including only 109 MiB of incremental memory-specific storage, representing a modest storage overhead. Moreover, AMG construction is an offline process that can run asynchronously in the background without affecting online retrieval. Routine maintenance can be performed incrementally as photos are added or updated, avoiding full reconstruction and keeping ongoing maintenance overhead low.

Overall, these additional experiments provide evidence that Mneme's performance gains are not explained solely by additional search trajectories or a larger turn budget. The compute-controlled comparison supports the contribution of memory-guided reconstruction, while the profiling makes its additional online and offline resource requirements explicit.

AuthorsOpenReview ↗

Pointwise Response (3/3)

Response to Q4: Analysis of Learned Search Skills

We thank the reviewer for emphasizing the importance of making the learned skill bank inspectable. We agree that examining the induced skills helps clarify their relationship with our Album Memory Graph (AMG) and whether they capture reusable procedures or benchmark-specific heuristics.

We view the learned skills as graph-grounded and task-specialized procedural knowledge. AMG provides structured evidence and navigation primitives, while the skills guide the agent on when and how to exploit these capabilities for different query structures.

Below, we reproduce three representative excerpts verbatim from the final skill bank:

  • Event Context Grounding

    Prioritize retrieving high-level event memories or narrative stories to establish the precise temporal and spatial scope before searching for specific visual subjects.
    

    This skill leverages Story Nodes to establish event-level context, particularly for queries involving temporal or same-event constraints.

  • Anchor Metadata Extraction

    First identify the reference photo and explicitly extract its precise temporal or location metadata to define a strict search scope for the remaining items.
    

    This skill exploits Photo Node metadata to refine the candidate scope based on a reference photo.

  • Semantic Exclusion Anchoring

    Prioritize semantic evidence search using distinctive descriptive phrases to identify the exclusion target and anchor the location context using descriptive memory captions.
    

    This skill leverages caption-based semantic evidence to handle exclusion constraints during retrieval.

Together, these skills guide the agent to establish event context, refine search scope through photo-level anchors, and enforce semantic constraints. The remaining skill, Narrative Scope Verification, further exploits photo–story relationships to verify the contextual consistency of retrieved candidates.

Importantly, these skills are specialized to recurring challenges in personal photo retrieval, such as temporal grounding, reference-based search, and exclusion constraints. However, they do not encode album-specific facts, photo identifiers, or ground-truth answers. We therefore view them as reusable procedures transferable across albums and queries within this task domain, rather than universally applicable search policies.

For transparency and reproducibility, we will include the complete final skill bank verbatim in the Appendix of the revised manuscript.

Response to Q5: Generalization Beyond Synthetic Skill-Induction Queries

We thank the reviewer for raising this important concern regarding potential structural biases in our skill-induction process.

We first clarify that skill induction and evaluation are conducted on disjoint albums. Specifically, synthetic queries are generated from 24 auxiliary YFCC100M albums, which are disjoint from the evaluation data in terms of users, photos, questions, and answers. Although both stages follow the same AMG construction paradigm, they operate on separate album graphs. The induced skill bank is also frozen before evaluation, without access to evaluation labels or trajectories.

Importantly, our CamRoll results provide evidence of transfer from synthetic induction to human-authored evaluation queries. Unlike our synthetic induction queries, the original CamRoll evaluation questions are human-authored. Mneme achieves 90.3% and 95.5% accuracy with Qwen3-VL-32B and Qwen3.5-27B, respectively, demonstrating system-level transfer beyond the synthetic query generation setting.

Furthermore, the learned skills focus on reusable procedures for exploiting AMG capabilities, such as story-level contextual grounding, photo-level metadata refinement, and semantic evidence verification, rather than memorizing album-specific content or ground-truth answers.

AuthorsOpenReview ↗

Thank You for Your Positive Reassessment

Thank you very much for your thoughtful reconsideration and for raising your rating following our response. We are glad that our additional experiments and clarifications helped address your concerns, and we truly appreciate your time and constructive feedback, which have significantly helped us improve our work.

Reviewer_KWYNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies context-dependent image-set retrieval in personal photo albums (DISBench) and camera-roll QA (CamRoll), where target photos must be found through clues that appear in other photos. MNEME has four components:

  1. Album Memory Graph (AMG) MA=(V,E)\mathcal{M}_A = (\mathcal{V}, \mathcal{E}), with V=VP∪VC∪VS\mathcal{V} = \mathcal{V}_P \cup \mathcal{V}_C \cup \mathcal{V}_S: an offline, per-album graph of photo nodes, recurring face/object clue nodes, and multi-scale story nodes, accessed through seven tools.
  2. Evidence-aware short-term memory Mt=(Et,Pt,Xt)M_t = (E_t, P_t, X_t): a query-local state for long trajectories.
  3. Self-Evolving Search Skills S(r)S^{(r)}: a textual skill bank induced from contrasts between successful and failed rollouts on synthetic questions built from 24 auxiliary YFCC100M albums.
  4. Dual-route inference: a memory-guided route and a memory-free direct route run in parallel. A VLM verifier then reconciles their candidates against raw images and metadata, R^=Vθ(Q,Rd,Rr,τ~d,τ~r,{E(p)}p∈C)\hat{R} = V_\theta(Q, R_d, R_r, \tilde{\tau}_d, \tilde{\tau}_r, \{E(p)\}_{p \in C}).

Results:

  • DISBench, Qwen3.5-27B: 23.0 EM / 41.8 F1, versus 14.8 / 35.6 for ImageSeeker (28/12228/122 vs. 18/12218/122 queries).
  • DISBench, Qwen3-VL-32B: 14.0 / 30.0, versus 9.0 / 24.7 for PhotoCraft.
  • CamRoll (200-question subset): MNEME leads on all reported metrics.
  • Cumulative ablation (EM): 14.8→18.0→14.8→19.7→23.014.8 \to 18.0 \to 14.8 \to 19.7 \to 23.0. The memory graph alone does not help; its benefit appears only when combined with short-term memory, learned skills, and the complementary direct route.
Critical strengths

1. A well-motivated design with consistent gains. The paper separates memory of album content (the AMG) from memory of the search procedure (skills), and keeps final decisions grounded in raw images and metadata through a closed candidate pool (R^⊆C\hat{R} \subseteq C). Several tool-level choices target the main failure mode of context-dependent retrieval, which is carrying anchor attributes over to targets:

  • story synopses do not expose photo IDs;
  • anchors are treated as context proposals rather than answers.

The full system outperforms all baselines under both backbones. The largest gain is on inter-event queries: 14/6514/65 vs. 7/657/65 for ImageSeeker with Qwen3.5-27B. This is the main reason I find the work promising.

2. A useful and candidly reported finding. Table 2 shows that adding the memory graph alone removes the gain from short-term memory (18.0→14.818.0 \to 14.8 EM). The authors also qualify their claims carefully:

  • the reconciliation gain is not attributed to reconciliation alone;
  • the coverage check is not album-wide;
  • the CamRoll findings are limited to the sampled subset.

The message that "richer memory does not by itself yield better search" is valuable for work on memory-augmented agents.

Critical weaknesses

1. The robustness of the main comparisons is not yet established.

  • DISBench has only n=122n = 122 queries, and each step in Table 2 changes the outcome of only 4–6 queries. For example, a gain of +4+4 queries with zero losses gives an exact McNemar p=2⋅0.54=0.125p = 2 \cdot 0.5^{4} = 0.125.
  • All results appear to come from single runs.
  • Figure 4 shows a 6-query EM change between 40 and 50 turns (25/122→19/12225/122 \to 19/122), which suggests run-to-run variation of a similar size.
  • The supplementary METHOD.md describes several intermediate fusion variants that were evaluated on DISBench. The paper does not explain how the final reconciliation design, or the number of skill generations (k=3k = 3), was chosen.

I am not assuming misuse, but clarifying the selection procedure and reporting paired significance tests would make the conclusions much more convincing.

2. The source of the gain is not fully isolated.

  • Dual-route inference adds a second full rollout plus a verifier call. Neither per-query cost nor a compute-matched control is reported, so it is hard to tell how much of the benefit comes from memory-guided reconstruction and how much from extra test-time computation.
  • In the Qwen3-VL-32B setting, the memory graph appears to be built with Qwen3.5 (Appendix A). The comparison is therefore not strictly within one backbone.

Since the paper's core claim is about reconstructive album memory, these points matter for interpreting the results.

If the authors can clarify Q1 and show that the main gains hold under paired testing and a compute-matched control (Q4–Q5), I would be inclined to raise my score.

Questions for authors

Clarifications

  1. Model selection. How were the final reconciliation design and the number of skill generations (k=3k = 3) chosen, and how do they relate to the variants described in the supplementary METHOD.md? If these choices were made on the skill-induction set or another non-test split, please state this explicitly. Figure 5 reports induction-set results only up to G2.
  2. Backbone consistency. In the Qwen3-VL-32B setting, which models were used for memory-graph construction, for skill induction, and as the final verifier VθV_\theta? Which retriever did the ImageSeeker baseline use?
  3. Setup details.
    • (a) On CamRoll, does MNEME use one route or both routes, and which skill bank?
    • (b) How often is the WebSearch tool (visible in Figure 3) used, and does it matter for the results?
    • (c) Why do the released skill-induction files differ slightly from Appendix C in per-album counts and in the intra/inter split?

Additional evidence (feasible within the rebuttal period)

  1. Paired significance. Please report per-query paired tests for the main comparison in Table 1 and for each step in Table 2, for example McNemar for EM and a paired bootstrap for F1. The existing predictions are sufficient for this. If feasible, please also report the mean ±\pm std over repeated runs of the full system.
  2. Cost and a compute-matched control. Please report per-query cost (model calls, tokens, latency) for MNEME and the baselines, together with one compute-matched control. An example is two memory-free direct routes reconciled by the same verifier, Vθ(Q,Rd(1),Rd(2),… )V_\theta(Q, R_d^{(1)}, R_d^{(2)}, \dots).

Suggestions for the final version (not required for the rebuttal)

  • Ablation: add a control without the memory graph, for example short-term memory plus skills.
  • CamRoll: evaluate on the full benchmark, and add a question-and-options-only baseline.
  • Related work: briefly discuss hierarchical and graph-based retrieval memory (RAPTOR, GraphRAG, HippoRAG) and classical photo-event clustering.
  • Ethics statement: add a short note on face clustering and on the use of external web search.
7 comments
AuthorsOpenReview ↗

Pointwise Response (1/4)

We sincerely thank the reviewer for the careful review and constructive feedback. We appreciate the positive assessment of our framework, its consistent improvements across different backbones, and the value of our reported findings. Our point-by-point responses are provided below.

Response to Q1: Model Selection and Evaluation Protocol

(1) Number of skill generations. We set k=3 as a fixed computational budget to investigate iterative skill evolution while balancing performance exploration and computational cost, rather than selecting the best-performing generation across benchmakrs. Specifically, our skill-induction set contains 72 queries, each with four independent rollouts, requiring 288 trajectories per generation and 864 trajectories across three generations, excluding the additional costs of trajectory reflection and skill revision.

Regarding Fig. 5, induction-set results end at G2 because each round of rollouts produces the next skill generation (No Skill → G1 → G2 → G3). As G3 is the final skill bank, no further induction rollouts were conducted to produce G4. Thus, G3 is evaluated only on the downstream benchmark. We will clarify this in the revision.

(2) Reconciliation design. Our reconciliation design follows Mneme's core motivation: combining complementary memory-guided and direct retrieval while grounding final decisions in raw visual evidence. It therefore merges candidates from both routes and verifies them within a closed candidate pool, rather than relying on benchmark-specific optimization.

The supplementary METHOD.md documents subsequent exploratory analyses of alternative fusion strategies, not the selection process for our reported method. These variants were not used to select the submitted configuration or produce the reported results.

AuthorsOpenReview ↗

Pointwise Response (2/4)

Response to Q4: Statistical Significance and Run-to-Run Stability

We thank the reviewer for suggesting paired statistical tests and repeated-run evaluation. Following this suggestion, we conduct exact McNemar tests for EM and paired bootstrap tests for F1 using per-query predictions. We also evaluate the full Mneme system with three independent random seeds (101, 202, and 303).

Main comparisons (Table 1). The results provide strong statistical evidence for Mneme's effectiveness. Mneme achieves significantly higher overall EM than all four baselines (p ≤ 0.031), with significant improvements on inter-event queries as well (p ≤ 0.039). For intra-event queries, the EM improvements are significant against Naïve RAG and PhotoCraft, although not against ImageSeeker and CamRoll-Agent. Notably, all paired-bootstrap 95% confidence intervals for F1 improvements are strictly above zero, demonstrating consistent F1 gains across all baselines and query types.

ComparisonIntra EM: pIntra ΔF1: 95% CIInter EM: pInter ΔF1: 95% CIOverall EM: pOverall ΔF1: 95% CI
Mneme vs. Naïve RAG0.013[+14.52, +37.84]<0.001[+12.59, +30.82]<0.001[+16.50, +31.12]
Mneme vs. ImageSeeker0.508[+1.18, +18.71]0.039[+2.51, +11.97]0.031[+0.07, +12.51]
Mneme vs. CamRoll-Agent0.375[+5.35, +17.06]<0.001[+12.24, +30.82]<0.001[+6.83, +21.44]
Mneme vs. PhotoCraft0.021[+11.96, +33.25]0.021[+2.33, +20.64]<0.001[+9.62, +23.64]

Ablation studies (Table 2). We further perform paired tests for each cumulative ablation step to assess the contribution of individual components. Although the EM differences do not reach statistical significance (all p ≥ 0.109), adding Short-Term Memory, Search Skills, and Dual-Route Search yields statistically significant F1 improvements, with all corresponding 95% confidence intervals above zero. In contrast, adding the Album Memory Graph alone does not produce a statistically significant improvement, highlighting the importance of combining structured memory with effective search skills and complementary retrieval mechanisms.

Cumulative ablation stepΔEM (pp)EM: McNemar pΔF1 (pp)ΔF1: Bootstrap 95% CI
Base Agent → + Short-Term Memory+3.30.219+1.4[+1.29, +7.65]
+ Short-Term Memory → + Album Memory Graph−3.30.344−2.2[−9.05, +4.38]
+ Album Memory Graph → + Search Skills+4.90.109+3.5[+2.81, +10.26]
+ Search Skills → + Dual-Route Search+3.30.289+3.5[+2.37, +8.42]

Run-to-run stability (Table 3). To assess reproducibility, we evaluate the full Mneme system across three independent random seeds (101, 202, and 303), with all 122 DISBench queries evaluated in each run. Mneme achieves 23.22 ± 0.47 EM and 41.61 ± 1.49 F1 overall, demonstrating consistent performance with limited variation across the three runs. The relatively small standard deviations, particularly for overall EM, further support the stability of our approach.

Query TypeQueries per SeedEM (%) Mean ± StdF1 (%) Mean ± Std
Intra-event5723.98 ± 1.0145.98 ± 0.75
Inter-event6522.56 ± 0.8937.77 ± 2.44
Overall12223.22 ± 0.4741.61 ± 1.49

Overall, these additional analyses provide complementary evidence for Mneme's effectiveness and reliability: its improvements over existing baselines are statistically significant in overall EM and consistently positive in F1, its key components yield significant F1 gains, and its performance remains stable across three independent runs. We will incorporate these statistical analyses and repeated-run results into the revised manuscript.

AuthorsOpenReview ↗

Pointwise Response (3/4)

Response to Q5: Inference Cost and Compute-Matched Control

To better isolate the contribution of Mneme's architecture, we introduce two additional controls on all 122 DISBench queries using Qwen3.5-27B.

First, we evaluate a single STM route with an extended 60-turn budget. Second, we evaluate two independent STM routes with 30 turns each, followed by the same reconciliation procedure used by Mneme. The latter preserves the dual-route execution and verification process while replacing the memory-guided route with another direct-search route.

MethodSearch BudgetReconciliationEMF1
STM, extended budget1 × 60 turnsNo18.0336.92
Two independent STM routes2 × 30 turnsSame verifier19.6738.82
Mneme2 × 30 turnsSame verifier22.9541.84

Mneme outperforms the two-route STM control by 3.28 EM and 3.02 F1 percentage points, and the extended single-route control by 4.92 points on both metrics. These results suggest that additional search turns and reconciliation alone cannot explain Mneme's performance gains.

We further profile inference time and token consumption across different configurations. The per-query averages are summarized below. Parallel execution latencies are calculated from per-query component-level timings rather than directly measured end-to-end wall-clock times.

ConfigurationTime / QueryModel CallsInput TokensOutput TokensTotal Tokens
ImageSeeker395.07 s20.63279,9459,521289,467
STM, 30 turns440.78 s19.19278,04811,386289,433
STM, 60 turns700.26 s31.48477,87218,011495,883
Two STM routes + verifier550.84 s39.37577,93423,189601,123
Mneme: STM route440.78 s19.19278,04811,386289,433
Mneme: STM + LTM route508.21 s19.69418,78712,269431,056
Mneme: Fusion22.58 s0.9920,31142020,731
Mneme: dual-route + Fusion542.53 s39.87717,14624,075741,221

Notably, compared with the two-route STM control, Mneme achieves 3.28 higher EM and 3.02 higher F1 with comparable model calls (39.87 vs. 39.37) and estimated latency (542.53 vs. 550.84 seconds per query). Although Mneme consumes 23.3% more tokens, its estimated latency is approximately 1.5% lower. Compared with the 60-turn single-route baseline, Mneme also achieves better retrieval performance with approximately 22.5% lower estimated latency, while using more tokens due to its dual-route architecture.

These results suggest that Mneme makes more effective use of inference-time computation by combining complementary memory-guided and direct search, rather than relying solely on longer memory-free trajectories. Fusion adds a relatively small amount of latency and requires approximately one multimodal model call per nonempty query-level candidate pool, rather than one call per candidate.

The additional controls provide evidence that Mneme's performance improvements are not merely attributable to a larger search-turn budget, but also reflect the benefits of memory-guided reconstruction and learned search skills.

Response to Q2: Backbone Consistency

We thank the reviewer for the careful reading and for pointing out this ambiguity.

In the Qwen3-VL-32B setting, we consistently use Qwen3-VL-32B for AMG construction, skill evolution, and the final Evidence Reconciliation Verifier. For a fair comparison, the memory representations of PhotoCraft and the CamRoll agent are also constructed using Qwen3-VL-32B.

We apologize for the ambiguity in the Appendix, where we presented the Qwen3.5-based implementation as the default Mneme configuration without explicitly distinguishing the backbone settings for the Qwen3-VL-32B experiments.

For album-level vector retrieval, both Mneme and ImageSeeker use Qwen3-VL-Embedding-8B to ensure a consistent retrieval backbone. For external WebSearch, we use Google Search through the Serper API

We will clarify these implementation details and explicitly distinguish the backbone configurations in the revised manuscript.

Reviewer_sHLhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MNEME, a memory-augmented vision-language agent for personal deep image search and camera-roll question answering aiming to resolve queries requiring connecting contextual clues across multiple photos. The proposed reconstructive search framework contains four core components: (1) an offline-constructed Album Memory Graph that links raw photo nodes with clustered face/object visual-clue nodes (2) an Evidence-Aware Short-Term Memory that separates active evidence reasoning from deterministic search-progress and photo-exposure provenance records (3) a frozen textual bank of Self-Evolving Search Skills distilled offline via within-generation and cross-generation success–failure trajectory on synthetic queries on auxiliary albums (4) a Dual-Route Reconstructive Search architecture at inference time that executes memory-guided and direct retrieval routes in parallel before reconciling pooled candidates against raw images and metadata. Experiments across two VLM backbones (Qwen3-VL-32B-Instruct and Qwen3.5-27B) on DISBench and a 200-question stratified sample of CamRoll demonstrate consistent improvements over existing retrieval and agentic memory baselines. Ablation studies show that the relational album graph is effective only when paired with learned navigation skills and complementary verification.

Critical strengths
  • I found the problem the paper aiming to address practically very meaningful. Using llm instead of embedding based method can incorporate more semantic understanding to the search process. This can facilitate the understanding of increasing complex user queries.
Critical weaknesses
  • Unmatched inference-time compute budgets: At test time, MNEME executes two parallel 30-turn trajectories plus a multimodal Evidence Reconciliation Verifier pass. This is more than doubling the online reasoning and visual inspection budget compared to the single-trajectory baselines. Please consider reporting quantitative offline/online compute metrics, evaluating baselines under matched two-rollout or 60-turn budgets, and adding non-cumulative ablations.
  • Limited base VLM/LLM models: The paper mainly evaluated their method on two Qwen models. Since the proposed method is main harness / prompt / workflow based agent design. Including more open and closed source VLM/LLM model can provide a more comprehensive experiment results.
  • Missing important hyperparameters and prompts: candidate-pool parameter, trace compression format, and the prompts for the verifier, skill reflection, and induced skill bank are omitted from the paper.
Questions for authors

The above mentioned three weakness are my main questions to the authors including improved inference-time compute budget control, addtional based VLM/LLM model, and clarification of missing parameters and prompts.

3 comments
AuthorsOpenReview ↗

Pointwise Response (1/3)

We sincerely thank the reviewer for the time and effort devoted to reviewing our work and for providing valuable feedback. We appreciate the recognition of the practical significance of our problem and the potential of VLM-based approaches for handling complex personal image search queries. Our detailed responses are provided below.

Response to Q1: Inference-Time Compute Budget and Computational Cost Analysis

We thank the reviewer for highlighting the potential confounding effect of additional inference-time computation. We agree that the original comparison does not fully isolate the benefits of Mneme's memory-guided reconstruction from the additional search budget. We therefore introduce compute-controlled experiments and provide a quantitative analysis of online inference costs.

(1) Search-budget-controlled evaluation. We conduct two additional experiments on all 122 DISBench queries using Qwen3.5-27B: (i) a single memory-free STM route with an extended 60-turn budget, and (ii) two independent memory-free STM routes with 30 turns each, followed by the same reconciliation procedure used by Mneme. The second control preserves the dual-route execution and verification stages while replacing the memory-guided route with another direct-search route.

MethodSearch BudgetReconciliationEMF1
STM, extended budget1 × 60 turnsNo18.0336.92
Two independent STM routes2 × 30 turnsSame verifier19.6738.82
Mneme2 × 30 turnsSame verifier22.9541.84

Mneme outperforms the two-route STM control by 3.28 EM and 3.02 F1 percentage points, and the extended single-route control by 4.92 points on both metrics. These results suggest that additional search trajectories and a larger single-route budget alone cannot explain Mneme's performance gains.

(2) Online inference costs. We further profile per-query model calls, token consumption, and latency across the baselines and different Mneme configurations.

ConfigurationTime / QueryModel CallsInput TokensOutput TokensTotal Tokens
ImageSeeker395.07 s20.63279,9459,521289,467
STM, 30 turns440.78 s19.19278,04811,386289,433
STM, 60 turns700.26 s31.48477,87218,011495,883
Two STM routes + verifier550.84 s39.37577,93423,189601,123
Mneme: STM route440.78 s19.19278,04811,386289,433
Mneme: STM + LTM route508.21 s19.69418,78712,269431,056
Mneme: Fusion22.58 s0.9920,31142020,731
Mneme: dual-route + Fusion542.53 s39.87717,14624,075741,221

Notably, compared with the two-route STM control, Mneme achieves higher retrieval performance with comparable model-call counts (39.87 vs. 39.37) and estimated latency (542.53 vs. 550.84 seconds per query), despite consuming 23.3% more tokens. Compared with the 60-turn single-route baseline, Mneme also achieves better retrieval performance with approximately 22.5% lower latency. These results indicate a favorable performance-cost trade-off, although Mneme incur additional token consumption.

Fusion itself introduces relatively modest overhead compared with executing another complete search route, averaging 0.99 model calls and 22.58 seconds per query.

Overall, these additional experiments provide evidence that Mneme's improvements are not merely a consequence of a larger search budget. The compute-controlled comparison demonstrates the effectiveness of combining memory-guided reconstruction with direct search, while the cost analysis makes the associated inference-time overhead explicit.

AuthorsOpenReview ↗

Pointwise Response (2/3)

Response to Q2: Generalization Across Different VLM Backbones

We thank the reviewer for pointing out the limited diversity of the evaluated VLM backbones. We agree that evaluating models from different families, parameter scales, and access paradigms is important for assessing the generality of our agent framework.

To address this concern, we extend our evaluation with two additional VLMs: Gemini-3-Flash-Preview, a proprietary closed-source model, and GLM-4.6V-Flash (9B), an open-weight model. Together with the two existing Qwen backbones (27B and 32B), these additions broaden our evaluation across both open-weight and proprietary models, as well as different model scales. We evaluate both Mneme and the corresponding ImageSeeker baseline using the same backbone on DISBench.

The results are summarized below:

BackboneMethodIntra EMIntra F1Inter EMInter F1Overall EMOverall F1
Qwen3-VL-32B-InstructImageSeeker10.523.06.217.38.220.0
Qwen3-VL-32B-InstructMneme17.530.710.829.514.030.0
Qwen3.5-27BImageSeeker19.336.210.835.214.835.6
Qwen3.5-27BMneme24.646.021.538.223.041.8
Gemini-3-Flash-PreviewImageSeeker15.842.49.231.912.336.8
Gemini-3-Flash-PreviewMneme22.849.620.038.221.343.5
GLM-4.6V-Flash (9B)ImageSeeker7.018.17.716.77.417.3
GLM-4.6V-Flash (9B)Mneme8.819.59.217.89.018.6

The results show consistent improvements across all four backbones and all reported metrics, despite substantial differences in model families and capabilities. Notably, Mneme achieves particularly strong gains with Gemini-3-Flash-Preview, improving overall EM and F1 by 9.0 and 6.7 percentage points, respectively. The gains are especially pronounced on inter-event queries, with a 10.8-point improvement in EM. Meanwhile, even with the smaller 9B GLM-4.6V-Flash, Mneme consistently outperforms ImageSeeker, improving overall EM and F1 by 1.6 and 1.3 percentage points, respectively.

These results demonstrate that Mneme's improvements are not limited to the Qwen model family or a particular model scale. Its effectiveness extends to both open-weight and proprietary VLMs, including a relatively small 9B model, while achieving substantial gains with Gemini. This provides additional evidence for the generality and backbone-agnostic nature of our memory-augmented agent framework.

AuthorsOpenReview ↗

Pointwise Response (3/3)

Response to Q3: Clarification of Hyperparameters and Prompt Configurations

We thank the reviewer for pointing out these omissions. We agree that the candidate-pool construction, trajectory representation, and prompt configurations should be documented more explicitly to ensure reproducibility. We clarify the implementation details below and will provide the complete prompts, frozen skill banks, and implementation code in the revised manuscript and supplementary material.

Candidate pool and trajectory representation. The reported single-pass reconciliation configuration uses a maximum candidate-pool size of 64. The pool combines answer hypotheses from both routes with trace-derived candidates, with duplicate photo IDs removed. The verifier receives the original query, anonymized hypotheses (A/B), compact execution traces, and candidate images with available camera metadata.

The trace representation is deterministically extracted from observable execution records, rather than generated as an additional LLM summary. It preserves tool names and arguments, referenced and returned photo IDs, execution status, and short result excerpts. Specifically, we retain up to 20 tool-call records (the first and last 10 when necessary), cap result excerpts at 700 characters, and limit the final-statement field to 2,400 characters. Embedded image payloads are omitted from textual traces, as candidate images are provided separately to the verifier.

The main prompts and learned artifacts serve the following purposes:

Prompt / ArtifactContent and Purpose
Evidence Reconciliation PromptSeparates anchor/context constraints from target-photo constraints and evaluates the hypotheses in terms of anchor validity, target validity, and completeness within the candidate pool. Direct visual evidence and exact metadata take precedence over trace claims. The verifier can select A, B, their union, a custom set, or an empty set, using only supplied photo IDs.
Skill Consolidation PromptConsolidates lessons by underlying reasoning behavior, merges redundant guidance, and produces a compact natural-language skill bank emphasizing transferable, conditional search policies rather than rigid tool-call sequences.
Skill Revision PromptUpdates the existing skill bank using new contrastive lessons, preserving supported rules, narrowing overly broad triggers associated with regressions, and introducing new guidance only when supported by behavioral evidence.
Induced Skill Bank and InjectionThe learned artifact consists of Markdown instructions describing applicable situations, search strategies, and verification conditions. The final bank covers event-scope grounding, semantic-caption filtering, temporal-anchor verification, and visual identity/exclusion checks. It is injected into the system prompt as conditional guidance and does not override the original query or direct evidence.

For transparency and reproducibility, we will include the complete Evidence Reconciliation, Skill Consolidation, and Skill Revision prompts, and the exact frozen skill banks used in each reported evaluation configuration in the revised manuscript's Appendix. We also provide the implementation code in the supplementary material, allowing readers to inspect the implementation and reproduce the proposed workflow.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response for Mneme

We sincerely thank all reviewers for their thoughtful reviews, constructive feedback, and valuable suggestions. We are encouraged by the reviewers' recognition of the practical significance of personal deep image search, the motivation and design of Mneme, its consistent improvements across different backbones and benchmarks, and the value of its carefully designed memory and search mechanisms. We particularly appreciate the recognition of our transparent reporting of findings and the complementary roles of structured album memory, learned search skills, and dual-route retrieval.

The reviewers' primary concerns centered on several key aspects of our work. To address these concerns, we conducted additional controlled experiments, statistical analyses, and ablation studies, and provided further implementation details and clarifications. The following table summarizes the main areas of concern, our corresponding actions, and the key findings from our additional experiments.

Focus AreaReviewersActions and Key Findings
1. Statistical robustness and reproducibilityKWYN, oderConducted paired McNemar and bootstrap tests, demonstrating significant overall EM improvements over all four baselines and positive F1 improvements across all comparisons. Three-seed evaluation further confirms stable performance, achieving 23.22 ± 0.47 EM and 41.61 ± 1.49 F1.
2. Compute-matched evaluation and inference costsHLh, KWYN, oderIntroduced compute-matched memory-free controls. Mneme outperforms the two-route STM control by 3.28 EM and 3.02 F1 points, with comparable model calls and estimated latency. It also surpasses the 60-turn single-route baseline by 4.92 points on both metrics, demonstrating that its gains are not merely attributable to additional search computation.
3. Backbone generalization and consistencysHLh, KWYNExtended evaluation to Gemini-3-Flash-Preview and GLM-4.6V-Flash. Mneme consistently outperforms ImageSeeker across all four backbones and all reported metrics, with particularly substantial gains on Gemini (+9.0 EM / +6.7 F1 points), demonstrating its generality across model families and scales.
4. Component contributions and ablation analysisKWYN, oderAdded AMG and non-AMG ablations across different skill generations. Results show that skill evolution consistently improves search performance, while AMG provides a higher performance ceiling when combined with evolved skills. Paired tests further confirm significant F1 improvements from learned skills and dual-route search.
5. Learned skill analysis and generalizationKWYN, oderProvided representative excerpts from the final skill bank, showing reusable procedures for contextual grounding, metadata extraction, and semantic exclusion. Clarified the separation between auxiliary induction and evaluation data. Results on human-authored CamRoll queries provide additional evidence of transfer beyond synthetic induction queries.
6. Implementation details and experimental transparencysHLh, KWYN, oderClarified candidate-pool construction, deterministic trajectory compression, prompt configurations, skill-generation protocol, CamRoll settings, WebSearch usage, and auxiliary dataset statistics. Specified key hyperparameters and provided additional implementation details to improve reproducibility.
7. Experimental setup and manuscript clarificationsKWYN, oderClarified the single-route configuration evaluated in Figure 4, the computational motivation for fixing three skill generations, and the distinction between exploratory fusion variants and the reported configuration. Will further expand related work and the ethics discussion, and pursue full CamRoll evaluation and an additional question-and-options-only baseline in the final version.

We believe these additional experiments and analyses substantially strengthen the empirical evidence and reproducibility of Mneme. Importantly, our results show that learned search skills and dual-route search enable the agent to fully exploit Album Memory Graph's structured contextual evidence, unlocking its potential and leading to substantially improved retrieval performance.

We sincerely thank all reviewers and the Area Chair for their time and constructive feedback. We hope our responses adequately address the concerns and provide further evidence of the effectiveness and generality of Mneme. We look forward to continuing the discussion.

Best regards,

The Authors of Mneme

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.