acceptodds
Withdrawn from ICLR 2027

Learning to Access Multimodal Agent Memory: Agentic Retrieval with State-Level On-Policy Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-term multimodal memory enables agents to leverage temporally distributed evidence across interleaved conversations and images. Nevertheless, prevailing approaches largely rely on fixed retrieval strategies or rule-based agentic workflows, offering limited flexibility to adapt retrieval decisions as the available evidence evolves. Effective evidence acquisition requires evaluating evidence sufficiency and identifying unresolved information gaps before taking the next action. Accordingly, we propose state-level on-policy self-distillation (OPSD), in which privileged information (PI) explicitly captures these two factors. At each intermediate state of the student rollout, a verifier reconstructs these diagnoses by comparing the current short-term memory (STM) with the reference answer, providing privileged context for a frozen teacher to guide subsequent memory-access decisions. OPSD then aligns the student’s action preferences with those of the privileged teacher, enabling evidence-dependent decisions on when to terminate and what evidence to acquire next without privileged information at inference time. To support this learning formulation, we develop a unified agentic memory-access framework that formulates evidence acquisition as sequential tool use, provides heterogeneous memory-access operations, and maintains a compact STM for consolidating intermediate evidence. We further construct MemVista, a training dataset comprising long-horizon multimodal conversations with temporally evolving facts and traceable supporting evidence. Experiments on Mem-Gallery demonstrate that our approach improves the mean LLM-judge score from 0.780 to 0.852 compared with the strongest external baseline. These results indicate that state-level OPSD with dynamic PI enables more effective memory access and substantially improves performance on long-term multimodal memory reasoning tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Long-Term Memory, On-Policy Self-Distillation, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6LgYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a multimodal memory-access policy through state-level on-policy self-distillation. Reference-based sufficiency and gap diagnoses guide a privileged teacher at student-visited states. The framework combines retrieval tools, compact short-term memory, and the synthetic MemVista dataset. Evaluation on Mem-Gallery and MemEye measures answer quality, evidence recall, and retrieval cost.

Critical strengths

Supervision is reconstructed at student-visited states, allowing teacher feedback to reflect the evidence currently available. The empirical efficiency gain is substantial: on Mem-Gallery, OPSD achieves 79.74% full evidence recall with 2.89 actions, versus 78.49% with 6.03 actions for GRPO.

Critical weaknesses
  1. Sufficiency labels may encode completeness information unavailable to the student. In exhaustive image search, identical observations and search history can correspond to a complete two-image set or an undiscovered third image. The reference answer distinguishes these cases and determines whether the teacher must STOP, potentially assigning opposing stopping targets to observationally identical states.

  2. The ablations conflate better stopping with better retrieval choices. With STM retained, dynamic PI improves Mem-Gallery accuracy by 1.40 percentage points while reducing actions by 32.9%, also consistent with gains driven mainly by termination. The reference-trajectory comparison changes both feedback content and format, while final evidence recall reflects retrieval choices and stopping jointly.

  3. STM retains at most 24 events, and only the final STM reaches the answer model. Queries requiring image IDs from more than 24 independently necessary events would exceed this capacity unless information survives eviction elsewhere. Below capacity, rank fusion may still discard low-ranked corrections or bridging facts in favor of relevant but redundant events.

Questions for authors

How often do sufficiency labels depend on completeness information that cannot be inferred from the public state?

Can sufficiency-only supervision match full PI in evidence gained per non-STOP action?

Can evicted evidence survive elsewhere, and how often does STM consolidation remove necessary supporting events?

Reviewer_4MzzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes state-level On-Policy Self-Distillation (OPSD) for adaptive multimodal memory retrieval. At each student-visited state, a verifier uses the current short-term memory (STM) and reference answer to identify evidence sufficiency and unresolved information gaps. These diagnoses condition a frozen teacher, whose action preferences are distilled into the student. The authors also introduce a unified memory-access framework and the MemVista training dataset. Experiments on Mem-Gallery and MemEye demonstrate improvements in retrieval accuracy, evidence coverage, and efficiency.

Critical strengths

1.Well-motivated state-level supervision. The proposed approach adapts privileged feedback to the student's evolving evidence state rather than relying on fixed retrieval trajectories. This is a meaningful extension of OPSD to sequential memory access.

2.Promising empirical results. The method achieves competitive performance on both benchmarks, with notable improvements in retrieval efficiency. In Table 4, state-level PI reduces average actions from 4.31 to 2.89 on Mem-Gallery and from 6.95 to 5.66 on MemEye, while improving accuracy. The evidence-recall results in Table 10 further support its effectiveness.

Critical weaknesses
  1. Insufficient validation of the core contribution. The proposed state-level PI combines evidence sufficiency and information-gap supervision, but Table 4 does not isolate their contributions. It remains unclear whether the gains primarily come from better stopping decisions or more effective evidence acquisition, weakening the novelty claim.

  2. Limited generalization and robustness evidence. The retrieval policy is trained only with Qwen3.5-4B, while different answer/judge settings do not establish cross-backbone generalization. Moreover, the improvements over reference-trajectory PI are relatively modest (Table 4), without uncertainty estimates to establish their reliability

Questions for authors

1.What is the independent contribution of information-gap supervision? Could the authors separate evidence sufficiency from information-gap supervision (Eq. 5, Table 4) to clarify whether the latter improves retrieval decisions beyond stopping?

2.How reliable is the privileged supervision? Given the reference-answer-conditioned verifier and heuristic leakage controls (Appendix C), could the authors quantify diagnostic accuracy and its impact on policy learning?

3.Does the proposed OPSD generalize across student backbones? Section 4.1 uses only Qwen3.5-4B for policy training. Could the authors evaluate another student backbone rather than varying only the downstream answer model and judge?

Reviewer_2KvwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-term multimodal memory access as a sequential decision-making problem. Given a user query and an external memory containing text, images, timestamps, and metadata, an agent repeatedly selects among heterogeneous retrieval and inspection tools, accumulates the retrieved evidence in a compact short-term memory (STM), and decides when to stop and produce an answer.

The paper proposes three main components. First, it provides a unified tool interface for lexical, semantic, visual, hybrid, and metadata-based retrieval, together with context expansion and source-image inspection. Second, it maintains an STM that reranks, merges, and deduplicates intermediate evidence. Third, it trains the access policy using state-level online policy self-distillation (OPSD). At each student-visited state, a verifier compares the current STM with the reference answer and constructs privileged information describing whether the evidence is sufficient and what information is still missing. A frozen privileged teacher receives this diagnosis, and its action distribution is distilled into the student policy. The paper also introduces the synthetic MemVista training dataset and evaluates the method on Mem-Gallery and MemEye.

Critical strengths
  1. The paper provides a relatively complete implementation of a multimodal memory-access agent. The unified interface covers lexical, semantic, visual, hybrid, and metadata retrieval, and the STM provides a practical mechanism for controlling context growth. This may be useful as an engineering framework.

  2. The paper attempts to provide state-dependent supervision rather than relying only on a terminal answer reward. Asking whether the current evidence is sufficient and what remains missing is intuitively relevant to sequential retrieval. The ablations also indicate that the proposed state-level signal affects both answer quality and the number of tool calls.

Critical weaknesses
  1. The paper does not define a sufficiently specific or novel agentic-memory problem. The stated goal is to perform multi-step retrieval, adapt the next action to the accumulated evidence, and stop when the evidence is sufficient. These are already inherent objectives of agentic memory and iterative retrieval systems. The paper does not identify a sharply defined failure mode, constraint, or setting that distinguishes its task from general agentic memory access. The motivating examples show that different queries may require different tools, but this observation alone does not establish a new research problem. Without a more precise problem definition, it is difficult to assess what capability the method is specifically designed to contribute.

  2. The method is primarily a combination of existing components, and the experiments do not establish the value of this particular combination. Heterogeneous retrieval tools, compact memory or evidence-state management, iterative retrieval, and OPSD with privileged feedback are all established techniques. The paper adapts them to multimodal memory access, but it does not demonstrate a new learning principle or show that the proposed components are necessary relative to simpler alternatives. In particular, the comparison against the base agent, GRPO, and reference-trajectory privileged information is not sufficient to determine whether the gains come from the expanded tool set, the STM update procedure, the verifier-generated diagnosis, the OPSD objective, or the synthetic MemVista data. The method also lacks comparisons against stronger recent multi-turn OPSD and memory-aligned distillation approaches. Consequently, the current evidence does not establish a clear methodological contribution or practical advantage beyond an engineering integration of prior ideas.

Questions for authors
  1. What is the precise research problem introduced by this paper beyond the standard objective of agentic memory access? Please define a concrete failure mode, constraint, or capability that is not already covered by iterative retrieval, evidence accumulation, adaptive tool selection, and learned stopping. Which existing agentic-memory methods are fundamentally unable to address this problem, and why?
  2. Can the authors isolate the contribution of each major component under a fully controlled experimental setup? In particular, please compare: the original versus expanded tool set; full history versus the proposed STM; standard RL or behavior cloning versus OPSD; and generic feedback versus the proposed sufficiency-and-information-gap diagnosis. All variants should use the same memory index, training data, backbone, answer model, and interaction budget. This would clarify whether the reported gains come from the claimed learning method or from changes to the surrounding system.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.