acceptodds
Under review as a conference paper at ICLR 2027

Task-Focused Memorization for Multimodal Agents

Abstract

Long-term memory is essential for multimodal agents to build coherent experience, accumulate world knowledge, and achieve continual learning. Beyond module design and basic fidelity, the key challenge lies in determining what to memorize. Multimodal agents, such as embodied agents, continuously perceive, reason, and act in real or virtual environments, receiving an unbounded stream of multimodal observations. From this combinatorial explosion of information, an agent must selectively retain content that is relevant to its role in the environment and valuable for future tasks. To this end, we frame memory generation as a learnable memorization policy and introduce TaskMem (Task-focused Memorization Policy Learning), a reinforcement-learning-based framework that enables the policy to dynamically adjust its focus to the demands of real tasks encountered in the environment. TaskMem adopts a two-phase training paradigm: Phase One learns how to memorize by optimizing memory quality under fundamental fidelity requirements; Phase Two occurs after deployment, where the agent learns what to memorize by tuning an adapter on its base MLLM, using recent environment tasks to define a reward model that guides the memorization policy toward task-relevant content. To evaluate our approach, we reformulate VideoMME, EgoLife, and EgoTempo into streaming benchmarks that simulate a realistic setting in which an agent processes streaming observations and handles tasks arriving online. Questions must be answered from memory alone, without access to raw video. Built on Qwen3-VL-30B-A3B, TaskMem improves VQA accuracy by 6.3%, 7.0%, and 5.8% on these benchmarks, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MultiModal Language Model, long-term memory, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wv4GOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TaskMem, which make memory generation for multimodal agents as a learnable memorization policy. The approach consists of two phases: Phase one employs reinforcement learning to generate high-quality episodic memory, phase two fine-tunes a lightweight adapter using pairwise preferences and DPO based on recently observed tasks, which aligns generated memories more closely with task requirements. Experiments demonstrate consistent performance improvements on the VideoMME, EgoLife, and EgoTempo benchmarks.

Critical strengths
  • This paper takes a fascinating approach by focusing on how to determine "what to remember" based on the type of current task.
  • The two-stage design is clear. Phase Two updates only the 2048-parameter adapter, making the method lightweight and feasible for actual deployment.
Critical weaknesses
  • The evaluation setup may be overly favorable to TaskMem. Questions are grouped by task type, and the first ten questions of each type are used for adaptation, followed by evaluation on later questions from the same type. This creates a highly stationary setting and leaves the method’s robustness to mixed or changing task distributions unclear.
  • The baseline comparison is not entirely fair. TaskMem employs Qwen3-VL-30B-A3B as its backbone and generates streaming memory using 10-second clips, whereas baselines such as HippoMM utilize Qwen2.5-VL-7B-Instruct and process videos using 30-second non-overlapping clips.
Questions for authors
  • Does TaskMem remain effective when there are sudden changes in task types (mixed tasks) or task distribution?
  • Is it possible to re-evaluate the baselines using the same backbone?
  • How much time and computational power does a single online update in Phase Two actually require?

I would be open to raising my score if this concern can be adequately addressed during rebuttal. It can better enhance the practical deployment value of the paper.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wv4G

We thank the reviewers for the constructive comments. In the revised manuscript, we have (1) added cross-task adaptation experiments showing that matched adapters improve accuracy while unmatched adapters cause no degradation, (2) re-run HippoMM with the same Qwen3-30B-A3B backbone and confirmed that TaskMem still outperforms it, and (3) reported the full cost of Phase-Two update. Details and discussion are provided below.

Q1: The evaluation setup may be overly favorable to TaskMem. Questions are grouped by task type, and the first ten questions of each type are used for adaptation, followed by evaluation on later questions from the same type. This creates a highly stationary setting and leaves the method’s robustness to mixed or changing task distributions unclear.

A1: Thank you for raising this important concern. We would like to clarify that TaskMem is designed to support repeated adaptation to mixed or changing task distributions, rather than a single adaptation to one task category. In real-world streaming settings, the memorization policy can be updated periodically (e.g., at fixed time intervals or after a certain number of interactions). At each update, TaskMem uses the most recent k tasks, or samples k tasks from a recent window, to adapt the policy to the current task distribution.

While evaluating long-horizon sequential adaptation is computationally expensive, our experiments provide evidence for TaskMem's behavior under two key scenarios:

  1. When the task distribution remains stable: As shown in Table 3, the task-adapted policy outperforms the original Phase One policy, demonstrating the benefits of adaptation to recent task feedback.

  2. When the task distribution changes: Before the next update, the policy may encounter tasks that differ from those used for adaptation. Table 4 shows that adapters trained on different task categories achieve performance comparable to the Phase One policy on unmatched tasks.

To further validate this observation, we conduct additional experiments on two evaluation tasks, each tested with three randomly selected task vectors from other task categories, providing stronger evidence that task-specific adaptation does not compromise performance on unseen task categories. Once the next update is triggered, TaskMem can adapt again using recent tasks to align with the new distribution.

Together, these results provide evidence for the benefits of task-specific adaptation and robustness to task distribution shifts, supporting TaskMem's potential to handle evolving task demands. We have revised the paper to include these additional results and discussion.

Evaluation Task 1: Counting problem

MethodAcc.Cov.Prec.
Phase One (no adapter)35.060.058.3
w/ Information Synopsis adapter37.362.359.8
w/ OCR Problems adapter36.865.955.9
w/ Temporal Reasoning adapter37.358.663.6
w/ Counting problem adapter (matched)43.266.864.6

Evaluation Task 2: Action Reasoning

MethodAcc.Cov.Prec.
Phase One (no adapter)64.881.080.0
w/ TaskMaster adapter65.281.580.0
w/ HabitInsight adapter65.779.183.1
w/ Action Recognition adapter67.181.082.9
w/ Action Reasoning adapter (matched)69.582.983.9

Q2: The baseline comparison is not entirely fair. TaskMem employs Qwen3-VL-30B-A3B as its backbone and generates streaming memory using 10-second clips, whereas baselines such as HippoMM utilize Qwen2.5-VL-7B-Instruct and process videos using 30-second non-overlapping clips.

A2: Thank you for your valuable comment. To address the concern about backbone differences, we conducted additional experiments using Qwen3-30B-A3B as the backbone for HippoMM with 30-second clips. The results show that the stronger backbone improves HippoMM's performance over its original Qwen2.5-VL-7B. TaskMem still outperforms HippoMM when using the same backbone, demonstrating the effectiveness of our proposed method. We will include these results in the revised paper.

ModelVideoMME Acc.VideoMME Cov.VideoMME Prec.EgoLife Acc.EgoLife Cov.EgoLife Prec.EgoTempo Acc.EgoTempo Cov.EgoTempo Prec.
HippoMM(Qwen2.5-7B)48.966.673.530.443.470.015.830.851.1
HippoMM(Qwen3-30B-A3B)65.478.883.032.050.962.821.342.150.7
TaskMem67.979.385.645.456.480.528.344.563.6
AuthorsOpenReview ↗

Response to Reviewer wv4G

Q3: Does TaskMem remain effective when there are sudden changes in task types (mixed tasks) or task distribution?

A3: Please refer to A1. The results in Table 4 and our supplementary experiments show that TaskMem remains robust to sudden changes in task types or distributions, achieving performance at least comparable to the general Phase One policy. Once the next update is triggered, TaskMem adapts to the new distribution by incorporating recent tasks.

Q4: Is it possible to re-evaluate the baselines using the same backbone?

A4: Please refer to A2.

Q5: How much time and computational power does a single online update in Phase Two actually require?

A5: We pre-collect video clips and sample n=8 candidate memories per clip offline. Once the update triggered, we construct preference pairs by selecting the highest- and lowest- ranked memories based on their relevance to the current tasks. This takes approximately 5 minutes per clip, but can be parallelized across clips to construct 640 preference pairs. We then perform 10-step DPO training on the 2048-parameter adapter with the backbone frozen, using 32 GPUs (80GB memory) and a batch size of 64. With the backbone already loaded, each training step takes approximately 110 seconds. Notably, these measurements were obtained without engineering optimization, leaving room for further efficiency improvements.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns and I am happy to raise my score.

Thank you for your careful assessment and constructive feedback. The additional experiments and clarifications are already included in the revised manuscript. We sincerely appreciate your time and effort in reviewing our work.

Reviewer_8Nf3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper presents TaskMem, which learns a video-to-memory policy in 2 phases: GSPO improves memory fidelity, then task-conditioned DPO training on preferences over cached memories. Evaluation on VideoMME, EgoLife, and EgoTempo into task-specific streams, with questions answered from memory alone. Accuracy improves over the base model by some points. Phase ablations, adapter swaps, and an alternative answer generator provide supporting evidence.

Critical strengths
  1. the idea and formulation is straightforward and informative, the whole paradigm seems to be effective for task-dependent memory construction
  2. The downstream eval result is comprehensive and supports the design of the system, which is encouraging
Critical weaknesses
  1. the biggest concern is about the generalization of the method, the evaluation only groups one known question type, resets to the Phase 1 policy, and updates once after ten questions. This can establish adaptation to one task category, but not multiple other adaptation, retention, or recovery when task demands change.
  2. the latest model the paper compares with is GPT-5.2, which doesn't establish the sota comparison, I'm curious to see the gap between the trained model and the SoTA models on these tasks
Questions for authors

n/a

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8Nf3

We sincerely appreciate your valuable and insightful feedback. We have carefully addressed each of your concerns to the best of our ability.

Q1: the biggest concern is about the generalization of the method, the evaluation only groups one known question type, resets to the Phase 1 policy, and updates once after ten questions. This can establish adaptation to one task category, but not multiple other adaptation, retention, or recovery when task demands change.

A1: Thank you for raising this important concern. We would like to clarify that TaskMem is designed to support repeated adaptation to evolving task distributions, rather than a single adaptation to one task category. In real-world streaming settings, the memorization policy can be updated periodically (e.g., at fixed time intervals or after a certain number of interactions). At each update, TaskMem uses the most recent k tasks, or samples k tasks from a recent window, to adapt the policy to the current task distribution.

While evaluating long-horizon sequential adaptation is computationally expensive, our experiments provide evidence for TaskMem's behavior under two key scenarios:

  1. When the task distribution remains stable: As shown in Table 3, the task-adapted policy outperforms the original Phase One policy, demonstrating the benefits of adaptation to recent task feedback.

  2. When the task distribution changes: Before the next update, the policy may encounter tasks that differ from those used for adaptation. Table 4 shows that adapters trained on different task categories achieve performance comparable to the Phase One policy on unmatched tasks.

To further validate this observation, we conduct additional experiments on two evaluation tasks, each tested with three randomly selected task vectors from other task categories, providing stronger evidence that task-specific adaptation does not compromise performance on unseen task categories. Once the next update is triggered, TaskMem can adapt again using recent tasks to align with the new distribution.

Together, these results provide evidence for the benefits of task-specific adaptation and robustness to task distribution shifts, supporting TaskMem's potential to handle evolving task demands. We have revised the paper to include these additional results and discussion.

Evaluation Task 1: Counting problem

MethodAcc.Cov.Prec.
Phase One (no adapter)35.060.058.3
w/ Information Synopsis adapter37.362.359.8
w/ OCR Problems adapter36.865.955.9
w/ Temporal Reasoning adapter37.358.663.6
w/ Counting problem adapter (matched)43.266.864.6

Evaluation Task 2: Action Reasoning

MethodAcc.Cov.Prec.
Phase One (no adapter)64.881.080.0
w/ TaskMaster adapter65.281.580.0
w/ HabitInsight adapter65.779.183.1
w/ Action Recognition adapter67.181.082.9
w/ Action Reasoning adapter (matched)69.582.983.9

Q2: the latest model the paper compares with is GPT-5.2, which doesn't establish the sota comparison, I'm curious to see the gap between the trained model and the SoTA models on these tasks

A2: Thanks for your question. We would like to clarify that our primary contribution is a method for improving memory utility through adaptation at deployment time.

We evaluate our approach using Qwen3-30B-A3B, a relatively small open-weight model compared to frontier proprietary models. Our experiments include GPT-5.2, one of the strongest foundation models available at the time of evaluation. Despite this differences in model scale, our method achieves stronger performance on VideoMME and EgoLife, highlighting the effectiveness of our approach.

Dear Reviewer 8Nf3,

We sincerely appreciate the time and effort you have invested in reviewing our work. As the deadline of the discussion phase approaches, we would like to kindly remind you of our rebuttal, which addresses each concern you raised in your feedback.

If you have any further questions or concerns, we would be happy and eager to address them promptly in the discussion period.

Thank you once again for your valuable feedback.

I would like to thank the reviwers for the reply, and providing the extra experimental results, which partially addressed my concerns in W1. However, I'm still not convinced that a stronger model (like gpt-6 sol, gemini 3.8 flash) as even a mere comparison contradicts the main idea of the paper. I thus will maintain my score of weak rejection.

We thank you for acknowledging that our rebuttal have addressed your W1 concerns.

Regarding W2 concern, we would like to further clarify that the primary goal of our work is not to establish SOTA performance over the strongest available proprietary models, but rather to propose a novel, model-agnostic framework, TaskMem, that improves memory utility by enabling adaptation to the environment during deployment. TaskMem is not tied to any specific foundation model or backbone, and it is designed to be applicable to different models.

We agree with you that comparisons with newer and stronger proprietary models can provide additional context for evaluating our approach, and such comparisons do not contradict the main idea of the paper. Our original evaluation with GPT-5.2 was intended to provide such reference. Following your suggestion, we have conducted additional experiments with Gemin-3.8-flash and GPT-6-astra. Given the limited time remaining in the rebuttal period, we were only able to evaluate these models on EgoLife and EgoTempo. We will incorporate results of VideoMME in the revised paper.

EgoLife

Acc.Cov.Prec.
Gemini-3.8-flash4351.683.3
GPT-6-Astra45.860.276.1
TaskMem45.456.480.5

EgoTempo

Acc.Cov.Prec.
Gemini-3.8-flash25.543.159.2
GPT-6-Astra42.766.364.4
TaskMem28.344.563.6

While newer foundation models may achieve stronger overall performance, this does not undermine our central finding: TaskMem provides a general memory adaptation framework that can enhance the memory utility, rather than relying on the capabilities of a particular backbone.

We sincerely appreciate your constructive feedback.

Reviewer_fWdkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TaskMem, a two-phase framework for learning what a multimodal agent should retain in episodic memory. Phase One trains a Qwen3-VL-30B-A3B memorization policy using GSPO with format, reasoning-length, quality, and richness rewards. Phase Two uses recent task questions to construct preferences over cached candidate memories, then applies DPO to a 2,048-parameter additive vector at one transformer layer while freezing the backbone.

Critical strengths
  1. The method separates two meaningful objectives and connects task feedback to memory generation. Basic fidelity and future usefulness are different requirements, and the two-phase design makes that distinction explicit. The Phase One analysis shows why optimizing quality alone can favor short, uninformative memories, while Phase Two converts a small set of task questions into preferences over cached candidates.
  2. Table 3 reports improvements from both phases. Table 4 shows that the matched object-recognition adapter improves object-recognition accuracy from 65.7% to 69.7%, whereas adapters trained for other tasks do not. Appendix G.2 also changes the answer generator while keeping memories fixed: the VideoMME gain over the base memorizer remains 5.6 points with Gemini-2.5-Pro, compared with 6.3 points with GPT-4o. Together with evaluation on three datasets, these analyses provide substantive positive evidence.
Critical weaknesses
  1. The benefit of Phase Two over task prompting could be isolated more clearly. In Table 3, the task-prompt baseline uses the original Qwen3-VL model, whereas TaskMem benefits from both Phase One training and Phase Two adaptation. A comparison between Phase One plus the same recent-task prompt and Phase One plus the learned adapter, under matched memory budgets and preprocessing, would better establish the additional value of parameter adaptation. The current phase ablations and task-matched adapter results provide useful positive evidence, but do not fully resolve this comparison.
  2. The EgoTempo scoring and metric aggregation need clarification. Appendix D.2 defines count-based metrics that imply accuracy = coverage × precision when computed over the same evaluated set. However, the reported TaskMem values of 44.1% coverage and 65.3% precision imply approximately 28.8% accuracy, rather than the reported 28.1%.
Questions for authors
  1. Could you compare Phase One plus the same recent-task prompt against Phase One plus the learned adapter, using identical feedback questions, memory budgets, and preprocessing?
  2. How are free-form answers and abstentions scored, and how are the metrics aggregated? Please explain how the reported accuracy/coverage/precision values of 28.1/44.1/65.3 relate to the definitions in Appendix D.2.
  3. Is adaptation performed only once after the first ten questions, or repeatedly within a stream? Please clarify whether the demonstrated setting assumes a stationary task distribution.
  4. Please clarify the source of the Phase Two rollout cache and its separation from evaluation videos, whether the main results use the 40-step or scaled 10-step adapter, and the end-to-end time and cost per update, including preference construction.
2 comments
AuthorsOpenReview ↗

Response to Reviewer fWdk

We sincerely appreciate your valuable and insightful feedback. We also appreciate your recognition of our contributions to meaningful two-phase memory design and substantial experimental results. We have carefully addressed each of your concerns to the best of our ability.

Q1: The benefit of Phase Two over task prompting could be isolated more clearly. In Table 3, the task-prompt baseline uses the original Qwen3-VL model, whereas TaskMem benefits from both Phase One training and Phase Two adaptation. A comparison between Phase One plus the same recent-task prompt and Phase One plus the learned adapter, under matched memory budgets and preprocessing, would better establish the additional value of parameter adaptation. The current phase ablations and task-matched adapter results provide useful positive evidence, but do not fully resolve this comparison.

A1: Thank you for this helpful suggestion. Following your suggestion, we conducted an additional experiment comparing Phase One with the same recent-task prompt against Phase One with the learned adapter (i.e., the full TaskMem framework).

The results show that adding the task prompt improves accuracy over Phase One alone across all three benchmarks. However, it still underperforms TaskMem with the learned adapter. These results further demonstrate the benefit of parameter adaptation over task prompting alone. We have added the new comparison to the revised paper.

MethodVideoMME Acc.VideoMME Cov.VideoMME Rec.EgoLife Acc.EgoLife Cov.EgoLife Rec.EgoTempo Acc.EgoTempo Cov.EgoTempo Rec.
Base Model (Qwen3-VL-30B-A3B)61.674.782.538.452.473.322.338.957.3
Base + TaskPrompt64.277.383.140.051.078.424.542.957.1
Phase One64.475.485.439.651.077.623.737.563.2
Phase One + TaskPrompt65.477.184.840.451.478.625.941.462.6
TaskMem67.979.385.645.456.480.528.344.563.6

Q2: The EgoTempo scoring and metric aggregation need clarification. Appendix D.2 defines count-based metrics that imply accuracy = coverage × precision when computed over the same evaluated set. However, the reported TaskMem values of 44.1% coverage and 65.3% precision imply approximately 28.8% accuracy, rather than the reported 28.1%.

A2: Thank you for pointing this out. Your understanding is correct: accuracy should equal coverage ×\times precision. The revised TaskMem results on EgoTempo are: Accuracy = 28.3%, Coverage = 44.5%, Precision = 63.6%. We have updated the corresponding results in the revised paper.

Q3: Could you compare Phase One plus the same recent-task prompt against Phase One plus the learned adapter, using identical feedback questions, memory budgets, and preprocessing?

A3: Please refer to A1.

Q4: How are free-form answers and abstentions scored, and how are the metrics aggregated? Please explain how the reported accuracy/coverage/precision values of 28.1/44.1/65.3 relate to the definitions in Appendix D.2.

A4: Please refer to A2.

AuthorsOpenReview ↗

Response to Reviewer fWdk

Q5: Is adaptation performed only once after the first ten questions, or repeatedly within a stream? Please clarify whether the demonstrated setting assumes a stationary task distribution.

A5: Thank you for the question. In our experiments, Phase Two adaptation is performed once after the first ten questions of each task stream. However, TaskMem is designed to support repeated adaptation and does not require a stationary task distribution. We discuss two practical deployment scenarios:

  1. Temporal adaptation under changing task distributions. In real-world streaming environments, the memorization policy can be updated periodically, e.g., at fixed time intervals or after a certain number of interactions. At each update, TaskMem uses the most recent k tasks (or samples k tasks from a recent window) to adapt the policy to the latest task distribution. Our experiments provide evidence supporting two important properties of this setting:

    • When the task distribution remains stable: As shown in Table 3, the task-adapted policy outperforms the original Phase One policy, demonstrating the benefits of adaptation to recent task feedback.

    • When the task distribution changes: Before the next update, the policy may encounter tasks that differ from those used for adaptation. Table 4 shows that adapters trained on different task categories achieve performance comparable to the Phase One policy on unmatched tasks. To further validate this observation, we conducted additional experiments on two evaluation tasks, each tested with three randomly selected task vectors from other task categories, providing stronger evidence that task-specific adaptation does not compromise performance on unseen task categories. Once the next update is triggered, TaskMem can adapt again using recent tasks to align with the new distribution.

  2. Horizontal adaptation across users. Another practical scenario is a multi-user setting, where a shared backbone memorization policy is maintained centrally, while each user has a lightweight, personalized adapter vector. Each user's adapter can be independently updated using feedback from their own tasks, enabling the memorization policy to capture user-specific task distribution without modifying the shared backbone. In summary, while our current evaluation uses a single adaptation step within each task-specific stream, the framework naturally supports both repeated temporal adaptation and user-specific horizontal adaptation. We will clarify the experimental protocol and these deployment scenarios in the revised paper.

Evaluation Task 1: Counting problem

MethodAcc.Cov.Prec.
Phase One (no adapter)35.060.058.3
w/ Information Synopsis adapter37.362.359.8
w/ OCR Problems adapter36.865.955.9
w/ Temporal Reasoning adapter37.358.663.6
w/ Counting problem adapter (matched)43.266.864.6

Evaluation Task 2: Action Reasoning

MethodAcc.Cov.Prec.
Phase One (no adapter)64.881.080.0
w/ TaskMaster adapter65.281.580.0
w/ HabitInsight adapter65.779.183.1
w/ Action Recognition adapter67.181.082.9
w/ Action Reasoning adapter (matched)69.582.983.9

Q6: Please clarify the source of the Phase Two rollout cache and its separation from evaluation videos, whether the main results use the 40-step or scaled 10-step adapter, and the end-to-end time and cost per update, including preference construction.

A6: Thank you for your questions. We clarify each point below:

  • The Phase-Two rollout cache is constructed offline using in-house videos drawn from the same distribution as the Phase-One training set. These videos are strictly separate from the evaluation videos.

  • Our main results use the scaled 10-step adapter, rather than the full 40-step adapter. As shown in Fig. 4, the scaled 10-step adapter achieves a comparable win ratio to the 40-step adapter, while requiring substantially less training time.

  • We pre-collect video clips and sample n=8 candidate memories per clip offline. Once the update is triggered, we construct preference pairs by selecting the highest- and lowest- ranked memories based on their relevance to the current tasks. This takes approximately 5 minutes per clip, but can be parallelized across clips to construct 640 preference pairs. We then perform 10-step DPO training on the 2048-parameter adapter with the backbone frozen, using 32 GPUs (80GB memory) and a batch size of 64. With the backbone already loaded, each training step takes approximately 110 seconds. Notably, these measurements were obtained without engineering optimization, leaving room for further efficiency improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.