- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TaskMem, which make memory generation for multimodal agents as a learnable memorization policy. The approach consists of two phases: Phase one employs reinforcement learning to generate high-quality episodic memory, phase two fine-tunes a lightweight adapter using pairwise preferences and DPO based on recently observed tasks, which aligns generated memories more closely with task requirements. Experiments demonstrate consistent performance improvements on the VideoMME, EgoLife, and EgoTempo benchmarks.
Critical strengths
- This paper takes a fascinating approach by focusing on how to determine "what to remember" based on the type of current task.
- The two-stage design is clear. Phase Two updates only the 2048-parameter adapter, making the method lightweight and feasible for actual deployment.
Critical weaknesses
- The evaluation setup may be overly favorable to TaskMem. Questions are grouped by task type, and the first ten questions of each type are used for adaptation, followed by evaluation on later questions from the same type. This creates a highly stationary setting and leaves the method’s robustness to mixed or changing task distributions unclear.
- The baseline comparison is not entirely fair. TaskMem employs Qwen3-VL-30B-A3B as its backbone and generates streaming memory using 10-second clips, whereas baselines such as HippoMM utilize Qwen2.5-VL-7B-Instruct and process videos using 30-second non-overlapping clips.
Questions for authors
- Does TaskMem remain effective when there are sudden changes in task types (mixed tasks) or task distribution?
- Is it possible to re-evaluate the baselines using the same backbone?
- How much time and computational power does a single online update in Phase Two actually require?
I would be open to raising my score if this concern can be adequately addressed during rebuttal. It can better enhance the practical deployment value of the paper.
4 comments
Response to Reviewer wv4G
Q3: Does TaskMem remain effective when there are sudden changes in task types (mixed tasks) or task distribution?
A3: Please refer to A1. The results in Table 4 and our supplementary experiments show that TaskMem remains robust to sudden changes in task types or distributions, achieving performance at least comparable to the general Phase One policy. Once the next update is triggered, TaskMem adapts to the new distribution by incorporating recent tasks.
Q4: Is it possible to re-evaluate the baselines using the same backbone?
A4: Please refer to A2.
Q5: How much time and computational power does a single online update in Phase Two actually require?
A5: We pre-collect video clips and sample n=8 candidate memories per clip offline. Once the update triggered, we construct preference pairs by selecting the highest- and lowest- ranked memories based on their relevance to the current tasks. This takes approximately 5 minutes per clip, but can be parallelized across clips to construct 640 preference pairs. We then perform 10-step DPO training on the 2048-parameter adapter with the backbone frozen, using 32 GPUs (80GB memory) and a batch size of 64. With the backbone already loaded, each training step takes approximately 110 seconds. Notably, these measurements were obtained without engineering optimization, leaving room for further efficiency improvements.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns and I am happy to raise my score.
Thank you for your careful assessment and constructive feedback. The additional experiments and clarifications are already included in the revised manuscript. We sincerely appreciate your time and effort in reviewing our work.
Response to Reviewer wv4G
We thank the reviewers for the constructive comments. In the revised manuscript, we have (1) added cross-task adaptation experiments showing that matched adapters improve accuracy while unmatched adapters cause no degradation, (2) re-run HippoMM with the same Qwen3-30B-A3B backbone and confirmed that TaskMem still outperforms it, and (3) reported the full cost of Phase-Two update. Details and discussion are provided below.
A1: Thank you for raising this important concern. We would like to clarify that TaskMem is designed to support repeated adaptation to mixed or changing task distributions, rather than a single adaptation to one task category. In real-world streaming settings, the memorization policy can be updated periodically (e.g., at fixed time intervals or after a certain number of interactions). At each update, TaskMem uses the most recent k tasks, or samples k tasks from a recent window, to adapt the policy to the current task distribution.
While evaluating long-horizon sequential adaptation is computationally expensive, our experiments provide evidence for TaskMem's behavior under two key scenarios:
When the task distribution remains stable: As shown in Table 3, the task-adapted policy outperforms the original Phase One policy, demonstrating the benefits of adaptation to recent task feedback.
When the task distribution changes: Before the next update, the policy may encounter tasks that differ from those used for adaptation. Table 4 shows that adapters trained on different task categories achieve performance comparable to the Phase One policy on unmatched tasks.
To further validate this observation, we conduct additional experiments on two evaluation tasks, each tested with three randomly selected task vectors from other task categories, providing stronger evidence that task-specific adaptation does not compromise performance on unseen task categories. Once the next update is triggered, TaskMem can adapt again using recent tasks to align with the new distribution.
Together, these results provide evidence for the benefits of task-specific adaptation and robustness to task distribution shifts, supporting TaskMem's potential to handle evolving task demands. We have revised the paper to include these additional results and discussion.
Evaluation Task 1: Counting problem
Evaluation Task 2: Action Reasoning
A2: Thank you for your valuable comment. To address the concern about backbone differences, we conducted additional experiments using Qwen3-30B-A3B as the backbone for HippoMM with 30-second clips. The results show that the stronger backbone improves HippoMM's performance over its original Qwen2.5-VL-7B. TaskMem still outperforms HippoMM when using the same backbone, demonstrating the effectiveness of our proposed method. We will include these results in the revised paper.