acceptodds
Under review as a conference paper at ICLR 2027

Distilling Across Tasks for Generalizable Memory Policy in Personalized Agents

Abstract

Personalized agents must use a user’s history across changing tasks and domains. Existing memory systems typically rely on hand-designed routines or task-specific training, leaving open how to learn a shared, transferable memory policy. We introduce MEMORYOPD (Memory On-Policy Distillation), which trains a small student across tasks using the token distributions of specialist teachers operating within a memory harness. On MemoryCD, this harness combines episodic evidence with a semantic user profile; distillation transfers the teachers’ memory-conditioned behavior on the student’s own generations, providing dense token-level supervision without a downstream task reward. Our analysis characterizes a geometric teacher consensus at a common context and gives sufficient conditions for a memory-conditioned policy to transfer across domains. We evaluate generalization through two complementary settings with separately trained policies. A policy trained jointly on MemoryCD tasks is reused without parameter updates on eight held-out product domains. Separately, a policy trained on LongMemEval is transferred to LoCoMo without further training. In domain, MEMORYOPD configurations outperform the seven comparison methods on all eight personalization metrics for both student families, including a 27% reduction in Qwen rating MAE (0.490 to 0.357). The transferred MemoryCD policies lead the eight-domain averages on every metric; the LongMemEval policy reaches 0.695 held-out accuracy (best baseline 0.629) and transfers successfully to LoCoMo. These results support learning memory behavior that balances multiple tasks and remains useful beyond its training distribution.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Personalized Agents, Personalization, User Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8EWtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemoryOPD, which distills trained per-task specialist teachers into a single student using reverse KL on-policy distillation within a memory harness. Experiments on MemoryCD and on a transfer task from LongMemEval to LoCoMo show improvements over hand-crafted and RL memory baselines.

Critical strengths
  • This paper addresses a relevant problem, learning a shared memory policy across tasks without a downstream reward.
  • The experiments are extensive with two student families, three teacher sizes, four personalization tasks, eight held out product domains and two conversational benchmarks, and evaluate both task performance and transfer.
Critical weaknesses
  • Limited novelty: on-policy distillation with reverse KL and multi-teacher on-policy distillation are already established in prior work the paper cites. The paper's main contribution is their application to memory-conditioned personalization.
  • The experiments are not sufficient to establish the contribution of distillation. The comparisons do not include the specialist teachers’ own scores, warm start model before distillation, and multi-task SFT starting from that same warm start. The only SFT comparison starts from the base model instead and yet performs close to MemoryOPD on MemoryCD tasks.
Questions for authors
  • Can you report the scores of the specialist teachers, the warm-start model before distillation, and multi-task SFT starting from the same warm start? These comparisons would be helpful to assess the contribution of distillation to the overall performance.
Reviewer_uvwAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces MEMORYOPD (Memory On-Policy Distillation), which trains a small student model from different teachers across different tasks. It tests the trained student model on hold-out domains or tasks.

Critical strengths

This work trains a smaller student model to distill the behavior of different teachers, improving generalization ability and computational efficiency during testing.

This work tests the teacher models at various scales, making the results more reliable.

Critical weaknesses

1.This paper is hard to follow; I can't understand the research problem. In the title and abstract, it mentions a personalized agent. But don't mention it again in the main text. So what is the personalized agent, and why this work investigate it? What is the benefit of the method for a personalized agent?

2.The definition of memory policy, memory harness, and memory instances is also unclear.

  1. Where the memory data from teachers are from? Are teacher models assigned different memory data or the same. If different dataset for each teacher, how about distilling over the whole and large dataset for each teacher.
Questions for authors

The experimental setting of baselines are unclearly. In figure one, the results under different teacher scales are similar, how to explain it? Why larger teacher don't create a improvement.

Reviewer_oarGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MEMORYOPD, an on-policy multi-teacher distillation procedure for learning one memory-conditioned student policy across heterogeneous personalization tasks. Task-specific specialist teachers operate over an episodic-plus-semantic memory harness, while the student is warm-started by merging task-specialist adapters and trained to match each teacher’s top-K next-token distribution on the student’s own generated rollouts using a confidence-weighted reverse KL objective.

The paper provides a fixed-context reverse-KL decomposition showing that the idealized multi-teacher target is the normalized weighted geometric mean of teacher distributions. It also gives a generic feature-based source-to-target transfer bound. Experiments use MemoryCD’s rating, ranking, summarization, and generation tasks with Qwen and Gemma students, evaluate transfer to eight held-out product domains, and separately train a LongMemEval policy that is transferred to LoCoMo. The paper reports strong numbers, including Qwen rating MAE improving from 0.490 to 0.357 in-domain and LongMemEval held-out accuracy of 0.695 versus 0.629 for the best listed baseline.

Critical strengths

Meaningful transfer evaluation. The paper evaluates a shared policy across four heterogeneous personalization tasks and then applies it without parameter updates to eight unseen product domains. It also includes a separate LongMemEval-to-LoCoMo transfer experiment, which is more informative than evaluating only on source-task performance.

Reasonably careful data isolation. The paper splits MemoryCD by users and LongMemEval by question IDs, and states that there is no test-based checkpoint selection. These choices reduce obvious leakage concerns.

Critical weaknesses

The benefit of OPD is not causally isolated. The full recipe uses a TIES merge of task-specific adapters as warm-start, 150 multi-task updates, 120 additional refinement updates, confidence weighting, EMA regularization, and a support-overlap gate. The main “−OPD” ablation instead uses SFT on base-model targets and starts from the base model. Thus initialization, targets, optimization schedule, and possibly effective training budget all change simultaneously. The paper cannot attribute the improvements specifically to on-policy multi-teacher reverse KL.

The theoretical and baseline comparisons do not support the strength of the claims. The top-K-renormalized objective does not control the full-policy gap; the paper itself introduces a residual ΩB\Omega_B that can become arbitrarily large, but does not measure it. Training uses top-p sampling while evaluation uses greedy decoding, whereas the transfer theorem is stated for a matched execution policy. In addition, MEMORYOPD uses a particular episodic-plus-semantic memory harness, while baselines use different memory interfaces and adaptation procedures. The large degradation after removing memory inputs therefore demonstrates the value of the harness, not necessarily the superiority of the OPD algorithm.

Questions for authors

Please rerun the main MemoryCD comparison with identical initialization, update count, optimizer settings, and compute for MEMORYOPD, multi-task SFT, teacher-generated SFT, forward-KL/ULD, and sequence-level KD. Please separately ablate the TIES warm-start, 120-step refinement stage, confidence weighting, EMA regularization, answer-span weighting, and support-overlap gate. How much of the headline gain remains after these controls?

Please report results over at least three independent random seeds, with per-task and per-domain confidence intervals and significance tests. Were all baselines evaluated with the same student backbone, memory evidence, prompt budget, training data, and compute? In particular, is the LoCoMo judge difference of 0.5351 versus 0.4994 statistically reliable?

Please quantify the top-KK residual ΩB\Omega_B, support mass, and tail mismatch on both training and test states, including a KK-sensitivity study and, if feasible, full-vocabulary KL. How should the transfer claims apply when training uses top-pp sampling but deployment uses greedy decoding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.