acceptodds
Under review as a conference paper at ICLR 2027

MemFlow: Learning Compact Soft Memory for Long-Context Personalization via On-Policy Optimization

Abstract

An assistant that serves the same user over a long horizon has to answer from what that user has revealed: which preferences still hold, which were revised, and which constraints apply now. Retaining that information is not the same as acting on it, and the two are usually optimized as if they were. Keeping the information as text makes the reader's input grow with the retained history, while compressing it into a fixed number of latent vectors bounds the interface but is typically trained to reconstruct text or imitate reference answers, both of which are scored on sequences the reader never produced. We present MemFlow, which optimizes a fixed-budget soft memory by the behavior it supports. A query-conditioned textual memory is compressed into continuous vectors that form the reader's memory interface, and the reader is then trained on its own rollouts under two complementary signals: group-relative rewards for task outcomes, and confidence-gated on-policy distillation in which a frozen textual-memory teacher re-scores the student's sampled tokens under the textual memory. The teacher is never sampled from, so supervision stays on the student's current distribution and adds no autoregressive decoding; at inference, it is removed entirely. Across three Qwen backbones, MemFlow attains the highest accuracy we measure on PersonaMem-32K and PersonaMem-128K, with margins that widen at the longer history length, and transfers to PrefEval and LongMemEval without target-domain training. Ablations attribute most of the task gain to the reward term and a smaller additional gain to the teacher signal, and memory interventions show that the reader depends on the instance-specific content of its soft memory.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Personalization, On-Policy Optimization, Memory

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ccpqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author proposes memflow, which is optimized according to the behavior supported by fixed budget soft memory. The text memory is compressed into k continuous vectors to form the memory of the reader. Then, the reader is trained with two complementary signals on its own sampling trajectory. The author tested personamem on three qwen backbone models and achieved good performance.

Critical strengths
  1. Strong empirical results. The results on PersonaMem are particularly impressive. The memory intervention experiments provide useful evidence that the model relies on instance-specific memory when answering questions. Evaluation across multiple backbones and cross-dataset transfer experiments further strengthen the empirical support for the approach.

  2. Clear presentation. The paper is well written and easier to follow, making the motivation and method straightforward to assess. Although the “our starting point” passage in the introduction could be explained more intuitively, the overall presentation is a strength.

Critical weaknesses
  1. The experimental settings limit the relevance of the empirical evidence. PersonaMem-v2 was released in December 2025 [1], yet the paper evaluates only on the original PersonaMem. Although the experiments include three backbones, I find the two Qwen2.5 configurations less informative. Using older models is not necessarily a problem, but for a context- and memory-focused paper, evaluating a recent backbone is particularly relevant: in my experience, the increasingly widespread use of linear attention can substantially affect memory behavior. A concurrent work, From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space [2], evaluates on PersonaMem-v2 with Qwen3.5-9B. So I think it is a normal requirement to experiment on the updated model and benchmark. The absence of such an evaluation leaves uncertainty about whether the reported benefits extend to current model architectures and the newer personalization benchmark.

  2. The methodological distinction from prior work and the baseline selection are insufficiently established. Section 3.3, “On-policy memory distillation,” appears closely related to On-policy context distillation [3], which does not appear to be cited. Both train on student-generated trajectories using token-level guidance from a context-conditioned teacher. While the proposed method is more than a standalone on-policy distillation objective, this also means that comparisons should not primarily center on GRPO and OPSD. The system should be compared against memory methods addressing the same problem. xRAG and AutoCompressor are relatively old compression baselines; the only additional memory-specific method evaluated is MemGen, whose Trigger is disabled, so the comparison does not cover its complete method. These omissions make it difficult to assess both the incremental methodological contribution and the system’s competitiveness.

References:

[1] Jiang, B., et al. PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory. December 2025. arXiv:2512.06688.

[2] Xu, Y., et al. From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space. July 2026. arXiv:2607.05794.

[3] Ye, T., Dong, L., Wu, X., Huang, S., and Wei, F. On-Policy Context Distillation for Language Models. February 2026. arXiv:2602.12275.

Questions for authors
  1. What is the core difference between Section 3.3 and on policy context Disintegration?

  2. Are there any results on personamem-v2 or qwen3.5 models that use mixed/linear attention?

  3. Why close the trigger of MemGen? Does this affect its performance? Why not compare more long-term personalized memory methods?

Reviewer_wwcmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MemFlow targets long-context personalization, where answers must reflect preferences that were revealed or revised over a long history. A memory writer turns the history visible before a query into a query-conditioned textual memory MM. The writer is the shared LoRA adapter, first trained on memories extracted by GLM-5.2. A Perceiver-style compressor maps MM into K=256K = 256 soft memory vectors, which the reader takes as input. After a multi-stage initialization, the reader is trained on its own rollouts with GRPO and with a confidence-gated on-policy distillation (OPD) term. In OPD, a frozen copy of the initialized reader reads MM and re-scores the student's sampled tokens, and the detached gate σ(β(ℓT−ℓθ))\sigma(\beta(\ell_T - \ell_\theta)) weights the student's log-likelihood on those tokens. Experiments use Qwen2.5-3B/7B-Instruct and Qwen3-4B, in-domain on PersonaMem-32K/128K, with transfer to PrefEval (from PersonaMem-32K) and to LongMemEval (from LoCoMo). MemFlow has the highest accuracy in every cell of Table 1 (with one tie on PrefEval), and its margins are largest on PersonaMem-128K. Table 2 attributes most of the gain to reader initialization and GRPO and a smaller gain to OPD. Shuffled- and null-memory interventions show that the reader depends on instance-specific memory.

Critical strengths
  1. The full system gives large and consistent gains on long histories.

    • On PersonaMem-128K, MemFlow reaches 88.4, 94.4 and 89.4 on the three backbones, against 66.1, 78.5 and 65.7 for the strongest baseline in each case (Table 1).
    • On LongMemEval, with checkpoints trained and selected only on LoCoMo, it is 5.8 to 11.4 points above the best baseline on every backbone.
    • The PrefEval margins are small (a tie on Qwen2.5-7B-Instruct; 15.2 vs. 13.8 on Qwen3-4B), and Section 6 says so.

    The result holds across backbones and benchmarks rather than in one favorable setting, and this is the main reason I lean toward acceptance.

  2. The controls are informative, and the claims are scoped to what they show.

    • The shuffled- and null-memory interventions (Figure 5) reduce accuracy from 84.0 and 89.4 to about 58–62. So the reader uses the memory built for each instance, and the authors note that this does not show which parts are used.
    • Figure 3 shows a broad useful range of KK.
    • Figure 4 and Section 6 state that reading the history to build the textual memory dominates end-to-end cost, so a fixed KK bounds the reader's input but not the total cost.
    • Appendix C states which properties of the OPD update hold in expectation and which do not.
    • Figure 1 is described as a comparison of complete configurations rather than a controlled ablation.

    The design is also economical: GRPO and OPD reuse the same student rollouts, and the teacher only re-scores them without generating. This makes it easy to check each claim against its evidence.

Critical weaknesses
  1. Table 1 does not show that the large gains come from the soft memory or the on-policy objectives rather than from the memory writer.

    • The baselines lack the textual memory. MemFlow's reader receives MM from a writer trained on GLM-5.2 extractions (Appendix A.1.1), a supervision signal that no baseline receives. Full Text, GRPO and OPSD read the full history (about 124K tokens at 128K). xRAG retrieves raw messages, and MemGen and AutoCompressor compress the raw history. Section 1 presents textual memory as the main alternative to latent memory, yet no textual-memory method appears in Table 1.
    • Table 2 suggests the textual-memory pathway carries most of the 128K gain. On PersonaMem-128K, the Text-Space Control, which reads MM directly and skips reader initialization, reaches 85.5. Initialization Only, with no on-policy training, reaches 70.8. Both are close to or above most Table 1 baselines at 128K (65.7–78.5), although Table 2 reports mean accuracy over 16 samples and does not name its backbone. The null-memory MemFlow reader already scores 60.0 and 61.6 (Figure 5, Qwen3-4B), above several Qwen3-4B baselines (Full Text 56.0 and 4.3, AutoCompressor 58.0 and 30.8). Part of the margin therefore comes from learning the task format.
    • Some 128K baselines appear to fail rather than degrade. Full Text scores 4.3 on Qwen3-4B at 128K. The paper does not say how histories longer than each backbone's context window are handled. The token counts are also hard to reconcile. On LongMemEval, MemFlow's #Tok is about 23K below that of Full Text (122,545 vs. 145,521 on Qwen2.5-3B-Instruct), although Table 7 counts the writer reading the full visible history plus the memory and soft tokens.

    Section 4.2 attributes the wider 128K margin partly to the reader receiving "the same K vectors in both settings", but a reader of MM would share this property. Section 5 itself concludes that fixed-budget compression "is not the primary bottleneck", and Figure 4 shows that end-to-end cost is dominated by the pass over the history. The distinct benefit of the soft interface, in accuracy or in total cost, is therefore not shown, although the abstract and the first contribution credit the fixed-budget soft memory and its on-policy optimization.

  2. The evidence for the confidence gate, the second contribution, is weak in the setting where it is measured.

    • The gate is not tested on long responses. Section 1 motivates OPD with long responses in which one recommendation violates a preference (Figure 6). But PersonaMem rollouts are capped at five tokens (Appendix A.2), so where Table 2 measures OPD, the gate mostly weights a sampled answer option. The Figure 6 responses come from PersonaMem-32K checkpoints whose rollouts never exceeded five tokens. OPD is not ablated on LoCoMo, the only setting with longer rollouts (64 tokens).
    • The teacher is weak. The teacher is θinit\theta_{init} reading MM as text, but no initialization stage trains the adapter to answer from MM as text: writer initialization trains it to write MM, and reader initialization trains it to answer from ZZ. On PersonaMem-32K, the teacher scores 46.0 (Figure 5). This is below the final student with null memory (60.0) and below untrained Full Text on the raw history (56.0, Table 1, Qwen3-4B). At 128K it scores 73.2, above null memory (61.6). Section 6 argues that the bounded gate keeps the teacher's lower accuracy from dominating the update. That argument explains why the teacher does limited harm, not why ℓT−ℓθ\ell_T - \ell_\theta points to what compression removed.
    • The measured gain is small and uncontrolled. The OPD gain in Table 2 is 3.5 and 1.0 points of mean accuracy on 50 and 233 test questions, with λOPD=0.02\lambda_{OPD} = 0.02 against λGRPO=0.3\lambda_{GRPO} = 0.3. No run-to-run variation is reported, and there is no comparison with other weightings under the same initialization and rollouts, such as a constant gate gˉ=1/2\bar g = 1/2 (the baseline term in Remark 1) or a signed log-ratio term.

    The data show that adding LOPD\mathcal{L}_{OPD} helps somewhat, not that it helps for the stated reason.

Questions for authors
  1. Could you add textual-memory baselines to Table 1 for all backbones and benchmarks, for example the backbone reading the writer's memory MM with and without GRPO?

    • How are histories longer than each backbone's context window handled for Full Text, GRPO, OPSD and the writer?
    • Why is MemFlow's #Tok on LongMemEval lower than that of Full Text?
    • What is the average length of MM, and what fraction of 128K baseline outputs could not be parsed?

    If textual memory matches MemFlow on PersonaMem-128K and LongMemEval, I would attribute the main gains to the memory writer and lower my assessment of the soft-memory contribution.

  2. Under the same θinit\theta_{init}, cached memories and rollout budget, how does OPD compare with a constant gate gˉ=1/2\bar g = 1/2 and with a signed log-ratio term, with λOPD\lambda_{OPD} tuned for each? Please report the mean and spread over at least three training seeds, for Table 2 and for the PersonaMem-32K column of Table 1.

    • What is the teacher's accuracy next to the student's at the start of on-policy training, on the same backbone and decoding?
    • Does OPD help on LoCoMo → LongMemEval, where rollouts are longer?

    If the constant gate recovers most of the 3.5-point gain, the teacher signal contributes much less than claimed.

  3. (Clarification)

    • Which backbone is used in Table 2?
    • Who generates the auxiliary reasoning targets sis_i in Appendix A.1.2, and does the generator see the gold answer?
    • What are the training and validation sizes, and are examples with the same shared-context identifier kept in one split?
    • What are the LoCoMo settings corresponding to Table 5?
4 comments
AuthorsOpenReview ↗

Clarification Regarding the Ethics Flag

Dear Reviewer wwcm,

Thank you for your thoughtful review. Regarding the ethics flag, we would like to clarify that all datasets used in our study are publicly available, open-source datasets. To the best of our knowledge, our research complies with the ICLR Code of Ethics. If you have any specific ethical concerns, we would appreciate your clarification and would be happy to address them.

Best regards,
The Authors

AuthorsOpenReview ↗

Follow-up to Reviewer wwcm

Dear Reviewer wwcm,

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to briefly summarize how the additional experiments address your concerns.

  1. Could the gains come from the writer alone? We evaluated text readers with and without GRPO using the same writer’s memories. Full MemFlow scored higher than both text baselines across all three backbones on PersonaMem-32K and PersonaMem-128K, with the margin over Text + GRPO varying by backbone.
  2. Does the confidence gate add value beyond standard distillation? In a matched comparison across three training seeds, the sigmoid gate achieved the highest mean accuracy at both PersonaMem context lengths compared with GRPO alone, a constant gate, and signed log-ratio OPD.
  3. Does gated OPD help when training allows longer responses? In the LoCoMo-to-LongMemEval-S transfer experiment, adding gated OPD improved accuracy over GRPO alone with a 64-token training rollout cap.

MemFlow learns a fixed-budget soft memory interface from the reader’s own answer rollouts to support downstream personalization. Our extension to on-policy distillation lets a teacher reading the textual memory guide a student reading its compressed representation. A bounded sigmoid gate uses the teacher–student confidence gap to weight supervision for each sampled token, and the teacher scores the same rollouts used for GRPO. As you highlighted, the full system delivers consistent gains across backbones and benchmarks, with the largest margins on PersonaMem-128K in the original comparison. The shuffled and null-memory controls show that the reader benefits from information specific to each instance.

Do you have any remaining concerns or questions about the new results? We would be happy to address them before the discussion closes.

Best regards,

Authors

Thank you for the shared-writer text baselines and the matched gate comparison; they answer my questions on the memory writer, the token accounting and reproducibility, and I keep my score. The gate's advantage over the constant gate (76.67 ± 2.31 vs. 73.33 ± 2.31 at 32K, 88.70 ± 0.50 vs. 87.84 ± 1.08 at 128K) is within about one standard deviation, so I would describe it in the revision as a small gain rather than a clear effect.

The ethics flag was set by mistake; I have no ethics concerns about this paper and have removed the flag.

Reviewer_LY1qOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemFold, a framework for long-context personalization using compact soft memory. The method first constructs query-conditioned textual memory from interaction history and compresses it into a fixed number of continuous vectors. The reader is then optimized using GRPO and a confidence-gated on-policy distillation (OPD) objective, where a frozen teacher conditioned on textual memory evaluates the student's sampled tokens. Experiments across three Qwen backbones show substantial improvements on PersonaMem and positive transfer to PrefEval and LongMemEval.

Critical strengths

The paper addresses a meaningful problem: retaining user information does not necessarily ensure that a language model uses it correctly when generating personalized responses. Optimizing the reader based on downstream behavior, rather than relying entirely on reconstruction or supervised answer imitation, is a reasonable direction. The use of a textual-memory teacher to guide a compressed-memory student is also a concrete extension of existing on-policy distillation methods.

Critical weaknesses
  1. Despite positioning MemFold as learning compact memory through on-policy optimization, both the compressor and the textual memory snapshots are frozen during this stage. Only the reader's LoRA parameters are updated. Consequently, the proposed method does not directly learn what information should be retained or how it should be compressed based on downstream rewards. It instead optimizes a reader conditioned on a previously constructed memory representation. Furthermore, the updated adapter is used to regenerate memories at inference, although the reader is trained exclusively on cached memories from the initialization checkpoint. This creates an unexamined training–inference mismatch. The paper should more clearly distinguish optimizing memory representations from optimizing a model to use fixed memory representations.

  2. The most important algorithmic contribution appears to be confidence-gated OPD. However, Table 2 shows that adding OPD to GRPO improves mean accuracy by only 1.0 percentage point on PersonaMem-128K (86.9% to 87.9%) and 3.5 points on PersonaMem-32K (71.9% to 75.4%). Most of the improvement is already obtained through initialization and standard GRPO. The smaller OPD gains are also reported without variation across independent training runs. More importantly, the large improvements over external baselines in Table 1 are difficult to interpret because MemFold benefits from an extensive supervised initialization pipeline that those baselines do not share. Appendix A.4 further states that the MemGen baseline disables its Trigger mechanism, meaning that its full procedure is not evaluated. These comparisons demonstrate the effectiveness of the overall system, but provide much weaker evidence for the necessity or superiority of the proposed optimization objective.

  3. The central motivation for OPD is that sequence-level rewards cannot identify individual tokens that fail to use relevant memory. However, the teacher–student log-probability gap in Eq. (3) only measures disagreement between two conditional distributions. It does not establish whether a token is incorrect or whether the disagreement results from missing memory information. This is particularly important because the teacher is frozen while the student changes during training, and Figure 5 shows that the textual-memory teacher can be substantially less accurate than the final student. Moreover, the main PersonaMem experiments use multiple-choice answers with a maximum generation length of five tokens. Such a setting does not directly test the localized preference violations in longer responses that motivate OPD. Figure 6 provides only one qualitative example. Without demonstrating that the confidence gate identifies memory-related errors or outperforms a simple constant-weight alternative, the proposed mechanism remains insufficiently supported.

  4. Although the reader receives only 256 soft tokens, the memory writer must still process the full interaction history for every query. Thus, the method does not eliminate the main cost of processing long histories, nor does it maintain a persistent compressed memory across interactions. In Table 1, MemFold requires 123,229 token equivalents compared with 124,317 for Full Text on PersonaMem-128K with Qwen2.5-3B. Furthermore, the text-space control in Table 2 already achieves 85.5% mean accuracy on PersonaMem-128K, compared with 87.9% for MemFold. These results raise a more fundamental question: what practical or methodological advantage does the additional soft-memory compression provide over directly using the query-conditioned textual memory?

Questions for authors
  1. Could the authors provide a controlled comparison between textual-memory and soft-memory readers using comparable supervised initialization and GRPO training?

  2. Does the confidence-gated OPD objective outperform a constant-weight self-training objective under otherwise identical conditions? Is there evidence that larger teacher–student confidence gaps correspond to actual memory-related or preference-following errors?

  3. Could the authors quantify preference violations in the existing open-ended evaluation and clarify the end-to-end computational costs, including memory construction?

Reviewer_hpctOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

MemFlow addresses long-context personalization by first extracting query-relevant textual memory and then compressing it into a fixed number of soft tokens. After initialization, the reader is trained using GRPO and confidence-gated on-policy distillation from a frozen textual-memory teacher. The teacher scores student-generated responses without generating additional trajectories. Results across three Qwen backbones show strong gains on PersonaMem, particularly at 128K, and transfer to PrefEval and LongMemEval. Ablations show that GRPO accounts for most of the improvement, with a smaller additional benefit from distillation.

Critical strengths
  1. The problem is well motivated. Remembering the user’s preference and using them properly in downstream response are different capabilities. The authors consider not only recovering the original history information from compressed memory, but also the downstream response quality.

  2. Good empirical results. The improvement on PersonaMem-128K is significant across all three backbones. The authors also evaluate their method at two different memory settings, strengthening the empirical result.

  3. With theoretical support. The paper provides a rigorous mathematical analysis in Appendix C, proving that the proposed OPD objective acts as a bounded, saturated reverse KL divergence toward the textual teacher.

Critical weaknesses
  1. OPD is not new: Contribution 1 of the paper mainly emphasizes the application of OPD to memory compression. However, since OPD is already a well-established method, this makes Contribution 1 somewhat weak.

  2. The contribution of the gate in OPD is unclear. The authors include the gated OPD into their contributions. However, there already exists some works about On-policy distillation and context-conditioned teachers. Table 2 shows only a small additional gain from gated OPD, and gated OPD alone does not improve over initialization.

  3. Unfair Baseline Comparisons. MemFlow benefits from structured memory extraction and substantial interface initialization, whereas the main GRPO and OPSD baselines use full histories and different initialization. It seems that the paper lacks a controlled experiment where baselines receive the same structured inputs.

  4. Little benefit for long-context interactive scenario. The paper introduces a narrative about fixing localized preference violations within long, detailed responses. However, PersonaMem is optimized using multiple-choice tasks with a maximum response length of only 5 tokens. According to Table 1, on the PrefEval benchmark (which requires long-detailed responses), MemFlow with Qwen2.5-7B only achieves an accuracy of 14.1%, which is almost identical to the uncompressed Full Text baseline (14.0%).

Questions for authors
  1. Under the same initialization, soft-memory interface, and GRPO objective, how does the sigmoid gated OPD compare with ordinary reverse-KL on-policy distillation?

  2. Could the authors explain whether the experiments are fair? If not, could the authors add a controlled experiment where baselines receive the same structured inputs, together with the existing soft-memory ablation.

  3. Given the 5-token limit during training, Could the authors explain how can MemFlow generalize its local gate to hundreds of tokens in long responses, where the Qwen2.5-7B accuracy on PrefEval only marginally increases by 0.1% (14.1% vs 14.0%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.