acceptodds
Under review as a conference paper at ICLR 2027

MemAny: Plug-and-Play Generative Latent Memory for Any Model

Abstract

Knowledge acquired through model training is typically tied to the model that learned it, requiring repeated training as new models emerge. We introduce MemAny, a framework that turns adapted knowledge into plug-and-play generative latent memory that can be trained once on a host model and reused across different recipient models. MemAny routes adaptation into an external memory weaver while keeping the host model frozen, and expresses its generated latent tokens as linear combinations of each recipient’s own token embeddings. This latent superposition enables knowledge transfer across tokenizers through coefficients aligned by shared vocabulary anchors, without updating either the recipient or the learned memory. Experiments combining MemAny with various training algorithms, e.g., supervised fine-tuning (SFT), reinforcement learning (RL), and on-policy distillation (OPD), demonstrate that memories learned with small host models yield transferable gains in mathematical/logical reasoning, code generation, and tool-use. In particular, OPD-trained memory from Qwen3-4B-Base transfers across model families and tokenizers to MiMo-V2-Flash-Base (309B) and HY3-Preview-Base (295B), improving their AIME24 accuracy by up to points. These results establish portable generative memory as a practical interface for sharing acquired capabilities, paving the way for their efficient and rapid dissemination across an evolving model ecosystem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Memory, Agent Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jirsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MemAny proposes a cross-model latent memory mechanism. A memory weaver is trained while the host LLM remains frozen, and generates K=8 input-dependent latent tokens. To transfer these tokens across model families and tokenizers, MemAny represents them as linear combinations of each recipient’s vocabulary embeddings and aligns source/recipient coefficient spaces using shared-token anchors. The method is evaluated with SFT, OPD, and RL on math, code, tool-use, and logic tasks, showing large gains on several cross-family transfers.

Critical strengths
  1. The method is easy to follow and novel to some extent. Representing latent interventions in recipient-native vocabulary space and aligning coefficient spaces across models is a clean solution to tokenizer and embedding-space mismatch.

  2. The method shows strong empirical effects across heterogeneous recipients. The reported cross-family gains are large in some settings, e.g. MiMo-7B MATH-500 38.4 to 76.2, HY3 AIME25 13.3 to 43.3, and MiMo-V2-Flash AIME24 16.7 to 50.0.

Critical weaknesses
  1. To me, MemAny is more like eliciting capabilities already present in the recipient. And the experiments do not establish that MemAny transfers newly acquired knowledge. MemAny is effectively an input-dependent soft-prompt generator. The fixed-prompt and CoT ablations show that the learned intervention is nontrivial, but do not distinguish knowledge transfer from more sophisticated elicitation. This concern is reinforced by the latent visualizations, which emphasize generic reasoning/code cues such as Math, Solution, Thinking, and Python, and by large changes in generation length after memory insertion.

  2. The motivation is unclear to me. Why we should use gradient-trained latent memory over training-free external memory is not established. MemAny avoids recipient-side optimization, but each memory requires training and running a Qwen3-4B-based weaver, and separate memories are trained for different domains. In contrast, explicit memories are naturally model-agnostic, cheap to write/update, and editable without gradient training. Table 4 shows better benchmark accuracy than several explicit-memory baselines, but does not compare the properties that motivate memory systems (e.g., editability, scaling with accumulated experience, plus-and-play without training or inference with one more trained LLM), and the supervision is not clearly matched across methods.

Questions for authors
  1. Why does the same memory improve some benchmarks on a recipient while degrading others? How often does negative transfer occur, and can it be predicted before deployment? Table 1 contains several substantial regressions. Is there a measure that predicts whether attaching a memory will help or hurt a new recipient? Without such a mechanism, the practical claim of plug-and-play transfer seems limited.

  2. What is the true end-to-end deployment cost including loading and executing the weaver?

1 comment

Thank you for the detailed rebuttal and additional analyses. I appreciate the clarifications on negative transfer and deployment cost, as well as the additional evidence that MemAny's improvements go beyond answer formatting.

However, my main concerns remain. The problem-specific latent representations and reasoning improvements are still consistent with an input-dependent capability elicitation mechanism, rather than demonstrating the transfer of newly acquired capabilities. Additionally, the practical advantages over explicit memory remain insufficiently established, particularly without matched supervision and a more systematic comparison.

Overall, I find the empirical results interesting, but these concerns prevent me from raising my score at this stage. I will maintain my rating of 2.

Reviewer_i1BZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces MemAny, a framework for creating portable generative latent memory. A memory weaver is trained on a frozen host model using standard adaptation objectives (SFT, OPD, or RL). The weaver emits continuous latent tokens expressed as linear combinations of vocabulary embeddings, aligned across tokenizers via PCA and orthogonal Procrustes alignment. At inference, these tokens condition frozen recipient models across different scales and model families without recipient-side updates.

Critical strengths
  1. Compelling Problem Formulation: Decoupling acquired capabilities from host model weights into a reusable, cross-tokenizer interface addresses a fundamental challenge in model adaptation.
  2. Broad Empirical Coverage: The framework is tested across multiple training paradigms (SFT, OPD, RL), domains (mathematics, code, logic, tool use), and recipient scales up to 300B parameters.
  3. Transparent Reporting: The authors document performance regressions on large code recipients and provide detailed specifications of prompts, hyperparameters, and compute costs.
Critical weaknesses
  1. Deflated Baselines on Competition Benchmarks

    • MiMo-7B-Base officially achieves 32.9% on AIME24, whereas Table 1 reports 3.3% for Base and 32.5% for +MemAny. The +MemAny score matches the official zero-memory baseline.
    • The authors acknowledge in Appendix B.4 (lines 1034–1039) an inability to reproduce official AIME scores under their Qwen-derived evaluation setup. When an evaluation pipeline reduces a base model's score by an order of magnitude, large headline gains (e.g., +31.7 on Ling-2.6, +33.3 on MiMo-V2-Flash) likely reflect the recovery of formatting and execution failures rather than genuine capability transfer.
    • On MATH-500, where baseline reproduction matches official figures, cross-model transfer is marginal or negative (e.g., Qwen3-8B drops from 68.4% to 67.0%, and Qwen3-30B-A3B drops on OlympiadBench).
  2. Confounded Prompting in the Primary Comparison

    • Table 1 compares Base under few-shot CoT demonstrations against MemAny under zero-shot instruction templates, altering prompt structure and memory simultaneously.
    • While Table 3 includes prompt ablations (CoT prompt, <think> prefix) on three models, the exact textual template for CoT prompt is not defined in Appendix B.6. Furthermore, the reverse condition (MemAny evaluated with few-shot demonstrations) is missing, and the main comparisons in Table 1 remain unaligned.
  3. Confounding with Test-Time Compute (Output Length)

    • Performance gains correlate strongly with large increases in generation length (+44% to +154% in Table 11).
    • According to lines 1044–1046, the few-shot baseline stops decoding whenever a new demonstration delimiter (Problem:) is generated, artificially truncating reasoning trajectories. MemAny uses instruction formatting without this stopping trigger, allowing unconstrained generation. The evaluation does not isolate knowledge transfer from the effect of increased test-time compute.
  4. Incomplete Method Realization and Restricted Scope

    • Section 4.1 formalizes a dynamic trigger TψT_\psi to selectively inject memory during decoding. However, the experimental setup bypasses trigger training entirely (lines 341–343) and inserts memory as a static 8-token prefix, reducing the framework to an input-dependent soft prompt.
    • The primary math and code benchmarks evaluate only base models. The paper does not demonstrate whether this mechanism benefits instruction-tuned or reasoning-tuned models, which represent the actual deployment target for external capability transfer.
Questions for authors
  1. Baseline Discrepancy: What causes MiMo-7B-Base to drop from its official 32.9% to 3.3% in Table 1? Please provide the exact failure breakdown (parsing errors, length limits, early delimiter stops) for the base model under this protocol.
  2. Controlled Prompting: Can you provide a version of Table 1 where Base and +MemAny are evaluated using identical prompt templates (both tested under the zero-shot instruction template, and both tested under the few-shot demonstration template)?
  3. Compute Matching: Does the performance advantage of +MemAny remain when base models are sampled with output length budgets and stopping conditions matched to +MemAny generations?
Reviewer_VRYrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method to transfer knowledge learnt by model A during fine-tuning on task T to model B directly (without fine-tuning) such that model B improves on task T. The authors propose a method to 1. learn continuous vectors during fine-tuning to improve task T 2. transfer these representations across models

The authors show this actually works on a bunch of tasks like math, code etc.

Critical strengths
  1. the problem is important and interesting
  2. the authors have very good results. The method works surprisingly well and is evaluated across a broad range of tasks.
Critical weaknesses
  1. Evaluation: Is the model learning formatting or actually improving reasoning? One hypothesis is the latent vectors learn the correct formatting like put answer in \boxed{} and this skill transfers across models.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.