- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MemAny proposes a cross-model latent memory mechanism. A memory weaver is trained while the host LLM remains frozen, and generates K=8 input-dependent latent tokens. To transfer these tokens across model families and tokenizers, MemAny represents them as linear combinations of each recipient’s vocabulary embeddings and aligns source/recipient coefficient spaces using shared-token anchors. The method is evaluated with SFT, OPD, and RL on math, code, tool-use, and logic tasks, showing large gains on several cross-family transfers.
Critical strengths
-
The method is easy to follow and novel to some extent. Representing latent interventions in recipient-native vocabulary space and aligning coefficient spaces across models is a clean solution to tokenizer and embedding-space mismatch.
-
The method shows strong empirical effects across heterogeneous recipients. The reported cross-family gains are large in some settings, e.g. MiMo-7B MATH-500 38.4 to 76.2, HY3 AIME25 13.3 to 43.3, and MiMo-V2-Flash AIME24 16.7 to 50.0.
Critical weaknesses
-
To me, MemAny is more like eliciting capabilities already present in the recipient. And the experiments do not establish that MemAny transfers newly acquired knowledge. MemAny is effectively an input-dependent soft-prompt generator. The fixed-prompt and CoT ablations show that the learned intervention is nontrivial, but do not distinguish knowledge transfer from more sophisticated elicitation. This concern is reinforced by the latent visualizations, which emphasize generic reasoning/code cues such as Math, Solution, Thinking, and Python, and by large changes in generation length after memory insertion.
-
The motivation is unclear to me. Why we should use gradient-trained latent memory over training-free external memory is not established. MemAny avoids recipient-side optimization, but each memory requires training and running a Qwen3-4B-based weaver, and separate memories are trained for different domains. In contrast, explicit memories are naturally model-agnostic, cheap to write/update, and editable without gradient training. Table 4 shows better benchmark accuracy than several explicit-memory baselines, but does not compare the properties that motivate memory systems (e.g., editability, scaling with accumulated experience, plus-and-play without training or inference with one more trained LLM), and the supervision is not clearly matched across methods.
Questions for authors
-
Why does the same memory improve some benchmarks on a recipient while degrading others? How often does negative transfer occur, and can it be predicted before deployment? Table 1 contains several substantial regressions. Is there a measure that predicts whether attaching a memory will help or hurt a new recipient? Without such a mechanism, the practical claim of plug-and-play transfer seems limited.
-
What is the true end-to-end deployment cost including loading and executing the weaver?
Thank you for the detailed rebuttal and additional analyses. I appreciate the clarifications on negative transfer and deployment cost, as well as the additional evidence that MemAny's improvements go beyond answer formatting.
However, my main concerns remain. The problem-specific latent representations and reasoning improvements are still consistent with an input-dependent capability elicitation mechanism, rather than demonstrating the transfer of newly acquired capabilities. Additionally, the practical advantages over explicit memory remain insufficiently established, particularly without matched supervision and a more systematic comparison.
Overall, I find the empirical results interesting, but these concerns prevent me from raising my score at this stage. I will maintain my rating of 2.