- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents ELM, which keeps a language model frozen and trains a separate soft prompt for each encountered problem. After answering a problem, it trains the prompt with GRPO using rewards based on agreement with the majority answer among sampled solutions, then stores it for later use. For subsequent problems, the system retrieves a stored prompt and generates answers both with and without it. If the answers differ, a process reward model selects the final response.
Critical strengths
-
Practical research question. The paper studies how to reuse experience from past problems to help a model solve future ones, which is relevant to models used over time.
-
Evidence of transfer across problems. The experiments show that some prompts trained on a single problem can help solve other problems, with further evidence from transfer between datasets.
-
Useful analysis of the results. The paper examines how memories help or hurt performance and how different answer selection methods affect the results. These analyses help explain both the gains and the failure cases.
Critical weaknesses
-
Unclear value of the extra training. ELM uses majority-vote answers to train its memories, but its main results on Qwen2.5-Math-7B are worse than simply using majority voting on all four benchmarks. This raises doubts about whether the extra memory training and retrieval are worth the effort.
-
Missing basic comparisons. The paper does not compare its method with randomly choosing a memory, always using the same memory, or using an untrained prompt. Without these comparisons, it is unclear whether creating a separate memory for each question and retrieving a relevant one are actually necessary.
-
Unclear advantage over textual memories. In Table 3, ELM uses a verifier to choose between two answers, but the textual memory baselines do not. Appendix E.4 reports that adding the same selection step allows textual memories to outperform ELM on Llama-3.1-8B-Instruct on MATH500 (52.9 vs. 50.9) and GPQA (27.4 vs. 19.7). The claimed advantage of latent memories is therefore less convincing when the methods use the same answer selection process.
-
Dependence on a separate verifier. For Qwen2.5-Math-1.5B, the input-retrieval version performs worse than the original model on both AMC23 and AIME24 without the verifier (Table 12). Asking the model to judge its own answers also fails to reproduce the gains obtained with the separate 7B PRM, leaving limited evidence that the model can improve reliably on its own.
-
Inconsistent numbers in Table 4. The zero-shot averages in Table 4 are 12.9 for Llama and 31.7 for Qwen, but averaging the AMC23 and AIME24 results in Tables 3 and 12 gives 11.65 and 30.0. These numbers do not match, which also affects the reported 3.8-point improvement without routing.
Questions for authors
See the weaknesses above.
I would appreciate the authors’ response to the main concerns raised in my review. These points are important to my assessment, and without further clarification, I may need to reconsider my current rating.