- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The research team proposed a new approach, Hash-Embedding Memory Injection, to transfer the domain knowledge from a smaller model to a larger model or transfer between different base models. To do so, they proposed three components: 1. anchor rendering: trying to provide a backbone-independent space for the memory, so different models can access the same information. 2. Differential injection, which use residual, gate, RMS balance, and adapter to inject the memory. 3. Critical frequency pruning, which is a signal-to-noise analysis of slot updates and provides a pruning threshold. The authors evaluated their method on the DeepSeek-R1-Distill-Qwen 1.5B, 7B, and 14B models across the fields of PubMed, Math, Code, and General web text.
Critical strengths
- Cross-backbone knowledge reuse is an important question and is very meaningful.
Critical weaknesses
- Theorem 2 is false under the stated assumptions. Let , , and Both matrices satisfy . Their column spaces are identical, so . However, taking the unit memory vector gives whereas the claimed upper bound is This contradicts the stated inequality.
The error occurs in the assertion that the eigenvalues of equal , where . The quantities are the singular values of , not generally the eigenvalues of its symmetric part. Here, has singular value but eigenvalue .
Could the authors specify an additional basis-alignment condition that excludes this counterexample and explain whether their actual rendering procedure satisfies it?
- line 989-990: [state the definition used in the code]; line 995: [fourth slot statistic; supply from the code]; line 1165-1166: pruning at f∗ = nctx [state the value of nctx, hence of f∗, used here] yields a validation loss degradation of <1%.
This weakens the work's soundness a lot.
- The article's writing quality is bad. It is hard to read and follow; some paragraphs seem purely AI-generated, with little human polishing.
Questions for authors
Same as weakness.
the evaluation protocol and what the table actually contributes
Both points are fair.
On evaluation: validation loss is the wrong yardstick for a reasoning-tuned model and we leaned on it too heavily. We will run the model-card benchmarks (AIME, MATH-500, LiveCodeBench, GPQA-Diamond) on 7B with and without the frozen table.
On attribution: we did the ablation, on the wrong backbone. Table~2 is 1.5B with 1k samples, not the 7B main protocol --- which is exactly what you are asking about. The PubMed ordering there is trained +0.339, shuffled +0.221, random +0.201, zero +0.138, so restoring the slot-to-n-gram correspondence is worth about +0.12 nats over a permuted table with identical statistics. On Math it goes the other way: trained +0.071 sits below zero +0.087, random +0.102 and shuffled +0.128. We will not average that away. On this protocol the table carries no usable signal for Math and the paper should say so where the number appears, not in a limitations paragraph.
On prompt tuning: you are right that it is competitive. At 65K it gets +0.244/+0.192/+0.170/+0.299 against our +0.247/+0.150/+0.182/+0.284, so it is ahead in three of four domains. What we can defend is narrower --- the table is trained once on the source and carried, prompt tuning is refit per target --- and that belongs in the abstract, not where we put it.
What we will add: trained table against zero, random and shuffled on 7B under the main protocol, same target-side optimisation. Section~4.3 already admits that comparison is missing. If the margin does not replicate on 7B, we will report that and drop the claim it supports.