acceptodds
Under review as a conference paper at ICLR 2027

HASH-EMBEDDING MEMORY INJECTION: CROSS SCALE TRANSFERABLE DIFFERENTIAL MEMORY FOR LARGE LANGUAGE MODELS

Abstract

Fine-tuning is expensive and adapters such as LoRA are tied to one base model, so domain knowledge is voided at every backbone upgrade. We propose Hash-Embedding Memory Injection (HEMI), which stores knowledge in a hash -gram table trained once on a source backbone and reused across backbones of the same family and pre-training lineage through each backbone's anchor basis. Three contributions: (1) a critical-frequency theory giving a closed-form and the table-size bound ; pruning at costs under held-out loss over configurations in domains; (2) differential injection of only the residual between stored memory and the hidden state, under evidential gating and RMS balancing—ablations show the gate and the adapter decide whether a transferred table helps or hurts (removing the adapter hurts in all three settings; removing the gate does so in two and leaves of the gain in the third), while dropping differencing or RMS costs – nats; (3) anchor rendering via thin QR factorization, plus a migration stability theorem bounding direction error by three additive sources: finite-frequency noise, Stage-2 drift, and anchor-subspace mismatch. On DeepSeek-R1-Distill-Qwen, a table trained on B improves the B target in four domains by to nats with target-side parameters ( of LoRA at ), on top of a M-parameter table trained once on the source; the gain does not survive on the B target ( on PubMed, on Math; against on B under a second protocol), where a zero-table control gains more than the trained table ( on PubMed, on Math), so the content of the table contributes nothing there.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
memory-augmented language models, parameter-efficient knowledge injection, cross-model knowledge transfer, n-gram hash memory, domain adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2UUsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The research team proposed a new approach, Hash-Embedding Memory Injection, to transfer the domain knowledge from a smaller model to a larger model or transfer between different base models. To do so, they proposed three components: 1. anchor rendering: trying to provide a backbone-independent space for the memory, so different models can access the same information. 2. Differential injection, which use residual, gate, RMS balance, and adapter to inject the memory. 3. Critical frequency pruning, which is a signal-to-noise analysis of slot updates and provides a pruning threshold. The authors evaluated their method on the DeepSeek-R1-Distill-Qwen 1.5B, 7B, and 14B models across the fields of PubMed, Math, Code, and General web text.

Critical strengths
  1. Cross-backbone knowledge reuse is an important question and is very meaningful.
Critical weaknesses
  1. Theorem 2 is false under the stated assumptions. Let D=2D=2, dm=1d_m=1, and ΦS=(1,0)⊤,ΦB=(−1,0)⊤.\Phi_S=(1,0)^\top,\qquad \Phi_B=(-1,0)^\top. Both matrices satisfy ΦS⊤ΦS=ΦB⊤ΦB=1\Phi_S^\top\Phi_S=\Phi_B^\top\Phi_B=1. Their column spaces are identical, so θmax⁡=0\theta_{\max}=0. However, taking the unit memory vector v^=1\hat v=1 gives ∥ΦSv^−ΦBv^∥2=2,\|\Phi_S\hat v-\Phi_B\hat v\|_2=2, whereas the claimed upper bound is 2∥v^∥2sin⁡(θmax⁡/2)=02\lVert\hat{v}\rVert_2\sin(\theta_{\max}/2)=0 This contradicts the stated inequality.

The error occurs in the assertion that the eigenvalues of (W+W⊤)/2(W+W^\top)/2 equal cos⁡θi\cos\theta_i, where W=ΦS⊤ΦBW=\Phi_S^\top\Phi_B. The quantities cos⁡θi\cos\theta_i are the singular values of WW, not generally the eigenvalues of its symmetric part. Here, W=[−1]W=[-1] has singular value 11 but eigenvalue −1-1.

Could the authors specify an additional basis-alignment condition that excludes this counterexample and explain whether their actual rendering procedure satisfies it?

  1. line 989-990: [state the definition used in the code]; line 995: [fourth slot statistic; supply from the code]; line 1165-1166: pruning at f∗ = nctx [state the value of nctx, hence of f∗, used here] yields a validation loss degradation of <1%.

This weakens the work's soundness a lot.

  1. The article's writing quality is bad. It is hard to read and follow; some paragraphs seem purely AI-generated, with little human polishing.
Questions for authors

Same as weakness.

Reviewer_wTb3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HEMI trains a hashed n-gram memory table on a small source model, freezes it, and reuses it for a larger target model through a QR basis derived from anchors and a small trained alignment package. The method combines differential injection, gating mechanisms, and frequency-based pruning, and proposes a transfer stability bound. Transfer from 1.5B to 7B is reported to improve domain language modeling loss with very few target-side parameters, but transfer to a 14B model is substantially degraded. The paper includes studies on injection, anchors, content, and pruning, but its proposed theory and some numerical evidence have substantive problems.

Critical strengths
  1. Reusing an explicit memory table across backbone widths is a useful problem. Table 1 reports positive 1.5B-to-7B loss gains in four domains using 65,811 target-side parameters, with the source table cost stated separately. This is a potentially useful design point between target fine-tuning and running an additional decoder at inference.

  2. The paper reports informative negative evidence: the 14B transfer boundary, gains from an empty table, and cases where prompt tuning or other reusable methods are competitive. It also distinguishes reduced-protocol Engram results from the main comparison. This transparency is valuable and the negative transfer result is not itself a reason for rejection.

Critical weaknesses
  1. The rendering bound in Appendix A (pp.13–14) is false under its stated assumptions: eigenvalues of (W+W^T)/2 need not equal the singular values of W. Even with the specified positive-diagonal QR and raw-anchor Procrustes construction, source/target anchor columns I_2 and [[1,1],[0,1]] give zero principal angles but a rendering mismatch of 0.4595 for e_1. Thus subspace agreement does not bound same-coordinate rendering, and Theorem 1 inherits an unsupported term. A corrected basis-difference bound or an additional justified alignment condition is needed.

  2. The pruning evidence cannot be reconciled as written. Table 10 (p.22) implies 31.25M total hits, yet its PubMed retained slots at threshold 200 alone require at least 38.444M; the other domains show the same conflict. B.5 (p.19) leaves gate features e3/e7 undefined, and C.2.1 leaves the actual n_ctx/f* value as a placeholder. These could be reporting errors, but they affect central components and require corrected definitions and auditable counts before the pruning claim can be assessed.

Questions for authors
  1. Theory clarification: Can you give a corrected rendering bound that accounts for coordinate-frame rotation, or state and justify an additional basis-alignment condition satisfied by the actual algorithm? Please address the positive-diagonal-QR/Procrustes counterexample above and explain the consequence for Theorem 1.

  2. Reporting clarification: What exactly are the frequency denominator and threshold convention in Table 10, and which entries must be corrected to reconcile the hit totals? Please also provide the missing gate-feature definitions and actual n_ctx/f* used for the 48-configuration experiment.

  3. Evidence request: For the successful 7B transfer under the main protocol, does the trained table outperform zero/random/shuffled tables with the same target-side optimization? Source-side attribution and the failed 14B setting do not settle whether transferred table content explains the central 7B gain.

Reviewer_5797OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HEMI, a method for transferring in-domain knowledge from one model to another, similar model. HEMI is an Engram-style n-gram hash table that is trained with DeepSeek-R1-Distill-Qwen-1.5B and mounted on the 7B model together with a trained alignment pack. Performance is measured as the reduction in validation loss.

Critical strengths
  1. The idea is good: when the backbone is upgraded, knowledge already trained with the current model can be carried over. The additional inference cost is also low.
Critical weaknesses
  1. The evaluation is unconventional. DeepSeek-R1-Distill-Qwen-1.5B already has reported results on well-recognized benchmarks, such as math (AIME, MATH-500), code (LiveCodeBench, CodeForces) and GPQA Diamond. Instead, the paper uses validation loss, which is rarely used to measure the performance of a reasoning or instruction-tuned model like DeepSeek-R1-Distill-Qwen-1.5B. The small changes in validation loss are not convincing evidence of downstream gains.

  2. Even on validation loss, the experiments do not show that the trained or transferred table actually helps the target. The prompt-tuning control is competitive on average, and a near-empty table beats the trained one in several settings. The 7B gain may therefore come mostly from the alignment pack fitting the domain.

Questions for authors
  1. Please evaluate the model on the standard benchmarks reported in its public model card.

  2. Please show a clear margin of the trained table over the zero table.

1 comment
AuthorsOpenReview ↗

the evaluation protocol and what the table actually contributes

Both points are fair.

On evaluation: validation loss is the wrong yardstick for a reasoning-tuned model and we leaned on it too heavily. We will run the model-card benchmarks (AIME, MATH-500, LiveCodeBench, GPQA-Diamond) on 7B with and without the frozen table.

On attribution: we did the ablation, on the wrong backbone. Table~2 is 1.5B with 1k samples, not the 7B main protocol --- which is exactly what you are asking about. The PubMed ordering there is trained +0.339+0.339, shuffled +0.221+0.221, random +0.201+0.201, zero +0.138+0.138, so restoring the slot-to-nn-gram correspondence is worth about +0.12+0.12 nats over a permuted table with identical statistics. On Math it goes the other way: trained +0.071+0.071 sits below zero +0.087+0.087, random +0.102+0.102 and shuffled +0.128+0.128. We will not average that away. On this protocol the table carries no usable signal for Math and the paper should say so where the number appears, not in a limitations paragraph.

On prompt tuning: you are right that it is competitive. At 65K it gets +0.244/+0.192/+0.170/+0.299+0.244/+0.192/+0.170/+0.299 against our +0.247/+0.150/+0.182/+0.284+0.247/+0.150/+0.182/+0.284, so it is ahead in three of four domains. What we can defend is narrower --- the table is trained once on the source and carried, prompt tuning is refit per target --- and that belongs in the abstract, not where we put it.

What we will add: trained table against zero, random and shuffled on 7B under the main protocol, same target-side optimisation. Section~4.3 already admits that comparison is missing. If the margin does not replicate on 7B, we will report that and drop the claim it supports.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.