acceptodds
Under review as a conference paper at ICLR 2027

NGM: A Plug-and-Play Training-Free Memory Module for LLMs

Abstract

Recent studies have introduced memory-augmented architectures that decouple knowledge storage from neural computation, enabling more direct access to stored representations. However, existing approaches often rely on learned memory embeddings or explicit memory and retrieval components, requiring additional training or increasing architectural complexity. To address these limitations, we propose N-gram Memory (NGM), a training-free, plug-and-play module that repurposes a backbone model's pretrained token embedding space as memory. NGM consists of a Causal N-Gram Encoder and a Cosine-Gated Memory Injector. The former constructs N-gram representations by directly aggregating pretrained token embeddings, while the latter uses cosine similarity with ReLU gating to adaptively control the amount of retrieved memory injected into contextual representations. Unlike conventional memory-augmented architectures, NGM requires neither additional embedding training nor a separately learned memory bank or external retrieval infrastructure. We evaluate NGM on ten benchmarks across six language-model backbones spanning two model families and multiple scales, including four Qwen3 models (0.6B, 1.7B, 4B, and 8B) and two Llama-3.2-Instruct models (1B and 3B). NGM consistently improves the ten-task average across all six backbones. Notably, it improves HumanEval by 4.26, HumanEval+ by 2.41, and GPQA-Diamond by 6.56 percentage points. Furthermore, NGM extends beyond text-only language models to vision-language models, with Qwen3-VL-2B achieving improvements across all five VLM benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_grw9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes N-gram Memory (NGM), a training-free module that enhances frozen language models by mean-pooling pretrained token embeddings over local windows and reinjecting them through cosine-similarity gating. The paper reports broadly consistent improvements across multiple model families and diverse tasks, with preliminary results extending to a vision-language model.

Critical strengths
  1. A simple and accessible intervention. NGM reuses frozen token embeddings and requires neither additional training nor a separate learned memory table. Its pooling, gating, and residual injection operations are straightforward to understand and implement. This makes the approach potentially useful when retraining is unavailable.

  2. Broad evaluation with generally consistent gains. The experiments cover multiple language model families and diverse tasks, with positive average gains for every evaluated text backbone. Preliminary experiments also extend the evaluation to a vision-language model. This breadth provides encouraging evidence of the method’s applicability across models and tasks.

Critical weaknesses

The experiments primarily compare NGM with the unmodified backbone, without representative training-free alternatives or relevant training-based methods. Although the gains are generally positive, the average improvements are modest, and it is difficult to assess NGM’s relative competitiveness without comparisons with representative alternatives. Results for representative methods under the same evaluation settings, together with their training costs, additional parameter counts, and inference costs, would better establish NGM’s competitiveness and the trade-off between performance and cost.

Questions for authors

The mean-pooled 2+3-gram injection can be rewritten exactly as a constrained, dynamically weighted sum of the three most recent token embeddings. What benefit does multi-scale pooling provide over directly gating and injecting these embeddings individually? A comparison using the same local window, injection layers, and comparable injection magnitude would clarify this design choice.

Reviewer_rocuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes N-gram Memory (NGM), a training-free module that reuses pretrained token embeddings to construct local N-gram representations and injects them into decoder hidden states through cosine-similarity-based gating. The method introduces no additional trainable parameters and demonstrates improvements across six language-model backbones and ten benchmarks, with preliminary extensions to vision-language models. While the simplicity and applicability of the approach are appealing, I have concerns regarding its methodological novelty, the lack of competitive comparisons with related approaches, and the limited evidence supporting the proposed memory mechanism. The empirical results are encouraging, but these limitations currently lead me to lean toward a clear reject.

Critical strengths
  1. The proposed method is simple and easy to integrate into existing pretrained language models. Reusing pretrained token embeddings without introducing additional trainable parameters or external retrieval infrastructure provides a practical approach to inference-time model augmentation.
  2. The experiments cover six backbones from the Qwen3 and Llama-3.2 families across ten benchmarks. The consistent improvements in average performance, particularly on HumanEval and GPQA-Diamond, provide encouraging evidence that the approach can benefit different model architectures and scales.
  3. The paper includes component ablations, representation-alignment analyses, and wall-clock latency measurements. These evaluations help characterize the behavior of the proposed module and provide useful insights into its design choices and practical overhead.
Critical weaknesses
  1. The methodological novelty appears somewhat limited. The core operation consists of averaging pretrained token embeddings over local windows and injecting the resulting representations through a similarity-weighted residual connection. Although Section 4.3 examines ReLU gating, fusion strategies, and N-gram window sizes, it does not isolate the benefits of the fundamental memory construction and injection operations. In particular, the "− ReLU gating" variant still retains cosine-similarity weighting. Comparisons with unigram-only injection and completely ungated N-gram injection would help clarify whether the improvements arise from the proposed memory mechanism or simpler embedding-based residual augmentation.
  2. The experimental comparison with existing methods is insufficient. Table 1 only compares each original backbone with its NGM-augmented counterpart, without evaluating other competitive model-augmentation approaches. Although Section 2 discusses recent memory-based methods such as Engram, SCONE, and L3, the paper provides no quantitative comparison to establish NGM's relative effectiveness. Including representative training-free alternatives under matched evaluation settings, along with comparisons or quantitative trade-off analyses against recent memory-augmented approaches, would substantially strengthen the experimental evidence. Since some of these methods require additional training or parameters, differences in training requirements, parameter costs, and inference overhead should be clearly accounted for to ensure fair comparisons.
  3. The mechanistic analysis does not fully substantiate the interpretation of NGM as an effective local-memory retrieval mechanism. Figure 1 shows stronger alignment for actual N-gram representations than for shuffled or random controls, while Figure 3 demonstrates diagonal dominance in cross-position similarity. However, these observations may partly reflect the inherent relationship between decoder hidden states and their corresponding input token embeddings, especially because each N-gram representation includes the current token. Additional controls that exclude the current token, together with downstream evaluations using shuffled or randomized memory candidates, would help distinguish meaningful N-gram retrieval from basic input-representation similarity.
  4. The reported performance improvements are generally modest, and their statistical reliability is not sufficiently established. Table 1 reports average gains of approximately 0.98–1.72 percentage points across the six backbones, while several individual improvements are small or negligible. Although standard deviations are provided for NGM over three runs, corresponding baseline variability and statistical significance analyses are not reported. Given the use of stochastic decoding, reporting repeated baseline measurements and paired statistical comparisons would help determine whether the smaller gains are robust.
  5. The effectiveness of the selected injection layers and output scale requires further investigation. Appendix Table 5 uses the same two injection layers and a fixed scale of 0.1 across models with substantially different depths, including the 16-layer Llama-1B and 36-layer Qwen3 models. Although the appendix explains that these placements are heuristic, the paper does not provide a sensitivity analysis to assess their robustness. Evaluating different injection positions, output scales, and numbers of injection layers would strengthen the practical applicability of the method.
  6. The connection between the proposed local-memory mechanism and the improvements on complex reasoning tasks remains insufficiently explained. The method is motivated by the reuse of local lexical and symbolic patterns, yet some of its largest gains occur on benchmarks such as GPQA-Diamond and HumanEval. The current analysis and qualitative examples do not establish how local N-gram aggregation contributes to these higher-level reasoning improvements. More targeted experiments on repeated identifiers, local factual associations, or controlled lexical dependencies would help clarify which capabilities benefit most directly from NGM.
  7. The practical efficiency claims are not fully supported by the current implementation. Section 3.5 describes an incremental caching strategy with constant additional decoding cost relative to prefix length, but Section 4.6 acknowledges that the evaluated implementation recomputes N-gram features over the full prefix. Consequently, the measured overhead reaches 16.0% for prefill and 9.9% for decoding at 2048 tokens. Implementing and benchmarking the proposed streaming cache, particularly for longer contexts and different batch sizes, would be important for validating the claimed efficiency and deployment benefits.
  8. The multimodal evaluation is preliminary and does not sufficiently support broad generalization claims. Table 4 evaluates only Qwen3-VL-2B, and two of the five reported benchmarks primarily assess language capabilities rather than visual understanding. Several improvements are also very small, such as 0.03 points on MMLU-Redux and 3 points out of 1000 on OCRBench, without repeated-run statistics. Furthermore, the paper acknowledges that the reported TruthfulQA results use a legacy evaluation metric that is not directly comparable with the main experiments. Additional multimodal backbones, more diverse vision-language tasks, and statistical analyses would provide stronger evidence of transferability.
Questions for authors

See Weaknesses.

Reviewer_riKXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes N-gram Memory (NGM), a plug-and-play and training-free memory module that improves the performance of pretrained LLMs without introducing additional trainable parameters or external memory infrastructure. Specifically, the authors first propose a Causal N-gram Encoder that constructs multi-scale local memory representations by averaging pretrained token embeddings within causal sliding windows, thereby reusing the existing embedding space without learning a separate N-gram memory table. Next, the authors introduce a Cosine-Gated Memory Injector that computes the cosine similarity between the constructed N-gram representations and the contextual hidden states, and applies ReLU gating to selectively retain positively aligned memory signals. The resulting memory representations are then aggregated and injected into selected Transformer layers through scaled residual connections, while keeping the backbone model parameters frozen. The authors further conduct alignment analyses and ablation studies to investigate the compatibility between N-gram representations and hidden states, as well as the effects of different memory injection configurations. Experiments across six LLM backbones and ten benchmarks demonstrate consistent improvements in average performance, with additional results suggesting that NGM can also be applied to vision-language models.

Critical strengths
  1. The authors propose NGM, a lightweight memory augmentation module that combines causal multi-scale N-gram representation construction with cosine-gated memory injection to enhance pretrained LLMs.
  2. The proposed method reuses pretrained token embeddings to construct local memory representations and employs a parameter-free gating mechanism for memory injection, enabling integration into frozen LLMs without additional training, learnable parameters, or external memory infrastructure.
  3. The authors conduct extensive experiments across six LLM backbones from two model families and ten benchmarks, demonstrating consistent improvements in average performance, and further provide ablation studies, mechanistic analyses, and preliminary experiments on vision-language models to evaluate the proposed method.
Critical weaknesses
  1. There is a gap between the motivation and the proposed methodology. The paper motivates NGM by arguing that standard Transformers lack explicit lookup primitives for stable local patterns, requiring such patterns to be reconstructed through dense computations. However, NGM introduces additional operations without reducing the original dense computations. Moreover, the paper does not sufficiently demonstrate that NGM effectively serves as a local lookup mechanism rather than merely enhancing local representations.
  2. Insufficient justification for treating averaged token embeddings as N-gram memory. NGM constructs N-gram representations by simply averaging pretrained token embeddings, which is permutation-invariant and cannot preserve token order. The paper does not sufficiently justify why these representations can serve as effective N-gram memory.
  3. Unclear source of performance improvements. The gains of NGM may arise from general residual enhancement or local feature smoothing rather than actual memory reuse. However, the paper lacks comparisons with simple alternatives, such as hidden-state averaging, residual scaling, and random memory injection, making it difficult to isolate the contribution of the proposed memory mechanism.
  4. Insufficient analysis of N-gram window sizes. NGM primarily uses 2-gram and 3-gram windows, which may be insufficient to capture longer lexical or symbolic patterns. Although the paper evaluates window sizes up to 4, it lacks experiments with larger windows and more diverse combinations (e.g., {1, 3, 7}).
  5. Limited analysis of memory injection layer placement. The injection layers are selected heuristically without a systematic sensitivity analysis. Since different backbones have different depths and representation characteristics, using the same absolute layer indices may not be optimal. The paper lacks comparisons across early, middle, and late layers to justify the generality of its injection strategy.
  6. Insufficient statistical reliability of experimental results. The main experiments report the mean and standard deviation of only three runs for NGM, while the baseline variance is not reported in the same manner. Given that some improvements are relatively small, more repeated evaluations and statistical significance analyses are needed to establish the reliability of the reported gains.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.