- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes N-gram Memory (NGM), a training-free module that enhances frozen language models by mean-pooling pretrained token embeddings over local windows and reinjecting them through cosine-similarity gating. The paper reports broadly consistent improvements across multiple model families and diverse tasks, with preliminary results extending to a vision-language model.
Critical strengths
-
A simple and accessible intervention. NGM reuses frozen token embeddings and requires neither additional training nor a separate learned memory table. Its pooling, gating, and residual injection operations are straightforward to understand and implement. This makes the approach potentially useful when retraining is unavailable.
-
Broad evaluation with generally consistent gains. The experiments cover multiple language model families and diverse tasks, with positive average gains for every evaluated text backbone. Preliminary experiments also extend the evaluation to a vision-language model. This breadth provides encouraging evidence of the method’s applicability across models and tasks.
Critical weaknesses
The experiments primarily compare NGM with the unmodified backbone, without representative training-free alternatives or relevant training-based methods. Although the gains are generally positive, the average improvements are modest, and it is difficult to assess NGM’s relative competitiveness without comparisons with representative alternatives. Results for representative methods under the same evaluation settings, together with their training costs, additional parameter counts, and inference costs, would better establish NGM’s competitiveness and the trade-off between performance and cost.
Questions for authors
The mean-pooled 2+3-gram injection can be rewritten exactly as a constrained, dynamically weighted sum of the three most recent token embeddings. What benefit does multi-scale pooling provide over directly gating and injecting these embeddings individually? A comparison using the same local window, injection layers, and comparable injection magnitude would clarify this design choice.