acceptodds
Under review as a conference paper at ICLR 2027

FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models

Abstract

Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as -grams, instead of reconstructing them through successive layers of computation. However, existing designs such as Engram treat each retrieved embedding as a monolithic unit. Each embedding is stored in its own hashed slot and modulated by a single scalar gate. As a result, polysemous patterns cannot selectively read out the components of their memory that are relevant to the context. Moreover, parameters are shared only through hash collisions, which are largely unrelated to semantics. We propose FactorEngram, a factorized -gram memory with basis-level contextual gating. FactorEngram retrieves sparsity-regularized coefficients over a dictionary of basis vectors shared across patterns, so related patterns can reuse common components. The same dictionary is also used for gating. The backbone hidden state is scored against each basis vector to gate the corresponding coefficient before reconstruction, which lets the context modulate each memory component individually. FactorEngram also covers both individual tokens and multi-token -grams, and we systematically study where the memory branch should be inserted. On 340M- and 1B-parameter Transformer backbones, FactorEngram improves language modeling and downstream task performance. Ablation studies confirm the contribution of each component and identify insertion before the attention sublayer in the middle layers as an effective configuration. All code, data, and models will be made publicly available upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, lookup-based memory, conditional memory, sparse factorization, contextual gating

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_qUooOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Authors propose an evaluate an improvement over Engram, named FactorEngram. The main idea behind FactorizedEngram that I thought is pretty cool is allow the different n-grams to share components. The authors evaluate the method on 340M and 1B Transformer backbones and report improvements in perplexity, downstream accuracy, and especially long-context retrieval.

  1. I like the motivation behind factorizing n-gram memories and allowing context-dependent access to individual components.
  2. The method is relatively simple and integrates cleanly with existing Transformers.
  3. The authors include useful ablations, including gating, retrieval, and memory placement.

However, I'm not satisfied by depth of discussion of experimental results, and hence this is a weak reject for me. Reading the paper, a few things do NOT make sense to me: What do we expect factor engrams to actually improve? Authors mention things such as polysemy, semantic sharing, some sense of sparse parameter sharing (since I believe there is an L1 loss) etc. However, the experimental results presented are language modeling, NIAH and downstream tasks. I somewhat accept language modeling benefits at 340M scale (but not 1M scale), and I definitely do not understand why NIAH is affected. Additionally, there is no qualitative study of what the FactorEngram actually learns (what are the factors). Consequently, I fear that the experiments section does not justify all claims made by the authors in this paper.

Critical strengths

The architecture is relatively simple and integrates cleanly with existing Transformers. The authors provide several useful ablations covering gating, unigram retrieval, sparsity regularization, and memory placement. The improvements on long-context retrieval are substantial.

Critical weaknesses

Experiments don't validate the motivation: The paper motivates FactorEngram through polysemy and semantic sharing, but doesn't actually test either. I'd like to see evidence that related n-grams share meaningful components and that gating helps with disambiguation.

NIAH results are confusing: It's unclear why local n-gram memory should help so much with long-context retrieval. The ablations show huge swings in NIAH performance, but there's little explanation for why. It's also strange that Engram performs worse than a plain Transformer on NIAH-3. These results need more analysis.

Ablations aren't entirely convincing: FactorEngram with scalar gating actually performs worse than Engram on several metrics. I'm not convinced that factorization itself is helping, rather than just fine-grained gating. I'd also like to see comparisons against alternatives like TN-gram.

Limited improvements and scale: At 1B, the perplexity and downstream improvements are almost nonexistent. The experiments are also relatively small, despite adding billions of lookup parameters. It's unclear whether the benefits justify the extra memory and compute costs.

Questions for authors

Can the authors demonstrate that semantically related n-grams share learned basis components, and that different contexts activate different components?

Why does FactorEngram improve NIAH so dramatically? Is there evidence that it improves attention or long-context information retrieval?

Why is Engram worse than the plain Transformer on NIAH-3, and why does FactorEngram reverse this?

How much of the improvement comes from factorization versus fine-grained gating? Could similar gains be achieved without factorization?

How does FactorEngram compare against other factorized memory approaches

2 comments

We thank reviewer qUoo for the diligent and thoughtful feedback. Below we provide our responses.


W1/Q1: Experiments don't validate the motivation: The paper motivates FactorEngram through polysemy and semantic sharing, but doesn't actually test either. I'd like to see evidence that related n-grams share meaningful components and that gating helps with disambiguation. / Can the authors demonstrate that semantically related n-grams share learned basis components, and that different contexts activate different components?

A1: We agree that the paper does not directly test polysemy or semantic sharing. The ablation in Table 2 gives indirect evidence: replacing basis-level gating with scalar gating lowers every metric. In the revised version, we will add a direct analysis and a qualitative study of the learned basis vectors, such as the n-grams with the largest coefficients on each basis.

W2/Q2/Q3: NIAH results are confusing: It's unclear why local n-gram memory should help so much with long-context retrieval. The ablations show huge swings in NIAH performance, but there's little explanation for why. It's also strange that Engram performs worse than a plain Transformer on NIAH-3. These results need more analysis. / Why does FactorEngram improve NIAH so dramatically? Is there evidence that it improves attention or long-context information retrieval? / Why is Engram worse than the plain Transformer on NIAH-3, and why does FactorEngram reverse this?

A2: The NIAH gains depend on three components of FactorEngram (Tables 2 and 4). On NIAH-3:

  • Replacing basis-level gating with scalar gating: 58.3 → 9.4
  • Removing unigram retrieval: 58.3 → 23.0
  • Inserting memory before the FFN instead of before attention: 58.3 → 18.8

Both models with scalar gating score below the Transformer on NIAH-3 (Engram 15.0, FactorEngram with scalar gating 9.4, Transformer 22.1). Basis-level gating raises the score to 58.3. This suggests that the gating method is the main factor. We will add an analysis of the underlying mechanism in the revised version.

W3/Q4/Q5: Ablations aren't entirely convincing: FactorEngram with scalar gating actually performs worse than Engram on several metrics. I'm not convinced that factorization itself is helping, rather than just fine-grained gating. I'd also like to see comparisons against alternatives like TN-gram. / How much of the improvement comes from factorization versus fine-grained gating? Could similar gains be achieved without factorization? / How does FactorEngram compare against other factorized memory approaches

A3: Our contribution lies in the joint design of factorized memory and basis-level gating, rather than factorization alone. As stated in our contributions (Section 1) and ablation discussion (Section 4.3), the gating reuses the reconstruction dictionary: factorization provides a shared basis space, and context-dependent gating modulates each basis component individually before reconstruction. The ablations highlight this interaction. With scalar gating, factorization does not outperform Engram on most metrics, while combining it with basis-level gating gives the best results on five of six metrics in Table 2. These results do not rule out other fine-grained gating designs without factorization, and we will add a variant with dense embeddings and fine-grained gating in the revised version.

TN-gram is concurrent work, and Table 5 summarizes how it differs from FactorEngram. We will add a direct comparison in the revised version.

W4: Limited improvements and scale: At 1B, the perplexity and downstream improvements are almost nonexistent. The experiments are also relatively small, despite adding billions of lookup parameters. It's unclear whether the benefits justify the extra memory and compute costs.

A4: We agree that the gains in perplexity and average downstream accuracy are small at 1B. Engram also shows no perplexity gain over the Transformer at this scale (WikiText 15.60 vs. 15.57). FactorEngram still improves LAMBADA perplexity (11.73 → 11.39) and all three NIAH tasks (by 9.3 to 27.6 points over Engram).

To test whether the extra parameters are worth it, we compared against E1, a 5B-A0.5B standard Transformer with full attention, MLA, and latent MoE. In E1-FactorEngram, we replace 1.3B of E1's sparse parameters with FactorEngram lookup-branch parameters, so the total parameter count is the same. Both of them are trained with 200B tokens.

Table R1. E1 vs. E1-FactorEngram.

BenchmarkE1E1-FactorEngram
C-Eval26.4028.80
CMMLU25.8030.20
MMLU-Pro16.4015.20
GSM8K4.803.80
CMATH29.4029.40
MATH4.204.60
MBPP10.9113.74
MBPP+16.2215.96
BBH23.0025.60
Average17.4618.59

Table R2. Cost comparison.

MetricE1E1-FactorEngramChange
Training throughput (tokens/s)363015342612−5.6%
Training peak memory (GiB per GPU)47.151.3+8.9%
Prefill throughput (tokens/s)8,6978,510−2.2%
Decode throughput (tokens/s)4038−5.0%
Inference peak memory (GiB)18.0518.84+4.4%
Inference FLOPs per token921M943M+2.4%

With the same total parameter count, moving 1.3B parameters from experts to the FactorEngram lookup branch raises the average score from 17.46 to 18.59, while prefill and decode throughput decrease by only 2.2% and 5.0% and inference peak memory increases by only 4.4% (Table R2).

Reviewer_cqztOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FactorEngram, a lookup‑based n‑gram memory module built upon prior Engram architecture. It factorizes n‑gram memory via shared dictionary and basis‑level gating, supporting unigram/bigram/trigram patterns. Empirical results show strong gains especially on NIAH long‑context retrieval for 340M and 1B models. The paper provides ablation studies on architectural components, sparsity weight and memory insertion positions.

Critical strengths
  1. Well‑motivated: points out critical limitation of original Engram: scalar gate cannot handle polysemous n‑gram patterns, and parameter sharing only via hash collision.
  2. Good ablation suite, investigating gating mechanism, unigram branch, sparsity regularization and insertion location.
  3. Large improvement on needle‑in‑haystack long‑context retrieval task for 340M model.
Critical weaknesses
  1. Experiments are not sufficient, limited to model scales of 340M and 1B parameters. Moreover, the empirical comparisons are restricted to Engram, lacking experimental evaluations against other Lookup-based memory architectures.
  2. In 1B setting, the downstream accuracy shows marginal improvement, raising concern whether gains transfer to larger LLMs, more discussions and experiments are necessary.
  3. Long‑context evaluation heavily relies on synthetic NIAH benchmark, lacking real‑world long‑generation benchmarks. The large improvement is not enough to support the method of FactorEngram.
  4. Missing latency and memory overhead analysis and the related comparison. Lookup‑based memory’s practical value heavily depends on computation and time cost.
  5. There is not experiments verifying polysemy disambiguation and long contexts claimed in motivation.
Questions for authors

Given that algorithm hyper‑parameters are obtained via ablations on the 340M model, could there be adaptation issues when applying them to 1B or larger models?

1 comment

We thank reviewer cqzt for the diligent and thoughtful feedback. Below we provide our responses.


W1: Experiments are not sufficient, limited to model scales of 340M and 1B parameters. Moreover, the empirical comparisons are restricted to Engram, lacking experimental evaluations against other Lookup-based memory architectures.

A1: For scale, we added a comparison with a larger model. Please see A2.

We use Engram as the main baseline because it is the most representative lookup-based n-gram memory and is used in DeepSeek-V4.1-Flash. FactorEngram is built directly on Engram, so the comparison isolates the effect of the new memory representation and gating. TN-gram is concurrent work, and Table 5 summarizes how it differs from FactorEngram. We will add direct comparisons with TN-gram and other lookup-based memory methods in the revised version.

W2: In 1B setting, the downstream accuracy shows marginal improvement, raising concern whether gains transfer to larger LLMs, more discussions and experiments are necessary.

A2: We agree that the gains in perplexity and average downstream accuracy are small at 1B. Engram also shows no perplexity gain over the Transformer at this scale (WikiText 15.60 vs. 15.57). FactorEngram still improves LAMBADA perplexity (11.73 → 11.39) and all three NIAH tasks (by 9.3 to 27.6 points over Engram).

To test a larger model, we compared against E1, a 5B-A0.5B standard Transformer with full attention, MLA, and latent MoE. In E1-FactorEngram, we replace 1.3B of E1's sparse parameters with FactorEngram lookup-branch parameters. Results are shown in Table R1. Both of them are trained with 200B tokens.

Table R1. E1 vs. E1-FactorEngram.

BenchmarkE1E1-FactorEngram
C-Eval26.4028.80
CMMLU25.8030.20
MMLU-Pro16.4015.20
GSM8K4.803.80
CMATH29.4029.40
MATH4.204.60
MBPP10.9113.74
MBPP+16.2215.96
BBH23.0025.60
Average17.4618.59

E1-FactorEngram raises the average score from 17.46 to 18.59.

W3: Long-context evaluation heavily relies on synthetic NIAH benchmark, lacking real-world long-generation benchmarks. The large improvement is not enough to support the method of FactorEngram.

A3: The NIAH gains appear at both 340M and 1B. We agree that real-world long-context tasks are missing. We will add them in the revised version and list this as a limitation.

W4: Missing latency and memory overhead analysis and the related comparison. Lookup-based memory's practical value heavily depends on computation and time cost.

A4: Neither our Engram reproduction nor FactorEngram implements the test-time sparse table loading used in the original Engram paper. Both keep all tables in GPU memory, so they are compared under the same conditions. FactorEngram's coefficient tables are addressed only by input tokens, so the same offloading scheme can also be applied in future updates.

Table R2 compares the training and inference cost of E1-FactorEngram and E1. The inference overhead is small: prefill and decode throughput decrease by 2.2% and 5.0%, inference peak memory increases by 4.4%, and FLOPs per token increase by 2.4%.

Table R2. Cost comparison.

MetricE1E1-FactorEngramChange
Training throughput (tokens/s)363015342612−5.6%
Training peak memory (GiB per GPU)47.151.3+8.9%
Prefill throughput (tokens/s)8,6978,510−2.2%
Decode throughput (tokens/s)4038−5.0%
Inference peak memory (GiB)18.0518.84+4.4%
Inference FLOPs per token921M943M+2.4%

We will report the parameter counts and FLOPs of the 340M and 1B models in the revised version, and add test-time sparse table loading in future work.

W5: There is not experiments verifying polysemy disambiguation and long contexts claimed in motivation.

A5: We agree that the paper does not directly test polysemy disambiguation. The ablation in Table 2 gives indirect evidence: replacing basis-level gating with scalar gating lowers every metric (e.g., NIAH-3 drops from 58.3 to 9.4). We will add a direct analysis in the revised version. For the long-context part of the motivation, please see A3.

Q1: Given that algorithm hyper-parameters are obtained via ablations on the 340M model, could there be adaptation issues when applying them to 1B or larger models?

A6: Both the 340M and 1B models have 24 layers, so the insertion layers (10 and 12) are at the same relative depth. We used the same settings at 1B, and FactorEngram still improves over Engram on LAMBADA perplexity and all three NIAH tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.