- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Authors propose an evaluate an improvement over Engram, named FactorEngram. The main idea behind FactorizedEngram that I thought is pretty cool is allow the different n-grams to share components. The authors evaluate the method on 340M and 1B Transformer backbones and report improvements in perplexity, downstream accuracy, and especially long-context retrieval.
- I like the motivation behind factorizing n-gram memories and allowing context-dependent access to individual components.
- The method is relatively simple and integrates cleanly with existing Transformers.
- The authors include useful ablations, including gating, retrieval, and memory placement.
However, I'm not satisfied by depth of discussion of experimental results, and hence this is a weak reject for me. Reading the paper, a few things do NOT make sense to me: What do we expect factor engrams to actually improve? Authors mention things such as polysemy, semantic sharing, some sense of sparse parameter sharing (since I believe there is an L1 loss) etc. However, the experimental results presented are language modeling, NIAH and downstream tasks. I somewhat accept language modeling benefits at 340M scale (but not 1M scale), and I definitely do not understand why NIAH is affected. Additionally, there is no qualitative study of what the FactorEngram actually learns (what are the factors). Consequently, I fear that the experiments section does not justify all claims made by the authors in this paper.
Critical strengths
The architecture is relatively simple and integrates cleanly with existing Transformers. The authors provide several useful ablations covering gating, unigram retrieval, sparsity regularization, and memory placement. The improvements on long-context retrieval are substantial.
Critical weaknesses
Experiments don't validate the motivation: The paper motivates FactorEngram through polysemy and semantic sharing, but doesn't actually test either. I'd like to see evidence that related n-grams share meaningful components and that gating helps with disambiguation.
NIAH results are confusing: It's unclear why local n-gram memory should help so much with long-context retrieval. The ablations show huge swings in NIAH performance, but there's little explanation for why. It's also strange that Engram performs worse than a plain Transformer on NIAH-3. These results need more analysis.
Ablations aren't entirely convincing: FactorEngram with scalar gating actually performs worse than Engram on several metrics. I'm not convinced that factorization itself is helping, rather than just fine-grained gating. I'd also like to see comparisons against alternatives like TN-gram.
Limited improvements and scale: At 1B, the perplexity and downstream improvements are almost nonexistent. The experiments are also relatively small, despite adding billions of lookup parameters. It's unclear whether the benefits justify the extra memory and compute costs.
Questions for authors
Can the authors demonstrate that semantically related n-grams share learned basis components, and that different contexts activate different components?
Why does FactorEngram improve NIAH so dramatically? Is there evidence that it improves attention or long-context information retrieval?
Why is Engram worse than the plain Transformer on NIAH-3, and why does FactorEngram reverse this?
How much of the improvement comes from factorization versus fine-grained gating? Could similar gains be achieved without factorization?
How does FactorEngram compare against other factorized memory approaches
2 comments
W4: Limited improvements and scale: At 1B, the perplexity and downstream improvements are almost nonexistent. The experiments are also relatively small, despite adding billions of lookup parameters. It's unclear whether the benefits justify the extra memory and compute costs.
A4: We agree that the gains in perplexity and average downstream accuracy are small at 1B. Engram also shows no perplexity gain over the Transformer at this scale (WikiText 15.60 vs. 15.57). FactorEngram still improves LAMBADA perplexity (11.73 → 11.39) and all three NIAH tasks (by 9.3 to 27.6 points over Engram).
To test whether the extra parameters are worth it, we compared against E1, a 5B-A0.5B standard Transformer with full attention, MLA, and latent MoE. In E1-FactorEngram, we replace 1.3B of E1's sparse parameters with FactorEngram lookup-branch parameters, so the total parameter count is the same. Both of them are trained with 200B tokens.
Table R1. E1 vs. E1-FactorEngram.
| Benchmark | E1 | E1-FactorEngram |
|---|---|---|
| C-Eval | 26.40 | 28.80 |
| CMMLU | 25.80 | 30.20 |
| MMLU-Pro | 16.40 | 15.20 |
| GSM8K | 4.80 | 3.80 |
| CMATH | 29.40 | 29.40 |
| MATH | 4.20 | 4.60 |
| MBPP | 10.91 | 13.74 |
| MBPP+ | 16.22 | 15.96 |
| BBH | 23.00 | 25.60 |
| Average | 17.46 | 18.59 |
Table R2. Cost comparison.
| Metric | E1 | E1-FactorEngram | Change |
|---|---|---|---|
| Training throughput (tokens/s) | 363015 | 342612 | −5.6% |
| Training peak memory (GiB per GPU) | 47.1 | 51.3 | +8.9% |
| Prefill throughput (tokens/s) | 8,697 | 8,510 | −2.2% |
| Decode throughput (tokens/s) | 40 | 38 | −5.0% |
| Inference peak memory (GiB) | 18.05 | 18.84 | +4.4% |
| Inference FLOPs per token | 921M | 943M | +2.4% |
With the same total parameter count, moving 1.3B parameters from experts to the FactorEngram lookup branch raises the average score from 17.46 to 18.59, while prefill and decode throughput decrease by only 2.2% and 5.0% and inference peak memory increases by only 4.4% (Table R2).
We thank reviewer qUoo for the diligent and thoughtful feedback. Below we provide our responses.
W1/Q1: Experiments don't validate the motivation: The paper motivates FactorEngram through polysemy and semantic sharing, but doesn't actually test either. I'd like to see evidence that related n-grams share meaningful components and that gating helps with disambiguation. / Can the authors demonstrate that semantically related n-grams share learned basis components, and that different contexts activate different components?
A1: We agree that the paper does not directly test polysemy or semantic sharing. The ablation in Table 2 gives indirect evidence: replacing basis-level gating with scalar gating lowers every metric. In the revised version, we will add a direct analysis and a qualitative study of the learned basis vectors, such as the n-grams with the largest coefficients on each basis.
W2/Q2/Q3: NIAH results are confusing: It's unclear why local n-gram memory should help so much with long-context retrieval. The ablations show huge swings in NIAH performance, but there's little explanation for why. It's also strange that Engram performs worse than a plain Transformer on NIAH-3. These results need more analysis. / Why does FactorEngram improve NIAH so dramatically? Is there evidence that it improves attention or long-context information retrieval? / Why is Engram worse than the plain Transformer on NIAH-3, and why does FactorEngram reverse this?
A2: The NIAH gains depend on three components of FactorEngram (Tables 2 and 4). On NIAH-3:
Both models with scalar gating score below the Transformer on NIAH-3 (Engram 15.0, FactorEngram with scalar gating 9.4, Transformer 22.1). Basis-level gating raises the score to 58.3. This suggests that the gating method is the main factor. We will add an analysis of the underlying mechanism in the revised version.
W3/Q4/Q5: Ablations aren't entirely convincing: FactorEngram with scalar gating actually performs worse than Engram on several metrics. I'm not convinced that factorization itself is helping, rather than just fine-grained gating. I'd also like to see comparisons against alternatives like TN-gram. / How much of the improvement comes from factorization versus fine-grained gating? Could similar gains be achieved without factorization? / How does FactorEngram compare against other factorized memory approaches
A3: Our contribution lies in the joint design of factorized memory and basis-level gating, rather than factorization alone. As stated in our contributions (Section 1) and ablation discussion (Section 4.3), the gating reuses the reconstruction dictionary: factorization provides a shared basis space, and context-dependent gating modulates each basis component individually before reconstruction. The ablations highlight this interaction. With scalar gating, factorization does not outperform Engram on most metrics, while combining it with basis-level gating gives the best results on five of six metrics in Table 2. These results do not rule out other fine-grained gating designs without factorization, and we will add a variant with dense embeddings and fine-grained gating in the revised version.
TN-gram is concurrent work, and Table 5 summarizes how it differs from FactorEngram. We will add a direct comparison in the revised version.