acceptodds
Under review as a conference paper at ICLR 2027

StreamOE: Scaling Over Encoding with N-Gram Streams

Abstract

Sparse parameter access enables large language models (LLMs) to expand their capacity with minimal computational overhead. Over Encoding (OE) uses this principle through N-gram embedding tables, introducing an additional scaling axis beyond the backbone architecture. Specifically, OE hashes local N-grams into table indices to retrieve embeddings that augment standard token representations. An open question is how to determine appropriate sizes for these N-gram tables and allocate capacity across different N-gram orders. To address this, we conduct extensive scaling measurements using StreamOE, an architecture that directly injects N-gram information into the network's Hyper-Connections (HC). Based on these experiments and statistical corpus analysis, we identify two key findings. First, useful table capacity is influenced by dataset scale, enabling N-gram slot counts calibrated on a small proxy model to transfer to larger backbones. Second, an asymmetric capacity allocation, such as the 1:6.75 reference ratio between 2-gram and 3-gram tables, can outperform conventional uniform splits under a fixed parameter budget. We further scale StreamOE to 80B-A3B and 309B-A15B Mixture-of-Experts (MoE) models, achieving substantial improvements in both average benchmark scores and long-context performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Over encoding, Scaling, LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tTojOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to size and allocate N-gram embedding tables for language models and introduces StreamOE, which injects token, bigram, and trigram embeddings into separate Hyper-Connections residual streams. Its main contributions are a coverage-based heuristic for transferring table capacity choices from smaller models across training budgets and backbone sizes, and evidence that allocating more capacity to trigrams can outperform uniform allocation under a fixed parameter budget. The authors validate these findings through capacity sweeps on 24B-A1B and 48B-A2B MoE models, then evaluate StreamOE on 80B-A3B and 309B-A15B models, reporting improvements in downstream benchmarks and long-context performance, alongside experiments on embedding-table quantization.

Critical strengths
  1. Actionable evidence for allocating N-gram memory capacity. The paper studies bigram and trigram capacity separately and compares their allocation under a fixed 18.6B-parameter budget, showing that an asymmetric split outperforms uniform allocation. This controlled comparison provides useful guidance for allocating sparse memory parameters and constitutes a meaningful empirical contribution.
  2. Validation at substantial model scales with a parameter-matched architectural comparison. Experiments extend to 80B-A3B and 309B-A15B backbones. On 80B-A3B, StreamOE improves over Vanilla OE without adding parameters, including a 5.91-percentage-point gain on RULER-v2 at 256k context. This comparison provides concrete evidence that separating N-gram representations across residual streams can offer practical benefits beyond simply adding embedding capacity.
Critical weaknesses
  1. The capacity prediction rule needs more direct validation. Table 2 reports gain fractions evaluated on fitted curves, but it is unclear whether models were actually trained at the predicted capacities. Were these predictions independently tested, and does coverage-based scaling predict suitable capacities better than simply keeping the slot count fixed? This matters because transferring capacity choices from a small proxy model is a central contribution.

  2. The large-model results do not clearly isolate the benefit of StreamOE. On 80B, StreamOE improves the benchmark average over Vanilla OE by approximately 0.48 points, without an estimate of variability. On 309B, there is no Vanilla OE baseline, and an additional OE injection is introduced at layer 41. Could the authors clarify how much of the improvement comes from separating the residual streams rather than adding N-gram memory or changing its placement? This limits how confidently the large-model gains can be attributed to the proposed architecture.

Questions for authors
  1. Were the capacities predicted in Table 2 directly tested, and how does their accuracy compare with keeping the slot count fixed?
  2. What evidence shows that the large-model gains come from separating residual streams rather than adding N-gram memory or changing its placement?
4 comments

Dear reviewer tToj,

We are grateful for the time and effort you have devoted to reviewing our manuscript. The revised text is shown in blue for your convenience.

Q1: Were the predicted capacities directly tested, and how accurate are they compared with fixed slots?

We did not train separate models at the exact predicted capacities of 2.487M and 3.559M slots. Table 2 evaluates these capacities on curves fitted to the 24B-A1B and 48B-A2B training runs at 500B tokens shown in Figure 3(b). Both predictions lie within the measured ranges, with observations on either side.

We select reference capacities corresponding to 80% and 90% of the fitted maximum loss reduction on 24B-A1B trained for 169B tokens. We then scale these slot counts by the change in estimated distinct N-gram coverage to predict capacities at 500B tokens. The target curves are used only to evaluate the predictions and do not determine or adjust the predicted capacities.

Figure 4 extends this assessment from individual operating points to full gain-fraction curves. Panels (a) and (c) predict the 24B-A1B responses at 500B tokens from its 169B-token experiments for 2-grams and 3-grams, respectively. Panel (b) uses 24B-A1B at 500B tokens as the reference to predict the 2-gram response on 48B-A2B at 500B tokens. The predictions closely track the independent target fits across the measured capacity ranges. The plotted loss reductions are expressed as fractions of each target setting's fitted plateau gain.

For the fixed-slots baseline, we retain the reference capacity for each of the same 80% and 90% gain targets when moving from 169B to 500B tokens. We evaluate both rules on the same target curves, expressing gains as fractions of each target setting's fitted maximum loss reduction.

Target gain fractionTarget model (500B tokens)Fixed slots: fitted gain fractionAbsolute error (percentage points)Coverage scaling: fitted gain fractionAbsolute error (percentage points)
80%24B-A1B64.80%15.2077.35%2.65
80%48B-A2B70.11%9.8982.05%2.05
90%24B-A1B77.55%12.4588.05%1.95
90%48B-A2B82.23%7.7791.43%1.43

Fixed slots fall short of the requested gain fractions in all four comparisons, with absolute errors of 7.77 to 15.20 percentage points. Coverage scaling reduces these errors to 1.43 to 2.65 percentage points. Its maximum deviations from the 80% and 90% targets are 2.65 and 1.95 percentage points, respectively. These comparisons of the fitted curves support approximate capacity transfer across the tested token budgets and backbones.

Q2: Do the gains come from separating streams or from adding and repositioning N-gram tables?

The 80B-A3B experiment in Table 3 isolates the effect of separating vocabulary and N-gram inputs across residual streams. Vanilla OE and StreamOE use the same N-gram table configurations and parameter counts, with OE only at the input (Section 5, StreamOE setup). Vanilla OE fuses the vocabulary and N-gram representations before copying them into the HC streams. StreamOE assigns these representations to separate streams.

In this comparison, StreamOE raises the 17-benchmark average from 61.75 to 62.23, a gain of 0.48 percentage points, and lowers byte perplexity on Pile-10K and Repo Code. After context extension, it also outperforms Vanilla OE on RULER-v2 at all five tested lengths. At 256k, the score increases from 29.97 to 35.88, a gain of 5.91 percentage points.

The 309B-A15B experiment tests whether StreamOE remains effective at a larger model scale. This run uses two independent OE modules, at the input and layer 41, within the available training budget. Compared with No-OE, the complete StreamOE configuration raises the 17-benchmark average from 68.63 to 71.56, a gain of 2.93 percentage points.

We also evaluate multi-layer StreamOE on 80B-A3B in Table 4 (Section 5.1). The ablation compares one, two, and three OE modules, each with independent 2-gram and 3-gram tables containing approximately 6.7B parameters. Each additional module increases the total table capacity, as shown below.

OE configurationTotal OE table parameters (B)Validation loss
Input layer6.71.47257
Input layer, layer 2513.41.46512
Input layer, layers 25, 3720.11.46133

W2: How consistent is the 80B-A3B gain, and what is its variability?

We compare the final three 80B-A3B checkpoints using the equally weighted mean of the 17 benchmark scores. Appendix A (Figure 6) provides the individual benchmark trajectories alongside the mean score difference.

Training stepVanilla OE average scoreStreamOE average scoreStreamOE gain (percentage points)
800057.6458.60+0.96
900060.8961.46+0.57
966661.7562.23+0.48

StreamOE leads at all three checkpoints. The mean gain is 0.67 percentage points, and the sample standard deviation of these three gains is 0.26 percentage points. StreamOE also has a higher three-checkpoint average than Vanilla OE on 12 of the 17 benchmarks. With the same table configuration and parameter count, its average advantage persists across these late-training checkpoints.

Dear reviewer tToj,

We are writing to follow up as the discussion period nears its end. We hope our revisions and responses have been satisfactory, and we welcome any further questions or suggestions you may have before the deadline.

Best regards, The Authors

Dear Reviewer tToj,

As the rebuttal deadline approaches, we would be grateful if you could review our reply. We have devoted substantial time and computational resources to this work, and we would greatly appreciate your feedback on whether our responses address your concerns.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_ySd4OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StreamOE, an Over Encoding (OE) architecture and scaling framework that augments large language models (LLMs) with hashed N-gram embedding tables injected into Hyper-Connections (HC) residual streams. The authors argue that existing OE methods lack principled rules for sizing N-gram tables across token budgets and backbones, and typically rely on suboptimal uniform capacity splits between 2-gram and 3-gram tables. To address this, the authors 1) propose StreamOE, which places vocabulary, 2-gram, and 3-gram embeddings into separate HC streams rather than averaging them prior to replication, 2) establish a coverage-based capacity calibration rule (S_N(T) = U_N(T) / K_N_ref) that uses distinct N-gram counts from corpus statistics and small proxy models to predict table slot counts for larger backbones and token budgets, and 3) show that an asymmetric capacity ratio (around 1:6 to 1:6.75 for 2-gram to 3-gram tables) outperforms uniform 1:1 allocation, validating the design up to 80B-A3B and 309B-A15B MoE models.

Critical strengths
  1. The paper studies an important and practical scaling question for sparse N-gram memory in large MoE LLMs, connecting corpus-level distinct N-gram growth curves directly to embedding table slot allocation rather than relying on ad-hoc table sizes.

  2. The proposed StreamOE stream-separation design is clean and parameter-free relative to Vanilla OE, and the large-scale validation on 80B-A3B and 309B-A15B MoE backbones (including 256k long-context extension and FP8/MXFP4 table quantization) shows consistent gains on reasoning, coding, and RULER-v2 long-context retrieval.

Critical weaknesses
  1. The coverage-based slot calibration rule in Section 4.2 (Table 2) is a very neat way to predict table capacity across token budgets and model sizes. Since the transfer validation in Table 2 focuses on 2-gram tables while 3-gram tables hold the majority of the capacity under the 1:6 split, it would be interesting to hear the authors' insights on whether 3-gram tables follow a similarly stable reference load ratio as training tokens scale.

  2. The large-scale results on the 80B-A3B and 309B-A15B MoE models in Table 3 are impressive and show clear gains over the baselines. Because the 309B configuration incorporates both the input-layer StreamOE design and an additional OE module at layer 41, it would be helpful if the authors could share some intuition or discussion on the relative contribution of the layer-41 module and how the optimal 2-gram to 3-gram ratio behaves at larger token budgets.

Questions for authors

Could the authors share their insights or any observations on how the constant reference load ratio transfers to 3-gram tables across token budgets and how the 2-gram to 3-gram ratio adapts as training tokens scale? Additionally, it would be helpful to hear more details on the role of the layer-41 OE module and the table configurations used in the 80B and 309B runs.

4 comments

Dear reviewer ySd4,

We are grateful for yours' thorough reading and helpful comments. The revised passages in the main text are highlighted in blue.

Q1: Does the sizing rule generalize to 3-grams and asymmetric allocation across budgets?

We extended the transfer evaluation to 3-grams by predicting the 500B-token table sizes from the 24B-A1B experiments at 169B tokens. For the 80% and 90% gain targets, the predicted capacities correspond to 44.51B table parameters (31.614M slots) and 63.68B table parameters (45.229M slots).

Target gain fractionBackbonePredicted slots (M)Fitted gain fractionGap to target (percentage points)
80%24B-A1B31.61479.62%−0.38
90%24B-A1B45.22989.73%−0.27

On an independent fit to the 500B-token measurements, these capacities reach 79.62% and 89.73% of the plateau gain. The two operating points stay close to their targets after the increase in training tokens, consistent with approximate transfer of the 3-gram sizing rule.

Figure 4(c) shows that the 3-gram curve predicted from the 169B-token reference nearly overlaps the independent fit to the 24B-A1B measurements at 500B tokens. Figure 4(a) tests the same 169B-to-500B transfer on 24B-A1B for 2-grams, while Figure 4(b) tests 2-gram transfer from 24B-A1B to 48B-A2B with both trained for 500B tokens. The 2-gram predictions also follow the target fits closely across the measured capacity ranges. The figure expresses target loss reductions as fractions of each target's fitted plateau gain.

For Question: How does the allocation ratio change as training tokens increase?

We transfer the two single-order reference capacities using their respective N-gram coverage growth factors. In our corpus, distinct 3-grams grow faster than distinct 2-grams as training tokens increase, so the predicted allocation shifts toward 3-grams. Starting from the 2-gram:3-gram reference ratio of 1:6.75 at 169B tokens, the rule predicts approximately 1:10.42 at 752B tokens. This transferred ratio provides a reference for dividing a fixed total OE budget between the two orders.

The time and compute available during rebuttal did not allow us to extend this evaluation to 48B-A2B at 500B tokens or to test asymmetric allocation at other training or table budgets.

Q2: Why use multiple OE modules, and what supports the two-module setup on 309B-A15B?

A multi-layer OE ablation on 309B-A15B would require more training time and compute than were available during the rebuttal period.

Instead, we report an ablation on 80B-A3B in Table 4 (Section 5.1) to examine the effect of adding OE at intermediate layers. We compare one, two, and three OE modules on the same MoE backbone with four mHC streams.

Starting from input-only OE, the two-module configuration adds OE at layer 25, and the three-module configuration adds it at both layers 25 and 37. Every module has independent 2-gram and 3-gram tables containing about 6.7B lookup-table parameters, with 1,030,798 and 2,241,445 slots per head, respectively. This per module size allows the three-module run to fit within the memory budget. As specified in Section 3, the intermediate modules add their outputs to the corresponding residual streams.

We held the data mixture and main training hyperparameters fixed across the three runs. All used Muon, 32K sequences, a global batch size of 2048, and the same learning-rate schedule.

ModelValidation loss
80B-A3B w/ StreamOE x 1 (entry)1.47257
80B-A3B w/ StreamOE x 2 (entry, layer 25)1.46512
80B-A3B w/ StreamOE x 3 (entry, layer 25 and layer 37)1.46133

Each added module lowers validation loss while also increasing the total table capacity. The 309B-A15B run combines an input module with a second module at layer 41, keeping training cost within the available budget.

Q3: What are the full OE configurations for the models in Table 3?

The table sizes and 2-gram/3-gram allocation are guided by the capacity-scaling analysis in Section 4.2 and the allocation analysis in Section 4.3. We use these analyses to balance performance gains against storage cost and assign more capacity to 3-grams, within each experiment's memory budget. The StreamOE setup paragraph in Section 5 reports the final configurations.

For the Table 3 experiments, each N-gram order uses four hash heads. A head retrieves a 512-dimensional embedding, so the combined lookup width is 2048. The slot counts below apply to one head at one insertion location. We report lookup-table parameters both per location and across the model.

Model & Method2-gram slots / head3-gram slots / headOE insertion locationsOE lookup-table parameters per location (B)Total OE lookup-table parameters (B)
80B-A3B Vanilla OE / StreamOE1,030,7982,241,445Input6.76.7
309B-A15B StreamOE1,050,00011,270,000Input and layer 4125.2350.46

The 80B-A3B experiment uses input-only OE with matched table configurations and parameter counts. It compares two ways of forming the HC streams: replicating a fused vocabulary/N-gram representation in Vanilla OE, and assigning the vocabulary and N-gram representations to separate streams in StreamOE.

The 309B-A15B total includes separate tables at the input and layer 41: approximately 2.15B 2-gram and 23.08B 3-gram parameters per module, or 25.23B each and 50.46B overall. We evaluate the complete StreamOE configuration at this larger scale against No-OE, which omits both modules.

Thank you for the reply! I I have no further concerns here and would like to keep my original positive rating.

We thank you for your careful review and thoughtful comments. We greatly appreciate the time and effort you have devoted to helping us improve our paper.

Reviewer_EH9iOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how to size and allocate hashed N-gram embedding tables for language models. It introduces StreamOE, which places vocabulary, 2-gram, and 3-gram embeddings into separate Hyper-Connections residual streams instead of fusing them before replication. Using N-gram statistics from a 10T-token corpus and capacity sweeps, the authors propose transferring candidate table sizes by preserving a calibrated ratio of distinct N-grams to embedding slots. They test 2-gram capacity transfer from 169B to 500B training tokens on 24B-A1B and 48B-A2B backbones, and show that asymmetric 2-gram/3-gram allocation improves validation loss at a fixed embedding-parameter budget. Further experiments on 80B-A3B and 309B-A15B models report benchmark and long-context improvements.

Critical strengths
  1. This paper presents a data-driven, transferable scaling laws for N-gram table capacity and asymmetric allocation.
  2. Strong empirical validation of StreamOE at frontier MoE scales and long context is provided.
Critical weaknesses
  1. Table 2 evaluates the sizing rule using fitted gain curves normalized by each setting’s fitted plateau. This provides some evidence of transfer, but uncertainty in the plateau and sensitivity of the reported gain fractions are not shown. Measured losses at or near the predicted sizes would help assess how robust the agreement with the 80% and 90% targets is. The transfer tests cover only 2-grams, one corpus/tokenizer, and the change from 169B to 500B tokens, with 48B-A2B as the largest target. The joint allocation study also uses only one 24B setting. Transfer of the sizing rule to 3-grams, and of the asymmetric allocation to other settings, has not been shown. This limits the broader design guidance, especially since 3-grams receive most of the joint-table budget.
  2. The per-order table configurations used in Table 3 and their connection to the Section 4 calibration are not clearly reported. This makes it difficult to tell whether the large-model runs validate the proposed sizing rule. A comparison with a 1:1 split at the same total OE budget on a larger model would help. At 309B, the comparison is only with No-OE, and StreamOE also includes OE at layer 41. The 2.93-point improvement therefore does not isolate the benefit of separate streams from adding N-gram memory or intermediate-layer injection. The parameter-matched 80B comparison does support the stream design, particularly on RULER-v2. However, uncertainty estimates would help assess the reliability of its 0.48-point average gain on standard benchmarks.
Questions for authors
  1. Have you tested capacity transfer for 3-grams, or the asymmetric split at another training or table budget? For the allocation comparison, please keep total OE parameters matched to the 1:1 baseline. If not, please distinguish the tested 2-gram result from the broader design guidance.
  2. Please list the 2-gram and 3-gram slot counts, embedding dimensions, and total OE parameters for the Table 3 models. How were these settings chosen from the Section 4 results, and were they adjusted after testing on the target models?
  3. Table 2 reports fitted gain fractions at 2.487M and 3.559M slots. What measurements are available at or near these sizes? Please give the fit equation and plateau uncertainty, and explain how they affect the reported agreement with the 80% and 90% targets.
  4. Table 3 compares 309B StreamOE with No-OE, and StreamOE also adds OE at layer 41. Do you have a Vanilla OE control with the same tables and insertion layers? A smaller-scale control with the same intermediate-layer design would also be useful.
  5. Could you report paired evaluation confidence intervals for the 80B benchmark and RULER-v2 gains over Vanilla OE? Please explain how the 17-benchmark average is handled. Separately, do repeated small-model runs support the 0.0052 loss improvement from changing the allocation?
5 comments

Dear reviewer EH9i,

Thank you for the careful review and constructive suggestions. Revisions to the main text are marked in blue.

Q1: Does the sizing rule transfer to 3-grams and joint allocation, and are allocation budgets matched?

We have added a 3-gram capacity-transfer experiment on 24B-A1B, from 169B to 500B training tokens.

Target gain fractionBackbonePredicted slots (M)Fitted gain fractionGap to target (percentage points)
80%24B-A1B31.61479.62%−0.38
90%24B-A1B45.22989.73%−0.27

Using the 169B-token setting as the reference, the rule predicts 44.51B table parameters (31.614M slots) and 63.68B table parameters (45.229M slots) at 500B tokens for the 80% and 90% gain targets. At these capacities, the target fit reaches 79.62% and 89.73% of its plateau gain. The predicted curve and independent target fit in Figure 4(c) also nearly overlap across the measured range of 3-gram capacities. These results support approximate transfer of the 3-gram sizing rule across the two training budgets.

Due to time and compute constraints during the rebuttal period, we have not yet tested 3-gram transfer on 48B-A2B at 500B tokens or joint allocation at other training or table budgets.

For question: Do the allocation comparisons use the same total OE parameter budget?

All allocation experiments, including the 1:1 baseline, use the same total OE budget of 18.6B parameters, as described in Section 4.3 and the caption of Figure 3(a).

Q2: What OE configurations are used in Table 3, and how were they selected?

The Table 3 configurations are as follows. We now describe them in the StreamOE setup paragraph in Section 5.

All slot counts are per hash head. Each N-gram order uses four heads with 512-dimensional embeddings, giving a combined lookup width of 2048. The parameter counts below refer to the lookup tables.

80B-A3B. Vanilla OE and StreamOE both use 1,030,798 2-gram slots and 2,241,445 3-gram slots per head, totaling approximately 6.7B table parameters. Both configurations apply OE only at the input and use the same lookup width. They differ in how vocabulary and N-gram representations enter the HC streams.

309B-A15B. Each OE module has 1,050,000 2-gram slots and 11,270,000 3-gram slots per head. With the same combined lookup width of 2048, the two tables contain approximately 2.15B and 23.08B parameters, or 25.23B per module. Independent modules at the input and layer 41 give a total of approximately 50.46B lookup-table parameters. The No-OE control has none of these additional parameters.

For both backbones, we fixed the slot counts, lookup widths, and insertion locations before target-model training and did not adjust them using target validation or test scores.

The 80B-A3B configuration uses smaller tables per module so that the supplementary experiments can accommodate up to three OE modules. For 309B-A15B, the 2-gram:3-gram parameter ratio is approximately 1:10.73, close to the rule's predicted ratio of 1:10.42.

Q3: Were the predicted capacities directly tested, and how sensitive are the results to the fitted plateau?

Table 2 reports fitted gain fractions at the predicted capacities. We did not train separate models at these two exact capacities, but both lie within the measured ranges. The nearest observations on either side are listed below. Figure 3(b) shows the full sweeps.

Model (500B tokens)Predicted capacity (M slots)Lower measured point: M slots / lossUpper measured point: M slots / loss
24B-A1B2.4872.130682 / 1.6552992.556818 / 1.654604
24B-A1B3.5592.556818 / 1.6546043.977273 / 1.652884
48B-A2B2.4871.704799 / 1.5791963.409040 / 1.578547
48B-A2B3.5593.409040 / 1.5785474.261161 / 1.574515

Figure 4 applies the same coverage-based rule to predict gain fractions across the measured capacity ranges. For 2-grams, we test transfer from 169B to 500B training tokens on 24B-A1B in panel (a), and from 24B-A1B to 48B-A2B with both trained for 500B tokens in panel (b). Panel (c) tests 3-gram transfer from 169B to 500B tokens on 24B-A1B. The predicted curves closely follow the independent fits to the target measurements. The target points show loss reductions divided by each target setting's fitted plateau gain.

For each setting, we independently fit the loss reduction relative to its zero-table baseline by least squares:

gi(S)=Ai(1−e−S/τi)g_i(S)=A_i\left(1-e^{-S/\tau_i}\right)

Here, SS and τi\tau_i are measured in million slots, and AiA_i is the fitted plateau gain. Table 2 reports the fitted gain fraction:

hi(S)=gi(S)Ai=Ai(1−e−S/τi)Ai=1−e−S/τi.h_i(S)=\frac{g_i(S)}{A_i}=\frac{A_i(1-e^{-S/\tau_i})}{A_i}=1-e^{-S/\tau_i}.

At fixed τi\tau_i, the amplitude AiA_i cancels from this fraction. Since we estimate AiA_i and τi\tau_i jointly, we also checked how both parameters change when we omit one nonzero-capacity observation at a time and refit.

SettingAiA_i fitted on the full dataτi\tau_i (M slots)Range of refitted AiA_i
24B-A1B / 169B tokens0.0163851.0867150.016014 to 0.017108
24B-A1B / 500B tokens0.0213841.6748700.019604 to 0.022324
48B-A2B / 500B tokens0.0130021.4482870.012302 to 0.013565

We then held the reference calibration and predicted capacities fixed while omitting each nonzero target observation in turn and refitting the target curve. At the original predicted capacities, the maximum deviations from the 80% and 90% targets were 5.75 and 4.35 percentage points. The fitted fractions remain close to the targets in these checks, supporting approximate transfer in the tested settings.

Q4: Is there a matched multi-layer Vanilla OE control, and what do additional OE modules contribute?

Given the time and compute constraints of the rebuttal period, we could not add a multi-layer Vanilla OE control matched to StreamOE in table configuration and insertion locations.

Instead, we report a StreamOE ablation on 80B-A3B in Table 4 (Section 5.1), comparing one, two, and three OE modules on the same backbone with four mHC streams. All configurations include OE at the input. The second module is at layer 25, and the third is at layer 37. Each module has independent 2-gram and 3-gram tables totaling approximately 6.7B parameters, with the configuration specified in Q2. Intermediate-layer outputs are added to the corresponding residual streams as described in Section 3.

The experiments use the same data mixture and main training hyperparameters: a 32K sequence length, a global batch size of 2048, the Muon optimizer, and the same learning-rate schedule.

ModelTotal OE table parameters (B)Validation loss
80B-A3B w/ StreamOE x 1 (entry)6.71.47257
80B-A3B w/ StreamOE x 2 (entry, layer 25)13.41.46512
80B-A3B w/ StreamOE x 3 (entry, layer 25 and layer 37)20.11.46133

The two comparisons in Table 3 address different questions:

The 80B-A3B experiment in Table 3 is an ablation of the multi-stream input design. Both methods apply OE only at the input, with the same table configuration and parameter count. Vanilla OE fuses vocabulary and N-gram representations before replicating them across HC streams, while StreamOE places them in separate streams.

The 309B-A15B experiment evaluates StreamOE at a larger model scale. We use two OE modules within the available training budget and compare the complete StreamOE configuration with No-OE.

Q5: How are benchmark scores averaged, and how reliable are the benchmark and allocation gains?

The 17-benchmark average is the arithmetic mean of the individual scores, with weight 1/17 per benchmark. It excludes PPL and RULER-v2.

We evaluate both methods on the same full test sets using each benchmark's standard protocol. To examine how the gains vary late in training, we compare the final three checkpoints. On 80B-A3B, the average gains over Vanilla OE at steps 8000, 9000, and 9666 are 0.96, 0.57, and 0.48 percentage points. Figure 6 in Appendix A shows the individual benchmark curves and their average score difference. On RULER-v2 at 256k, the gains at steps 1200, 1400, and 1490 are 5.91, 4.20, and 5.91 percentage points. Both comparisons show gains across all three late-stage checkpoints.

For allocation, several non-uniform splits improve on the 1:1 baseline at the same total OE budget. The 1:6, 1:7, 1:9, and 1:10 splits reduce loss by 0.00520, 0.00394, 0.00368, and 0.00249, respectively. The gain therefore occurs across several ratios. We have not repeated these 24B-A1B runs with different seeds. The runs use the same seed and training setup, varying only the capacity split between the 2-gram and 3-gram tables.

Comments on the paper

3 comments

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close soon, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

Dear Reviewers,

Thank you again for your thorough review!

As the discussion period is nearing its end, we would like to gently follow up to confirm whether our responses have adequately addressed your concerns. We remain available for any additional questions you may have.

Best regards, The Authors

Dear all,

Thank you all for your careful reviews. We appreciate your participation in the discussion and the time and effort you have devoted to the review process.

We summarize our contributions and our responses to reviewers' questions during rebuttal below.

Recent models from Qwen, DeepSeek, LongCat, and Hunyuan have adopted N-gram memory (engram, Over Encoding(OE), etc). How to size these tables as training data and model scale grow, and how to allocate capacity across N-gram orders, remains under explored. We study OE scaling laws through corpus statistics and experiments with StreamOE, which assigns vocabulary and N-gram embeddings to separate Hyper-Connections streams. Our main findings are:

  1. Useful table capacity depends on data coverage, with diminishing returns from further expansion under a fixed training budget. Scaling slot counts with estimated distinct N-gram coverage allows calibration on a small proxy model to guide capacity selection across training budgets and backbone sizes.

  2. Asymmetric allocation between 2-gram and 3-gram tables can outperform a uniform split under a fixed parameter budget. Our measurements favor a larger share for 3-grams, consistent with their substantially greater diversity in the training data.

We also conducted scaling experiments with StreamOE. StreamOE improves average benchmark scores and long-context performance on 80B-A3B and 309B-A15B models.

During rebuttal, we extended capacity-transfer evaluation to 3-grams and compared predicted response curves with independent target fits. We added fixed-slot comparisons and sensitivity checks, alongside multi-layer ablations and evidence that benchmark gains persist across the final three checkpoints. We also clarified the full OE configurations and the distinct roles of the matched 80B comparison and the 309B evaluation of the complete configuration.

We appreciate the feedback that helped us improve the paper and thank you for your participation.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.