- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to size and allocate N-gram embedding tables for language models and introduces StreamOE, which injects token, bigram, and trigram embeddings into separate Hyper-Connections residual streams. Its main contributions are a coverage-based heuristic for transferring table capacity choices from smaller models across training budgets and backbone sizes, and evidence that allocating more capacity to trigrams can outperform uniform allocation under a fixed parameter budget. The authors validate these findings through capacity sweeps on 24B-A1B and 48B-A2B MoE models, then evaluate StreamOE on 80B-A3B and 309B-A15B models, reporting improvements in downstream benchmarks and long-context performance, alongside experiments on embedding-table quantization.
Critical strengths
- Actionable evidence for allocating N-gram memory capacity. The paper studies bigram and trigram capacity separately and compares their allocation under a fixed 18.6B-parameter budget, showing that an asymmetric split outperforms uniform allocation. This controlled comparison provides useful guidance for allocating sparse memory parameters and constitutes a meaningful empirical contribution.
- Validation at substantial model scales with a parameter-matched architectural comparison. Experiments extend to 80B-A3B and 309B-A15B backbones. On 80B-A3B, StreamOE improves over Vanilla OE without adding parameters, including a 5.91-percentage-point gain on RULER-v2 at 256k context. This comparison provides concrete evidence that separating N-gram representations across residual streams can offer practical benefits beyond simply adding embedding capacity.
Critical weaknesses
-
The capacity prediction rule needs more direct validation. Table 2 reports gain fractions evaluated on fitted curves, but it is unclear whether models were actually trained at the predicted capacities. Were these predictions independently tested, and does coverage-based scaling predict suitable capacities better than simply keeping the slot count fixed? This matters because transferring capacity choices from a small proxy model is a central contribution.
-
The large-model results do not clearly isolate the benefit of StreamOE. On 80B, StreamOE improves the benchmark average over Vanilla OE by approximately 0.48 points, without an estimate of variability. On 309B, there is no Vanilla OE baseline, and an additional OE injection is introduced at layer 41. Could the authors clarify how much of the improvement comes from separating the residual streams rather than adding N-gram memory or changing its placement? This limits how confidently the large-model gains can be attributed to the proposed architecture.
Questions for authors
- Were the capacities predicted in Table 2 directly tested, and how does their accuracy compare with keeping the slot count fixed?
- What evidence shows that the large-model gains come from separating residual streams rather than adding N-gram memory or changing its placement?
4 comments
W2: How consistent is the 80B-A3B gain, and what is its variability?
We compare the final three 80B-A3B checkpoints using the equally weighted mean of the 17 benchmark scores. Appendix A (Figure 6) provides the individual benchmark trajectories alongside the mean score difference.
| Training step | Vanilla OE average score | StreamOE average score | StreamOE gain (percentage points) |
|---|---|---|---|
| 8000 | 57.64 | 58.60 | +0.96 |
| 9000 | 60.89 | 61.46 | +0.57 |
| 9666 | 61.75 | 62.23 | +0.48 |
StreamOE leads at all three checkpoints. The mean gain is 0.67 percentage points, and the sample standard deviation of these three gains is 0.26 percentage points. StreamOE also has a higher three-checkpoint average than Vanilla OE on 12 of the 17 benchmarks. With the same table configuration and parameter count, its average advantage persists across these late-training checkpoints.
Dear reviewer tToj,
We are writing to follow up as the discussion period nears its end. We hope our revisions and responses have been satisfactory, and we welcome any further questions or suggestions you may have before the deadline.
Best regards, The Authors
Dear Reviewer tToj,
As the rebuttal deadline approaches, we would be grateful if you could review our reply. We have devoted substantial time and computational resources to this work, and we would greatly appreciate your feedback on whether our responses address your concerns.
Thank you for your time and consideration.
Best regards, The Authors
Dear reviewer tToj,
We are grateful for the time and effort you have devoted to reviewing our manuscript. The revised text is shown in blue for your convenience.
Q1: Were the predicted capacities directly tested, and how accurate are they compared with fixed slots?
We did not train separate models at the exact predicted capacities of 2.487M and 3.559M slots. Table 2 evaluates these capacities on curves fitted to the 24B-A1B and 48B-A2B training runs at 500B tokens shown in Figure 3(b). Both predictions lie within the measured ranges, with observations on either side.
We select reference capacities corresponding to 80% and 90% of the fitted maximum loss reduction on 24B-A1B trained for 169B tokens. We then scale these slot counts by the change in estimated distinct N-gram coverage to predict capacities at 500B tokens. The target curves are used only to evaluate the predictions and do not determine or adjust the predicted capacities.
Figure 4 extends this assessment from individual operating points to full gain-fraction curves. Panels (a) and (c) predict the 24B-A1B responses at 500B tokens from its 169B-token experiments for 2-grams and 3-grams, respectively. Panel (b) uses 24B-A1B at 500B tokens as the reference to predict the 2-gram response on 48B-A2B at 500B tokens. The predictions closely track the independent target fits across the measured capacity ranges. The plotted loss reductions are expressed as fractions of each target setting's fitted plateau gain.
For the fixed-slots baseline, we retain the reference capacity for each of the same 80% and 90% gain targets when moving from 169B to 500B tokens. We evaluate both rules on the same target curves, expressing gains as fractions of each target setting's fitted maximum loss reduction.
Fixed slots fall short of the requested gain fractions in all four comparisons, with absolute errors of 7.77 to 15.20 percentage points. Coverage scaling reduces these errors to 1.43 to 2.65 percentage points. Its maximum deviations from the 80% and 90% targets are 2.65 and 1.95 percentage points, respectively. These comparisons of the fitted curves support approximate capacity transfer across the tested token budgets and backbones.
Q2: Do the gains come from separating streams or from adding and repositioning N-gram tables?
The 80B-A3B experiment in Table 3 isolates the effect of separating vocabulary and N-gram inputs across residual streams. Vanilla OE and StreamOE use the same N-gram table configurations and parameter counts, with OE only at the input (Section 5, StreamOE setup). Vanilla OE fuses the vocabulary and N-gram representations before copying them into the HC streams. StreamOE assigns these representations to separate streams.
In this comparison, StreamOE raises the 17-benchmark average from 61.75 to 62.23, a gain of 0.48 percentage points, and lowers byte perplexity on Pile-10K and Repo Code. After context extension, it also outperforms Vanilla OE on RULER-v2 at all five tested lengths. At 256k, the score increases from 29.97 to 35.88, a gain of 5.91 percentage points.
The 309B-A15B experiment tests whether StreamOE remains effective at a larger model scale. This run uses two independent OE modules, at the input and layer 41, within the available training budget. Compared with No-OE, the complete StreamOE configuration raises the 17-benchmark average from 68.63 to 71.56, a gain of 2.93 percentage points.
We also evaluate multi-layer StreamOE on 80B-A3B in Table 4 (Section 5.1). The ablation compares one, two, and three OE modules, each with independent 2-gram and 3-gram tables containing approximately 6.7B parameters. Each additional module increases the total table capacity, as shown below.