acceptodds
Under review as a conference paper at ICLR 2027

The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation

Abstract

The scaling relationship of compute and performance for language models (LM) has been extensively studied; more recently, models are being designed to scale memory use, as well. In particular, adding input embeddings for n-grams scales memory separately from compute use, because only individual rows from the embedding table are read during each pass. However, existing methods train embeddings from scratch, which needs expensive GPU memory for fast reading and writing. Meanwhile, the ordinary memory of the host system is cheap and underutilized by these training methods. To scale on host memory, we introduce COUNT (Counting Offline Unlocks N-gram Transformers). COUNT scales n-gram representations while adding zero learned parameters by storing precomputed n-gram counts on host. We use the corpus frequencies to weight together the LM's own unembeddings of the next words most often following the current n-gram history. At a fixed compute budget, COUNT can be added as an additional input embedding for pretraining 1B-parameter LMs to improve validation loss as much as an LM with 58% more compute and task loss as much as 24% more compute. If GPU memory is limited, using COUNT to let backbone parameters fill GPU memory gives better performance than splitting memory with additional learned embeddings. Beyond efficiency, COUNT offers a new affordance: switching in a different table of n-gram counts from a domain-specific corpus at inference time to adapt to a new task without any training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language modeling, Language model architectures, Count-based language models, Hybrid parametric–nonparametric models, N-gram embeddings, Domain adaptation, Training-free adaptation, Language model pretraining, Representation learning, Efficient language model scaling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EfffOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Paper suggests a novel way to introduce data statistics information to a language model. These parameters are configured via ngram counts on language data and are not learned during training. Although the resulting table is very large in terms of memory, it can reside in host memory which is considerably cheaper than the gpu memory used for training. The authors show how the use of these ngram counts to weight model's own unembedding matrix leads to gains over equivalent compute with no added learned parameters and adaptability to other domains when these tables are modified post training. The authors also show this table can be shrinked in size with graceful degradation in accuracy.

Critical strengths

the paper introduces a new way to scale models using cheap host memory with little impact on performance. This scaling is novel and shows ability to adapt to new domains.

Critical weaknesses
  • seems like the count table was calculated on much larger set of data than was used for training. To reduce the risk for data contamination, I would like to see an experiment where the used data for ngram count is limited to the same data used for training.
Questions for authors
  • Did the authors think of a way to introduce the ngram data to the model in way more complaint with distributed embeddings? For example, decompose the table (similar to log-pmi decompositons) and introduce the added information as continues vectors rather than count-weighted sums of embeddings. This will allow tunable parameters that can be interpolated, quantized, used in other ways similar to neural embeddings.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Efff

Thank you for noting the novelty of scaling language models with n-gram counts stored in host memory.

Does improvement rely on counts from unseen data?

“I would like to see an experiment where the used data for ngram count is limited to the same data used for training.” (Efff W.1)

Responding to the concern that improvements required n-gram counts from tokens unseen by the pretraining token budget of our models, we have built count tables from the exact 1.9B, 3.8B, and 7.6B tokens used to train the model at D/N = 10, 20, and 40, respectively. When the count table is only a subset of the pretraining data (i.e. D/N = 20 and 40 for 1.9B tokens and D/N = 40 for 3.8B tokens in the experiments below) performance is similar to our headline results. Interestingly when the count table and pretraining tokens overlap completely, generalization is worse. Practically this is easy to avoid by cheaply counting unused data (in the small model infinite data regime of our experiments) or by fitting count tables on only a subset of the training data (with relatively little performance degradation) (Figure 19).

Model or count corpusLM loss at D/N = 10LM loss at D/N = 20LM loss at D/N = 40
Standard Transformer3.443.293.18
Full-corpus COUNT3.343.223.12
1.9B counted tokens COUNT3.913.253.13
3.8B counted tokens COUNT3.883.833.14
7.6B counted tokens COUNT3.843.793.77

Count-based dense representations

“Did the authors think of a way to introduce the ngram data to the model in way more complaint with distributed embeddings? For example, decompose the table (similar to log-pmi decompositons) and introduce the added information as continues vectors rather than count-weighted sums of embeddings.” (Q.1)

Good idea! Similar to your suggestion, in early experiments we tried fitting dense representations to empirical next-token distributions and using them to initialize learned n-gram embeddings, but this did not improve on random initialization. Now, we try this again but use the initialization as fixed rather than learned embeddings and compare to COUNT. Specifically, before starting pretraining we fit the 190M 1:1 n-gram embeddings to supervision from the empirical next-token distributions with early stopping on the lowest validation KL divergence. We then froze the embeddings but left the projections in the learned embedding architecture trainable. While this does improve loss beyond the standard transformer, it does not do so as well as COUNT (Figure 24).

D/N190M modelLM lossTask BPB
10Standard Transformer3.4381.659
10COUNT3.3361.634
10Count-initialized frozen embeddings3.3941.666
20Standard Transformer3.2931.576
20COUNT3.2231.542
20Count-initialized frozen embeddings3.2771.583
40Standard Transformer3.1761.507
40COUNT3.1241.478
40Count-initialized frozen embeddings3.1681.501

I thank authors for their answers and additional results. The experiment where the n-gram count tables are built from the exact training set directly addresses my main concern about contamination. I would encourage the authors to include this experiment. The additional experiment on count-initialized frozen dense embeddings also answers my question. As authors noted, this alternative improves over the baseline but does not match COUNT. Overall, my assessment remains positive.

Reviewer_kRh2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces COUNT (Counting Offline Unlocks N-gram Transformers). This method aims to enhance the scalability of language models by leveraging host memory (RAM) to store precomputed N-gram statistics, without requiring any additional learnable parameters. Specifically, COUNT aggregates the model’s unembedding vectors based on corpus-derived "next-word" frequency statistics to construct N-gram representations, which are then integrated into the model’s input embeddings.

The authors evaluated COUNT on language models ranging from 60M to 1B parameters. Experimental results demonstrate that, compared to standard Transformer models and baselines utilizing learnable N-gram embeddings, COUNT achieves improvements in both language modeling loss and performance on downstream tasks. Furthermore, the method enables "training-free" domain adaptation, which can be accomplished simply by replacing or interpolating the N-gram statistics during the inference phase.

In summary, this work explores a significant avenue for scaling language models: utilizing host memory as an additional resource, alongside GPU VRAM and computational power.

Critical strengths
  1. A Key Issue in Language Model Scaling. This paper explores a critical yet relatively under-researched aspect of scaling language models: how to effectively leverage host memory to enhance model performance without significantly increasing GPU VRAM requirements or computational costs. This represents an important area of ​​inquiry, especially given that resource efficiency is becoming increasingly vital for the scaling of language models.

  2. An innovative and intriguing approach. COUNT offers a novel way to integrate N-gram statistics into language models without introducing any additional learnable parameters. The method cleverly reuses the model’s unembedding representations and stores corpus statistics in the host machine’s RAM—a highly ingenious design choice. Furthermore, it enables training-free domain adaptation by simply swapping out the N-gram table, a feature with significant potential.

  3. Clear writing and presentation. This paper is generally well-written and clearly structured. The research motivation, methodological details, and experimental setup are all articulated with clarity, making the proposed approach easy to understand.

Critical weaknesses
  1. Insufficient evaluation of actual downstream task performance. Although this paper evaluates language modeling loss, task-specific loss, and accuracy on multiple-choice questions, it lacks a comprehensive assessment of the correctness of the model's generated outputs—particularly in tasks involving code generation and mathematical reasoning. Improvements in likelihood-based metrics do not necessarily translate to enhanced capabilities in solving real-world tasks. I strongly recommend that the authors incorporate evaluation metrics based on generated outputs—such as Pass@1 or execution-based correctness for code generation, and answer accuracy for mathematical reasoning—and compare these results against baseline models, ensuring that the training and inference resource budgets remain consistent. Such an evaluation would provide more compelling evidence for the performance gains offered by COUNT in practical applications.

  2. The substantial host RAM requirements raise concerns about overall resource efficiency. The N-gram table requires approximately 797 GB of host RAM, which may limit practical applicability despite introducing no additional learnable parameters and requiring almost no extra GPU memory. Although the paper reports throughput and experiments with smaller tables, a comprehensive analysis of total memory usage, hardware requirements, and end-to-end cost-efficiency is needed to substantiate its claimed scaling advantages.

  3. Limited evaluation at larger model scales. The models involved in these experiments range in size from 60M to 1B parameters. Although the results demonstrate a positive scaling trend, these models are relatively small compared to many modern language models currently in widespread use. It remains unclear whether the advantages provided by COUNT will persist or diminish as the backbone models grow significantly larger and more capable. Evaluating this approach on larger-scale models—for instance, those with 2B or 4B parameters—would further substantiate the paper’s claims regarding scalability and demonstrate the method's applicability to more realistic, practical scenarios.

  4. Dependence on corpus statistics may limit generalization to data-scarce domains. COUNT relies on the size and domain distribution of its reference corpus. Domain adaptation experiments show sensitivity to the interpolation ratio between general-domain and domain-specific statistics, with some configurations degrading performance. More systematic analysis of data requirements, corpus composition, and interpolation sensitivity is needed to establish robustness under data scarcity and distribution shifts.

  5. Some figures are difficult to read. Several figures are relatively small (Figure 1), making it difficult to distinguish individual curves, markers, and numerical annotations. Enlarging these figures or improving their layout would make the experimental results easier to interpret.

Questions for authors
  1. How does COUNT compare with the baseline in end-to-end hardware and memory costs? Beyond the smaller-table inference results in Figure 5, could the authors compare performance, throughput, and total resource costs across RAM budgets representative of practical deployments?
  2. How does COUNT differ from N-gram-guided decoding, and how does it relate to constrained decoding or other decoding-time guidance methods? Could the authors clarify these connections and provide relevant comparisons to establish COUNT’s methodological novelty and practical advantages?
  3. Could the authors report Pass@1 for code generation and final-answer accuracy for mathematical reasoning, comparing COUNT with standard Transformers and learned N-gram embedding baselines? This would establish whether the reported loss reductions translate into downstream performance gains.
  4. Could the authors evaluate COUNT on larger backbones, such as 2B or 4B models, to assess whether its scaling benefits extend beyond the model sizes studied?
5 comments
AuthorsOpenReview ↗

Response to Reviewer kRh2 (1/n)

Thank you for your generous description of COUNT’s design choice of reusing the model’s existing token representations to incorporate n-gram counts without adding learned parameters as “highly ingenious.”

Is COUNT robust to a smaller count corpus?

“Dependence on corpus statistics may limit generalization to data-scarce domains.” (kRh2 W.4)

For math adaptation, we have now made 100M and 1B domain adaptation tables. We evaluate these with the same 25% interpolation weight in with the 1B COUNT as used in the submitted results. We can see that loss gracefully degrades with smaller count data but still improves over the unadapted model (Figure 22).

Math corpusPacked tableLoss difference (BPB)
100M tokens1.386 GB−0.00091
1B tokens10.436 GB−0.00923
Full math (10.66B tokens)83.398 GB−0.01491

We also tested limiting the domain-general count table. Counting just 100M tokens improves LM loss and task loss over the standard transformer. The full-corpus table gives better losses, but much of the benefit remains with this small sample (Figure 20).

Model or count corpusLM loss (D/N = 20)Task BPB (D/N = 20)
Standard Transformer3.291.58
Full-corpus COUNT3.221.54
100M counted tokens COUNT3.261.57

How does COUNT compare to decoding-time methods?

“How does COUNT differ from N-gram-guided decoding, and how does it relate to constrained decoding or other decoding-time guidance methods?” (Q.2)

Since submission, we compared 1B COUNT against two inference-time methods: additive probability interpolation, which follows the fusion method of kNN-LM (Khandelwal et al., 2020) and infini-gram (Liu et al., 2025), and a multiplicative combination, which follows Li et al. (2022). COUNT performs better than our implementations of inference time-only methods (Figure 9). Unlike inference-only methods, COUNT needs pretraining with count features, but adds no GPU RAM or compute overhead for learned parameters (lines 478–486). If the reviewer has another specific decoding time method in mind, we would be happy to look at it.

ModelLM loss (D/N = 20)Task BPB (D/N = 20)
Full-corpus COUNT 1B2.661.24
Additive late fusion 1B (inference only)2.691.26
Multiplicative late fusion 1B (inference only)2.741.27

What are the deployment throughput and memory costs?

“Although the paper reports throughput and experiments with smaller tables, a comprehensive analysis of total memory usage, hardware requirements, and end-to-end cost-efficiency is needed to substantiate its claimed scaling advantages.” (W.2)

“How does COUNT compare with the baseline in end-to-end hardware and memory costs? Beyond the smaller-table inference results in Figure 5, could the authors compare performance, throughput, and total resource costs across RAM budgets representative of practical deployments?” (Q.1)

The reviewer asks for throughput and memory analysis beyond Figure 5: We also report training, prefill, and decoding throughput for several deployment configurations in Figure 6 and Tables 7–9 and Table 10 reports peak GPU memory for the same measurements.

How do sensitive is COUNT to data scarcity and distribution shifts?

“More systematic analysis of data requirements, corpus composition, and interpolation sensitivity is needed to establish robustness under data scarcity and distribution shifts.” (W.4)

The 100M-token count table experiment above shows that counts estimated from a small corpus still help (Figure 20). Figure 12 shows how task loss changes across all four domain-table weights for math, biomedicine, and law. Our selected 25% domain interpolation is robust when holding out each domain or group of related tasks (lines 402–403). All three domain folds and all ten task-group folds selected the same 25% weight.

Continued in thread...

AuthorsOpenReview ↗

Response to Reviewer kRh2 (2/2)

Larger models and generated-answer evaluation

“Although this paper evaluates language modeling loss, task-specific loss, and accuracy on multiple-choice questions, it lacks a comprehensive assessment of the correctness of the model's generated outputs—particularly in tasks involving code generation and mathematical reasoning.” (W.1)

“Evaluating this approach on larger-scale models—for instance, those with 2B or 4B parameters—would further substantiate the paper’s claims.” (W.3)

“Could the authors report Pass@1 for code generation and final-answer accuracy for mathematical reasoning, comparing COUNT with standard Transformers and learned N-gram embedding baselines?” (Q.3)

“Could the authors evaluate COUNT on larger backbones, such as 2B or 4B models, to assess whether its scaling benefits extend beyond the model sizes studied?” (Q.4)

At the same D/N ratio, 2B and 4B models require roughly four and sixteen times the compute of a 1B model, which is beyond our budget (Equation 4 gives the compute model). On generation tasks, a standard transformer at 1B and D/N = 20 achieves a pass@1 of only 2.16% on GSM8K and 0.93–2.27% across the seven MATH subjects. Scores this low are uninformative as they give little room to measure differences between treatments and simple evaluation noise. We therefore use likelihood-based evaluations for small-scale pretraining experiments following Llama 3 (Dubey et al., 2024, §3.2.1), DataDecide (Magnusson et al., 2025), and Signal and Noise (Heineman et al., 2025) (lines 211–215).

Figure readability

“Several figures are relatively small (Figure 1)” (W.5)

Thank you for the suggestion and for rating the paper’s presentation 3/3. We have increased the font size of the figures in the updated pdf (Figures 1–6).

Thank you for conducting additional experiments and providing detailed, thoughtful explanations in response to my questions.

I understand that models with a 1B parameter scale may struggle to perform well on challenging benchmarks like GSM8K and MATH, resulting in lower Pass@1 scores. However, it remains unclear whether the improvements in task loss achieved by COUNT translate into better performance on downstream tasks.

The authors only report generation accuracy for the standard Transformer architecture and do not provide corresponding results for COUNT. I suggest evaluating the existing model checkpoints on simpler generation tasks. Furthermore, applying COUNT to more powerful, publicly available pre-trained models (such as Qwen3 or Gemma4) would help assess the impact of COUNT on generation quality. I have increased the Soundness, but the rating will remain unchanged.

Reviewer_JizqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The starting point is simple: instead of learning n-gram embeddings from scratch, why not just count? The authors go through the Dolma 3 corpus ahead of time and count n-grams up to length six, scoring continuations with good old stupid backoff. For every history they keep, the table stores the top-K continuations and their scores — K = 32 in the stored table, K = 8 actually used per order during training and inference.

What the model does with this at each position is the clever part. It finds the longest suffixes of the current context that appear in the table, takes the stored continuations for each, and builds an extra input vector by averaging the model's own unembedding rows for those continuation words, weighted by their exponentiated count scores. That vector is added to the ordinary token embedding, and the sum goes through the embedding amplification trick from LongCat before entering the transformer. The table itself never gets trained. It has no gradients and no optimizer state, so it doesn't need to live on the GPU at all — it sits in host RAM. In the main experiments that's a 797 GB table for the 370M to 1B models; the smaller backbones use a much smaller 17.1 GB one

Critical strengths
  • the construction is elegant, and the offloading argument actually holds up. A history is represented by the unembedding rows of its most likely next words, weighted by their counts. That reuses a matrix the model already trains, adds no learned parameters

  • the pretraining comparison is broad, and the effect is consistent. 7 model sizes times 3 token budgets, on LM loss, task loss, and accuracy, with COUNT ahead of the plain transformer at almost every budget

  • the learned-embedding baseline is not a strawman. The authors built real optimizations for it that they didn't have to: sparse optimizer updates for just the rows that get retrieved, and a host-offloaded variant for inference that runs about as fast as COUNT

  • the systems numbers are measured, and the inconvenient results are in the paper. Throughput and peak GPU memory are measured at 1B for training, prefill, and decode, with full settings

  • The table-shrinking experiment is clean and useful in practice. All the tables share one format, so the authors can rerun the full eval suites with the 797 GB table swapped for 459, 142, 68, or 17 GB versions

Critical weaknesses
  • Matched on compute, not on data: the table is counted over the full Dolma 3 corpus, but each run trains on at most D/N = 40 tokens per parameter, against several trillion counted. At the D/N = 10 and 20 endpoints the table even contains statistics from tokens the run hasn't trained on yet. A baseline at the same C = 6ND gets none of this. The trade may be legitimate but then counted tokens are a third resource, and the paper never accounts for them: no token counts per table, no build cost for the 797 GB table and no run with counts restricted to the trained tokens. Without that run, I can't tell how much of the 44–82% equivalence is the method and how much is extra data. This is the main omission.

  • headline numbers from five-parameter fits, single seeds, no uncertainty

  • No control shows the counts themselves do the work. Every ablation changes the machinery (amplification, max order, inference-time K and J); none changes the signal. A shuffled-history or mismatched-corpus table at 190M would separate "the frequencies carry the gain" from "any well-scaled extra input vector helps."

Questions for authors
  • How many tokens were counted to build the 17.1 GB and 797 GB tables, and what did building the big table cost in CPU time? More importantly: at one scale (190M or 370M is enough), how much of the gain survives if the table is counted only on the tokens that run actually trained on, instead of the full corpus?

  • Can you put some uncertainty on the headline numbers? The 44–82% and 8–58% compute-equivalence ranges come from scaling-law fits where every COUNT run is a single seed — a bootstrap over the fit, or a second seed at a couple of ladder points, would do. While you're at it: the fit treats the table as constant, but it jumps from 17.1 GB to 797 GB between 190M and 370M. Does the fit change if that's accounted for? And what statistic are the abstract's 58% and 24%, exactly?

  • Two quick controls at one scale. What happens with a shuffled table — same size, same score distribution, wrong histories? And where does COUNT land if Fig. 3 is redrawn against total memory (GPU plus host), with the learned baseline allowed to grow beyond the 1:1 ratio?

  • For the domain swap: are the domain corpora subsets of what the checkpoints trained on (and of what the general table counted)? Was lambda = 0.25 chosen on the tasks you report, and do the gains hold if it's picked on held-out domain tasks instead? And does a domain table built from something genuinely small — say 100M to 1B tokens — still help?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jizq (1/n)

Thank you for highlighting the work we put into making the learned-embedding baseline “not a strawman” through implementing optimizations to make it more competitive with our approach (lines 452–458).

Is COUNT robust to a smaller count corpus?

“And does a domain table built from something genuinely small — say 100M to 1B tokens — still help?” (Jizq Q.4)

For math adaptation, we have now made 100M and 1B domain adaptation tables. We evaluate these with the same 25% interpolation weight in with the 1B COUNT as used in the submitted results. We can see that loss gracefully degrades with smaller count data but still improves over the unadapted model (Figure 22).

Math corpusPacked tableLoss difference (BPB)
100M tokens1.386 GB−0.00091
1B tokens10.436 GB−0.00923
Full math (10.66B tokens)83.398 GB−0.01491

We also tested limiting the domain-general count table. Counting just 100M tokens improves LM loss and task loss over the standard transformer. The full-corpus table gives better losses, but much of the benefit remains with this small sample (Figure 20).

Model or count corpusLM loss (D/N = 20)Task BPB (D/N = 20)
Standard Transformer3.291.58
Full-corpus COUNT3.221.54
100M counted tokens COUNT3.261.57

Does improvement rely on counts from unseen data?

“no run with counts restricted to the trained tokens” (Jizq W.1)

“how much of the gain survives if the table is counted only on the tokens that run actually trained on, instead of the full corpus?” (Jizq Q.1)

Responding to the concern that improvements required n-gram counts from tokens unseen by the pretraining token budget of our models, we have built count tables from the exact 1.9B, 3.8B, and 7.6B tokens used to train the model at D/N = 10, 20, and 40, respectively. When the count table is only a subset of the pretraining data (i.e. D/N = 20 and 40 for 1.9B tokens and D/N = 40 for 3.8B tokens in the experiments below) performance is similar to our headline results. Interestingly when the count table and pretraining tokens overlap completely, generalization is worse. Practically, this is easy to avoid by cheaply counting unused data (in the small model “effectively infinite data” regime of our experiments) or by fitting count tables on only a subset of the training data (with relatively little performance degradation) (Figure 19).

Model or count corpusLM loss at D/N = 10LM loss at D/N = 20LM loss at D/N = 40
Standard Transformer3.443.293.18
Full-corpus COUNT3.343.223.12
1.9B counted tokens COUNT3.913.253.13
3.8B counted tokens COUNT3.883.833.14
7.6B counted tokens COUNT3.843.793.77

Do count-based next-token predictions help beyond embedding n-grams?

“No control shows the counts themselves do the work.” (W.3)

“What happens with a shuffled table — same size, same score distribution, wrong histories?” (Q.3)

The reviewer asks whether the gain comes from count-based next-token priors or simply from adding an embedding for each n-gram. To test this we add the following 190M control experiment: for each of the histories in the original 17.1 GB table, we randomly sampled k=8 distinct indices of the unembedding matrix and assigned equal weight to each. This is effectively learning n-gram embeddings from scratch without adding new parameters by instead overloading the LM-head. This worsens LM loss and task loss compared with full COUNT (Figure 21). The model still improves LM loss over the standard transformer, suggesting that some benefit comes from simply letting the LM-head learn to represent n-grams, but this is not the optimal approach.

BudgetMethodΔ LM loss (nats)Δ Task loss (BPB)
0.5×CFull COUNT−0.102−0.024
0.5×CRandom next-token embeddings−0.062−0.010
1×CFull COUNT−0.070−0.034
1×CRandom next-token embeddings−0.033−0.012
2×CFull COUNT−0.052−0.029
2×CRandom next-token embeddings−0.018+0.011

What are the count table building costs?

“no token counts per table, no build cost for the 797 GB table” (W.1)

“How many tokens were counted to build the 17.1 GB and 797 GB tables, and what did building the big table cost in CPU time?” (Q.1)

Both generalist tables count approximately 5.9T Dolma tokens (lines 164–165). Offline count extraction and table preparation required approximately one week of 96-vCPU EC2 instance time, based on reconstructed runtimes (lines 171–172).

Continued in thread...

AuthorsOpenReview ↗

Response to Reviewer Jizq (2/2)

How to account for uncertainty in scaling fits?

“headline numbers from five-parameter fits, single seeds, no uncertainty” (W.2)

“Can you put some uncertainty on the headline numbers?” (Q.2)

We prioritize more ladder compute budgets instead of repeated runs at the same budgets so that additional runs increase the support as well as statistical power of our scaling fits (lines 159–161). We do have 4 comparable seeds for the 190M standard transformer: at D/N = 20, the standard errors were 0.35 percentage points for accuracy, 0.0023 nats for LM loss, and 0.0033 BPB for task loss (lines 243–245). We added a range of fitted curves under leave-one-checkpoint-out resampling of the scaling fits in the updated PDF, which show that the different treatments are still cleanly separated (Figures 1–2 and lines 264–266).

“what statistic are the abstract's 58% and 24%, exactly?”(Q.2)

These are the extra transformer compute needed to match COUNT’s LM and task losses at D/N = 20 at 1B parameters, as estimated by our scaling fit on the standard baseline. Note that with the updated baseline in the new PDF specific quantities have changed, but our claims hold (Figures 1–2).

“While you're at it: the fit treats the table as constant, but it jumps from 17.1 GB to 797 GB between 190M and 370M. Does the fit change if that's accounted for?” (Q.2)

We chose table size to fit the RAM available on the hardware used to train a given model size (lines 167–172). So we don’t have a range of table sizes available across all model sizes to fit a table size parameter in our scaling law. We swept table sizes at 190M (Figure 23). Larger tables consistently improve LM loss, while task loss varies across table sizes.

Packed tableΔ LM loss (nats)Δ Task loss (BPB)
0.535 GB−0.035+0.004
4.279 GB−0.053−0.003
17.114 GB−0.070−0.034
68.458 GB−0.083−0.035
142.037 GB−0.099−0.053
459.222 GB−0.115−0.015

How do larger learned embeddings compare on the memory-performance tradeoff?

“And where does COUNT land if Fig. 3 is redrawn against total memory (GPU plus host), with the learned baseline allowed to grow beyond the 1:1 ratio?” (Q.3)

If GPU and host RAM are considered equivalent costs both learned embeddings and COUNT underperform simply scaling dense backbone parameters in a standard transformer. We argue that this is not a fair representation, because COUNT uses host RAM that is not already being used by backbone parameters (lines 269 and 303–304). When learned embeddings are larger than the 1:1 ratio, dense backbone params are still a better use of GPU RAM. At D/N = 40, a 17GB learned embedding 190M backbone model trains on 7.61B tokens and gets LM loss 3.08, task BPB 1.47, and accuracy 33.06%. The 760M dense Transformer at D/N = 10 trains on 7.58B tokens and gets 2.95, 1.38, and 35.16%, respectively. It has only 0.91B total learned parameters, compared with 8.83B for the learned-table model (lines 315–320).

Domain interpolation configurations

“Was lambda = 0.25 chosen on the tasks you report, and do the gains hold if it's picked on held-out domain tasks instead?” (Q.4)

Yes, we selected 25% domain interpolation weight as the only setting in our grid that lowered loss across all domains we report (Figure 12). We have repeated selection while holding out each domain or group of related tasks (lines 402–403). All three domain folds and all ten task-group folds selected the same 25% weight.

“are the domain corpora subsets of what the checkpoints trained on (and of what the general table counted)?” (Q.4)

The math table uses Dolmino Math midtraining data not in Dolma 3. The biomedical and law tables use the health and crime-and-law subsets of Dolma 3’s olmOCR science-PDF source also used for generalist counts (lines 365–369 for domain sources and lines 164–165 for the generalist corpus).

I am keeping my rating at weak accept. This was a thorough rebuttal, and I want to start with what it resolved, because it is a lot.

The build cost is now a real number. Both generalist tables count about 5.9T Dolma tokens, at roughly a week on a 96-vCPU instance. The abstract's 58% and 24% are pinned down, and the leave-one-out resampling ranges are a fair answer on fit uncertainty. The lambda selection holds up better than I expected: every held-out domain fold and task-group fold re-selects 25%. I was wrong about the math table. Dolmino Math is not part of Dolma 3, so that one is a genuine out-of-corpus adaptation. The small-table results also help: a general table counted on just 100M tokens still improves on the standard transformer (3.26 against 3.29). And the random-continuation control is informative — it recovers part of the gain, so some of the effect is the pathway itself, but the counts add clearly on top.

What keeps me at weak accept is the restricted-count table itself, which I read as cutting both ways. Where the counted tokens are a subset of what the run trains on, the gain survives almost intact: the 1.9B table at D/N = 20 and 40 gives 3.25 and 3.13, against 3.22 and 3.12 for the full-corpus table. But where the counted set matches or covers the training tokens, it falls apart — the same 1.9B table at D/N = 10 gives 3.91 against the standard transformer's 3.44, and the 7.6B table at D/N = 40 gives 3.77 against 3.18. You name the mechanism, and the workaround is cheap, but the boundary between "the method works" and "the table has memorized the training data" is exactly the data-exposure question my review asked about, and so far it is mapped only coarsely by these three tables, not by a controlled sweep of the overlap. I do credit you for reporting the bad cells instead of hiding them, not every rebuttal would

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reviewer Response Summary

We thank the reviewers for their questions. Since the submission we have run additional experiments, including most of those requested by the reviewers. These are detailed fully in the updated PDF and highlights are provided in the reviewer responses as relevant to the reviewers questions. Unfortunately, the requested scaling of our experiments to 2B and 4B parameter runs is beyond our compute budget. Our new experiments show:

  1. COUNT improves relative to the standard transformer even when the count table is built from just 100M tokens (Figure 20). We also rerun domain adaptation in math with 100M and 1B token count tables and still find improvement (Figure 22).
  2. Counts from a subset of the pretraining data still help. Held-out loss rises when the count corpus exactly matches the tokens seen in training, then recovers as training continues beyond those tokens (Figure 19).
  3. Assigning random next-token embeddings with equal weights to each n-gram history rather than using count-based scoring worsens performance, showing that corpus counts do matter for improvement (Figure 21).
  4. COUNT improves performance more than two inference-only methods that combine model predictions with n-gram counts at decoding time (Figure 9).
  5. We corrected the batch size configuration for the standard transformer baseline. The paper’s main conclusions still hold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.