acceptodds
Under review as a conference paper at ICLR 2027

PAT: Bending Parameter and Context Scaling Curves through Efficient Context Rewrites

Abstract

Causal full self-attention (FA) caches token representations without revising them when later evidence arrives. This limits efficient storage and relationship composition. We analyze local context rewriting, which converts completed chunks into memory states trained through future-token prediction. These states compose relationships in parallel (A→B and B→C yield A→C) and remove redundant distinctions in the cache, freeing capacity for prediction. Our analysis shows that, under locality assumptions, rewriting reduces the expected layers needed to combine a chain of relationships from linear to logarithmic in its length. The Predictive Abstraction Transformer (PAT) implements these context rewrites within FA’s compute budget by attending to a smaller cache. Its KV cache grows at about one-third FA’s rate, training is fully parallel, and measured training throughput exceeds NanoGPT’s. Experiments show a widening quality advantage over FA as context length and model size grow, while PAT’s relative compute falls with context length. At matched compute within each context length, PAT’s perplexity reduction grows from 4.2% at 2K to 8.0% at 8K. At approximately 97 EFLOPs, 379M PAT achieves lower perplexity than an FA model with 2.3× as many parameters (861M). PAT also improves recall at lengths beyond training: after training on sequences up to 256 tokens, it achieves 95.3% multi-query associative-recall (MQAR) accuracy at 1,024 tokens, versus FA’s 31.9% and 31.7% for KDA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention mechanisms, language modeling, Transformer architectures, long-context performance, latent representations, memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hqTEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PAT (Predictive Abstraction Transformer), a KV-cache-efficient architecture that periodically compresses local tokens into compact representations. The paper frames the compression operation as ‘rewriting’ the given context into a more compact and informative representation. The proposed architecture showcases strong empirical performance in language modeling capabilities, consistently achieving improved perplexity compared to the variant with full self-attention despite being more efficient. PAT also showcases comparable downstream performance to full self-attention variant while using significantly less KV cache.

Critical strengths

The paper addresses an important problem of fundamentally reducing the KV cache requirements of LLMs. The empirical results are strong, with PAT consistently achieving lower perplexity than the full self-attention baseline despite using smaller KV cache, and shows comparable performance on downstream tasks with better efficiency. The paper also provides a thorough theoretical analysis to back up the approach.

Critical weaknesses

I did not find any critical weakness.

Questions for authors

I do not have any further questions that may materially affect my assessment.

1 comment

Thank you for your analysis of our work and the positive review!

Some of our other contributions are:

  • PAT bends the parameter and context scaling curves, which shows that as we scale the model size and increase the context length, the performance gain of PAT over FA increases.
  • In our improved theoretical analysis we show that PAT's attention remains selective as the context length grows with a fixed parameter count, while full attention needs more parameters.
  • On synthetic tasks, PAT generalizes to unseen lengths and layout changes, achieving significantly higher average accuracy than all five other models evaluated, including full attention, two memory-based methods, and Kimi Delta Attention.

Please let us know if there is anything else we can clarify or answer for you.

Reviewer_E93zOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the Predictive Abstraction Transformer (PAT), which chunks the input sequence and stores memory tokens per chunk which future tokens attend to (outside of a sliding window of a fixed size), in contrast to full attention (FA) which attends to all previous tokens. This enables the model to compose associations between tokens within a chunk in the compressed memory representation, including later tokens in the chunk. This also means that the model only has to maintain a fraction of FA's KV-cache, improving inference efficiency. In addition, the paper provides a thorough theoretical analysis of local context rewriting. PAT is evaluated against FA on language modeling perplexity and long-context tasks, and against FA, KDA and GSA on synthetic recall, and shown to achieve lower language-modeling perplexity and improved long context performance.

Critical strengths
  • Well motivated idea with an intuitive design and implementation.
  • Thorough and transparent reporting of data.
Critical weaknesses

The main weaknesses are in the experiment methodology:

  • No learning rate or batch size tuning. Every model from 124M to 1.7B, both PAT and FA use the same learning rate and batch size. Different sizes will prefer different learning rates, and more importantly, there's a chance that different architectures will have different optimal learning rates too.
  • No baselines from the reported related work on language modeling or long context tasks. Section 2 covers several related methods, but it is not clear where PAT empirically stands compared to other existing memory token/compression methods.
  • All language-modeling runs use a single seed, with conclusions drawn in many cases with only small perplexity gaps (for example, "379M PAT outperforms an 861M FA model" with a difference of ~1%, equal token gaps of ~1-4%, data-slice gap of 2.2-3.1%, and most ablations under ~2%). It is not clear how much of the difference is a result of the architecture versus pre-training noise [1].
  • A lot of the models are undertrained; a 100B corpus was put together, but at most 33B tokens were used for any model. The 1.7B model wasn't trained for more than ~11B tokens, which is among the most severely undertrained, and all the long-context benchmarks use it. FA gets 46.9% multikey at 16K vs 97.1% for Qwen3 1.7B base, so it's unclear whether the gap persists against a well-trained FA model. In the 1.7B data-scaling fit, both the PAT and FA models are early in training. The gap widens from 2.2% to 3.1% but might reverse with longer training, and a 0.003 difference in slope without error bars can't tell either case apart.
  • Matching compute gives PAT a small advantage with it receiving ~7-14% more training tokens at the largest budget (and up to ~23% for the smaller models), potentially masking differences purely due to architecture. With context scaling, matching compute within each window costs FA's longer windows training tokens, which might explain "FA's loss does not improve from 4K to 8K". In addition, table 8 shows that FA's perplexity does improve from 4k to 8k at equal tokens. Moreover, measured training throughput practically matches that of GPT (Section 6.5), so the FLOP savings may not exist in practice.

[1] Zeyuan Allen-Zhu. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.

Questions for authors
  • Can the authors sweep the learning rate (at least 2 or 3 LRs) for PAT and FA with their setup?
  • Table 12's "causal within-chunk memory writer" and "writer without historical reads" ablations seem to be similar to how UniGist and KSA are described in the related work section respectively, but differ by only ~0.1-0.2% in perplexity from PAT. Could the authors compare PAT against the actual methods (UniGist and KSA) empirically?
  • Can the authors report at least 2-3 pre-training seeds for the main comparisons and for the Table 12 ablations?
  • With the 100B corpus available, does the performance gap hold for the 1.7B model at 50B+ tokens?
6 comments

Thank you for reviewing our work.


General clarification

In this work, our goal is to show that local context rewriting improves the performance of full attention and has asymptotic gains over full attention, in parameter efficiency and in the use of long context. We show this theoretically and empirically.

  • Scaling laws predict performance as parameters, data or compute scale, which makes them the standard way to compare new architectures. PAT's gain widens with model size and context length, which matters most at very large scale; its nearly constant-factor gain with data is also valuable, as it shows the gains hold with more training.
  • Our scaling laws, 121 checkpoints plus 19 new, give PAT and FA identical learning rate, batch size and seed at every size and training length. This ensures a fair, matched comparison of how the architectures scale. We ran new experiments on your concerns, varying these settings: PAT has lower perplexity in every paired comparison.
  • Our five-seed synthetic study now includes KSA; PAT is the best and most stable of six models.
  • We also trained the paper's 254M models longer, to 50B tokens, and its 1.7B models to 20B; PAT's gain holds, as our data-scaling curves predict.

Learning rate and batch size tuning

"No learning rate or batch size tuning."

By design, both architectures share one recipe in the scaling laws, so that only the architecture differs; in new runs, PAT has lower perplexity at 0.5x, 1x and 2x the original learning rate and 5x the original batch size.

  • Same hyperparameters. PAT and FA share every training hyperparameter, including the original paper's learning rate of 2.5×10⁻⁴. Each run's learning-rate schedule matches its training horizon, the established practice that gives the best final loss at every model size [1].
  • Critical batch size. The critical batch size marks where a larger batch starts to need more training tokens to reach the same loss. Theory and experiments show that batch sizes well below it reach a given loss with almost the same number of training tokens [2], [3, Section 5.1]. It depends only on the loss the training reaches, not on the model size, and by Kaplan et al.'s fit [3] is about 1-2M tokens at the paper's compute-optimal losses. The original paper's batch size of about 100K tokens is well below it.
  • New results. PAT's advantage holds at every tested learning rate, batch size and seed. At matched tokens, PAT's perplexity is 1.4%, 2.0% and 3.4% lower than FA's at half, the original and double the learning rate (3.9%, 4.1% and 5.4% at matched compute), and 3.2% lower at five times the batch size (5.8% at matched compute).

Table 1: Learning rate and batch size (new). 161M models at 13.3B tokens (curve B of Figure 1(C)); iso-FLOP: FA trains on 11.1B tokens. FA and PAT columns: iso-token perplexity; gains from unrounded values (Appendix A.1, Table 2). OG: the original paper's setting.

SettingFAPATIso-token gainIso-FLOP gain
Learning rate 1.25e-4 (0.5x)18.77618.5151.4%3.9%
OG (2.5e-4; 100K-token batch)18.28817.9142.0%4.1%
Learning rate 5e-4 (2x)18.26217.6373.4%5.4%
Batch 500K tokens (5x)19.49918.8783.2%5.8%

Comparison with related methods

"No baselines from the reported related work on language modeling or long context tasks"

The original paper compared PAT with four other methods, including the memory-based GSA and KDA, a strong long-context attention alternative, on 12 synthetic tasks that include long-context extrapolation and robustness to layout and noise changes. We added KSA to this comparison. UniGist has no released code.

  • Synthetic comparison. PAT leads KSA by 30 points in average accuracy and trains 2x as fast (Tables 2 and 3). Synthetic tasks are the field's standard way to diagnose specific capabilities of new architectures, such as recall, denoising and the composition of multi-token keys [4]-[8]. The sweep has 12 tasks: MAD's six, two of MQAR's long-context settings and four in-context-recall (ICR) denoising and robustness tests.
  • Best of six models. PAT has the highest average accuracy, the most stable performance and the shortest training time of the six models; its standard deviation over seeds is 1.4 points against KSA's 5.1, averaged over the tasks both solve (Appendix D.1, Table 10). The other four are two full-attention implementations (FA and nanoGPT), GSA and KDA.
  • UniGist and KSA. UniGist compresses a pretrained model's KV cache rather than training from scratch. KSA compresses its distant context, each 8-token chunk into one token, and keeps the keys and values of the nearest 1,024 tokens uncompressed, yet nearby tokens matter most for composition [14, 15]. PAT instead rewrites every chunk into a large bidirectional memory latent space, which is expanded in the middle layers, and each token sees at most K=16K = 16 lexical tokens. Additionally, KSA's end-to-end implementation ran 2x slower than PAT on the same hardware.

Table 2: Synthetic accuracy (%), mean over five seeds (the paper's Table 1; the KSA column is new).

PATFAnanoGPTKDAGSAKSA
12-task average83.068.533.963.745.652.6
ICR robustness (four tests)80.256.34.061.143.217.0
MQAR at 4x training length95.331.947.431.710.540.0

Table 3: Training time per synthetic run (new) on one H100: PAT in minutes, the others as multiples of PAT's time; KSA in its faster released mode (Appendix D.2, Table 11).

PATFAnanoGPTKDAGSAKSA
46.0 min1.38x1.65x1.99x2.14x2.01x

Stability across seeds

"All language-modeling runs use a single seed"

Scaling laws are robust comparisons because they span many model sizes and training horizons: here, 9 sizes trained to 2-7 horizons each, 140 checkpoints in total, with PAT achieving lower perplexity in every matched comparison. We found no prior work that repeats a full architecture-comparison scaling law over seeds. New experiments with a second pretraining seed show that PAT's gains hold.

  • Stable scaling laws. Our scaling-law results are stable: no single run drives their trends. Refitting Figure 1(B)'s curves without any one model size or training length, including the ends, leaves PAT's curve steeper than FA's in all twelve refits (Appendix B).
  • Initialization and width. External work shows that the effect of the initialization seed diminishes with width: a trained network's output varies less with its random initialization as the network widens [9].
  • Second pretraining seed. PAT's gain holds with the second seed you requested, run for the 161M models (Table 4).
  • Five synthetic seeds. PAT is the best and most stable of the six models in the synthetic experiments, repeated over five seeds (Table 2). Here, stability is the standard deviation of accuracy over the initialization seeds [10], averaged over the tasks both models of a pair solve, so that failed runs [11], which can show near-zero spread, do not count as stable. PAT's standard deviation is lower in all five pairs (Appendix D.1, Table 10).

Table 4: Second pretraining seed (seed 43; new). 161M models at 13.3B tokens (curve B of Figure 1(C)); iso-FLOP: FA trains on 11.1B tokens. Validation perplexity; gains from unrounded values (Appendix A.1, Table 2).

Model (seed 43)TokensPerplexityPAT's gain
PAT13.3B17.819—
FA, iso-token13.3B18.3623.0%
FA, iso-FLOP11.1B18.7795.1%

Longer training

"A lot of the models are undertrained"

Our data scaling shows a nearly constant-factor gain, which predicts stable gains with longer training. We trained the paper's 1.7B models to 20B tokens and its 254M models to 50B; both show that PAT's advantage remains stable.

  • IsoFLOP curves. Large models trained on few tokens are part of Chinchilla scaling curves [1], from which larger runs are predicted [12], [13]. Each IsoFLOP curve fixes the training FLOPs and sweeps from small models trained on many tokens to large models trained on few [1]. In Figure 1(C), PAT's highest-compute curve runs from the 379M model at 32.6B tokens to the 1.7B model at 7.5B.
  • Data scaling. Predicting how the gain changes as a fixed-size model keeps training is the purpose of data-scaling curves. The paper's curves (Figure 1(B)) show a nearly constant-factor gain. Thus, with longer training, the established trend is that FA needs 10-18% more training tokens to reach PAT's loss.
  • New longer training. We trained the paper's checkpoints for longer to provide more evidence for this prediction and reran the long-context downstream evaluations (Tables 5 and 6): PAT's gain remains 2.8% at both 24.1B and 50B tokens for the 254M models, and is 3.1% at 9.5B tokens and 3.0% at 20B for the 1.7B models. At 16K context, PAT's long-context lead over FA increased from 10.6 points at 9.5B tokens to 12.3 points at 20B.

Table 5: Longer training of the paper's checkpoints (new), as far as our compute and time allowed. OG: the original paper's checkpoints (the paper's Table 4). Validation perplexity; gains from unrounded values (Appendix B.1, Table 7).

ModelComparisonTokens (FA / PAT)FAPATGain
254Miso-token, OG24.1B / 24.1B15.62015.1892.8%
254Miso-token50B / 50B14.60214.1942.8%
1.7Biso-token, OG9.5B / 9.5B13.62213.2003.1%
1.7Biso-token20B / 20B12.05711.6913.0%
1.7Biso-FLOP, OG7.0B / 7.5B14.42313.8194.2%
1.7Biso-FLOP18.65B / 20B12.18311.6914.0%

Table 6: Long-context evaluations of the 1.7B models, iso-token, where FA uses 7.3% more compute: equal-weight average of seven tasks (%). 9.5B: the original paper's checkpoints; 20B: longer training (new), which the revision reports in place of the 9.5B results. Leads (PAT minus FA) in points, from unrounded values (Appendix C, Table 8).

Context9.5B FA9.5B PAT9.5B lead20B FA20B PAT20B lead
4K25.9636.66+10.727.5237.64+10.1
8K20.9730.85+9.924.3434.67+10.3
16K14.2324.80+10.616.6228.95+12.3

Matched compute and matched training tokens

"Matching compute gives PAT a small advantage"

PAT has lower perplexity in all 62 of the revised paper's paired comparisons, including all 24 token-matched comparisons.

  • Token-matched comparisons. PAT has lower perplexity in all 24 token-matched comparisons in the paper: context scaling at four model sizes (Appendix B); the data scaling of the 1.7B models; the Kaplan parameter scaling at 24.1B tokens (Figure 1(B)); and 2 from the longer training added for this response (Appendix B.1). These results do not depend on analytical FLOP counts. The gain widens with model size: PAT's fitted slope is steeper, −0.066 against FA's −0.058. It also widens with context length, from 2K to 8K at all four sizes (379M: 4.3% to 6.2%).
  • Compute-matched comparisons. Matching compute is a fair and standard comparison: analytical FLOP counts track the cost of training and scaling up an architecture more closely than training-token counts do, and scaling laws measure that cost in FLOPs [1], [3]. When PAT is trained on more tokens at matched compute, it is because each of its tokens costs fewer FLOPs, which is the efficiency the comparison measures.

Thank you again for your time and the detailed review. We hope this answers your questions and concerns. Please let us know if you have any further questions.

Reviewer_YcMkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors claim that this rewriting mechanism enables better long-context language modeling. However, full rewriting requires (O(N^3)) attention and is impractical for long sequences. The paper therefore introduces a local rewriting mechanism that is computationally more practical. The proposed PAT architecture achieves lower perplexity than full-attention models while using a smaller KV cache.

Critical strengths
  1. A useful empirical result. Bidirectional memory lets PAT reduce loss with a much smaller KV cache. Many KV-cache compression methods lose quality when the cache shrinks. This result is therefore interesting.

  2. A detailed theory. The paper explains why local rewriting may help language modeling. It is not only an empirical architecture.

  3. Broad scaling experiments. The experiments show that PAT beats FA across model sizes, compute budgets, and context lengths.

Critical weaknesses
  1. The forward process is unclear. Section 4 and Figure 2 only give a high-level view. Important details are missing: the exact attention masks, how token and memory queries are formed, and how memory states pass between layers. The authors should add equations or pseudocode.

  2. There is no direct ablation. The paper does not compare bidirectional memory updates with causal-only updates, or rewriting with no rewriting, under matched parameters and compute. This ablation would show whether the gains come from rewriting or from extra memory parameters and compute. A fully-rewriting, locally-rewriting and no-rewriting comparisons will make the conclusion more strong.

  3. The training budget may weaken the scaling results. Standard practice scales tokens with model size. This paper uses a fixed budget, so larger models get fewer tokens. This may limit their performance. For example, two classical settings are a 400M model with 15B tokens and a 1.5B model with 100B tokens [1].

[1]: Parallelizing Linear Transformers with the Delta Rule over Sequence Length

Questions for authors

see weakness

5 comments

Thank you for reviewing our work.


General clarification

In this work, our goal is to show that local context rewriting has a widening gain over full attention, where the method's advantage grows as model size and context length increase. We show this theoretically and empirically.


Forward pass and attention masks

"The forward process is unclear."

Section 4 now gives the forward pass as equations (3)-(4), consolidating the original Sections 1 and 4, Figure 2 and Appendix I.1.

  1. Input. The sequence X1M1X2M2⋯X_1M_1X_2M_2\cdots interleaves each chunk XcX_c of K=16K=16 lexical tokens with its group McM_c of memories, copies of one special vocabulary item.
  1. Queries, keys and values. Lexical and memory states hih_i use the same projections, (qi,ki,vi)=(WQhi,WKhi,WVhi)(q_i,k_i,v_i)=(W_Qh_i,W_Kh_i,W_Vh_i); a group's memory keys share one rotary position and its queries take distinct rotary offsets, so identical copies can attend differently.
  1. Attention. Each query ii reads R(i)R(i) (Table 1) in one softmax,

    zi=hi+WO∑j∈R(i)softmaxj∈R(i)(qi⊤kj),vj,z_i=h_i+W_O\sum_{j\in R(i)}\mathrm{softmax}_{j\in R(i)}(q_i^{\top}k_j)\\,v_j,

    with output projection WOW_O; prediction stays causal because R(t)R(t) excludes McM_c, which has read the whole chunk.

Table 1: The attention mask, equation (3). WtW_t is lexical token tt's KK most recent lexical tokens and M<cM_{\lt c} the memories of all earlier chunks; a memory reads its own group McM_c bidirectionally.

QueryLayer 1 readsLater layers read, R(i)R(i)
Lexical token t∈Xct\in X_cWtW_tWt∪M<cW_t\cup M_{\lt c}
Memory m∈Mcm\in M_cXc∪McX_c\cup M_cXc∪Mc∪M<cX_c\cup M_c\cup M_{\lt c}
  1. Update, equation (4). Memories skip the FFN, so a memory's next-layer state is its state after attention, hm′=zmh'_m=z_m.
  1. Next layer. The memories' keys and values alone carry forward the context beyond each token's window, replacing those of older lexical tokens; this replacement enables the context rewrite.
  1. Middle expansion (2-8-2). Groups hold 2 memories in the first and last quarters of the layers and 8 in the middle half; learned maps resize them between stages.
  1. Training. PAT trains fully in parallel across positions and chunks, with only the layers sequential, as in full attention, because every query reads preceding-layer states; only lexical tokens receive the next-token loss.

Architectural ablations

"There is no direct ablation."

Ablations were in the appendix; the results have now been moved to the main paper. These experiments isolate rewriting as the source of PAT's gain, and show how each of PAT's implementation designs contributes to the gains.

  • Rewriting. Full attention (FA) is PAT without rewriting, with the same backbone, data and optimizer; its perplexity is 5.8% higher at matched parameters and compute.
  • Bidirectional memory. In PAT, memories of the same chunk attend to each other bidirectionally. Making this attention causal raises perplexity by 0.2%; each memory still reads its whole assigned chunk and prior memories.
  • Memory allocation. More memories per chunk lower perplexity monotonically at equal tokens, by 6.4% from 1-4-1 to 16 in every layer; at equal compute, the default 2-8-2 is best (the revision's Table 14).
  • The revision brings both ablations into the main text (Section 6.5) from the original Appendix E (Table 2).

Table 2: Ablations in the original paper (Table 12 of the revision), 161M models at 16K context and matched parameters. Each PAT variant changes one component and trains on PAT's 6.73B tokens; FA matches PAT's compute with 5.62B tokens. Rewriting: memories replace older lexical tokens' keys and values. Attention and positions are those among a chunk's memories, and memory counts are per chunk. Fixed averages: of contiguous groups of the chunk's token states. Historical reads: memories read earlier chunks' memories.

PAT's designAblationPerplexity increase
RewritingFA, iso-FLOP+5.8%
Bidirectional memory attentionCausal memory attention+0.2%
Shared positionDistinct positions+1.7%
Learned memoriesFixed averages+1.1%
Middle expansion: 2-8-2No middle expansion: 5 per layer+0.4%
Historical readsNo historical reads+0.1%

No extra memory parameters or compute

"...whether the gains come from rewriting or from extra memory parameters and compute."

Every paired comparison matches parameters and either compute or training tokens.

  1. Parameters. PAT and FA have the same parameter count at every model size (the revision's Table 3). Memories do not add any parameters: they reuse the lexical tokens' projection parameters and are copies of one learned embedding when packed in the data. The expansion of memories in the middle layers adds a negligible 0.00006% of the 1.7B model (Appendix A).
  1. Compute. All compute-matched comparisons set the token horizon of each architecture so that their analytical FLOP counts are equal (Appendix I). The analytical compute calculation of PAT adds the cost of the memories' projections and subtracts the attention pairs saved. These pairs are saved because fewer memories are allocated per chunk than the chunk size (on average 5 memories for each 16 tokens), which yields a constant-factor reduction in the number of attention pairs compared with full attention.
  1. Comparisons. PAT has lower perplexity than FA in all 62 paired comparisons: 38 at matched compute and 24 at matched training tokens (Section 6.1; Appendix B, Tables 4, 6 and 7).

Local rewriting, full rewriting and full attention

"A fully-rewriting, locally-rewriting and no-rewriting comparisons will make the conclusion more strong."

Full rewriting is effectively not runnable due to its cubic cost in sequence length. PAT implements local rewriting hardware-efficiently, and no rewriting is equivalent to our FA baseline.

  • Full rewriting. Re-encoding the whole prefix for every prediction makes its attention cost cubic in context length: the 1.7B model would need at least 7 months on 8 H100s (Appendix I.2), so it cannot be trained. PAT leverages locality of language [1, 2] to implement context rewrites efficiently with faster runtime than NanoGPT. Moreover, external results show that predictive information and compositions are concentrated in the local window, with gains from longer context following a power law [4, 5]. Therefore, the gains of full context rewriting would have diminishing returns.
  • Full attention. FA shares PAT's implementation and model backbone (normalization, FFNs, etc.) without rewriting, and PAT's perplexity reduction over FA widens with context and model scale (Figure 1; Appendix B).
  • Memories alongside full attention. We did an ablation between FA and PAT where memory tokens are added per chunk and are read at the next layer causally, along with all lexical tokens' keys and values, without replacing them. This experiment gave 19.838 perplexity, 1.3% higher than the corresponding PAT control with a uniform allocation of five memories in all layers, and 1.63% behind the baseline PAT. This control matches PAT's training tokens and uses about 1.5x PAT's compute, yet it is still behind PAT; therefore, rewriting the context gives PAT its lead.

Training budgets

"Standard practice scales tokens with model size. This paper uses a fixed budget"

PAT maintains its advantage in both matched-token and matched-compute comparisons. Various training horizons were tested as part of the scaling laws.

  • Growing compute. Each curve in Figure 1(C) fixes the training compute and sweeps model size [3]; tracing the starred points down the curves gives a line along which the training tokens increase with model size, the scaling you asked about. PAT's advantage holds as compute grows 11x from curve A to curve D: all 25 paired comparisons on the four curves favor PAT, and PAT's best model has 5.5-6.7% lower perplexity than FA's best on each.
  • Context scaling at matched tokens. PAT's advantage holds at every size and context length when Figure 1(A)'s compute-matched comparisons are repeated at matched tokens (Appendix B, the revision's Table 6); at 379M it grows from 4.3% at 2K to 6.2% at 8K.
  • One model size, several horizons. PAT maintains its advantage as the same model trains 11x longer: on the four curves, the 1.3B PAT model trains to 0.88B, 1.75B, 4.62B and 9.88B tokens, and its gain over FA of the same size and compute is 3.9%, 4.4%, 5.6% and 5.4%.
  • Longer training. We trained the paper's 1.7B models longer, from 9.5B to 20B tokens, for this response. As expected, the gain holds, at 3.1% and 3.0%. At 16K context, PAT's long-context lead over FA increased from 10.6 points at 9.5B tokens to 12.3 points at 20B (Table 3).

Table 3: Longer training of the paper's 1.7B models (Appendices B.1 and C, Tables 7 and 8). Both architectures train on the same tokens or compute (iso-FLOP) in each row; OG is the original paper's checkpoint (the original paper's Tables 6 and 9). Gain is PAT's perplexity reduction; long context is the average over seven RULER and BABILong tasks at 16K, in percent, and lead is PAT's lead in points.

TokensPerplexity, FA / PATGainLong context, FA / PATLead
9.52B (OG)13.622 / 13.2003.1%14.23 / 24.8010.6
20B (longer)12.057 / 11.6913.0%16.62 / 28.9512.3
18.65B / 20B (iso-FLOP)12.183 / 11.6914.0%——

To make key results easier to find, the revised paper brings them from the appendix into the main text and is clearer throughout; its improved theory parameterizes locality instead of assuming a power law.

Thank you again for taking the time to review our work. We hope this answers your questions and concerns. Please let us know if you have any further questions.


References

[1] Khandelwal et al., Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context, ACL, 2018. link

[2] Sun et al., Do Long-Range Language Models Actually Use Long-Range Context?, EMNLP, 2021. link

[3] Hoffmann et al., Training Compute-Optimal Large Language Models, arXiv, 2022. link

[4] Hahn et al., Modeling Word and Morpheme Order in Natural Language as an Efficient Tradeoff of Memory and Surprisal, Psychological Review, 2021. link

[5] Xiong et al., Effective Long-Context Scaling of Foundation Models, NAACL, 2024. link

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.