acceptodds
Under review as a conference paper at ICLR 2027

ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory

Abstract

What determines whether a language model extrapolates beyond its training length? We study attention–memory interactions and checkpoint variability using ATMA, a 378M-parameter hybrid combining Polar Attention with gated-delta recurrent memory. Polar uses a normalized direction, a bounded participation-ratio magnitude, and learned length scaling. In a complete 120-cell, 1B-token factorial, memory improves Polar's teacher-forced target-token retrieval at 64K in all 20 matched cells, by 47.8 percentage points on average. We then train matched NoPE, RoPE, and Polar models for 9.816B tokens at length 2K. The primary Polar checkpoint retains 62.7% exact FinePDFs retrieval at 8K and 34.4% target-token accuracy and 9.0% exact five-token accuracy overall at 256K, with all exact successes on synthetic haystacks. Two fresh paired runs retain a retrieval advantage over NoPE but reach only 0.0% and 3.83% exact accuracy. A post-hoc intervention caps the most retentive head in each checkpoint: primary NoPE and Polar 256K bits per byte fall from 8.097 to 1.595 and from 1.825 to 1.501. Both NoPE replications recover the likelihood improvement; the Polar replications have no comparable gate outlier and remain essentially unchanged by the cap. A separately optimized TDA hybrid reaches 0.2% token retrieval but 40% adapted BABILong accuracy at 256K, exposing distinct retrieval and reasoning trade-offs. Prospective controls resolve attribution: across three matched initialization triplets, Polar strictly outperforms temperature-matched softmax on natural-text exact retrieval by an average of 22.18 percentage points, and retains a 37.18-point advantage across 30 independent documents. The one-head retention cap rescues likelihood without conferring a general retrieval boost, delineating the boundaries of retention stability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Context Extrapolation, Hybrid Architectures, Polar Attention, Gated-Delta Recurrent Memory, Test-Time Memorization, Associative, Recall & Retrieval, Attention-Memory Interactions, Retention Horizon, Checkpoint Variability, Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ihsTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents ATMA, a 378M-parameter hybrid language model combining local gated convolutions, Polar Attention, and gated-delta recurrent memory. Polar Attention uses a normalized direction channel, a bounded participation-ratio magnitude channel, learned length-dependent temperature scaling, and a virtual null competitor. The study examines how attention and recurrent memory interact during length extrapolation and why checkpoints with similar short-context validation losses can exhibit different long-context behavior. The experimental program includes a 120-cell factorial study, matched NoPE/RoPE/Polar models trained on 9.816B tokens at a sequence length of 2K, additional paired training runs, and temperature-matched component controls. Evaluation extends to 256K tokens and covers teacher-forced retrieval, fixed-target likelihood, adapted BABILong reasoning, short-context tasks, and serving performance. Separately optimized Raven and TDA models provide additional reference points. The reported results show that memory improves Polar’s retrieval performance in the factorial study and that Polar outperforms the tested NoPE and temperature-matched controls on selected retrieval evaluations. Extreme-length accuracy varies across training runs, with exact retrieval successes at 256K confined to synthetic haystacks. The paper also investigates near-unit retention gates through an inference-only, single-head retention cap. This intervention improves long-context likelihood in the NoPE replications, while its effects on retrieval and Polar checkpoints vary, distinguishing retention stability from associative retrieval capability.

Critical strengths
  • The complete factorial study and additional paired training runs provide useful empirical evidence.
  • The single-head intervention identifies a reproducible likelihood failure in NoPE checkpoints.
  • The paper explicitly reports negative results and distinguishes synthetic retrieval from natural-text retrieval.
Critical weaknesses

1. Important length-extrapolation baselines are missing.
The principal matched comparisons use NoPE and vanilla RoPE. Relevant alternatives such as ReRoPE and YaRN are absent, leaving unclear whether redesigning attention is preferable to existing extrapolation techniques. Training-free methods and methods requiring adaptation should be evaluated separately, with matched additional training budgets. Raven/TDA change architecture, optimization, and other factors simultaneously and cannot substitute for these controls. The temperature-matched comparison is useful but covers only the 1B-token regime, not the headline 9.816B-token setting.

2. The component-level conclusions exceed the evidence.
E1 establishes that the complete Polar configuration outperforms temperature-matched softmax; it does not establish that direction normalization, the null competitor, and the magnitude channel are individually necessary. Indeed, Table 7 shows that fixing the length-dependent null slope near zero improves 64K retrieval from 77.50% to 91.25%, while direction-only retains 71.25%. This directly challenges the empirical justification for the full proposed recipe. The 120-cell factorial does not replace multi-seed ablations of these defining components. The conditional theoretical bounds likewise do not establish preservation of relevant information.

3. Extreme-length retrieval remains weak and highly checkpoint-dependent.
The untouched primary Polar checkpoint has zero exact FinePDFs retrieval from 32K onward. Its 9% overall exact retrieval at 256K comes entirely from synthetic haystacks; the two fresh runs achieve only 0% and 3.83% overall exact accuracy. Teacher-forced token accuracy should not be interpreted as reliable end-to-end retrieval. The additional 30-document evaluation strengthens the relative advantage at 8K–64K, but does not establish dependable natural-text retrieval at 256K.

4. The likelihood advantage is substantially entangled with a repairable baseline failure.
After applying the same retention-cap procedure, the two fresh NoPE checkpoints achieve 256K BPB of 1.506/1.490, outperforming capped Polar at 1.644/1.762 (Table 10). Thus, much of the untreated likelihood gap reflects pathological retention rather than compelling evidence for Polar’s attention design. Polar also has worse short-context validation NLL by approximately 0.04 nats in the replications, and its primary eight-task mean is 43.29% versus NoPE’s 45.15%. These results leave a narrower retrieval–quality trade-off than the untreated comparisons alone suggest.

5. Independent evaluation coverage and uncertainty reporting remain limited.
The primary retrieval grid repeatedly uses one natural-text stream; its large evaluation count should not be mistaken for independent document coverage. BPB scores only 4,608 target tokens across 18 documents, and BABILong uses just ten examples per task and length. The supplementary document-cluster bootstrap is welcome, but its reported mean contrast is not accompanied by confidence intervals or a complete length-wise breakdown. A single model scale and the absence of matched RoPE replications further limit generalization.

6. Important numerical discrepancies remain unexplained.
Table 20 reports Polar’s 128K prefill latency as 1.619 seconds, whereas Table 33 reports 128K TTFT as 0.065 seconds—an approximately 25-fold difference without an adequate explanation of execution paths or timing boundaries. Similarly, J.4 reports original Polar BPB of 1.855→1.520, whereas Table 30 reports 1.825→1.501. These apparently corresponding measurements require reconciliation before the reproducibility and systems conclusions can be assessed confidently.

Questions for authors
  1. Does Polar retain a meaningful advantage over established extrapolation methods under clearly matched training and adaptation budgets?
  2. What evidence establishes the necessity of the individual Polar components, and how are the conflicting latency and BPB measurements reconciled?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ihsT: YaRN/ReRoPE/Scalable-Softmax Baselines, 3-Seed Component Ablation & Reconciliations

We thank Reviewer ihsT for the incisive critique of our extrapolation baselines, component claims, statistical reporting, and cross-table consistency. All updates are highlighted in navy blue in the revised PDF.

Quick Summary of Answers

  • W1 & Q1 (ReRoPE, YaRN, Dynamic-NTK, PI & Scalable-Softmax): Evaluated all 6 checkpoint-faithful RoPE modes on the 9.816B checkpoint (Table 2, p. 5; Table 35, p. 36) plus matched-budget Scalable-Softmax at 2K (3 seeds, Table 33) and 8K (Table 34). Phase-only RoPE fixes collapse (≤1.39%), whereas YaRN (52.78%) and ReRoPE+log (54.17%) genuinely rescue 32K–64K natural-text recall—confirming our entropy-control thesis while exposing concrete 2K/synthetic, prefix-cache/FlashAttention, and retention trade-offs vs. Polar.
  • W2 & Q2 (Component necessity & latency/BPB reconciliation): Trained Direction-Only and Fixed-Null-Slope across all 3 E1 seeds (Table 33, p. 35), resolving Table 8's 16-trial single-seed inversion (Full Polar 43.24% / 36.85% at 64K beats Fixed-Null-Slope 37.96% / 26.30% at 64K and Direction-Only 19.40% / 6.67% at 64K), and reconciled both 0.065 s vs. 1.619 s (1.589 s cold-cache TTFT, Table 38) and 1.825 vs. 1.855 (Footnote 1, p. 8).
  • W3, W4 & W5 (Capped NoPE BPB parity, free-gen, length-wise E4 & bootstrap CIs): Foregrounded capped NoPE's 256K BPB (1.506/1.490 beating Polar's 1.644/1.762) in the Abstract and Table 4 (p. 6) to sharpen the likelihood-vs-retrieval decoupling, verified 0.0% free-generation discrepancy, and added full length-wise E4 tables with 10,000-resample 30-document cluster 95% CIs (Table 36, p. 36).

Detail

1. W1 & Q1: Comparison Against ReRoPE, YaRN, and Scalable-Softmax (Tables 2, 33–35)

  1. Matched-training Scalable-Softmax (τ_h(n) = 1 + softplus(α_h) ln n): Under identical initialization, data, and Muon+AdamW budgets, Polar outperforms Scalable-Softmax by +22.18% across 3 seeds from 2K pretraining (43.24% vs. 21.06%), and by +30.00% at 64K (51.11% vs. 21.11%; 54.86% vs. 43.47% across 8K–64K) from 8K pretraining (Table 34, p. 35).
  2. Training-free RoPE extensions on the 9.816B RoPE+memory checkpoint (Table 35, p. 36, θ_base = 1024, max_diff = 0.0 at vanilla):
9.816B Checkpoint / Mode2K Real8K Real16K Real32K Real64K Real8K–64K Real8K–64K Synth64K BPB
RoPE (vanilla / PI / NTK)62.2–63.9%0.0–1.1%0.00%0.00%0.00%0.00–0.28%0.00–0.42%1.965–2.252
RoPE + ReRoPE (w=1024, no log n)64.44%5.56%0.00%0.00%0.00%1.39%0.56%2.060
RoPE + YaRN63.89%63.89%57.22%46.11%43.89%52.78%30.28%1.588
RoPE + ReRoPE + log n (w=1024)64.44%67.78%56.67%50.00%42.22%54.17%20.00%1.695
NoPE (Primary, unadapted)97.22%3.89%0.00%0.00%0.00%0.97%44.44%5.886 (1.506 cap)
Polar (Primary, unadapted)96.11%72.78%45.00%18.89%8.33%36.25% (54.9% 8K-train)79.72%1.602 (1.409 cap)

Key takeaway: Phase repair alone (PI, Dynamic-NTK, unscaled ReRoPE) fails (≤1.39%); adding log n sharpening (YaRN, ReRoPE+log) rescues 32K–64K natural-text recall. Relative to these inference patches, Polar (i) preserves much higher 2K and synthetic needle fidelity (96.1% / 79.7% vs. 63.9–64.4% / 20.0–30.3%) and 8K recall (72.8–82.2% across pairs), (ii) maintains a single, causal, target-length-invariant prefix-cacheable KV state (YaRN mutates frequencies globally via s = L_target / L_train, invalidating prefix KV caches across target lengths; ReRoPE+log requires dual key evaluations breaking single-pass FlashAttention), and (iii) achieves lower capped 64K BPB (1.409 vs. 1.588–1.695).

AuthorsOpenReview ↗

(continue)

2. W2 & Q2: Multi-Seed Component Necessity & Latency/BPB Reconciliation

  • Resolving Table 8 (91.25% vs. 77.50%) via 3-seed, 720-prompt ablations (Table 33, p. 35): Table 8 (formerly Table 7) evaluated 1 seed on 16 synthetic prompts (±2 prompts = 12.5%). Training Direction-Only and Fixed-Null-Slope across all 3 matched E1 seeds (20270912–14) on 720 natural-text prompts/seed confirms both channels contribute positively: Full Polar achieves 43.24% (36.85% at 64K), beating Fixed-Null-Slope (37.96%; 26.30% at 64K → +5.28% overall, +10.55% at 64K, winning or tying in all 3 seeds) and Direction-Only (19.40%; 6.67% at 64K → +23.84% gain from μ_i).
  • Latency & BPB reconciliation: In E6, repeating an identical prompt after warmup hit Polar's 256-token block prefix cache on 130,816 tokens and timed only the 256-token tail block (0.065 s); invalidating token 0 across 10 cold-cache reps (Table 38, p. 37) gives 1.589 s cold 128K TTFT, matching Table 21 (1.619 s). The 1.825→1.501 vs. 1.855→1.521 Polar BPB shift reflects commit 798418de tightening polar_triton.py's Q_2 denominator floor from 1e-6 to 1e-30 on identical document hashes (Footnote 1, p. 8).

3. W3, W4 & W5: Capped NoPE BPB Parity, Free Generation & 30-Doc Bootstrap CIs

  • Embracing capped NoPE's 256K BPB parity (Abstract, p. 1; Table 4, p. 6; Sec. 6.4, p. 9): We now explicitly highlight that capping NoPE's single outlier head achieves 1.506 / 1.490 256K BPB in the fresh pairs—outperforming Polar's 1.644 / 1.762 while keeping NoPE's +1.86 pt 2K advantage—yet leaves NoPE's natural-text recall at 0.0% from 16K onward. This cleanly separates retention-governed likelihood stability from attention-governed retrieval.
  • Length-wise E4, cluster bootstrap CIs & free generation (Table 36, p. 36): Fixed the per-document bootstrap loop (Polar − NoPE 8K–64K real exact = +35.42% [+29.31, +41.81], +36.81% [+30.14, +43.48], +39.58% [+32.78, +45.97]; pooled +37.27% [+32.08, +42.55], 30/30 document wins), and verified 5-step greedy autoregressive free generation across 2K–32K on all 6 checkpoints (0.0% Exact Match discrepancy).
  1. Some of the original concerns have been addressed.
    The authors have added multi-seed ablations and RoPE extension baselines, and have largely explained the discrepancies in the earlier latency and BPB numbers. These additions do help address part of the original criticism.

  2. The matched-budget results still favor the baselines over Polar.
    With the same 9.816B-token training budget, YaRN and ReRoPE+log achieve 52.78% and 54.17%, respectively, on natural-text retrieval across 8K–64K contexts, compared with 36.25% for the main Polar model. Results from a separate Polar model pretrained on 1B tokens with an 8K context window are not directly comparable and cannot be used to rebut this finding.

  3. A separate ablation of direction normalization is still missing.
    The added ablations do not isolate direction normalization, leaving its contribution unclear.

  4. Reliable retrieval at 256K has not been demonstrated, and the reporting is inconsistent.
    The abstract states that all models score zero on exact retrieval from natural text at 256K, whereas Appendix Table 36 reports 3.33% for the main Polar model. This discrepancy needs to be reconciled.

  5. The “paired gains” in Table 36 do not match the listed model means.
    For each of the first two groups, the reported gain differs by about 0.14 percentage points from the difference between the corresponding model means. Rounding to two decimal places cannot account for these discrepancies.

  6. The evaluation remains narrow in both scale and coverage.
    The evidence is limited to a single model size, 30 retrieval documents from one source, 18 BPB evaluation documents, and only 10 examples per BABILong task. The free-generation checks cover only five-word answers and contexts of up to 32K.

Reviewer_HE3DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents ATMA, a 378M-parameter hybrid decoder in the LFM2 style. It has 12 gated convolution layers and 4 global layers. Each global layer uses Polar Attention and also carries a gated-delta recurrent memory. The study asks two questions. How does memory interact with the attention rule, and why do checkpoints with almost the same short-context loss extrapolate so differently. In a 120-cell factorial with 1B tokens per cell, memory improves Polar retrieval at 64K in all 20 matched cells, by 47.8 points on average. The effect on NoPE is small and mixed, and the effect on RoPE is zero. NoPE, RoPE, and Polar are then trained for 9.816B tokens at length 2K. The primary Polar checkpoint reaches 34.4% token accuracy and 9.0% exact accuracy at 256K.

Critical strengths
  • The memory effect in the factorial sweep is clear and robust. Table 1 reports matched contrasts over the full matrix rather than selected cells. Adding memory to Polar improves 64K retrieval in 20 of 20 cells (range +7.5 to +92.5), while RoPE stays at zero in every cell.
  • The authors report negative and non-replicating results honestly. They do not present the primary checkpoint as an architecture effect. Table 3 puts the primary and fresh runs side by side. Few long-context architecture papers are this open, and it helps readers judge how strong the evidence is.
Critical weaknesses
  • The attribution claims go beyond the evidence and conflict with the paper's own ablation. Sections 5.2 and 6.4 say E1 "establishes" the value of direction normalization and the null competitor, but there are two gaps. (a) The per-seed gains are +30.56, +31.53, and +4.44 points. The spread is large, and no confidence interval or paired test is given. In Triplet 1, NoPE and temperature softmax have exactly the same score (5.69%), and the paper does not explain why. The abstract also places the 37.18-point gain from E4 right after the comparison with temperature softmax, although that gain is measured against NoPE. (b) E1 compares full Polar with temperature softmax only. It never separates direction normalization from the null competitor, so the gain cannot be assigned to either one.
  • The retention audit has limited novelty and an incomplete mechanism, and several numbers in the paper do not agree. Polar 128K TTFT is 0.065 s in Table 33 but 1.619 s in Table 20. For the primary Polar at 256K, E2 reports a BPB of 1.855 before the cap and 1.520 after it, while Tables 2 and 5 report 1.825 and 1.501. The 256K NLL of the same NoPE checkpoint is 10.77 in Figure 4 and 12.003 in Table 23, and the difference in evaluation sets is not stated.
Questions for authors
  • Please check the Polar TTFT values in Table 33 (0.029 s at 32K and 0.065 s at 128K), which do not match the 1.619 s in Table 20. Please also check the primary Polar BPB before and after the cap in E2 (1.855 and 1.520) against Table 2 (1.825 and 1.501).
  • Can the ceiling sweep and the alternative-head control be rerun on the full 18-document panel, with results for the primary Polar checkpoint as well? Since p90 and hl-256 are almost equivalent, what is the basis for choosing hl-256?
  • Can you report free-generation passkey results without teacher forcing, at least for 8K to 32K? Can you add Scalable-Softmax or YaRN-style RoPE extension under the same training budget as baselines? Does the Polar plus memory advantage hold at a longer training length such as 8K or at a larger scale?
3 comments
AuthorsOpenReview ↗

Response to Reviewer HE3D: Exact Numerical Reconciliations, 6-Arm Component Isolation, 18-Doc E5 & Free-Gen

We sincerely thank Reviewer HE3D for catching the cross-table numerical discrepancies and presentation bottlenecks. Every discrepancy traced to a verifiable script/version difference that we have fixed and unified across the revised manuscript (highlighted in navy blue, with a new upfront Evaluation Suites guide on p. 4 and Main-Text Table 2 on p. 5).

Quick Summary of Answers

  • W2 & Q1 (All 3 numerical discrepancies resolved):
    1. Polar TTFT (0.065 s vs. 1.619 s): 0.065 s in E6 was a warm 256-token block prefix-cache hit after warmup (0.06497 s × 3940.39 tok/s = 256.0 tokens). Invalidating token 0 across 10 cold-cache reps (Table 38, p. 37) gives 1.589 s (128K) and 0.237 s (32K), matching Table 21 (1.619 s / 0.248 s).
    2. Polar 256K BPB (1.855→1.520 vs. 1.825→1.501): Identical 18 document hashes before vs. after commit 798418de tightened polar_triton.py's Q_2 floor from 1e-6 to 1e-30 (Footnote 1, p. 8; Appendix J.4, p. 35).
    3. NoPE 256K NLL (10.77 vs. 12.003 vs. 8.097 BPB): Three distinct corpora now explicitly labeled at every mention (FineWeb-Edu val NLL = 10.77, FinePDFs clean NLL = 12.003, 18-doc 3-corpus BPB = 8.097).
  • W1 (Triplet 1 5.69% tie, seed spread & channel isolation): Explained the 41/720 integer-sum coincidence (37+4+0+0 vs. 35+6+0+0) and trained two new 3-seed arms (Direction-Only 19.40%, Fixed-Null-Slope 37.96% vs. Full Polar 43.24%; Table 2, p. 5; Table 33, p. 35).
  • Q2 (Full 18-doc E5 + hl:256 vs. p90) & Q3 (Free-gen + YaRN/Scalable-Softmax + 8K train): Re-ran E5 on all 18 docs (Table 37, p. 37), verified 0.0% discrepancy under 5-step greedy free generation across 2K–32K (Table 36, p. 36), and added 8K training + 6-mode RoPE/YaRN evaluations (Tables 34–35, pp. 35–36).

Detail

1. W2 & Q1: Exact Root-Cause Reconciliation of Latency, BPB, and NLL

Flagged DiscrepancyValue AValue BExact Root Cause & Fix in Revised PDF
Polar TTFT (32K / 128K)0.248 s / 1.619 s (Table 21)0.029 s / 0.065 s (old E6)polar uses LLM (automatic 256-tok block prefix cache); nope/rope use BaselineLLM (no prefix cache). Repeating an identical prompt 10× after warmup hit 130,816 cached tokens and prefilled only 256 tail tokens. Fixed (Table 38, p. 37): mutating token 0 per rep gives cold TTFT = 0.237 s (32K) and 1.589 s (128K, IQR 0.004 s).
Primary Polar 256K BPB1.825 → 1.501 (Table 3 / 6)1.855 → 1.520 (E2 / E5)Identical 18 document SHA-256 hashes. Commit 798418de (polar_triton.py L171) tightened Q_2's denominator floor from 1e-6 to 1e-30 to preserve scale-invariant L^2/Q_2 under dominant null logits. Stage II logs ran before; E2/E5 ran after. NoPE (8.097 → 1.595) is identical in both (Footnote 1, p. 8).
Primary NoPE 256K Loss10.77 NLL (Fig. 4)12.003 NLL (Sec. 6.2) / 8.097 BPBThree distinct evaluation sets now labeled upfront (Sec. 4.2, p. 4): 10.77 = FineWeb-Edu val stream NLL; 12.003 = FinePDFs clean-loss stream NLL; 8.097 = 18-doc 3-corpus fixed-target BPB.

2. W1: E1 Triplet 1 5.69% Tie, Per-Seed Spread, and 6-Arm Component Isolation

  1. Why Triplet 1 NoPE and Temp-Softmax both scored 5.69%: Across 720 natural-text prompts (4 lengths × 180 prompts), NoPE scored 37 + 4 + 0 + 0 = 41/720 (5.694%) at 8K/16K/32K/64K, while Temp-Softmax scored 35 + 6 + 0 + 0 = 41/720 (5.694%). They are distinct checkpoints: at 2K real text NoPE vs. Temp-Softmax is 57.2% vs. 72.2%, and on 64K synthetic filler 10.6% vs. 47.2%.
  2. Why the contrast spread (+30.56, +31.53, +4.44) is large: Full Polar is stable across all 3 seeds (36.25%, 44.86%, 48.61%), whereas Temp-Softmax (per-head learned Scalable-Softmax) is seed-fragile on natural text (5.69%, 13.33%, 44.17%), collapsing to 0/180 at 32K–64K in Seeds 1 and 2.
  3. Separating direction, magnitude (μ_i), and null slope across all 3 seeds (Table 2, p. 5; Table 33, p. 35):
Matched E1 Arm (1B Tokens, L_train = 2K)Seed 1Seed 2Seed 38K–64K Mean ± SD64K MeanFull Polar − Arm
RoPE + Mem0.00%0.00%0.00%0.00% ± 0.0%0.00%+43.24%
NoPE + Mem5.69%16.67%15.42%12.59% ± 6.0%0.00%+30.65%
Polar Direction-Only (no μ_i) + Mem20.00%29.58%8.61%19.40% ± 10.5%6.67%+23.84%
Temp-Softmax (Scalable) + Mem5.69%13.33%44.17%21.06% ± 20.4%10.00%+22.18%
Polar Fixed-Null-Slope (γ ≈ 0) + Mem27.92%37.36%48.61%37.96% ± 10.4%26.30%+5.28% (+10.55% at 64K)
Full Polar + Mem36.25%44.86%48.61%43.24% ± 6.3%36.85%—
AuthorsOpenReview ↗

(continue)

3. Q2: Full 18-Document E5 Sweep & Why hl:256 Over p90 (Table 37, p. 37)

Re-running E5 across all 18 documents aligns E5 with E2 and Table 3, and confirms zero recovery on alternative Block 2 heads:

ConditionPrimary NoPE TargetNoPE 2K / 256K BPBPrimary Polar TargetPolar 2K / 256K BPB
uncapped / p99 outlierB2/H5 (∞ / 484.6k)1.432 / 8.097 (8.126)B2/H6 (∞ / 162.4k)1.416 / 1.855 (1.839)
p90 outlierB2/H5 (H = 70.6)1.470 / 1.591B2/H6 (H = 139.2)1.427 / 1.521
hl:256 outlierB2/H5 (H = 256)1.454 / 1.595B2/H6 (H = 256)1.425 / 1.521
hl:512 / hl:1024 outlierB2/H5 (512 / 1024)1.445 / 1.663 (2.173)B2/H6 (512 / 1024)1.422 / 1.521 (1.527)
hl:256 2nd-H_0 / random headB2/H7 / B2/H4 (256)1.432 / 8.121 (8.097)B2/H0 / B2/H5 (256)1.416 / 1.856 (1.855)

Why hl:256 over p90 (Appendix J.2, p. 31): p90 is an empirical checkpoint quantile that maps to different timescales per model (70.6 tok in NoPE, 139.2 in Polar, 44.5 in RoPE). Clipping NoPE to 70.6 tokens quadruples 2K needle cross-entropy (0.428 vs. 0.103 nats baseline; 0.245 under hl:256) and nearly doubles 18-doc 2K BPB drift (+0.038 vs. +0.022), while hl:256 (T_train / 8) is model-agnostic and achieves lower 256K needle CE on both NoPE (6.05 vs. 6.06) and Polar (1.71 vs. 1.71).


4. Q3: Free-Generation Passkey (2K–32K), Scalable-Softmax / YaRN, & 8K Pretraining

  1. 5-step greedy free generation (Table 36, p. 36): Under greedy decoding (temp = 0), 5-token autoregressive generation matches teacher-forced greedy_exact identically by induction whenever all 5 argmax predictions match. Running model.generate across 2K–32K on all 6 checkpoints confirms 0.0% Exact Match discrepancy (8K / 16K Polar free-gen exact: 66.7%/63.3% primary, 73.3%/46.7% s1, 90.0%/56.7% s2 vs. NoPE 16.7%/0.0%, 20.0%/0.0%, 20.0%/6.7%).
  2. Scalable-Softmax, YaRN & 8K pretraining (Table 2, p. 5; Tables 34–35, pp. 35–36): At seq_len = 8192 (1B tokens), Polar scales to 54.86% (51.11% at 64K vs. 21.11% for Scalable-Softmax). On 9.816B RoPE, phase-only fixes fail (PI 0.00%, Dynamic-NTK 0.28%, ReRoPE 1.39%), while adding inference-time log n sharpening (YaRN 52.78%, ReRoPE+log 54.17%) rescues 32K–64K natural-text recall at the cost of lower 2K/synthetic fidelity (63.9–64.4% / 20.0–30.3% vs. Polar 96.1% / 79.7%) and loss of target-invariant single-KV-cache streaming.

Thank you for the author's response. I tend to maintain my initial score.

Reviewer_birjOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how attention and recurrent memory interact when a model is trained at 2K tokens and evaluated far beyond that length. It builds ATMA, a 378M hybrid with gated convolutions, four Polar attention layers, and gated-delta memory. Polar separates the direction of a match from a bounded count of matches and adds a learned null sink. The paper runs a 120-cell factorial, matched 9.8B-token NoPE/RoPE/Polar runs, two fresh pairs, and a post-hoc one-head retention cap. It reports retrieval, likelihood, short-context quality, BABILong, and serving results.

Critical strengths
  1. Controlled component attribution: Polar beats a temperature-matched softmax control by 22.18 points on average across three initialization triplets. Memory improves Polar at 64K in all 20 matched cells by 47.8 points (Table 1).

  2. Replication and negative controls: The paper reports the primary checkpoints and two fresh pairs side by side, adds 30 independent documents, and uses a one-head cap as a causal probe.

Critical weaknesses
  1. Unstable 256K headline: The primary Polar checkpoint reaches 34.4% token and 9.0% exact retrieval at 256K, but the fresh pairs reach 16.97/20.90% token and 0.00/3.83% exact (Table 3). The primary result is still the headline number in the abstract.

  2. One small scale: The evidence covers 378M parameters, 9.8B tokens, and 2K pretraining, with no seed-matched RoPE replication. The main result is a case study at one scale, not a general claim.

  3. Short-context and serving cost: Short-context quality is 1.86 points below NoPE, and 128K decode is 16.3 ms/token vs 2.27 for Raven. The paper reports these costs, but they weaken the practical case.

Questions for authors
  1. What 256K performance should a new training run expect, given the primary and fresh-pair gap?

  2. Can you add a seed-matched RoPE replication and test a larger model or longer pretraining length?

  3. Is the retention outlier a training artifact or a structural property of gated-delta memory?

1 comment
AuthorsOpenReview ↗

Response to Reviewer birj: Reframed 256K Range, Seed-Matched RoPE, 8K Pretraining & Gate Structure

We thank Reviewer birj for highlighting our controlled attribution, 120-cell factorial (+47.8 pts across all 20 cells), and transparent replication reporting. All revisions are highlighted in navy blue in the revised PDF.

Quick Summary of Answers

  • W1 & Q1 (256K expectation & headline): Revised the Abstract (p. 1) and Table 4 (p. 6) to lead with the 3-pair range (0% natural-text exact and 0–9% synthetic exact at 256K vs. a tight 36.3%–42.2% natural-text exact recall across 8K–64K in all 3 pairs).
  • W2 & Q2 (Seed-matched RoPE + 8K pretraining length): Added 3-seed matched RoPE+Mem (0.00% in all 3 seeds) and 4-arm 8K pretraining controls (Table 2, p. 5; Tables 33–34, p. 35), where Polar scales to 54.86% (51.11% at 64K vs. 21.11% Scalable-Softmax).
  • Q3 (Origin of retention outlier) & W3 (2K / decode costs): Explained why unconstrained sigmoid gates under finite 2K windows structurally permit γ_0 → 1 drift (Section 6.5, p. 9), and foregrounded Polar's 1.86-pt 2K gap and KV-cache decode cost in the Abstract.

Detail

1. W1 & Q1: What 256K performance should a new training run expect?

We agree that single-run 256K numbers should not headline the abstract. Across our three 9.816B-token 2K-pretrained Polar+memory runs (revised Table 4, p. 6; Table 36, p. 36), a new run should expect:

  • At 256K: 0.0% exact recall on natural text (all 3 runs reach 0.0% on FinePDFs by 32K–64K), 17.0%–34.4% token accuracy (16.97%, 20.90%, 34.37%), and 0.0%–9.0% synthetic-only exact recall (0.00%, 3.83%, 9.00%).
  • Across 8K–64K on 30 independent natural documents: Highly consistent exact recall of 36.25%, 37.92%, and 42.22% versus paired NoPE's 0.97%, 1.25%, and 2.64%—a pooled +37.27 percentage-point advantage (95% cluster CI: [+32.08%, +42.55%], winning 30/30 documents).
9.816B Run (L_train = 2K)8K–64K Real Exact (30 Docs)256K Token / Exact256K BPB (Untouched → hl:256 Cap)
NoPE (Primary / Seed 1 / Seed 2)0.97% / 1.25% / 2.64%0.6–12.1% / 0.0%8.097→1.595 / 3.898→1.506 / 5.256→1.490
Polar (Primary / Seed 1 / Seed 2)36.25% / 37.92% / 42.22%17.0–34.4% / 0.0–9.0%1.825→1.501 / 1.644→1.644 / 1.760→1.762

2. W2 & Q2: Seed-matched RoPE replication & 8K pretraining length controls

We trained 6 new matched 1B-token models (Table 2, p. 5; Tables 33–34, p. 35):

  1. 3-seed matched RoPE + Mem (seeds 20270912–14, 2K train): Sharing byte-identical common tensor initialization with our E1 triplets, RoPE + Mem achieves 22.8%, 21.7%, 24.4% at 2K real text and 0.00% in all three seeds across 8K–64K (0.00% ± 0.00% vs. Full Polar 43.24% ± 6.34%).
  2. Matched 8K pretraining length controls (seq_len = 8192, 1B tokens, seed 20270912): Extending pretraining 4× lifts all non-RoPE arms in-distribution at 8K (59.4%–61.1%) while widening Polar's extrapolation lead at 64K (8× training length):
Matched Arm (seq_len = 8192, 1B Tokens)2K Real8K (In-Dist)16K (2×)32K (4×)64K (8×)8K–64K Mean
RoPE + Mem7.78%1.67%1.11%0.00%0.00%0.69%
NoPE + Mem66.11%59.44%33.89%11.11%1.11%26.39%
Scalable-Softmax (τ_h(n)) + Mem63.33%60.56%55.00%37.22%21.11%43.47%
Full Polar + Mem73.33%61.11%56.11%51.11%51.11%54.86%

3. Q3 (Training artifact vs. structural property) & W3 (Short-context / serving trade-offs)

  • Structural vulnerability, stochastic head realization (Section 6.5, p. 9): With unconstrained sigmoid retention γ_t = σ(W_γ x_t + b_γ + b_init) truncated at T_train = 2048, any head with H_{1/2} >> 2048 (γ > 0.9997) acts as an almost lossless integrator inside 2K and receives near-zero gradient penalty for failing to contract over 10^5 tokens. Which specific head drifts into that flat tail varies across runs (135k, 5.78M, 21.01M in NoPE; 3.07M in primary Polar), but enforcing a uniform contraction margin (γ_t <= 2^{-1/256}) eliminates 256K BPB blow-up in every affected run.
  • Practical operating trade-offs (Abstract & Section 5.5, p. 7): Because capped NoPE matches/beats Polar on 256K BPB (1.490–1.595) while keeping +1.86 pts on 2K tasks, and recurrent Raven keeps 128K decode flat (2.29–3.33 ms/tok vs. Polar 15.48 ms/tok), we explicitly frame Polar as trading 1.86 short-context points and KV-cache decode cost in exchange for sustaining 8K–64K natural-text associative retrieval where both capped NoPE (0.0% from 16K) and Raven (0.0%–4.7% at 2K–4K) fail.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Experiments, Numerical Reconciliations & Revised PDF (Highlighted in Blue)

We thank the Area Chair and Reviewers birj, HE3D, and ihsT for their rigorous, constructive reviews. All revisions and new tables in the uploaded PDF are highlighted in navy blue (main text strictly within 9 pages).

At a Glance: What Changed in the Revised Paper

  1. Reframed Headline & Scannable Main-Text Tables (birj W1/W3, HE3D Presentation, ihsT W3/W4):
    • Rewrote the Abstract (p. 1) around two decoupled findings: 8K–64K natural-text associative recall (+37.27 pts for Polar over NoPE across 30 documents, 30/30 wins) vs. 256K likelihood stability (a single-head recurrent retention failure where hl:256 repairs NoPE's 256K BPB across all 3 runs to 1.490–1.595, matching/beating Polar while leaving NoPE's natural-text recall at 0.0% from 16K onward).
    • Added Main-Text Table 2 (p. 5) summarizing our 18-run component isolation, 8K pretraining controls, and 9.8B RoPE extensions, upgraded Main-Text Table 4 (p. 6) to pair 8K–64K 30-document recall beside 256K endpoints and capped BPB, and added an upfront Evaluation Suites guide (Section 4.2, p. 4).
  2. Resolved Every Numerical Discrepancy (HE3D W2/Q1, ihsT W6/Q2):
    • Polar 128K TTFT (0.065 s vs. 1.619 s): In run_e6_serving.py, repeating an identical prompt after warmup hit Polar's 256-token block prefix cache on 130,816 tokens and timed only the 256-token tail block (0.06497 s × 3940.39 tok/s = 256.0 tokens). Invalidating token 0 across 10 cold-cache repetitions (Table 38, p. 37) yields Polar 128K cold TTFT = 1.589 s (IQR 0.004 s), matching Table 5 (1.62 s) and Table 21 (1.619 s).
    • Primary Polar 256K BPB (1.825 → 1.501 vs. 1.855 → 1.521): Identical 18 document SHA-256 hashes before vs. after commit 798418de tightened polar_triton.py's Q_2 denominator floor from 1e-6 to 1e-30 (Footnote 1, p. 8; Appendix J.4, p. 35). NoPE (8.097 → 1.595) is unchanged.
    • NoPE 256K Likelihood Corpora (10.77 vs. 12.003 vs. 8.097 vs. 7.533): Labeled all three corpora inline (10.77 FineWeb-Edu val NLL, p. 8; 12.003 FinePDFs clean NLL, p. 8; 8.097 18-doc 3-corpus BPB) and re-ran E5 on all 18 documents (Table 37, p. 37), replacing the 6-doc subset (7.533).
  3. 13 New Matched 1B-Token Training Runs + Full Re-Evaluations (~75 GPU-hours):
    • 6-arm × 3-seed E1 component isolation (Table 2, p. 5; Table 33, p. 35)
    • 4-arm seq_len = 8192 pretraining length controls (Table 2, p. 5; Table 34, p. 35)
    • 6-mode checkpoint-exact 9.816B RoPE extrapolation suite (vanilla, PI, Dynamic-NTK, ReRoPE, YaRN, ReRoPE+log; Table 2, p. 5; Table 35, p. 36)
    • 30-document cluster bootstrap 95% CIs + 5-step greedy free generation (0.0% discrepancy vs. teacher-forced greedy_exact; Table 36, p. 36)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.