- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents ATMA, a 378M-parameter hybrid language model combining local gated convolutions, Polar Attention, and gated-delta recurrent memory. Polar Attention uses a normalized direction channel, a bounded participation-ratio magnitude channel, learned length-dependent temperature scaling, and a virtual null competitor. The study examines how attention and recurrent memory interact during length extrapolation and why checkpoints with similar short-context validation losses can exhibit different long-context behavior. The experimental program includes a 120-cell factorial study, matched NoPE/RoPE/Polar models trained on 9.816B tokens at a sequence length of 2K, additional paired training runs, and temperature-matched component controls. Evaluation extends to 256K tokens and covers teacher-forced retrieval, fixed-target likelihood, adapted BABILong reasoning, short-context tasks, and serving performance. Separately optimized Raven and TDA models provide additional reference points. The reported results show that memory improves Polar’s retrieval performance in the factorial study and that Polar outperforms the tested NoPE and temperature-matched controls on selected retrieval evaluations. Extreme-length accuracy varies across training runs, with exact retrieval successes at 256K confined to synthetic haystacks. The paper also investigates near-unit retention gates through an inference-only, single-head retention cap. This intervention improves long-context likelihood in the NoPE replications, while its effects on retrieval and Polar checkpoints vary, distinguishing retention stability from associative retrieval capability.
Critical strengths
- The complete factorial study and additional paired training runs provide useful empirical evidence.
- The single-head intervention identifies a reproducible likelihood failure in NoPE checkpoints.
- The paper explicitly reports negative results and distinguishes synthetic retrieval from natural-text retrieval.
Critical weaknesses
1. Important length-extrapolation baselines are missing.
The principal matched comparisons use NoPE and vanilla RoPE. Relevant alternatives such as ReRoPE and YaRN are absent, leaving unclear whether redesigning attention is preferable to existing extrapolation techniques. Training-free methods and methods requiring adaptation should be evaluated separately, with matched additional training budgets. Raven/TDA change architecture, optimization, and other factors simultaneously and cannot substitute for these controls. The temperature-matched comparison is useful but covers only the 1B-token regime, not the headline 9.816B-token setting.
2. The component-level conclusions exceed the evidence.
E1 establishes that the complete Polar configuration outperforms temperature-matched softmax; it does not establish that direction normalization, the null competitor, and the magnitude channel are individually necessary. Indeed, Table 7 shows that fixing the length-dependent null slope near zero improves 64K retrieval from 77.50% to 91.25%, while direction-only retains 71.25%. This directly challenges the empirical justification for the full proposed recipe. The 120-cell factorial does not replace multi-seed ablations of these defining components. The conditional theoretical bounds likewise do not establish preservation of relevant information.
3. Extreme-length retrieval remains weak and highly checkpoint-dependent.
The untouched primary Polar checkpoint has zero exact FinePDFs retrieval from 32K onward. Its 9% overall exact retrieval at 256K comes entirely from synthetic haystacks; the two fresh runs achieve only 0% and 3.83% overall exact accuracy. Teacher-forced token accuracy should not be interpreted as reliable end-to-end retrieval. The additional 30-document evaluation strengthens the relative advantage at 8K–64K, but does not establish dependable natural-text retrieval at 256K.
4. The likelihood advantage is substantially entangled with a repairable baseline failure.
After applying the same retention-cap procedure, the two fresh NoPE checkpoints achieve 256K BPB of 1.506/1.490, outperforming capped Polar at 1.644/1.762 (Table 10). Thus, much of the untreated likelihood gap reflects pathological retention rather than compelling evidence for Polar’s attention design. Polar also has worse short-context validation NLL by approximately 0.04 nats in the replications, and its primary eight-task mean is 43.29% versus NoPE’s 45.15%. These results leave a narrower retrieval–quality trade-off than the untreated comparisons alone suggest.
5. Independent evaluation coverage and uncertainty reporting remain limited.
The primary retrieval grid repeatedly uses one natural-text stream; its large evaluation count should not be mistaken for independent document coverage. BPB scores only 4,608 target tokens across 18 documents, and BABILong uses just ten examples per task and length. The supplementary document-cluster bootstrap is welcome, but its reported mean contrast is not accompanied by confidence intervals or a complete length-wise breakdown. A single model scale and the absence of matched RoPE replications further limit generalization.
6. Important numerical discrepancies remain unexplained.
Table 20 reports Polar’s 128K prefill latency as 1.619 seconds, whereas Table 33 reports 128K TTFT as 0.065 seconds—an approximately 25-fold difference without an adequate explanation of execution paths or timing boundaries. Similarly, J.4 reports original Polar BPB of 1.855→1.520, whereas Table 30 reports 1.825→1.501. These apparently corresponding measurements require reconciliation before the reproducibility and systems conclusions can be assessed confidently.
Questions for authors
- Does Polar retain a meaningful advantage over established extrapolation methods under clearly matched training and adaptation budgets?
- What evidence establishes the necessity of the individual Polar components, and how are the conflicting latency and BPB measurements reconciled?
3 comments
(continue)
2. W2 & Q2: Multi-Seed Component Necessity & Latency/BPB Reconciliation
- Resolving Table 8 (
91.25%vs.77.50%) via 3-seed, 720-prompt ablations (Table 33, p. 35): Table 8 (formerly Table 7) evaluated 1 seed on 16 synthetic prompts (±2 prompts = 12.5%). TrainingDirection-OnlyandFixed-Null-Slopeacross all 3 matched E1 seeds (20270912–14) on 720 natural-text prompts/seed confirms both channels contribute positively:Full Polarachieves43.24%(36.85%at 64K), beatingFixed-Null-Slope(37.96%;26.30%at 64K →+5.28%overall,+10.55%at 64K, winning or tying in all 3 seeds) andDirection-Only(19.40%;6.67%at 64K →+23.84%gain fromμ_i). - Latency & BPB reconciliation: In E6, repeating an identical prompt after warmup hit Polar's 256-token block prefix cache on 130,816 tokens and timed only the 256-token tail block (
0.065 s); invalidating token 0 across 10 cold-cache reps (Table 38, p. 37) gives1.589 scold 128K TTFT, matching Table 21 (1.619 s). The1.825→1.501vs.1.855→1.521Polar BPB shift reflects commit798418detighteningpolar_triton.py'sQ_2denominator floor from1e-6to1e-30on identical document hashes (Footnote 1, p. 8).
3. W3, W4 & W5: Capped NoPE BPB Parity, Free Generation & 30-Doc Bootstrap CIs
- Embracing capped NoPE's 256K BPB parity (
Abstract, p. 1; Table 4, p. 6; Sec. 6.4, p. 9): We now explicitly highlight that capping NoPE's single outlier head achieves1.506 / 1.490256K BPB in the fresh pairs—outperforming Polar's1.644 / 1.762while keeping NoPE's+1.86pt 2K advantage—yet leaves NoPE's natural-text recall at0.0%from 16K onward. This cleanly separates retention-governed likelihood stability from attention-governed retrieval. - Length-wise E4, cluster bootstrap CIs & free generation (
Table 36, p. 36): Fixed the per-document bootstrap loop (Polar − NoPE8K–64K real exact =+35.42% [+29.31, +41.81],+36.81% [+30.14, +43.48],+39.58% [+32.78, +45.97]; pooled+37.27% [+32.08, +42.55],30/30document wins), and verified 5-step greedy autoregressive free generation across 2K–32K on all 6 checkpoints (0.0%Exact Match discrepancy).
-
Some of the original concerns have been addressed.
The authors have added multi-seed ablations and RoPE extension baselines, and have largely explained the discrepancies in the earlier latency and BPB numbers. These additions do help address part of the original criticism. -
The matched-budget results still favor the baselines over Polar.
With the same 9.816B-token training budget, YaRN and ReRoPE+log achieve 52.78% and 54.17%, respectively, on natural-text retrieval across 8K–64K contexts, compared with 36.25% for the main Polar model. Results from a separate Polar model pretrained on 1B tokens with an 8K context window are not directly comparable and cannot be used to rebut this finding. -
A separate ablation of direction normalization is still missing.
The added ablations do not isolate direction normalization, leaving its contribution unclear. -
Reliable retrieval at 256K has not been demonstrated, and the reporting is inconsistent.
The abstract states that all models score zero on exact retrieval from natural text at 256K, whereas Appendix Table 36 reports 3.33% for the main Polar model. This discrepancy needs to be reconciled. -
The “paired gains” in Table 36 do not match the listed model means.
For each of the first two groups, the reported gain differs by about 0.14 percentage points from the difference between the corresponding model means. Rounding to two decimal places cannot account for these discrepancies. -
The evaluation remains narrow in both scale and coverage.
The evidence is limited to a single model size, 30 retrieval documents from one source, 18 BPB evaluation documents, and only 10 examples per BABILong task. The free-generation checks cover only five-word answers and contexts of up to 32K.
Response to Reviewer ihsT: YaRN/ReRoPE/Scalable-Softmax Baselines, 3-Seed Component Ablation & Reconciliations
We thank Reviewer
ihsTfor the incisive critique of our extrapolation baselines, component claims, statistical reporting, and cross-table consistency. All updates are highlighted in navy blue in the revised PDF.Quick Summary of Answers
≤1.39%), whereasYaRN(52.78%) andReRoPE+log(54.17%) genuinely rescue 32K–64K natural-text recall—confirming our entropy-control thesis while exposing concrete 2K/synthetic, prefix-cache/FlashAttention, and retention trade-offs vs. Polar.Direction-OnlyandFixed-Null-Slopeacross all 3 E1 seeds (Table 33, p. 35), resolving Table 8's 16-trial single-seed inversion (Full Polar43.24%/36.85%at 64K beatsFixed-Null-Slope37.96%/26.30%at 64K andDirection-Only19.40%/6.67%at 64K), and reconciled both0.065 svs.1.619 s(1.589 scold-cache TTFT, Table 38) and1.825vs.1.855(Footnote 1, p. 8).1.506/1.490beating Polar's1.644/1.762) in the Abstract and Table 4 (p. 6) to sharpen the likelihood-vs-retrieval decoupling, verified0.0%free-generation discrepancy, and added full length-wise E4 tables with 10,000-resample 30-document cluster 95% CIs (Table 36, p. 36).Detail
1. W1 & Q1: Comparison Against ReRoPE, YaRN, and Scalable-Softmax (
Tables 2, 33–35)τ_h(n) = 1 + softplus(α_h) ln n): Under identical initialization, data, and Muon+AdamW budgets, Polar outperforms Scalable-Softmax by+22.18%across 3 seeds from 2K pretraining (43.24%vs.21.06%), and by+30.00%at 64K (51.11%vs.21.11%;54.86%vs.43.47%across 8K–64K) from 8K pretraining (Table 34, p. 35).Table 35, p. 36,θ_base = 1024,max_diff = 0.0atvanilla):vanilla/PI/NTK)ReRoPE(w=1024, nolog n)YaRNReRoPE + log n(w=1024)Key takeaway: Phase repair alone (
PI,Dynamic-NTK, unscaledReRoPE) fails (≤1.39%); addinglog nsharpening (YaRN,ReRoPE+log) rescues 32K–64K natural-text recall. Relative to these inference patches, Polar (i) preserves much higher 2K and synthetic needle fidelity (96.1%/79.7%vs.63.9–64.4%/20.0–30.3%) and 8K recall (72.8–82.2%across pairs), (ii) maintains a single, causal, target-length-invariant prefix-cacheable KV state (YaRNmutates frequencies globally vias = L_target / L_train, invalidating prefix KV caches across target lengths;ReRoPE+logrequires dual key evaluations breaking single-pass FlashAttention), and (iii) achieves lower capped 64K BPB (1.409vs.1.588–1.695).