acceptodds
Under review as a conference paper at ICLR 2027

MANTA: Linearizing Large Language Models with Multi-Center Taylor Attention

41 views

Abstract

Softmax attention gives large language models flexible access to the preceding context, but its quadratic computation and growing key–value cache limit efficient long-context deployment. Converting a pretrained softmax Transformer into a fixed-state model offers a practical alternative to pretraining a new architecture from scratch. However, Taylor-based linearization typically expands all keys around one global center. This approximation is inaccurate when pretrained keys form broad, anisotropic distributions. We introduce MANTA (Multi-Center Linear Attention via Taylor Approximation), which uses query-aware routing to make the softmax approximation local. MANTA maintains an additive recurrent state for each routed group, while the total state size remains independent of context length. It projects the memory-dominant matrix in each state onto a learned low-rank basis to reduce memory. Finally, joint normalization combines the compressed history with sliding-window attention. Experiments show that MANTA recovers the language-modeling quality of the original softmax model, with an average score of across six benchmarks. Compared with the softmax baseline, MANTA improves decoding efficiency and achieves up to the throughput. By making the Taylor approximation local and memory-efficient, MANTA provides a practical path from pretrained softmax Transformers to accurate fixed-state models with linear total computation and memory independent of context length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Linear-Complexity Sequence Modeling, Linear Attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_MR5QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MANTA, a method for converting pretrained softmax Transformers into fixed-state linear attention models with limited additional training. To address the approximation errors caused by single-center Taylor expansion, MANTA introduces query-aware multi-center routing to better approximate softmax attention. It further employs learned low-rank projections to reduce recurrent-state memory overhead and combines compressed historical information with exact sliding-window attention through joint normalization. Experiments on Llama-3.1-8B demonstrate that MANTA largely preserves the original model's accuracy on standard benchmarks while achieving substantial improvements in decoding throughput. The authors also evaluate its effectiveness on long-context tasks and report improvements over existing linearization methods.

Critical strengths
  1. Efficient linearization with limited additional training. The proposed method enables the conversion of pretrained softmax Transformers into linear attention models with relatively low training overhead. It achieves nearly lossless performance on standard benchmarks while providing substantial inference efficiency gains.

  2. Insightful mathematical analysis. The paper provides a thorough and insightful mathematical analysis of the limitations of single-center Taylor approximation, offering a well-motivated foundation for the proposed multi-center approach.

  3. Memory-efficient design. The authors effectively address the additional memory overhead introduced by multiple centers through low-rank recurrent state compression, making the proposed approach more practical for deployment.

Critical weaknesses
  1. Limited evaluation across model architectures and scales: The experiments are conducted exclusively on a single model size from one model family. This is insufficient to demonstrate the generalizability of the proposed approach. I would strongly encourage the authors to extend their evaluation to different model architectures and parameter scales. The generalizability of MANTA is my primary concern.

  2. Lack of direct evaluation of attention approximation error: A central motivation of MANTA is to reduce the approximation error relative to standard softmax attention. However, the authors primarily validate this claim through downstream benchmark accuracy, without directly quantifying the approximation error between MANTA attention and exact softmax attention. A more direct analysis would provide stronger evidence supporting the proposed method.

  3. Insufficient analysis of training costs: Although MANTA aims to provide an efficient conversion of pretrained Transformers, the paper lacks a detailed analysis of the training costs associated with the conversion process. Reporting the computational resources and training time required would help clarify the practical efficiency of the proposed approach.

Questions for authors
  1. Performance on long-generation reasoning tasks: I appreciate the inclusion of long-context benchmarks. However, most of these tasks involve long inputs but relatively short outputs. I am interested in how the converted model performs on challenging mathematical reasoning and coding tasks that require extended reasoning or long generation sequences. In autoregressive generation, approximation errors may propagate or accumulate over successive decoding steps, potentially leading to substantial performance degradation. Could the authors evaluate MANTA on such long-generation tasks to better demonstrate its effectiveness and robustness?

  2. Robustness of the learned low-rank projection under domain shift: Regarding the learnable projection matrix (P) introduced for low-rank recurrent state compression, how sensitive is its effectiveness to domain shifts between the training data and deployment scenarios? I would be interested in understanding how much performance degradation may occur when applying a projection learned from one data distribution to a substantially different domain, while keeping the model configuration fixed.

  3. Comparison between practical single-center and multi-center configurations. Section 5.2 provides a detailed analysis of the benefits of increasing the number of Taylor centers. However, I would find a comparison between the following two practical configurations more convincing. These two configurations represent realistic deployment choices for the proposed method and its single-center counterpart:

    • Multi-center + SWA + Low-rank compression: The proposed MANTA configuration.
    • Single-center + SWA, without Low-rank compression: A simpler baseline configuration.
6 comments
AuthorsOpenReview ↗

Responses to W1, W2 and W3

We thank the reviewer for the careful assessment of our work and the constructive comments. Below, we address each concern with additional experimental evidence and clarifications.

W1

Reviewer comment

The experiments are conducted exclusively on a single model size from one model family. This is insufficient to demonstrate the generalizability of the proposed approach.

Response

We have extended our Llama-3.1-8B experiments to Qwen3-4B and Qwen3-8B, adding a second model family and a second parameter scale. We compare Liger-GLA, GDN, and MANTA with the original full-softmax models using identical evaluation settings. MANTA uses four centers and rank 16. SWA retains 8 sink and 56 recent tokens for MANTA/GDN; Liger uses its original window of 64 preceding tokens plus self, without sinks.

MANTA achieves the highest average among converted models at both scales. Its gains over GDN are 1.53 and 1.13 percentage points, and over Liger are 5.19 and 3.40 points, on 4B and 8B, respectively. The corresponding differences from full softmax are +0.43 and -0.20 points. Together with the existing Llama results, this consistent average advantage on both Qwen3 scales supports MANTA's generalization across model families and parameter sizes.

BackboneMethodPIQAARC-EasyARC-ChallengeHellaSwagWinoGrandeMMLUAvg.
Qwen3-4BSoftmax (full)75.0880.3954.1068.4765.9870.1769.03
Qwen3-4BLiger-GLA74.9278.5850.5167.7262.9850.9564.28
Qwen3-4BGDN75.1480.4754.0167.8466.6163.5567.94
Qwen3-4BMANTA77.7582.1555.1270.5866.3864.8069.46
Qwen3-8BSoftmax (full)76.5083.6356.6674.8968.1174.9372.45
Qwen3-8BLiger-GLA78.0281.6958.1173.6468.7552.9368.86
Qwen3-8BGDN76.4483.4656.1474.4367.8868.4271.13
Qwen3-8BMANTA78.4082.9157.0876.3668.9069.8872.26

W2

Reviewer comment

A central motivation of MANTA is to reduce the approximation error relative to standard softmax attention. However, the authors primarily validate this claim through downstream benchmark accuracy, without directly quantifying the approximation error between MANTA attention and exact softmax attention.

Response

Using frozen pretrained Q/K/V activations from Llama-3.1-8B and Qwen3-8B, we directly compare first-order Taylor attention with one center (R=1) and four centers (R=4) against exact causal softmax. Routers are calibrated on 128 Alpaca-cleaned texts; evaluation uses 64 held-out LongBench inputs at 1K, 4K, and 8K lengths, sampling 128 causal query positions per input.

We report relative Frobenius errors of effective attention weights and per-head attention outputs (before the output projection), normalized by the corresponding exact-softmax norm and averaged over inputs, layers, and query heads. Lower is better.

Four centers reduce output error by 26.48–31.34% on Llama and 7.03–13.12% on Qwen, and weight error by 11.41–19.51% and 2.61–7.44%, respectively. These controlled measurements directly verify that multiple centers reduce approximation error.

BackboneContextWeight error, R=1Weight error, R=4Output error, R=1Output error, R=4
Llama-3.1-8B1K0.96820.77932.19571.5076
Llama-3.1-8B4K0.98360.84172.00991.4437
Llama-3.1-8B8K0.98860.87581.96851.4473
Qwen3-8B1K0.95790.88671.40281.2188
Qwen3-8B4K0.97630.93511.29741.1837
Qwen3-8B8K0.98260.95691.26971.1804

W3

Reviewer comment

Reporting the computational resources and training time required would help clarify the practical efficiency of the proposed approach.

Response

For Llama-3.1-8B with MANTA (four centers, rank 16, cache size 128), the total training time is 2.87 hours on one 80GB H100 and 5.21 hours on one 32GB RTX 5090. These measurements demonstrate that MANTA can adapt an 8B model using only a single GPU and a few hours of fine-tuning, including on consumer GPU hardware.

AuthorsOpenReview ↗

Responses to Q1, Q2 and Q3

Q1

Reviewer question

Could the authors evaluate MANTA on such long-generation tasks to better demonstrate its effectiveness and robustness?

Response

We evaluate Qwen3-8B on the full GSM8K test set (mathematical reasoning) and MBPP-sanitized test set (coding). MANTA and GDN use cache size 128 (8 sink + 120 recent tokens).

MANTA outperforms GDN by 18.12 percentage points on GSM8K and 7.78 points on MBPP, supporting its effectiveness and relative robustness to attention conversion across mathematical reasoning and coding.

ModelGSM8K numeric-answer accuracy (%)MBPP-sanitized pass@1 (%)
Original softmax92.3477.04
GDN58.0763.42
MANTA76.1971.21

Q2

Reviewer question

How sensitive is its effectiveness to domain shifts between the training data and deployment scenarios?

Response

We learn P on Alpaca-cleaned and keep it fixed on 64 LongBench inputs at 1K–8K lengths, without domain-specific recalibration. Using identical frozen Q/K/V from each trained MANTA checkpoint, rank-16 compression yields relative increases in mean attention-output error of 0.06–3.25%, compared with the uncompressed operator. These results support transfer of the learned projection with limited additional local error in this setting.

Q3

Reviewer question

I would find a comparison between the following two practical configurations more convincing.

Response

On Llama-3.1-8B with the same training budget and cache size 64 (8 sink + 56 recent tokens), MANTA achieves 58.3% five-shot MMLU accuracy, versus 55.0% for single-center + SWA without compression. This 3.3-percentage-point improvement supports the practical advantage of multi-center attention with compressed recurrent states.

Thank you for the additional experiments. These results provide useful evidence of MANTA's performance.

Could the authors report the number of generated tokens for GSM8K and MBPP, along with the maximum generation length used in these evaluations? It would be helpful to clarify whether these tasks involve sufficiently long generations to meaningfully evaluate this concern.

I would also appreciate a direct inference efficiency comparison between MANTA and GDN under matched hardware and workload settings. Gated DeltaNet has already been adopted in practical large-scale architectures such as Qwen3.5, making it a particularly relevant efficiency baseline.

Reviewer_Au6KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes method named MANTA, which uses multi-center Taylor approximations and compressed memory states to make attention layer more efficient, while retaining exact attention for a small set of tokens. Experiments on Llama-3.1-8B show faster decoding and lower memory use, with an average score close to the original model across six benchmarks.

Critical strengths

The multi-center construction is a natural extension of Taylor-based attention that preserves fixed-size recurrent memory. In the dense ablation without sliding-window attention, increasing the number of centers from one to four improves the five-task average.

Critical weaknesses
  1. MANTA groups keys using pre-RoPE representations, while its Taylor approximation uses post-RoPE attention scores. The connection between these two choices needs clearer justification. In particular, it would help to measure whether the proposed grouping reduces the actual attention approximation error.

  2. In Table 2, sliding-window attention and joint normalization are added together. This makes it difficult to determine how much of the improvement comes from joint normalization rather than from retaining exact attention over recent tokens.

  3. The LongBench experiments compare MANTA with Liger but do not include the original Llama-3.1-8B model with full softmax attention. This makes it difficult to assess how much of the original model’s long-context performance is preserved.

Questions for authors
  1. What evidence shows that grouping keys before RoPE improves the attention approximation after RoPE? Can the authors provide direct measurements of the error relative to exact softmax attention as the number of centers increases?

  2. How does MANTA compare with the original Llama-3.1-8B model using full softmax attention on LongBench, under the same evaluation protocol and comparable fine-tuning? This comparison would help establish how much long-context performance is preserved.

5 comments
AuthorsOpenReview ↗

Responses to W1, W2 and Q1

We thank the reviewer for the careful assessment of our work and the constructive comments. Below, we address each concern with additional experimental evidence and clarifications.

W1 and Q1

Reviewer comment — W1

MANTA groups keys using pre-RoPE representations, while its Taylor approximation uses post-RoPE attention scores. The connection between these two choices needs clearer justification. In particular, it would help to measure whether the proposed grouping reduces the actual attention approximation error.

Reviewer question — Q1

What evidence shows that grouping keys before RoPE improves the attention approximation after RoPE? Can the authors provide direct measurements of the error relative to exact softmax attention as the number of centers increases?

Response

Pre-RoPE keys determine the routing weights, while each Taylor expansion center is the routing-weighted mean of the corresponding post-RoPE keys. The approximation therefore targets the actual post-RoPE attention scores.

Increasing the number of centers reduces both attention-weight and attention-output errors with pre-RoPE grouping, whereas random soft grouping provides essentially no improvement.

Attention-weight relative L2 error versus exact causal softmax (8K context; mean; lower is better):

Routing representation1 center2 centers4 centers8 centers
Pre-RoPE0.93490.89470.81360.6649
Post-RoPE0.93490.83330.77310.6708
Random soft routing0.93490.93480.93510.9372

Attention-output relative L2 error versus exact causal softmax (8K context; mean; lower is better):

Routing representation1 center2 centers4 centers8 centers
Pre-RoPE2.05291.85441.63561.2950
Post-RoPE2.05291.75471.56531.1061
Random soft routing2.05292.05292.05292.0539

We also report results for post-RoPE grouping, which yields lower frozen-model approximation error at four centers. However, pre-RoPE routing serves our goal of separating content-based grouping from the current layer’s positional rotation, while retaining positional information in the post-RoPE states and expansion centers. Moreover, subsequent language-model fine-tuning adapts both the router and the LoRA parameters, so lower frozen-model approximation error does not necessarily translate into higher downstream accuracy.

We therefore complement the approximation-error analysis with a downstream comparison of both routing choices following the same experimental protocol.

Routing representationPIQAARC-EasyARC-ChallengeHellaSwagWinoGrandeMMLUAvg.
Pre-RoPE80.28 ± 0.1982.41 ± 0.2558.02 ± 0.7780.62 ± 0.0273.99 ± 0.4357.88 ± 0.2572.20 ± 0.25
Post-RoPE80.22 ± 0.3582.13 ± 0.7856.40 ± 0.8580.90 ± 0.0274.03 ± 0.0857.21 ± 0.4971.82 ± 0.23

Post-RoPE provides no consistent downstream advantage in these runs; the six-task mean favors pre-RoPE by 0.38 percentage points, supporting our choice of content-based pre-RoPE routing.

W2

Reviewer comment — W2

In Table 2, sliding-window attention and joint normalization are added together. This makes it difficult to determine how much of the improvement comes from joint normalization rather than from retaining exact attention over recent tokens.

Response

To isolate the effect of normalization, we retain the same compressed-history and exact-SWA branches, but normalize them independently and combine their outputs with fixed 0.5/0.5 weights. The exact cache remains unchanged at 8 sink and 56 recent tokens.

SettingPIQAARC-EasyARC-ChallengeHellaSwagWinoGrandeMMLUAvg.
No SWA (A3, Table 2)76.573.045.968.763.124.958.7
SWA + separate normalization, equal mixing79.878.951.574.072.148.467.5
SWA + joint normalization (A4, Table 2)80.382.358.881.173.858.872.5

With the exact-SWA branch unchanged, joint normalization improves the six-task mean by approximately 5.0 percentage points over fixed equal mixing, supporting its additional contribution beyond retaining exact attention for the selected tokens.

AuthorsOpenReview ↗

Responses to Q2 and W3

W3 and Q2

Reviewer comment — W3

The LongBench experiments compare MANTA with Liger but do not include the original Llama-3.1-8B model with full softmax attention. This makes it difficult to assess how much of the original model's long-context performance is preserved.

Reviewer question — Q2

How does MANTA compare with the original Llama-3.1-8B model using full softmax attention on LongBench, under the same evaluation protocol and comparable fine-tuning? This comparison would help establish how much long-context performance is preserved.

Response

We now include the original Llama-3.1-8B with full softmax attention under the same evaluation protocol on the seven LongBench tasks reported in the paper.

MANTA + SFT reaches 79.7% of the original model's seven-task mean and outperforms Liger + SFT by 7.4 points. This shows substantial retention on the reported tasks.

TaskOriginal Llama-3.1-8BMANTA + SFTLiger + SFT
Average (7 tasks)34.927.920.5
Qasper12.717.116.2
MultiFieldQA-EN32.725.221.7
HotpotQA11.928.715.5
2WikiMQA13.922.319.9
MuSiQue8.510.66.3
TREC73.528.026.5
TriviaQA91.363.137.1

Dear Reviewer Au6K,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we wanted to follow up on our response. If you have a chance, we would welcome any feedback on our clarifications and would be happy to answer any remaining questions.

Best regards,

The Authors

Reviewer_bgToOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MANTA, a method that converts softmax attention in pretrained LLMs into linear attention using a first order Taylor approximation around multiple centers. It combines soft key assignment, a query aware distance metric, low rank state compression, and exact attention for sink and recent tokens. The conversion involves two training stages. The experiments evaluate model performance after conversion, inference efficiency, and performance on long contexts. Ablation studies examine the contributions of the main components.

Critical strengths
  1. The paper proposes a multi-center approximation method that effectively uses the original LLM weights to efficiently convert softmax attention into linear attention. The ablation studies show that the proposed strategies effectively mitigate the performance degradation after conversion.

  2. The efficiency gains after linearization are substantial. By combining compressed recurrent states with a constant exact attention cache, MANTA keeps memory requirements independent of context length. The reported experiments demonstrate higher decoding throughput and greater request capacity, highlighting its practical value for serving long sequences under limited GPU memory.

Critical weaknesses
  1. The ablation studies show that sink SWA is crucial for maintaining performance, but a baseline using only sink SWA is missing. Without this comparison under the same cache size and training budget, it is difficult to determine how much the proposed linear branch contributes to the final performance.

  2. The long context evaluation does not clearly establish how much of the original model’s capability MANTA retains. Results for the original Llama model and a comparison with GDN are missing, and Liger’s conversion training before SFT is insufficiently documented. SFT uses only 512 mixed examples, leaving it unclear whether both models have sufficiently converged and whether the performance gap would persist with more training data. Moreover, MANTA performs poorly on passage retrieval and counting even after SFT. These gaps limit the evidence supporting its long context effectiveness.

Questions for authors
  1. Could the authors consider evaluating MANTA on a more recent open source LLM? The current experiments are limited to Llama-3.1-8B, and results on newer models would strengthen the evidence for its broader applicability.

  2. Could the authors discuss whether MANTA’s approximation could support conversion from softmax attention to linear architectures with explicit memory management, such as Gated DeltaNet or Kimi DeltaNet? Although such an extension may be future work, understanding its feasibility would help clarify the broader practical value of the method.

If the author can address my questions, I will raise my rating.

9 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the careful assessment of our work and the constructive comments. Below, we address each concern with additional experimental evidence and clarifications.

W1

Reviewer comment

The ablation studies show that sink SWA is crucial for maintaining performance, but a baseline using only sink SWA is missing. Without this comparison under the same cache size and training budget, it is difficult to determine how much the proposed linear branch contributes to the final performance.

Response

We have trained a sink SWA-only baseline on Llama-3.1-8B and compared it with MANTA. Both models retain the same 64-token exact KV cache (8 sink tokens and 56 recent tokens). MANTA improves MMLU accuracy by 3.9 percentage points, supporting the contribution of its linear-attention branch.

ModelExact KV cacheMMLU accuracy (%)
Sink SWA-only8 sink + 56 recent tokens54.4
MANTA W64 (4 centers, rank 16)8 sink + 56 recent tokens58.3

Once a non-sink token leaves the recent window, sink SWA-only can no longer attend directly to that token. MANTA additionally summarizes older non-sink tokens in a compressed recurrent state and uses query-dependent multi-center readouts to access this history. This can help preserve information from earlier demonstrations and question context in five-shot MMLU, providing a plausible explanation for the observed improvement.

AuthorsOpenReview ↗

Response to W2

W2

Reviewer comment

The long context evaluation does not clearly establish how much of the original model’s capability MANTA retains. Results for the original Llama model and a comparison with GDN are missing, and Liger’s conversion training before SFT is insufficiently documented. SFT uses only 512 mixed examples, leaving it unclear whether both models have sufficiently converged and whether the performance gap would persist with more training data. Moreover, MANTA performs poorly on passage retrieval and counting even after SFT. These gaps limit the evidence supporting its long context effectiveness.

Response

We now compare GDN + SFT, MANTA + SFT, Liger + SFT, and the original Llama-3.1-8B using identical LongBench settings. On the seven main-text tasks, MANTA exceeds GDN and Liger by 1.75 and 7.40 points, respectively.

ModelAverage (21 tasks)Average (main 7)PassageRetrieval-ENPassageRetrieval-ZHPassageCount
GDN + SFT20.4926.124.505.003.00
MANTA + SFT21.4527.865.503.501.00
Liger + SFT15.4520.474.251.760.23
Original Llama-3.1-8B33.6134.9425.7566.940.88

Validation response-token NLL is measured on the same 32 held-out examples (7,974 response tokens). Over the final 64 updates, validation NLL decreases by only 0.54% for MANTA and 0.66% for Liger, suggesting that both models have largely converged under this training setting.

ModelStep 0Step 64Step 128Step 192Step 256
MANTA2.07911.80871.72441.68741.6783
Liger8.78032.18062.02751.97561.9626

To directly test whether the performance gap persists with more data, we repeated SFT from each model's original pre-SFT checkpoint using 2,048 unique examples (4×). We evaluate both on the same seven main-text LongBench tasks. Both models improve by approximately 2.15 points, while MANTA's 7.40-point advantage persists. Thus, additional SFT data benefits both models without closing the observed performance gap in this experiment.

SFT examplesUpdatesMANTA (main 7 avg.)Liger (main 7 avg.)MANTA − Liger
51225627.8620.47+7.40
2,0481,02430.0122.61+7.40

We acknowledge the retrieval and counting limitations: fixed-size state compression may blur passage identities and indices. Similar retrieval difficulties occur in pretrained recurrent models: Yuan et al. [1] report LongBench PassageRetrieval scores of 1.2 (Mamba-2.8B), 4.1 (Mamba2-2.7B), and 4.5 (RWKV-5-World-7B). Together with the GDN and Liger results above, this suggests a shared challenge for fixed-state recurrent architectures. We will qualify our long-context claims accordingly.

References

[1] Yuan, Jiayi, et al. "Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.

AuthorsOpenReview ↗

Response to Q1

Q1

Reviewer question

Could the authors consider evaluating MANTA on a more recent open source LLM? The current experiments are limited to Llama-3.1-8B, and results on newer models would strengthen the evidence for its broader applicability.

Response

We additionally evaluate Qwen3-4B and Qwen3-8B, comparing MANTA with reproduced GDN/Liger baselines and the original softmax models under the same evaluation settings. MANTA/GDN retain 8 sink and 56 recent tokens; Liger retains 64 preceding tokens plus the current token. MANTA achieves the highest six-task average among converted models at both sizes, exceeding GDN by 1.53 and 1.13 points, and Liger by 5.19 and 3.40 points, on 4B and 8B, respectively. These consistent average gains support its applicability to newer models and different parameter scales.

BackboneMethodPIQAARC-EasyARC-ChallengeHellaSwagWinoGrandeMMLUAvg.
Qwen3-4BOriginal softmax75.0880.3954.1068.4765.9870.1769.03
Qwen3-4BLiger-GLA74.9278.5850.5167.7262.9850.9564.28
Qwen3-4BGDN75.1480.4754.0167.8466.6163.5567.94
Qwen3-4BMANTA77.7582.1555.1270.5866.3864.8069.46
Qwen3-8BOriginal softmax76.5083.6356.6674.8968.1174.9372.45
Qwen3-8BLiger-GLA78.0281.6958.1173.6468.7552.9368.86
Qwen3-8BGDN76.4483.4656.1474.4367.8868.4271.13
Qwen3-8BMANTA78.4082.9157.0876.3668.9069.8872.26
Reviewer_yMqKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MANTA, a method to convert a pretrained softmax Transformer into a fixed-state recurrent model by replacing one global Taylor center with several local ones. It conducted experiments on Llama-3.1-8B and basically matches the short-context accuracy while decoding much faster.

  • background

Softmax attention is expensive at long context because key-value cache grows with the sequence. Pretraining from scratch with linear and recurrent methods is expensive. But converting an pretrained softmax model is hard because a first-order Taylor expansion is accurate only near its expansion point, and that why prior Taylor linearization expanding every key around one global mean is not ideal (Kuzina et al.).

  • method

MANTA makes many local expansions. It chooses centers using pre/post-RoPE keys. Each center keeps its stats. The memory-heavy moment matrix in each state is then projected onto a shared rank-r basis. Training is two-stage: first the dense multi-center model and router, then the low-rank basis and Q/K/V/O LoRA while the router is frozen.

  • evaluation

MANTA evaluates a linearized Llama-3.1-8B in multiple downstream tasks and compares with multiple works. The accuracy is close to the baseline with 64-token setting and exceeds the baseline and prior linearization works with 128-token setting.

Critical strengths
  • The paper is easy to follow, and the explanation of the idea is straightforward, though many math details are involved.

  • The evaluation setup is simple yet to the point, and covers multiple prior works.

Critical weaknesses
  • The technical contribution seems limited regarding the scope and the improvement over existing works.
Questions for authors

Suggestion (I am not an expert):

  • Pretraining Llama from scratch is hard, but maybe you can use a small benchmark like OLMo to compare with works that require pretraining.

  • In eval, MMLU drops for all including MANTA. Why is that?

  • Is it possible to rerun some prior work rather than using their numbers?

2 comments
AuthorsOpenReview ↗

Responses to Q1 and Q2

We thank the reviewer for the careful assessment of our work and the constructive comments. Below, we address each concern with additional experimental evidence and clarifications.

Q1

Reviewer comment

Pretraining Llama from scratch is hard, but maybe you can use a small benchmark like OLMo to compare with works that require pretraining.

Response

Thank you for this suggestion. We have conducted an additional from-scratch pretraining comparison between MANTA and Gated DeltaNet (GDN) using a small OLMo Hybrid configuration.

Both models use an eight-layer backbone with hidden size 384 and a 3:1 ratio of linear-attention to full-attention layers. To closely match GDN's parameter budget, we align MANTA's projection widths and add gating and causal short convolutions. Both models are trained from random initialization on the same 9.19B-token Dolma3 subset, using four H100 GPUs, a sequence length of 8,192, a global batch size of 196,608 tokens (24 sequences), and a peak learning rate of 2.9 × 10⁻³.

The table first reports the macro-average over all 23 length-normalized accuracy entries, followed by every individual entry. Prompt-format labels (MC-fast and RC) and test/validation splits are shown separately. MANTA achieves comparable accuracy to GDN under matched model capacity and identical training-token budgets, with a 0.39 percentage-point advantage on this aggregate. We interpret this as evidence that MANTA is also effective for from-scratch pretraining in small-scale settings.

Evaluation entryGDNMANTA
Macro-average (all 23 entries)30.2030.59
ARC-Challenge — test, RC22.6123.12
ARC-Easy — test, RC44.4041.88
CommonsenseQA — validation, RC37.6734.97
HellaSwag — RC27.9026.90
MMLU Humanities — test, MC-fast24.7824.55
MMLU Humanities — test, RC25.2324.55
MMLU Humanities — validation, MC-fast24.5224.90
MMLU Humanities — validation, RC25.1025.48
MMLU Other — test, MC-fast27.4525.60
MMLU Other — test, RC28.1928.22
MMLU Other — validation, MC-fast25.6324.51
MMLU Other — validation, RC27.8930.99
MMLU Social Sciences — test, MC-fast25.6429.70
MMLU Social Sciences — test, RC28.3427.98
MMLU Social Sciences — validation, MC-fast24.6328.49
MMLU Social Sciences — validation, RC24.0425.82
MMLU STEM — test, MC-fast26.1125.55
MMLU STEM — test, RC25.1824.25
MMLU STEM — validation, MC-fast23.6825.55
MMLU STEM — validation, RC24.6128.35
PIQA — validation, RC60.4560.23
Social IQa — validation, RC40.0740.94
WinoGrande — validation, RC50.5151.07

Q2

Reviewer question

In eval, MMLU drops for all including MANTA. Why is that?

Response

We interpret this shared decline as incomplete recovery of pretrained capabilities after attention conversion. Replacing softmax attention perturbs computations learned during pretraining, and the comparatively limited conversion training may not fully restore them.

This recovery gap is particularly relevant to five-shot MMLU, which jointly requires broad subject knowledge, in-context retrieval, and mapping the correct answer content to its option label. These abilities may remain insufficiently recovered when the conversion data and objectives do not adequately cover the relevant knowledge and interaction patterns.

This explanation is consistent with LoLCATs (Appendix B.5.2–B.5.3) [1], which examines the mismatch between linearization training and the retrieval demands of five-shot MMLU. Adding similarly formatted CommonsenseQA examples to its conversion data improves MMLU from 52.8% to 54.5%, supporting the role of conversion-data coverage and task format in capability recovery.

References

[1] Zhang, Michael, et al. "Lolcats: On low-rank linearizing of large language models." International Conference on Learning Representations. Vol. 2025. 2025.

AuthorsOpenReview ↗

Responses to Q3 and W1

Q3

Reviewer question

Is it possible to rerun some prior work rather than using their numbers?

Response

Following this suggestion, we reran the GDN and Liger implementations on Llama-3.1-8B, Qwen3-4B, and Qwen3-8B.

On Llama-3.1-8B, GDN nearly matches the reported mean (71.30% versus 71.34%), while Liger's mean is within 1.10 percentage points of the published reference (67.20% versus 68.3%), although its MMLU score is lower (39.43% versus 46.9%). MANTA consistently achieves the highest six-task mean among the converted models on all three backbones, exceeding reproduced GDN by 1.10, 1.53, and 1.13 percentage points, respectively.

BackboneMethodSourceMeanPIQAARC-EARC-CHellaSwagWinoGrandeMMLU
Llama-3.1-8BLiger-GLAPublished reference68.380.581.855.675.469.546.9
Llama-3.1-8BLiger-GLAOur reproduction67.2080.0381.2354.2776.5971.6739.43
Llama-3.1-8BGDNPublished reference71.3479.0781.9754.9278.7974.1159.17
Llama-3.1-8BGDNOur reproduction71.3079.0581.9755.0978.8874.2758.51
Llama-3.1-8BMANTAOur main paper72.480.382.958.080.674.458.3
Qwen3-4BLiger-GLAOur reproduction64.2874.9278.5850.5167.7262.9850.95
Qwen3-4BGDNOur reproduction67.9475.1480.4754.0167.8466.6163.55
Qwen3-4BMANTAOurs69.4677.7582.1555.1270.5866.3864.80
Qwen3-8BLiger-GLAOur reproduction68.8678.0281.6958.1173.6468.7552.93
Qwen3-8BGDNOur reproduction71.1376.4483.4656.1474.4367.8868.42
Qwen3-8BMANTAOurs72.2678.4082.9157.0876.3668.9069.88

W1

Reviewer comment

The technical contribution seems limited regarding the scope and the improvement over existing works.

Response

As mentioned in Q3, we have broadened the evaluation to Qwen3-4B and Qwen3-8B, including reproduced Liger and GDN baselines and the original softmax models. The from-scratch OLMo Hybrid experiment in Q1 further provides initial evidence for MANTA's potential as a pretraining attention mechanism.

MANTA achieves the highest six-task mean among the converted models at both sizes, exceeding GDN by 1.53 and 1.13 percentage points, respectively. MANTA also achieves a higher average length-normalized accuracy than GDN in the from-scratch OLMo Hybrid experiment (Q1): 30.59% versus 30.20%. These average gains are consistent across the tested conversion and pretraining settings.

Reviewer_R8ZBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MANTA converts a pretrained softmax Transformer into a model with a fixed-size recurrent state. It routes keys into several groups, applies a local Taylor approximation within each group, and compresses the resulting states with a learned low-rank projection. Joint normalization combines this compressed history with exact attention over sink and recent tokens. The evaluation uses Llama-3.1-8B and covers downstream accuracy, LongBench, and autoregressive serving throughput.

Critical strengths
  1. The main design choices have direct experimental support.

  2. The reported quality–throughput trade-off is useful.

Critical weaknesses
  1. The average score hides a meaningful loss on MMLU: 58.3 with the 64-token cache and 62.4 with the 128-token cache, compared with 65.3 for softmax (in table 1). The appendix does report all 21 LongBench tasks, but only compares MANTA with Liger. Without the original softmax model under the same protocol, it is difficult to judge how much long-context capability survives conversion. The claim of recovering the original model's quality needs this qualification.

  2. Table 2 does not isolate joint normalization. The change from A3 to A4 adds both exact attention and joint normalization, raising the average from 58.7 to 72.5. This supports the combined hybrid, but does not show how much the normalization rule contributes. A comparison using the same exact cache with separately normalized branches would address that attribution directly.

Questions for authors
  1. How does the original softmax Llama-3.1-8B model perform with the same LongBench prompting, truncation, and scoring, particularly on the full-suite average and passage retrieval?

  2. With the exact cache, centers, and training budget fixed, how does joint normalization compare with separately normalized branches and a tuned mixing weight?

1 comment
AuthorsOpenReview ↗

Responses to Reviewer R8ZB

We thank the reviewer for the careful assessment of our work and the constructive comments. Below, we address each concern with additional experimental evidence and clarifications.

W1 and Q1

Reviewer comment — W1

The average score hides a meaningful loss on MMLU: 58.3 with the 64-token cache and 62.4 with the 128-token cache, compared with 65.3 for softmax (in table 1). The appendix does report all 21 LongBench tasks, but only compares MANTA with Liger. Without the original softmax model under the same protocol, it is difficult to judge how much long-context capability survives conversion. The claim of recovering the original model's quality needs this qualification.

Reviewer question — Q1

How does the original softmax Llama-3.1-8B model perform with the same LongBench prompting, truncation, and scoring, particularly on the full-suite average and passage retrieval?

Response

We agree that the six-task mean masks a meaningful MMLU loss: the 64-token and 128-token configurations fall below softmax by 7.0 and 2.9 percentage points, respectively. A plausible explanation is that limited conversion training does not fully restore the pretrained computations supporting the knowledge access, in-context retrieval, and answer-to-option mapping required by five-shot MMLU.

We additionally evaluated the original softmax Llama-3.1-8B on all 21 LongBench tasks.

ModelAverage (all 21 tasks)PassageRetrieval-ENPassageRetrieval-ZH
Original softmax Llama-3.1-8B33.6125.7566.94
MANTA18.825.504.00
MANTA + SFT21.455.503.50

The passage retrieval remains a limitation that the existing 16K SFT does not resolve. However, similar passage-retrieval difficulties have been reported for other fixed-state recurrent models, suggesting a shared challenge across these architectures. The following published scores are from [1].

Given an abstract, passage retrieval requires identifying its source passage among many distractors and returning the corresponding paragraph index. Compressing distant passages into a shared fixed-size state can blur passage-specific details and their associations with paragraph indices, making the correct passage harder to distinguish when the abstract is presented.

ModelReported LongBench PassageRetrieval score
Mamba-2.8B1.2
Mamba2-2.7B4.1
RWKV-5-World-7B4.5

We will qualify the claim of recovering the original model's quality to refer specifically to the average accuracy on the reported downstream benchmarks, while explicitly acknowledging the remaining MMLU and long-context retrieval losses. We will also add the original softmax baseline for all 21 LongBench tasks to the appendix.

W2 and Q2

Reviewer comment — W2

Table 2 does not isolate joint normalization. The change from A3 to A4 adds both exact attention and joint normalization, raising the average from 58.7 to 72.5. This supports the combined hybrid, but does not show how much the normalization rule contributes. A comparison using the same exact cache with separately normalized branches would address that attribution directly.

Reviewer question — Q2

With the exact cache, centers, and training budget fixed, how does joint normalization compare with separately normalized branches and a tuned mixing weight?

Response

To isolate normalization, we retain A4's branch architectures, four centers, and exact cache of 8 sink and 56 recent tokens, with the same training pipeline. We compare joint normalization with two independently normalized variants: fixed 0.5/0.5 mixing and learned static mixing, using one sigmoid coefficient per layer/query head, initialized to 0.5.

SettingPIQAARC-EasyARC-ChallengeHellaSwagWinoGrandeMMLUAvg.
Separate normalization + fixed equal mixing79.8278.9151.4574.0572.1448.3567.45
Separate normalization + learned static mixing79.9879.6352.2276.2672.3052.0968.75
Joint normalization (A4, Table 2)80.382.358.881.173.858.872.5

Learning the mixing weights improves the six-task mean by 1.29 percentage points, but joint normalization retains an approximately 3.75-point advantage, including 6.71 points on MMLU. These results support an additional contribution from joint normalization beyond retaining the exact cache and optimizing static branch weights. The learned far-branch coefficients have a final mean of 0.425 and range from 0.227 to 0.578, confirming that the weights adapted during training.

References

[1] Yuan, Jiayi, et al. "Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.