- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MANTA, a method for converting pretrained softmax Transformers into fixed-state linear attention models with limited additional training. To address the approximation errors caused by single-center Taylor expansion, MANTA introduces query-aware multi-center routing to better approximate softmax attention. It further employs learned low-rank projections to reduce recurrent-state memory overhead and combines compressed historical information with exact sliding-window attention through joint normalization. Experiments on Llama-3.1-8B demonstrate that MANTA largely preserves the original model's accuracy on standard benchmarks while achieving substantial improvements in decoding throughput. The authors also evaluate its effectiveness on long-context tasks and report improvements over existing linearization methods.
Critical strengths
-
Efficient linearization with limited additional training. The proposed method enables the conversion of pretrained softmax Transformers into linear attention models with relatively low training overhead. It achieves nearly lossless performance on standard benchmarks while providing substantial inference efficiency gains.
-
Insightful mathematical analysis. The paper provides a thorough and insightful mathematical analysis of the limitations of single-center Taylor approximation, offering a well-motivated foundation for the proposed multi-center approach.
-
Memory-efficient design. The authors effectively address the additional memory overhead introduced by multiple centers through low-rank recurrent state compression, making the proposed approach more practical for deployment.
Critical weaknesses
-
Limited evaluation across model architectures and scales: The experiments are conducted exclusively on a single model size from one model family. This is insufficient to demonstrate the generalizability of the proposed approach. I would strongly encourage the authors to extend their evaluation to different model architectures and parameter scales. The generalizability of MANTA is my primary concern.
-
Lack of direct evaluation of attention approximation error: A central motivation of MANTA is to reduce the approximation error relative to standard softmax attention. However, the authors primarily validate this claim through downstream benchmark accuracy, without directly quantifying the approximation error between MANTA attention and exact softmax attention. A more direct analysis would provide stronger evidence supporting the proposed method.
-
Insufficient analysis of training costs: Although MANTA aims to provide an efficient conversion of pretrained Transformers, the paper lacks a detailed analysis of the training costs associated with the conversion process. Reporting the computational resources and training time required would help clarify the practical efficiency of the proposed approach.
Questions for authors
-
Performance on long-generation reasoning tasks: I appreciate the inclusion of long-context benchmarks. However, most of these tasks involve long inputs but relatively short outputs. I am interested in how the converted model performs on challenging mathematical reasoning and coding tasks that require extended reasoning or long generation sequences. In autoregressive generation, approximation errors may propagate or accumulate over successive decoding steps, potentially leading to substantial performance degradation. Could the authors evaluate MANTA on such long-generation tasks to better demonstrate its effectiveness and robustness?
-
Robustness of the learned low-rank projection under domain shift: Regarding the learnable projection matrix (P) introduced for low-rank recurrent state compression, how sensitive is its effectiveness to domain shifts between the training data and deployment scenarios? I would be interested in understanding how much performance degradation may occur when applying a projection learned from one data distribution to a substantially different domain, while keeping the model configuration fixed.
-
Comparison between practical single-center and multi-center configurations. Section 5.2 provides a detailed analysis of the benefits of increasing the number of Taylor centers. However, I would find a comparison between the following two practical configurations more convincing. These two configurations represent realistic deployment choices for the proposed method and its single-center counterpart:
- Multi-center + SWA + Low-rank compression: The proposed MANTA configuration.
- Single-center + SWA, without Low-rank compression: A simpler baseline configuration.
6 comments
Responses to Q1, Q2 and Q3
Q1
Reviewer question
Could the authors evaluate MANTA on such long-generation tasks to better demonstrate its effectiveness and robustness?
Response
We evaluate Qwen3-8B on the full GSM8K test set (mathematical reasoning) and MBPP-sanitized test set (coding). MANTA and GDN use cache size 128 (8 sink + 120 recent tokens).
MANTA outperforms GDN by 18.12 percentage points on GSM8K and 7.78 points on MBPP, supporting its effectiveness and relative robustness to attention conversion across mathematical reasoning and coding.
| Model | GSM8K numeric-answer accuracy (%) | MBPP-sanitized pass@1 (%) |
|---|---|---|
| Original softmax | 92.34 | 77.04 |
| GDN | 58.07 | 63.42 |
| MANTA | 76.19 | 71.21 |
Q2
Reviewer question
How sensitive is its effectiveness to domain shifts between the training data and deployment scenarios?
Response
We learn P on Alpaca-cleaned and keep it fixed on 64 LongBench inputs at 1K–8K lengths, without domain-specific recalibration. Using identical frozen Q/K/V from each trained MANTA checkpoint, rank-16 compression yields relative increases in mean attention-output error of 0.06–3.25%, compared with the uncompressed operator. These results support transfer of the learned projection with limited additional local error in this setting.
Q3
Reviewer question
I would find a comparison between the following two practical configurations more convincing.
Response
On Llama-3.1-8B with the same training budget and cache size 64 (8 sink + 56 recent tokens), MANTA achieves 58.3% five-shot MMLU accuracy, versus 55.0% for single-center + SWA without compression. This 3.3-percentage-point improvement supports the practical advantage of multi-center attention with compressed recurrent states.
Thank you for the additional experiments. These results provide useful evidence of MANTA's performance.
Could the authors report the number of generated tokens for GSM8K and MBPP, along with the maximum generation length used in these evaluations? It would be helpful to clarify whether these tasks involve sufficiently long generations to meaningfully evaluate this concern.
I would also appreciate a direct inference efficiency comparison between MANTA and GDN under matched hardware and workload settings. Gated DeltaNet has already been adopted in practical large-scale architectures such as Qwen3.5, making it a particularly relevant efficiency baseline.
Responses to W1, W2 and W3
We thank the reviewer for the careful assessment of our work and the constructive comments. Below, we address each concern with additional experimental evidence and clarifications.
W1
Reviewer comment
Response
We have extended our Llama-3.1-8B experiments to Qwen3-4B and Qwen3-8B, adding a second model family and a second parameter scale. We compare Liger-GLA, GDN, and MANTA with the original full-softmax models using identical evaluation settings. MANTA uses four centers and rank 16. SWA retains 8 sink and 56 recent tokens for MANTA/GDN; Liger uses its original window of 64 preceding tokens plus self, without sinks.
MANTA achieves the highest average among converted models at both scales. Its gains over GDN are 1.53 and 1.13 percentage points, and over Liger are 5.19 and 3.40 points, on 4B and 8B, respectively. The corresponding differences from full softmax are +0.43 and -0.20 points. Together with the existing Llama results, this consistent average advantage on both Qwen3 scales supports MANTA's generalization across model families and parameter sizes.
W2
Reviewer comment
Response
Using frozen pretrained Q/K/V activations from Llama-3.1-8B and Qwen3-8B, we directly compare first-order Taylor attention with one center (R=1) and four centers (R=4) against exact causal softmax. Routers are calibrated on 128 Alpaca-cleaned texts; evaluation uses 64 held-out LongBench inputs at 1K, 4K, and 8K lengths, sampling 128 causal query positions per input.
We report relative Frobenius errors of effective attention weights and per-head attention outputs (before the output projection), normalized by the corresponding exact-softmax norm and averaged over inputs, layers, and query heads. Lower is better.
Four centers reduce output error by 26.48–31.34% on Llama and 7.03–13.12% on Qwen, and weight error by 11.41–19.51% and 2.61–7.44%, respectively. These controlled measurements directly verify that multiple centers reduce approximation error.
W3
Reviewer comment
Response
For Llama-3.1-8B with MANTA (four centers, rank 16, cache size 128), the total training time is 2.87 hours on one 80GB H100 and 5.21 hours on one 32GB RTX 5090. These measurements demonstrate that MANTA can adapt an 8B model using only a single GPU and a few hours of fine-tuning, including on consumer GPU hardware.