acceptodds
Under review as a conference paper at ICLR 2027

MANTA: A Multi-level Aggregation Transformer

Abstract

Large language models are increasingly deployed on tasks that require access to large amounts of context, including document collections, code repositories, and long-running agent trajectories. Processing such extensive information with transformers is computationally and memory intensive: during autoregressive generation, the key-value (KV) cache grows linearly with sequence length, increasing accelerator memory demands and limiting throughput. The challenge is therefore to reduce the memory required to represent the full sequence history without compromising the information available for future predictions. We introduce MANTA, a Multi-level Aggregation Transformer that progressively aggregates distant context while retaining recent tokens at full resolution. MANTA maintains a window size at each aggregation level, resulting in logarithmic KV-cache growth while retaining information spanning the full sequence history. We evaluate MANTA by adapting the Llama 3.2 1B base architecture and compare it with a dense-attention baseline. Despite its substantially smaller KV cache, MANTA achieves competitive language-modeling performance and retains strong performance on long-context tasks. In a typical configuration, MANTA reduces the KV-cache size by 97.5% at a context length of 8K tokens. We further show that, with a natural modification, aggregation can replace tokenization, and that MANTA can be readily adapted to perform nearly lossless token compression. Overall, multi-level aggregation offers substantial architectural flexibility while directly addressing the memory capacity and bandwidth bottlenecks that increasingly constrain large-scale LLM inference.

Primary area:
generative models
Keywords:
Long-context, KV-Cache Reduction, Multi-Level Attention, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_9JUiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MANTA represents sequence history using recursively aggregated memory tokens and a causal attention mask with bounded windows at each level, yielding logarithmic KV-cache growth. Experiments train 1B language models from scratch and explore hybrid attention, autoencoding, and byte-level variants. Results demonstrate substantial cache reductions and competitive standard language modeling, but considerable retrieval losses for the logarithmic-cache configurations.

Critical strengths
  1. The paper presents a coherent architecture that unifies hierarchical aggregation and retrieval through standard attention and a timestamp-based mask. The resulting cache bound is clear, and the design offers a concrete mechanism for retaining representations of distant history with limited memory.

  2. The 1B-scale, 200B-token experiments provide substantive evidence of feasibility. The aggregation, window, positional-encoding, and hybrid studies explore meaningful design choices. In particular, the unbounded-top and hybrid variants recover substantial retrieval quality while retaining considerable cache savings.

Critical weaknesses
  1. The main comparisons use dense attention and a 16-token window, while representative MANTA configurations retain around 170 KV states per layer. Without memory-matched windows or learned-compression baselines, it is unclear how much recursive aggregation improves the quality–memory–compute trade-off.

  2. Long-context performance remains a concern. At 8K, logarithmic-cache variants score 0.176 – 0.180 on RULER versus 0.585 for dense attention. The strongest recovery comes from variants with linear cache growth. Evaluation beyond 8K and clearer throughput measurements would help establish where the logarithmic design is practically useful.

Questions for authors
  1. Additional evidence: How does MANTA compare with a memory-matched window, single-level learned memory, and an established compression method, accounting for the extra training compute from memory tokens?

  2. Additional evidence: Could you report quality, cache size, and throughput beyond 8K for the bounded, unbounded-top, and hybrid variants?

  3. Clarification: What hardware, precision, kernels, and timing procedure were used for Figure 2? How are memory-token computation, cache maintenance, and prefill accounted for?

  4. Clarification: Does MANTA update the decoder through LoRA while 500x keeps it frozen? Are adaptation budgets comparable? Please also qualify “nearly lossless” using Table 7’s exact reconstruction rates across sequence lengths.

2 comments

We thank the reviewer for the assessment of our work. We particularly appreciate the concrete suggestions regarding memory-matched baselines.

W1/Q2: To compare the performance of MANTA with that of other methods that reduce KV cache requirements, we added sliding-window attention, Activation Beacons, and H2O (Heavy-Hitter Oracle) as new baselines. Under the same KV cache budget, MANTA outperforms these baselines (see Table 1 and Figure 3).

W2/Q1: Figure 11 now compares the KV-cache sizes of the bounded, unbounded-top, and hybrid variants across sequence lengths for Llama 3.2 1B. In addition, we have expanded the analysis beyond 8K by adding evaluations at context lengths of 16K and 32K, reported in Figure 12. Together, these results provide a clearer picture of the quality–memory trade-off.

Q3: The throughput measurements in Figure 2 were obtained using one H100 GPU, BF16 precision, and FlexAttention with the Triton backend and compiled attention. We timed 32 consecutive decode steps with CUDA-synchronized wall-clock timing at the respective sequence length and averaged over five repetitions. Throughput counts only original (non-memory) tokens. Memory-token computations and cache maintenance are included, whereas prefill is excluded. We've added this description to the study accordingly. MANTA’s KV cache is organized into level-specific segments, each acting as a circular buffer that retains the most recent window states at the respective hierarchy level. Thus, the cache size is fixed per level and grows only with the number of hierarchy levels. We have added these implementation and measurement details to the revised manuscript.

Q4: The 500x compressor keeps the decoder frozen. The MANTA autoencoder does not contain a separate encoder and decoder but is built upon one pretrained language model. The MANTA autoencoder and 500x compressor were trained with the same number of LoRA parameters and the same token budget. Table 7 shows that, with 31 visible memory tokens, exact sequence reconstruction is 97%, 77%, and 11% at lengths 128, 256, and 512, respectively, and is lower with smaller memory budgets. Although exact sequence reconstruction decreases with sequence length, BLEU scores remain high, as reported in Figure 6. We have quantified what "nearly lossless" means in the study.

Reviewer_9JUiOpenReview ↗

Response to the authors

I thank the authors for their detailed responses to my review and the additional experiments. After considering these clarifications and results, I have decided to raise my rating.

Reviewer_8wmTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a hierachial transformer architecture that massively reduce use of memory while maintaining perfrmance.

Critical strengths

The idea is elegant tackle of a key bottleneck in LLM architecture.

Critical weaknesses
  • Efficient architecture has evolved significantly in recent years. However, the paper mainly compares to dense transformer.
  • The tasks that the paper evaluate are not representative of today's language model use case anymore
Questions for authors
  • How would authors see the potential role the architecture proposed in today's LLM developement, given that efficient architectures are not widely used, or mostly used as hybrid regime.
  • Today's transformer, while theoretically non-efficient has become very efficient with hardware-aware optimizaiton. How would the proposed architecture, with sparse attention mask, be efficient on hardware given challenges such as poor block utilization and non contiguous memory access.
2 comments

We thank the reviewer for highlighting the tension between theoretical efficiency and practical inference. MANTA contributes to this discussion by addressing KV-cache requirements, a key obstacle to scalable long-context inference. It provides a simple, flexible approach that requires only modest changes to existing Transformers and can be combined with other attention mechanisms in hybrid architectures. By retaining standard attention while introducing structured multi-scale aggregation, MANTA remains applicable across diverse Transformer-based settings.

Q1: KV-cache reduction techniques are increasingly being adopted in modern LLMs, including Kimi K3, DeepSeek V4, and Mistral architectures. Given MANTA’s simple inference implementation and compatibility with existing Transformers, we view it as a simple and flexible approach that can work in tandem with other strategies, particularly in hybrid architectures, as demonstrated in Section 5. Its unbounded variants further provide access to the entire context at a fixed aggregation resolution while substantially reducing KV-cache requirements compared with dense attention. Beyond conventional LLM applications, we envision MANTA’s flexible multi-scale aggregation mechanism being useful for compression, image processing, and extremely long sequences, such as genomic data and weather data.

Q2: Modern hardware is highly optimized for dense tensor operations, particularly through Tensor Cores. MANTA’s KV cache can be implemented as a single cache of fixed or slowly growing size, with individual segments acting as circular buffers for the respective hierarchy levels. Compared with standard inference, the primary modification is the level-dependent RoPE angle. During training, we already achieve competitive performance using FlexAttention, with substantial potential for further optimization because the sparsity pattern is predefined. At inference time, MANTA can be implemented in a manner compatible with highly optimized Transformer implementations.

W1: In the revised manuscript, we substantially broadened our comparison beyond dense attention by adding several efficient KV-cache reduction baselines: causal window attention (KV cache matched), Activation Beacons, and Heavy-Hitter Oracle (H₂O). We match these baselines to MANTA’s KV-cache size to enable a fair comparison. We also evaluate hybrid MANTA architectures that combine MANTA and dense-attention layers, reflecting the increasing use of hybrid attention mechanisms in modern LLMs. These results are reported in Table 1 and Section 5.

W2: As this study provides an initial investigation into the capabilities of the proposed architecture, we primarily focus on established benchmarks that enable controlled comparisons with existing approaches. Evaluating instruction-tuned MANTA models on contemporary application-level tasks remains an important direction for future work.

Thank you for your response! I have decided to maintain my existing score.

Reviewer_MotuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MANTA, a Transformer architecture that progressively aggregates historical context into a hierarchy of learned memory tokens while retaining recent information at higher resolution. Through hierarchical causal attention, MANTA achieves logarithmic KV-cache growth with sequence length.

The authors train 1B-scale models from scratch and evaluate language modeling, long-context retrieval, KV-cache usage, and decoding throughput. The results demonstrate substantial memory savings and decoding speedups, with quality degradation in long-context retrieval performance. The paper also explores architectural variants that trade memory efficiency for improved retrieval quality, as well as extensions to autoencoding and byte-level modeling.

Critical strengths

MANTA uses a relatively intuitive attention mechanism to progressively aggregate historical information into learned memory tokens at different resolutions. Memory tokens aggregate earlier representations, while ordinary tokens can attend to those memories after the original tokens leave the local window. The unified timestamp-based causal attention mask allows the model to aggregate information and access compressed history using standard Transformer attention, while limiting the number of cached KV states at each level. This is a useful architectural design that achieves logarithmic KV-cache growth without requiring separate compression or retrieval modules.

The experiments demonstrate substantial KV-cache savings (around 98% at 8K context length) and decoding speedups. The authors also explore different aggregation settings and architectural variants, showing how retrieval quality can be improved by allocating more memory. These results provide useful insights into the trade-off between memory efficiency and model quality.

Critical weaknesses

The main evaluation compares MANTA mostly against dense and sliding-window attention, making it difficult to assess the benefit of the proposed hierarchical compression mechanism over existing learned-memory approaches.

The effect of increasing context length on information retention is not sufficiently characterized. Figure 3 shows that MANTA benefits from additional context, but the retrieval results are primarily reported at 8K. Evaluating retrieval across context lengths under a fixed configuration would clarify how reliably information remains accessible as length increases.

Questions for authors

See weakness

2 comments

We thank the reviewer for their suggestions on how to improve our evaluation of MANTA’s performance.

  • To better assess MANTA against existing KV cache reduction methods, we added comparisons with Activation Beacons (learned compression) and H2O (Heavy-Hitter Oracle, KV cache eviction) at matched KV cache budgets. As shown in the revised Table 1, MANTA outperforms both methods on long-context benchmarks. We also added Activation Beacons to the perplexity plot in Figure 3.
  • To better characterize information retention across context lengths, we added RULER evaluations at 2K, 4K, 8K, 16K, and 32K, presented in Figure 12 in the appendix. The evaluations at 16K and 32K exceed the training context length of 8K. We leave the study of fine-tuning methods for extending MANTA’s context length to future work.

Thank you for your response. I will keep my score.

Reviewer_RRZhOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

Transformers produce KV cache that grows linearly with the sequence length. In long context scenarios, this causes significant increase in memory usage and slower generation. To address this, the paper introduces MANTA, which retains recent tokens at full resolution while recursively aggregating older context into a hierarchy of learned memory tokens. This causes the KV cache to grow logarithmically. The experiments are done with a 1B model and the results show substantial cache reduction with a strong language-modeling performance but degradation in long-context retrieval tasks when compared to dense attention. The paper also show experiments with hybrid attention variants, context compression, and byte-level modeling.

Critical strengths
  • The KV cache savings are significant. About 97.5% of KV cache memory reduction at 8K context length.
Critical weaknesses
  • Significantly low scores on RULER, despite evaluating within the pretraining context length (8K). This shows that representing the full history does not ensure effective access to its contents. This limitation cannot be attributed to context-length extrapolation.
  • The SWA baseline uses a substantially smaller cache than MANTA. A larger-window SWA baseline with matched cache memory would be a much fair comparison to demonstrate whether hierarchical aggregation provides benefits over SWA.
  • The hybrid experiments lack a matched SWA-dense baseline. So its not clear whether hierarchical aggregation provides additional benefits beyond those supplied by the dense layers.
  • Other methods, such as Activation Beacon (mentioned in related work), also reduce KV-cache memory through learned compression. Comparisons with these alternatives at matched cache budgets would help understand if MANTA offers a better memory-accuracy trade-off.
Questions for authors

Please see weaknesses

2 comments

We thank the reviewer for the careful review and valuable criticism. We fully agree with the reviewer’s main concern regarding the drop in performance when using MANTA instead of dense attention, and we appreciate the suggestions for additional baselines. Reducing the KV cache for LLM inference is increasingly relevant as the number of LLM applications continues to grow. With MANTA, we aim to contribute to this discussion through a simple and flexible approach that allows hierarchical memory to be learned by a single unified transformer, rather than relying on manual tuning of the memory mechanism.

W1:

  • We acknowledge that MANTA’s RULER performance remains below that of dense attention. This reflects the difficulty of preserving fine-grained information under aggressive KV cache reduction. However, the NIAH attention visualizations in Figure 4 demonstrate that MANTA can successfully locate and retrieve information through multiple aggregation levels. Furthermore, retrieval performance continues to improve throughout training, suggesting that alternative training strategies or additional training could further improve information retention. As shown in Section 5, MANTA also offers flexible trade-offs between KV cache size and retrieval performance.

W2/W4:

  • To provide fairer comparisons, we added a sliding-window attention (SWA) baseline with a window size of 175, matching MANTA’s KV cache budget. At 8K context, MANTA (2/16) achieves a RULER score of 0.180, compared with 0.064 for SWA (see Table 1 and Figure 3). This demonstrates the benefit of hierarchical aggregation over using a sliding attention window alone.
  • We also added Activation Beacons and H2O (Heavy-Hitter Oracle) as baselines, both with matched KV cache budgets (see Table 1 and Figure 3). MANTA outperforms both.

W3:

  • We agree that a matched SWA–dense hybrid baseline is important for isolating the contribution of hierarchical aggregation beyond that of the dense layers, and we have added a figure to the appendix. Interestingly, in this setting, windowed intermittent layers outperform MANTA layers. We tentatively interpret this as suggesting that MANTA layers may not receive a sufficiently strong training signal for compression when the model also has access to more direct information pathways. Unfortunately, a full analysis is beyond the scope of this rebuttal. The chosen configuration, c=4c=4, w=16w=16, provides even stronger compression than the configurations reported in Table 1.

Thank you for adding the matched hybrid comparison in Figure 10. Standalone MANTA (2/16) improves over cache-matched SWA on 8K RULER, although it remains well below dense attention even within the training context length. Figure 4 shows attention to the memory tokens containing the needle in one example, but the gap on the broader 8K RULER benchmark remains.

In Figure 10, while adding full-attention layers improves retrieval over pure MANTA, at the same cache budget and dense attention layer count, hybrid-SWA achieves higher scores. With 8 full attention layers, it even matches the fully dense baseline. The SWA-hybrid results therefore leave me unconvinced that hierarchical aggregation provides an additional retrieval benefit in this setting.

I appreciate the added comparisons with other baselines at comparable cache budgets. These experiments address my concerns about the missing baselines, and I have revised my rating accordingly. I still have significant concerns for the reasons discussed above.

Reviewer_K5wvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MANTA, a Multi-level Aggregation Transformer that progressively compresses distant context into hierarchical memory tokens while preserving recent tokens at full resolution, achieving logarithmic KV-cache growth with competitive language modeling performance.The proposed method employs a recursive aggregation mechanism to compress distant historical context into Memory Tokens at different levels while preserving the full representations of recent tokens. By introducing a hierarchical causal attention mask, MANTA enables information aggregation and retrieval across multiple levels, reducing the space complexity of the KV cache.

Critical strengths
  • The paper addresses an important challenge in efficient long-context modeling and presents a simple yet effective hierarchical aggregation mechanism.
Critical weaknesses
  • My main concern is that the core idea of MANTA is quite similar to previous works such as MELODI and AdmTree, which are also cited in the paper. In particular, AdmTree also uses recursive hierarchical memory representations, while MELODI explores multi-level memory compression. Although MANTA introduces a unified hierarchical attention mask, it is still unclear what unique advantages this design brings over existing methods. More direct comparisons and ablation studies would help clarify its contributions.

  • While MANTA achieves impressive KV-cache compression, its long-context retrieval performance is still noticeably worse than dense attention. This raises concerns about how much information is lost during compression. Moreover, the paper does not compare MANTA with common KV-cache compression methods, such as token eviction, KV selection, or KV compression. I think comparisons under the same memory budget, along with a memory–accuracy Pareto frontier, would help better demonstrate the practical benefits of MANTA.

Questions for authors
  • What are the key differences between MANTA and existing hierarchical memory methods such as MELODI and AdmTree? Could the authors provide additional comparisons or ablation studies to demonstrate the benefits of the proposed attention mask?
  • How does MANTA compare with existing KV-cache compression methods under the same memory budget?
2 comments

We thank the reviewer for the careful review and thoughtful criticism. The reviewer is absolutely right to mention the many methods that follow similar ideas. Reducing the footprint of LLM inference is an important topic and with MANTA we try to make a contribution to the discussion.

W1/Q1: Regarding the mentioned methods:

  • MELODI has a fixed two-level hierarchical memory, and uses blockwise aggregation, while MANTA uses a sliding window. The sliding window approach reglarizes the memory access at inference time, providing the options for more efficiency. MELODI also separates long-term and short-term memory by layer, whereas MANTA’s mechanism handles both through a unified mechanism making the method simpler.
  • AdmTree builds a hierarchical tree of representations like MANTA. However, agregation is applied differently. For a node at a higher level, AdmTree computes self-attention over all children of this node and then averages the resulting representations to obtain the node’s representation. Mantas advantage here lies again in its simplicity: A single mechanism, the attention, performs all tasks. This has direct consequences also for efficiency: Our implementation leverages the efficiency of FlexAttention leading to high performance without manual optimization.
  • UniGist and Activation Beacons are most similar and can be interpreted as MANTA with an unbounded level 1 with fixed chunks and blockwise attention.

We believe that the main advantages of MANTA lie in its simple, flexible architecture that drastically reduces the KV cache with a single unified attention mechanism. MANTA can also adapt to many different applications and allows for customized trade-off between KV cache size and performance, as seen in Section 5 on MANTA’s variations.

W2/Q2: As a new comparison, we added Activation Beacons, as well as a second sliding window variant, both with KV cache budgets matched to MANTA. MANTA outperforms both new baselines (see Table 1 and Figure 3). MANTA is a different axis to KV compression methods such as TurboQuant, which could bring additional savings or allow for better performance through less compression. KV cache eviction and KV selection could also be realized on higher-level memory tokens for combined additional savings.

Thank you for your response. I decide to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.