- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MANTA represents sequence history using recursively aggregated memory tokens and a causal attention mask with bounded windows at each level, yielding logarithmic KV-cache growth. Experiments train 1B language models from scratch and explore hybrid attention, autoencoding, and byte-level variants. Results demonstrate substantial cache reductions and competitive standard language modeling, but considerable retrieval losses for the logarithmic-cache configurations.
Critical strengths
-
The paper presents a coherent architecture that unifies hierarchical aggregation and retrieval through standard attention and a timestamp-based mask. The resulting cache bound is clear, and the design offers a concrete mechanism for retaining representations of distant history with limited memory.
-
The 1B-scale, 200B-token experiments provide substantive evidence of feasibility. The aggregation, window, positional-encoding, and hybrid studies explore meaningful design choices. In particular, the unbounded-top and hybrid variants recover substantial retrieval quality while retaining considerable cache savings.
Critical weaknesses
-
The main comparisons use dense attention and a 16-token window, while representative MANTA configurations retain around 170 KV states per layer. Without memory-matched windows or learned-compression baselines, it is unclear how much recursive aggregation improves the quality–memory–compute trade-off.
-
Long-context performance remains a concern. At 8K, logarithmic-cache variants score 0.176 – 0.180 on RULER versus 0.585 for dense attention. The strongest recovery comes from variants with linear cache growth. Evaluation beyond 8K and clearer throughput measurements would help establish where the logarithmic design is practically useful.
Questions for authors
-
Additional evidence: How does MANTA compare with a memory-matched window, single-level learned memory, and an established compression method, accounting for the extra training compute from memory tokens?
-
Additional evidence: Could you report quality, cache size, and throughput beyond 8K for the bounded, unbounded-top, and hybrid variants?
-
Clarification: What hardware, precision, kernels, and timing procedure were used for Figure 2? How are memory-token computation, cache maintenance, and prefill accounted for?
-
Clarification: Does MANTA update the decoder through LoRA while 500x keeps it frozen? Are adaptation budgets comparable? Please also qualify “nearly lossless” using Table 7’s exact reconstruction rates across sequence lengths.
2 comments
Response to the authors
I thank the authors for their detailed responses to my review and the additional experiments. After considering these clarifications and results, I have decided to raise my rating.
We thank the reviewer for the assessment of our work. We particularly appreciate the concrete suggestions regarding memory-matched baselines.
W1/Q2: To compare the performance of MANTA with that of other methods that reduce KV cache requirements, we added sliding-window attention, Activation Beacons, and H2O (Heavy-Hitter Oracle) as new baselines. Under the same KV cache budget, MANTA outperforms these baselines (see Table 1 and Figure 3).
W2/Q1: Figure 11 now compares the KV-cache sizes of the bounded, unbounded-top, and hybrid variants across sequence lengths for Llama 3.2 1B. In addition, we have expanded the analysis beyond 8K by adding evaluations at context lengths of 16K and 32K, reported in Figure 12. Together, these results provide a clearer picture of the quality–memory trade-off.
Q3: The throughput measurements in Figure 2 were obtained using one H100 GPU, BF16 precision, and FlexAttention with the Triton backend and compiled attention. We timed 32 consecutive decode steps with CUDA-synchronized wall-clock timing at the respective sequence length and averaged over five repetitions. Throughput counts only original (non-memory) tokens. Memory-token computations and cache maintenance are included, whereas prefill is excluded. We've added this description to the study accordingly. MANTA’s KV cache is organized into level-specific segments, each acting as a circular buffer that retains the most recent window states at the respective hierarchy level. Thus, the cache size is fixed per level and grows only with the number of hierarchy levels. We have added these implementation and measurement details to the revised manuscript.
Q4: The 500x compressor keeps the decoder frozen. The MANTA autoencoder does not contain a separate encoder and decoder but is built upon one pretrained language model. The MANTA autoencoder and 500x compressor were trained with the same number of LoRA parameters and the same token budget. Table 7 shows that, with 31 visible memory tokens, exact sequence reconstruction is 97%, 77%, and 11% at lengths 128, 256, and 512, respectively, and is lower with smaller memory budgets. Although exact sequence reconstruction decreases with sequence length, BLEU scores remain high, as reported in Figure 6. We have quantified what "nearly lossless" means in the study.