- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Foldable Maps, an efficient attention architecture that combines shared latent compression, expert-specific representations, and sparse KV-cache access. The motivation is that conventional Multi-Head Latent Attention (MLA) reduces KV-cache storage but limits expert-level sparsity, since multiple attention heads share the same latent representation. The proposed method introduces a shared latent and a private latent for each expert. Each expert selects 6 out of 24 learnable linear maps to transform its private latent into a higher-dimensional workspace. The resulting representation is combined with the shared latent to construct attention keys and values.
The key mechanism is that these conditional linear transformations can be absorbed into the query projections during inference. The authors additionally introduce an attention-concentration auxiliary loss and a lightweight Glance indexer to support sparse token retrieval.
Critical strengths
The combination of shared/private latents and conditional Foldable Maps addresses an important limitation of MLA under expert-level sparse attention. The exact algebraic absorption enables richer representations during training without requiring the high-dimensional workspace during inference.
Empirically, the Twin versus Foldable Maps comparison provides convincing evidence for the effectiveness of conditional mappings. The substantial NIAH improvements and reduced cache read traffic with Glance demonstrate promising practical benefits for conducting long-context inference.
Critical weaknesses
Although Foldable Maps substantially improves NIAH accuracy, its passkey accuracy is significantly lower than Twin and the uncompressed sparse baseline. Its perplexity also remains worse than the uncompressed model. These results suggest a potentially important trade-off between semantic retrieval, exact copying, and general language modeling quality.
Questions for authors
Why does Foldable Maps substantially improve NIAH retrieval while degrading passkey accuracy relative to Twin? Can the authors explain whether conditional mappings introduce a trade-off between semantic retrieval and exact information preservation?
2 comments
Thank you for the clarification and the additional MLA baseline experiment. I agree that the observed trade-off between improved NIAH retrieval and reduced exact-copying accuracy does not necessarily destroy the practical value of Foldable Maps. Also, from my perspective, the additional comparison in a wider MLA latent also provides more evidence.
Overall, I would say the paper presents a meaningful architecture with non-trivial empirical improvements, and I would keep my rating and lean towards this paper.
Thanks for the review, the passkey results do suggest exact copy capabilities are somewhat worse than the twin model. The most plausible explanation is that the increased NLP expressiveness of the maps (demonstrated in the NIAH and perplexity tests) comes at the cost of less stable exact copy capabilities. So as you put it, a tradeoff between semantic retrieval and exact information preservation. For most model purposes, this would be a highly desirable trade. The uncompressed model should be expected to perform better across the board, so the NIAH differential is a surprisingly positive result.
We trained one more MLA model with a 256 width latent. Even with a better than perfect indexer (64 width indexer to match 4:1 ratio of DSA with token level topk for timing, allowed per head granularity instead of per token with perfect selection for quality testing) and over double the latent width, it still performed worse than the glance on all tests besides passkey. The passkey is the least relevant of the four tests for natural language modeling (and the hybrid outperformed MLA at matched latent width), providing strong evidence of superiority to the current baseline.