acceptodds
Under review as a conference paper at ICLR 2027

Foldable Maps Enable Expressive and Efficient Attention

Abstract

Long-context inference is bound by the KV cache. Linear attention on its own struggles with precise recall while dense attention has higher precision but becomes extremely expensive for long sequences. Shared latent compression (MLA) improves decode bandwidth by reading once across all heads, but that leaves no benefit for sparse head access. In this paper we introduce Foldable Maps, a conditional linear map system which pairs a shared latent with per-expert latents that utilize a foldable workspace. The expert-level latent expands using 6 of 24 map projections per-expert per-token, which simplifies to one once selected. This enables the training-time workspace to fold with absorption, making the per-expert cost independent of workspace and head width for inference. At 815M parameters with token-matched arms and a 4.57x KV compression ratio over its parent model, this format matches or exceeds both uncompressed baselines on NIAH retrieval averaged over 4k-100k contexts (+20% over parent) and more than doubles the score of the next best measured compression format at equal latent width. The perplexity is the best of the measured compressed formats, but is still  5% worse than the uncompressed parent at this compression ratio. We also train a 24-byte per-token per-layer indexer which maintains the full NIAH retrieval edge (0.423 vs 0.426) with  12x less read traffic compared to the parent model at 100k context length. Partitioning by consumer makes expert count a free axis for attention capacity, aligning the training and serving benefits of attention-side MoE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
architecture, attention, kv compression

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_N2RVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Foldable Maps, an efficient attention architecture that combines shared latent compression, expert-specific representations, and sparse KV-cache access. The motivation is that conventional Multi-Head Latent Attention (MLA) reduces KV-cache storage but limits expert-level sparsity, since multiple attention heads share the same latent representation. The proposed method introduces a shared latent and a private latent for each expert. Each expert selects 6 out of 24 learnable linear maps to transform its private latent into a higher-dimensional workspace. The resulting representation is combined with the shared latent to construct attention keys and values.

The key mechanism is that these conditional linear transformations can be absorbed into the query projections during inference. The authors additionally introduce an attention-concentration auxiliary loss and a lightweight Glance indexer to support sparse token retrieval.

Critical strengths

The combination of shared/private latents and conditional Foldable Maps addresses an important limitation of MLA under expert-level sparse attention. The exact algebraic absorption enables richer representations during training without requiring the high-dimensional workspace during inference.

Empirically, the Twin versus Foldable Maps comparison provides convincing evidence for the effectiveness of conditional mappings. The substantial NIAH improvements and reduced cache read traffic with Glance demonstrate promising practical benefits for conducting long-context inference.

Critical weaknesses

Although Foldable Maps substantially improves NIAH accuracy, its passkey accuracy is significantly lower than Twin and the uncompressed sparse baseline. Its perplexity also remains worse than the uncompressed model. These results suggest a potentially important trade-off between semantic retrieval, exact copying, and general language modeling quality.

Questions for authors

Why does Foldable Maps substantially improve NIAH retrieval while degrading passkey accuracy relative to Twin? Can the authors explain whether conditional mappings introduce a trade-off between semantic retrieval and exact information preservation?

2 comments

Thanks for the review, the passkey results do suggest exact copy capabilities are somewhat worse than the twin model. The most plausible explanation is that the increased NLP expressiveness of the maps (demonstrated in the NIAH and perplexity tests) comes at the cost of less stable exact copy capabilities. So as you put it, a tradeoff between semantic retrieval and exact information preservation. For most model purposes, this would be a highly desirable trade. The uncompressed model should be expected to perform better across the board, so the NIAH differential is a surprisingly positive result.

We trained one more MLA model with a 256 width latent. Even with a better than perfect indexer (64 width indexer to match 4:1 ratio of DSA with token level topk for timing, allowed per head granularity instead of per token with perfect selection for quality testing) and over double the latent width, it still performed worse than the glance on all tests besides passkey. The passkey is the least relevant of the four tests for natural language modeling (and the hybrid outperformed MLA at matched latent width), providing strong evidence of superiority to the current baseline.

Thank you for the clarification and the additional MLA baseline experiment. I agree that the observed trade-off between improved NIAH retrieval and reduced exact-copying accuracy does not necessarily destroy the practical value of Foldable Maps. Also, from my perspective, the additional comparison in a wider MLA latent also provides more evidence.

Overall, I would say the paper presents a meaningful architecture with non-trivial empirical improvements, and I would keep my rating and lean towards accepting\textbf{accepting} this paper.

Reviewer_ab75OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Foldable Maps, a conditional KV compression mechanism that combines shared and expert-specific latents with dynamically selected linear projections. By exploiting weight absorption, the expanded training-time representations can be folded into efficient inference-time operations without materializing the intermediate workspace. The method further incorporates a lightweight per-head indexer to reduce KV cache accesses during long-context decoding. Experiments on 815M-parameter models demonstrate improved long-context retrieval over several compressed attention baselines, achieving a 4.57× KV compression ratio while maintaining competitive perplexity.

Critical strengths
  1. Interesting architectural design. The proposed Foldable Maps effectively combines conditional representations with weight absorption, improving the expressiveness of compressed KV representations without introducing substantial inference-time computation.

  2. Strong long-context retrieval performance. The method achieves notable improvements on RULER NIAH benchmarks, outperforming both compressed attention variants and uncompressed baselines across the evaluated context lengths.

  3. Comprehensive architectural comparisons. The paper evaluates multiple variants, including Dense MLA, Routed MLA, Twin, and Expert-only Maps, providing useful evidence for the effectiveness of the proposed design.

Critical weaknesses
  1. The motivation remains relatively generic, and the necessity of Foldable Maps is not well established. What specific representational bottlenecks of existing KV compression methods motivate the proposed conditional mapping mechanism?

  2. The proposed method reduces KV cache accesses but does not consistently improve decoding throughput (116 vs. 247 tokens/s at batch size 1). Can the authors demonstrate actual end-to-end speedups and justify the additional routing complexity?

  3. Insufficient justification of expressiveness. Figure 2 illustrates combinatorial diversity, but this does not necessarily imply increased effective representational capacity. Can the authors provide theoretical analysis or controlled comparisons against parameter-matched conditional projection baselines to substantiate the claimed advantages?

Questions for authors

Please see weakness above.

3 comments

We appreciate the review, let us address the weaknesses you raised:

1.There are two very large bottlenecks the method tackles. First, MoE in attention does not save any time at inference if the whole layer shares a single latent. Since the initial draft, we trained an MLA model with 256 total width compared to 112 with the hybrid, and it was still outperformed in quality. This demonstrates how reorganizing the latent structure can lead to both speed and quality improvements. Second, at 4/16 sparsity MLA has 4x the attention mixing flops of the sparse models. In long contexts like 100k, attention mixing becomes dominant for both total flops and wall clock. So while MLA is effectively free at inference, training flops are considerably reduced with the introduction of sparsity. Those two factors are large enough to encourage development on their own; the hybrid outperforming latent width matched MLA on all measured tasks is a surprising secondary benefit.

2.Batch 1 is slower due to additional overhead and small per token latent sizes, though it has been improved substantially with the new kernels. However, for a model of this size a batch size of 1 would almost never be deployed, making the performance with larger batches more relevant. With a larger model, launch overhead and tensor sizes would be proportionally larger, decreasing the batch 1 disparity.

3.The best visualization for the expressiveness increase is figure 1. Without Foldable Maps, the private latent could only learn a single line to project with (16d subspace rather than line in our implementation) instead of over a hundred thousand subspace combinations. But this paper is past the point of theorized expressiveness, it is directly demonstrated with the twin vs hybrid comparison for the maps and hybrid vs mla for compression in general at matched latent width. In both cases, the hybrid performed better by a substantial margin on perplexity, NIAH single, and NIAH multi. Those are the tasks in our suite most relevant to natural language modeling, so the hybrid performing best across lengths with identical training conditions demonstrates improved NLP expressiveness.

I thank the authors for their response and the effort they have put into addressing my comments. However, their response is limited to textual explanations and does not adequately address the complexity analysis, theoretical analysis, and experimental comparisons I requested in the Weaknesses section. As a result, my main concerns remain unresolved. Furthermore, the paper lacks a related work section and therefore does not clearly distinguish its contributions from existing methods. Overall, I do not believe the paper meets the standard for acceptance at ICLR, and I will maintain my score.

Most of the introduction discusses related work and the discussion section connects our results to the most relevant baselines. Taken in the context of the paper those sections highlight how our method is both original and distinct while fitting into the body of related research . Although the twin model does not match stored parameters, the collapsing nature of the linear maps makes the effective active parameters identical between the twin and hybrid. Thus, the twin is the active-parameter-matched, unconditional counterpart to the hybrid, isolating the effect of conditional selection. We added a few lines at the end of section 2.2 to directly address your concern about representational capacity:

"The end-of-training cosine similarity between maps was 0.83 after folding. While a set of 24 identical maps produces a model equal to the twin, a similarity below 1.0 alongside the measured diverse map selections guarantees no twin can express the hybrid model over every token while the hybrid could still express any twin. This demonstrates a real increase in expressivity."

The newest set of results are especially strong, with the hybrid outperforming MLA with over double the latent width in every test besides passkey. This goes against the expectation that the latent with higher compression would perform worse. With the concrete measurements guaranteeing that no twin can represent the hybrid maps under conditional selection, the large quality improvements that have already been directly measured should be easier to attribute.

Reviewer_8i7dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines routed KV heads (attention MoE, 4 of 16 heads per token) with MLA-style latent compression. With MLA's single shared latent, head-level sparsity (routing or per-head token selection) doesn't save decode memory reads, since the whole latent is read anyway. The paper instead uses a small shared latent plus a small private latent per active head. To make the private latent expressive, it is expanded during training by a token-dependent sum of 6 of 24 learned maps ("foldable maps"), which are absorbed into the query at inference so the cost stays small. A per-head indexer ("glance") then selects 2048 tokens per head. At 815M params, it gets better NIAH than the uncompressed parent and GQA, with ~5% worse perplexity and ~12x fewer cache reads at 100K.

Critical strengths
  • Clear observation: MLA's shared latent doesn't benefit from head-level sparsity. Per-head private latents make routing / per-head selection save memory reads at decode.
  • The folding trick is neat: token-dependent maps during training that collapse to a small cost at inference.
  • Good controls (e.g. the "twin" arm isolates the maps), and measured decode kernels at 100K context.
Critical weaknesses
  • Missing key baseline: MLA already has the smallest KV cache (Table 7), so the claimed benefit relies on per-head token selection being better than per-token selection. The natural baseline is MLA + a per-token indexer as in DeepSeek-V3.2 (DSA), where all heads share a top-k token set. This is not compared. The MLA baseline used is also weak (112-dim latent for 16 heads, 0.18 NIAH).
  • Small scale and noisy evaluation: 815M params, one run per arm, low NIAH accuracies for all models, only 20 docs per length for MK-NIAH, and inconsistent rankings across tasks (e.g. on passkey the hybrid is much worse than twin and the parent). The 4.57x compression is only on the routed heads; the whole-model KV cache is ~1.7x smaller than the parent.
  • Many components at once (NoPE routed heads, SSMax, anchors, concentration aux, indexer), some only in some arms, so the gains are hard to attribute (e.g. the pruning result in Table 1 compares the hybrid with aux to the parent without it).
Questions for authors
  • How does glance compare to MLA + a DSA-style per-token indexer at the same read budget, and to MLA with a larger latent?
  • Do the gains hold at larger scale and on full RULER / LongBench?
1 comment

Thanks for the review, here are quick answers to your concerns before more details:

1.This is now trained with results in the table below.

2.The listing of 20 documents was an error, we actually tested 100 per length and have now added 95% confidence-interval half-widths to each table. The KV cache is over 4.5x smaller at 100k context length, the table numbers you are referring to were mostly due to inefficient kernels which have now been fixed. Including metadata, that figure is closer to 4x for some models.

3.The concentration aux was a legitimate confound, so we trained an otherwise identical sparse model with it on to more accurately gauge its effects. To completely remove the confound an aux and non aux version would need to be run on every model, but the current testing is enough to have confidence in the results. At least for perplexity the models with aux would only be underselling their results and all our raw perplexity claims are in the opposite direction. The other components you listed are shared by all non GQA models and the indexer is post training so none would affect any comparisons. We crafted the arms to isolate one mechanism apiece, but if you see any more confounding factors we would like to know so they can be addressed.

And the questions:

1.The glance hybrid model was better in both speed beyond small batches and quality compared to the 256 width MLA model with a size 64 oracle indexer. This maintains the 4:1 ratio of 512:128 in the original DSA paper. It uses best possible per head indexing for quality testing but topk token level for speed testing, essentially better than perfect indexing. We used that as the comparison instead of training a DSA style indexer as it removes any doubt the indexing quality or pipeline is to blame for the MLA performance.

2.We have not tested at larger scale and most non-NIAH RULER and LongBench tasks are near chance with these models. This would have to be addressed with a future suite of models.

We equalized latents between models to gauge performance at fixed budget, but you’re right that if MLA suffers more from the aggressive compression than the other methods this would not be a fair comparison. We have now trained an MLA model under the same conditions (token matching, otherwise equal architecture) with a size 256 latent. Here are the headline results:

| Arm | 1 | 8 | 32 | 64 | 128 | 192 | 256 |

| glance | 684 | 4652 | 9876 | 12428 | 14013 | 15029 | – |

| hybrid_maps | 782 | 4095 | 7022 | 7945 | 8419 | 8579 | 8708 |

| dense_mla | 1547 | 6545 | 10313 | 11375 | 11711 | 11612 | 12406 |

| big_mla | 1337 | 4148 | 5389 | 5668 | 5852 | – | – |

| oracle_mla | 1186 | 5335 | 10009 | 11785 | – | – | – |

Whole-model decode throughput (tokens/s) by batch size at T=102400 on a single GH200; – = exceeds the 92.7 GB memory budget.

| Model | 4K | 8K | 16K | 32K | 64K | 100K |

| big_mla | 30.29 | 27.25 | 27.25 | 25.47 | 26.85 | 27.97 |

| oracle_mla | 30.30 | 27.26 | 27.32 | 25.61 | 27.11 | 28.35 |

| hybrid_maps | 30.13 | 27.07 | 27.07 | 25.21 | 26.57 | 27.70 |

| glance | 30.13 | 27.07 | 27.10 | 25.29 | 26.79 | 27.98 |

Perplexity at various context-lengths by model. Big MLA has 256 latent width compared to 112 for hybrid

Kernel speed roughly tripled compared to the initial draft, for reference the best performing model before (glance) was roughly 5600 tokens per second. The new kernels for MLA, GQA, and the sparse model are near the roofline, so the custom kernels exceeding their throughput can be taken as true superior inference performance.

Altogether, this has better quality than a perfect indexer at a principled speed cost. But even with all these advantages, the falloff for MLA is still greater than with the glance due to differing attention patterns aided by concentration aux. The glance model is faster beyond small batches and performs better in all tasks besides passkey against the MLA model with a better than perfect indexer and over double the latent width.

Reviewer_mKvwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Foldable Maps, a conditional linear-map system for KV-latent compression in a hybrid linear/sparse attention model. Each expert-level latent (size 16) expands through 6-of-24 learned projections into a size-128 workspace at training time; at decode, MLA-style absorption (Eqs. 1–10) folds the workspace so per-query cost depends only on latent widths, not workspace size. The design goal is to make attention-side MoE routing actually reduce memory traffic.

Critical strengths
  • The Foldable Maps idea is interesting, and NIAH performance compared to the twin more than doubled on the average measured context, even exceeding the GQA and uncompressed sparse models.
  • The twin ablation is carefully designed, with identical latent widths, single controlled difference.
Critical weaknesses

The paper has three structural weaknesses: (1) the headline 4.57× compression ratio is inconsistent with the paper's own measured KV memory (Table 7 implies 1.69× over the sparse parent); (2) all results are single-run at 815M params / 10B tokens with no error bars, while several key claims (e.g., glance 0.423 vs hybrid 0.426) rest on differences that are plausibly within seed noise; (3) the strongest baseline (dense MLA) performs far below expectations in ways the authors attribute to domain transfer, so "2× the next best compressed format" is established against a possibly-crippled baseline.

Questions for authors
  • Table 3 GQA 4k = 0.88 dropping to 0.17 at 100k looks like the expected small-model cliff. Why does GQA degrade so steeply?
  • Table 6: hybrid 0.66 average vs twin 0.97 — and the hybrid is better at 64k–100k than at 4k, which is the opposite of every other arm's degradation pattern. Could you provide some explainations?
2 comments

Thanks for the review, we’ll go through your points and questions one by one:

1.The 4.57x compression is the correct number for KV compression, Table 7 was based on inefficient kernels which have now been corrected. Including metadata, some models are closer to a 4x compression.

2.Single run is a necessity with this number of models, but for the testing we have added 95% confidence-interval half-widths where applicable and corrected the number of documents used (previously listed 20, actually tested 100). For the combined NIAH + MK-NIAH, the glance and hybrid models are well beyond variance above the other models in performance. While the glance and hybrid themselves are within each other’s confidence intervals, that is in line with our claims because we did not assert that one or the other was superior.

  1. Although not totally airtight, we were able to isolate the greater train/test differential for GQA and MLA to the every token nature of those mechanisms. The slowest moving group in the linear layers has a decay of 0.99985. This corresponds to a half life of 4621 tokens which is small in comparison to the 100k context length. This does not compromise any results because it is identical in the models that use it, but it does give MLA and GQA an advantage since they are the only models that view every token with every query and is thus best suited to make use of long range gist. Between these two datasets, retrieval performance endured better than long range gist which resulted in greater performance dropoffs for MLA and GQA than other models since they took advantage of that role the most.

For your questions:

1.We expanded RoPE theta to accommodate long lengths, the falloff for GQA is natural and expected due to the small model cliff you noted. The NoPE heads being position blind can be a benefit at long distances even if it makes short ranges more difficult, as there is no rotation to change the character of keys. This effect is most visible in the passkey test where GQA is among the worst models at 100k.

  1. We have now isolated this to two factors. First is that the hybrid model picked the correct key as the second choice in most of its misses while other models did not. We did not give partial credit in any tests, so it would be unfair to rerun in this case. Second, passkey uses repeated filler text, and with the sparse models that can lead to very uneven key distributions. At the same context length, a passkey could be placed in a row that is almost empty or one with thousands of entries, and which is dependent on the individual model’s routing. These sources of variance are not an attempt to say the hybrid is equal or above the twin at this task, but hopefully it makes the unusual curve compared to the others more sensible. In all likelihood, most of the gap is real, suggesting the increased NLP expressiveness (demonstrated in the NIAH and perplexity tests) compared to the twin comes at the cost of reduced exact copy stability. Also, it is worth noting that although the hybrid trails the twin, it does far better than the standard compression technique (MLA) at the same latent width.

I thank the authors for their thorough rebuttal and clarifications. I welcome the corrected kernel measurements for Table 7. Even so, my primary reservations remain. The explanation for degraded dense MLA performance via token-level decay and long-range gist loss is plausible, yet comparisons against this impaired baseline still weaken the core “2x next best compressed format” claim. Therefore, I maintain my original score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview of changes

We addressed the points everyone raised individually, but for convenience we will outline the major changes here.

  1. An otherwise identical sparse model was trained with concentration aux to directly measure the training costs compared to pruning benefit (Table 1).

  2. More efficient kernels were written for most models, doubling or tripling performance in most cases (Table 8). KV cache per model was corrected to use the actual values rather than those from the earlier inefficient kernels. True KV compression is 4.57x, but table figures include metadata which brings that closer to 4.0x for some models.

  3. An MLA model with a 256 width latent was trained to ensure we weren’t comparing against a crippled baseline. The hybrid still outperforms it in all tasks besides passkey while the inference speed is reduced by over half compared to the MLA with a 112 width latent.

  4. We compare a better than perfect indexer for the MLA model (assumed standard 4:1 latent:indexer ratio and token level topk for speed assessment, but graded quality on head level granularity with perfect selections). Even with this beyond perfect indexer and more than double the latent space, the implemented glance still outperforms in raw perplexity, perplexity lost to pruning, and wall clock inference speed beyond small batches.

  5. Error bars were added to relevant tests, number of samples corrected to 100 per context length from 20.

Tables highlighting new results:

| Arm | 1 | 8 | 32 | 64 | 128 | 192 | 256 |

| glance | 684 | 4652 | 9876 | 12428 | 14013 | 15029 | – |

| hybrid_maps | 782 | 4095 | 7022 | 7945 | 8419 | 8579 | 8708 |

| dense_mla | 1547 | 6545 | 10313 | 11375 | 11711 | 11612 | 12406 |

| big_mla | 1337 | 4148 | 5389 | 5668 | 5852 | – | – |

| oracle_mla | 1186 | 5335 | 10009 | 11785 | – | – | – |

Whole-model decode throughput (tokens/s) by batch size at T=102400 on a single GH200; – = exceeds the 92.7 GB memory budget.

| Model | 4K | 8K | 16K | 32K | 64K | 100K |

| big_mla | 30.29 | 27.25 | 27.25 | 25.47 | 26.85 | 27.97 |

| oracle_mla | 30.30 | 27.26 | 27.32 | 25.61 | 27.11 | 28.35 |

| hybrid_maps | 30.13 | 27.07 | 27.07 | 25.21 | 26.57 | 27.70 |

| glance | 30.13 | 27.07 | 27.10 | 25.29 | 26.79 | 27.98 |

Perplexity at various context-lengths by model. Big MLA has 256 latent width compared to 112 for hybrid

We made some final edits to the paper text focused in the abstract and general grammar. The roughly tripled custom kernel speeds and favorable comparison to the MLA model with over double the latent width make our overall results much stronger, so we updated the abstract to reflect that.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.