acceptodds
Under review as a conference paper at ICLR 2027

WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing

Abstract

When generating text, a Transformer produces representations of past tokens at every layer, but each layer can normally use only representations from the same depth. This restriction prevents the model from fully reusing information it has already computed. We introduce WhiteMatter, which allows every layer to draw on past-token representations from any depth. A learned mixer selects the most useful depths for the current context and combines their representations into shared key–value (KV) cache channels. Sharing these channels across layers can reduce the cache size. Given the same number of training tokens, WhiteMatter with a full-size cache performs comparably to a standard Transformer with 50% more layers. With half the KV cache, WhiteMatter outperforms matched standard Transformers at two model scales, up to 1.3B parameters. Cross-layer connections, however, introduce dependencies that slow training and prompt processing. We address this problem with cyclic iteration, which updates interleaved groups of tokens in turn while processing the tokens within each group in parallel. On a reference model trained with exact autoregressive execution, cyclic iteration converges 12.5x faster than standard Jacobi iteration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
deep-to-shallow feedback, cross-layer connection, KV-cache compression, natural language processing, transformer architecture

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LAaDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This is the summary of “WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing”. This study proposes a novel LLM architecture in which all layers share a KV cache pool and a routing mechanism. This study also provides a modified optimization method to train the proposed architecture. The authors claim that the proposed method has better PPL compared with a vanilla transformer.

Critical strengths

This paper is impressive and I have found many strength in this paper.

  1. This paper proposes a novel KV cache mechanism in which all layers share one KV pool. This flexible architecture is very impressive, and I believe this method is novel.

  2. The proposed method reduce over 30% of memory on the inference.

  3. The paper is well organized and well written. The Figures are very beautiful.

  4. The authors also provide a modified training method based on the Gauss–Seidel iteration method. This is a major breakthrough in this kind of research because complicated KV cache structures require huge computational cost. Over 10x acceleration is very important, and the insight from this training method can be a contribution to the community.

Critical weaknesses

My main concern is only one, but it is critical.

From Appendix E, the proposed method requires double the FLOPS for training compared with Vanilla Transformers. The main claim of the proposed method is better PPL compared with Vanilla Transformer; however, this comparison seems not to be fair with respect to FLOPS size. The authors should report PPL with FLOPS-matched experiments.

Minor

The main text has half-page room; I think the authors should move the experimental settings or some figures to the main text. For example, Figure 8 may be one.

Questions for authors

If authors provide FLOPS matched results and this is consistent with the authors' claim, I am willing to support the acceptance of this paper.

Minor

I could not read intuition of Figure 8 and introduction. This does not affect the understanding of the proposed architecture. However, better explanation and polish introduction may help reader.

Reviewer_rgKFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

WhiteMatter constructs shared KV cache channels by dynamically mixing hidden states from different layers, allowing each layer to reuse past-token information from multiple depths. It also introduces a cyclic Gauss–Seidel iteration schedule to improve the efficiency of training and prefill with these feedback connections.

Critical strengths
  1. Mixing hidden states from multiple layers into distinct shared KV channels is an intuitive and useful design. At approximately 1.3B parameters, WhiteMatter reduces held-out perplexity by 4.3% while using half the KV cache of the standard Transformer

  2. WhiteMatter improves prefill throughput over the evaluated deep-to-shallow feedback baselines, achieving 1.78× that of LCKV and 2.92× that of Feedback Transformer. This helps reduce a practical cost of feedback architectures.

Critical weaknesses
  1. Extending a single shared cross-layer summary to multiple content-dependent summaries is intuitive, and the experiments support its effectiveness. However, this mainly adds flexibility to existing cross-layer aggregation and KV-sharing ideas, rather than introducing a new principle for designing feedback models. I therefore view the work as incremental, though a very effective increment.

  2. Training efficiency remains a key challenge for deep-to-shallow feedback models, and I do not think this paper substantially resolves it. The cyclic iteration scheme is useful, but I see it primarily as a way to make the proposed multi-channel KV framework practical to train, rather than a major efficiency contribution. The training FLOPs reported in Appendix Table 4 suggest a modest improvement over comparable LCKV configurations, rather than a fundamental change in efficiency.

  3. The abstract’s emphasis on “12.5× faster” convergence over standard Jacobi iteration may give a misleading impression of the paper’s main contribution. Although the reference-model setting is specified, this is a limited solver comparison and does not demonstrate a corresponding training-speed improvement. I suggest placing less emphasis on this number and presenting better KV representations as the central contribution.

Questions for authors
  1. How does the proposed cyclic iteration differ from LRT’s interleaved training scheme, beyond performing multiple refinement sweeps and studying convergence? Please clarify the methodological contribution relative to this concurrent work.

  2. Could you report actual training throughput for WhiteMatter and LCKV under the same KV-cache budget and training settings? The key question is whether the FLOPs savings in Table 4 translate into faster training. Including other feedback models would also be helpful. If training-speed improvements are a central claim, this comparison should appear in the main text rather than only in the appendix.

2 comments

Thank you for your review!

Extending a single shared cross-layer summary to multiple content-dependent summaries is intuitive, and the experiments support its effectiveness. However, this mainly adds flexibility to existing cross-layer aggregation and KV-sharing ideas, rather than introducing a new principle for designing feedback models. I therefore view the work as incremental, though a very effective increment.

We agree that our architecture modifications are simple but proven to be effective and thank you for recognizing this too! We would like to highlight that these modifications addressed the key limitations that makes prior feedback architectures underperform compared to vanilla. For instance, LCKV performs worse than vanilla when more than 14 of the 22 vanilla layers are replaced with LCKV layers[1], whereas WhiteMatter becomes more expressive as more layers receive the added connections.

[1] Figure 8, Layer-Condensed KV Cache for Efficient Inference of Large Language Models

Training efficiency remains a key challenge for deep-to-shallow feedback models, and I do not think this paper substantially resolves it.

Please see General Response 1: WhiteMatter can improve performance over vanilla at matched training compute.

The training FLOPs reported in Appendix Table 4 suggest a modest improvement over comparable LCKV configurations, rather than a fundamental change in efficiency.

There are two reasons why the FLOPs savings in the main experiments are small.

First, faster convergence offered by an iteration schedule can manifest as either less compute at matched quality or higher quality at matched compute. The 12.5x speedup happens when we match quality across schedule, where Jacobi iteration needs 53 passes to reach the convergence criteria. This is not feasible in practice, so LCKV uses only 9 passes, which likely contributes to the performance gap between it and WhiteMatter.

Second, LCKV keeps vanilla warmup layers to maintain performance due to the limitations mentioned earlier. WhiteMatter addresses these limitations, so it can give all layers feedback connections to achieve higher performance, although it also remains compatible with warmup layers. If we match the number of feedback layers, the FLOPs difference would be larger.

The abstract’s emphasis on “12.5× faster” convergence over standard Jacobi iteration may give a misleading impression of the paper’s main contribution

Thank you for this feedback. We will make the distinction between convergence speed and training speed clearer.

How does the proposed cyclic iteration differ from LRT’s interleaved training scheme, beyond performing multiple refinement sweeps and studying convergence?

Besides the use of multiple cyclic passes, the two methods also differ in how initial feedback signals are obtained. LRT performs a full sequence forward pass to initialize the recurrent memory before a subsequent interleaved pass. We instead initialize the KV directly from token embeddings so that the first forward pass can be cyclic, which we empirically found to be more cost effective.

LRT was published in EMNLP in October 2026, which is considered contemporaneous according to the Reviewer Guide.

Could you report actual training throughput for WhiteMatter and LCKV under the same KV-cache budget and training settings?

Because of the difference in warmup layers and convergence quality, we do not claim an increase in training speed in the main experiment. However, we can see such an improvement when we control those.

We measured the training time per step of LCKV, WhiteMatter, and Feedback Transformer, excluding the LM head. For WhiteMatter, we use 4 warmup layers, similar to LCKV. We use cosine similarity between the gradient obtained from the iteration methods and the gradient from autoregressive execution as a proxy for convergence quality.

(Note: this measure is not ideal for cross model comparison because it is heavily influenced by architecture and weights. Comparing iteration methods on the same model is preferred.)

ArchitectureWarmup LayersIter. MethodPassesTimeGrad. Similarity
LCKV4Jacobi7+2777 ms0.374415
LCKV4Jacobi9+41,226 ms0.576702
WM4Cyclic2+1585 ms0.838872
FT0AutoregressiveN/A3.55 sN/A

Passes X+Y means X no gradient passes followed by Y gradient passes.

With matched 4 warmup layers, WhiteMatter with cyclic iteration has higher gradient similarity than LCKV while using 24.7% less time. If LCKV use more passes for a closer gradient similarity, the saving becomes 52.3%.

Thank you for the clarifications and the additional training-time measurements. The response is helpful, and my overall assessment remains unchanged. I maintain my rating of 3.

Reviewer_KWWrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce WhiteMatter, a way of reducing the KV-cache memory by constructing a small number of cache channels from dynamic mixtures of representations across layers. In particular, this allows the layers to use hidden states from deeper layers for past tokens, which increases expressivity but prevents parallelism. While previous works use Jacobi iteration to approximate the model, the authors here propose a Gauss-Seidel-style iteration to improve convergence and reduce the overall compute cost. Results show improved quality and lower KV-cache memory, while keeping decoding throughput comparable to a standard Transformer (equal decoding throughput, but about halved prefill throughput).

Critical strengths

1 The approach is clearly presented and seems natural and well motivated.

2 Clear improvements over previous works: the model runs faster and achieves better perplexity.

Critical weaknesses

1 The architectural choices are not sufficiently justified. In particular, assigning layer ℓ\ell to channel ℓmodk\ell mod k seems arbitrary. The paper explains why each layer should read only one channel, for efficiency, but not why the modulo assignment is appropriate. Comparing it with random or learned assignments would strengthen the method. The effect of strictly causal attention and the learned boundary entry is also not isolated; a vanilla baseline using the same setup would clarify whether they contribute to the gains. More broadly, the paper could distinguish more clearly between its two contributions: the cross-layer parameterization, which changes model expressivity, and cyclic iteration, which makes training and prefill practical.

2 The analysis feels somewhat rushed. Section 5 presents useful ablations, but the discussion is often limited to a few short sentences reporting the result. More interpretation of the learned routing, channel specialization, and sensitivity to design choices would help explain why WhiteMatter works, rather than only showing that it works.

3 The headline prefill speedup may overstate the practical gain. The roughly 16x result is measured against exact autoregressive feedback prefill on a small controlled model, not against standard Transformer prefill. In the main-scale experiment, WhiteMatter prefill remains substantially slower than vanilla. This distinction should be made more prominent, and the convergence benchmark should ideally be repeated on the main model.

Questions for authors

1 How does the prefill trade-off translate to practical deployments? In which settings do the KV-memory savings outweigh the slower prefill compared with a vanilla Transformer?

2 Can existing pretrained Transformers be converted to WhiteMatter through continued training, or does the architecture require pretraining from scratch?

3 Did you experiment with alternative layer-to-channel assignments, such as contiguous, random, or learned assignments? If so, how did they compare with the modulo assignment?

1 comment

Thank you for your review!

Did you experiment with alternative layer-to-channel assignments, such as contiguous, random, or learned assignments?

Modulo assignment is a simple choice that balances channel usage and provides diversity across consecutive layers, but it is not required. Learned soft assignment gives the best performance, but requires reading all channels at each layer, which is very expensive during decoding.

We trained models with different assignment methods and report perplexity below:

Assignmentk=2k=4k=8Average
Modulo51.66951.44050.77551.295
Contiguous51.99951.37550.85151.408
Balanced random51.78851.66651.47651.643
Learned soft51.64350.94650.77051.120

The effect of strictly causal attention and the learned boundary entry is also not isolated

More broadly, the paper could distinguish more clearly between its two contributions

Changing the vanilla model to use strictly causal attention led to a small 0.4% reduction in perplexity, while the dummy token had little impact. In comparison, WhiteMatter's perplexity was 12.07% lower. The dummy token is added for compatibility with the FlashAttention kernel and is not required by our method.

We also isolated the effects of the architectural changes and the iteration method. A single-channel model (k=1) with static mixing weights, trained with Jacobi iteration, performed worse than vanilla. With cyclic iteration, its perplexity became 5.31% lower than vanilla's. Adding the architectural changes further reduced perplexity to 12.07% below vanilla.

ModelStrictly CausalDummyIteration MethodPPL
VanillaNoNoNA40.268
VanillaYesYesNA40.108
VanillaYesNoNA40.102
WhiteMatter k1 staticYesYesJacobi (7+2)42.413
WhiteMatter k1 staticYesYesCyclic (1+2)38.128
WhiteMatter k16YesYesCyclic (1+2)35.408

More interpretation of the learned routing, channel specialization, and sensitivity to design choices would help explain why WhiteMatter works

We analyzed the learned mixing patterns in the 16-layer, 16-channel model, and found the following:

  1. The model uses both shallower and deeper states. On average, 44.2% of the weight goes to shallower states, 7.5% to the reader’s input, and 48.3% to deeper states.
  2. Routing shows a reversed depth preference. For keys, layers 1–2 assign an average of 45.8% of their weight to outputs from layers 14–16, while layers 13–16 assign 47.8% to outputs from layers 3–4. For values, layers 14–16 assign 34.4% to embeddings. Middle layers (5–12) distribute their weights more broadly but prefer layer 10.
  3. Signed mixing supports computing the difference between consecutive-layer states. For example, key channels 13 and 14 assigned opposing signs to the outputs of layers 3 and 4 in 99% of observations.

We will add these findings and a routing-weight heatmap to the manuscript.

the convergence benchmark should ideally be repeated on the main model.

We performed the convergence benchmark on a small model trained with AR, instead of the main model, mainly to avoid advantaging our own method. A model can adapt to the iteration method it is trained with and require fewer passes during evaluation. On the main WhiteMatter model, cyclic iteration converged 15.1x faster than AR and 34x faster than Jacobi iteration.

In which settings do the KV-memory savings outweigh the slower prefill compared with a vanilla Transformer?

The reduction in KV-cache size when k<Lk<L mainly saves GPU VRAM, which is not directly comparable to the increase in prefill latency. This can reduce bandwidth requirements when reading/writing the KV cache from disk, support larger batch sizes, or enable inference on GPUs with less memory. The reduction in KV projections can also reduce compute, but this is generally not sufficient to outweigh the compute added by multiple iterations.

Can existing pretrained Transformers be converted to WhiteMatter through continued training?

Yes. All weights from a pretrained vanilla Transformer can be transferred into a WhiteMatter model with a full cache for continued training. Models whose attention includes the current token need to adapt to WhiteMatter’s strict causal mask.

We converted the vanilla model trained on 100B tokens in General Response 1 into a WhiteMatter model and continued training on 10B additional tokens. It recovered the original perplexity after 2.65B tokens and finished with a perplexity 2.9% below the original model’s.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.