- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This is the summary of “WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing”. This study proposes a novel LLM architecture in which all layers share a KV cache pool and a routing mechanism. This study also provides a modified optimization method to train the proposed architecture. The authors claim that the proposed method has better PPL compared with a vanilla transformer.
Critical strengths
This paper is impressive and I have found many strength in this paper.
-
This paper proposes a novel KV cache mechanism in which all layers share one KV pool. This flexible architecture is very impressive, and I believe this method is novel.
-
The proposed method reduce over 30% of memory on the inference.
-
The paper is well organized and well written. The Figures are very beautiful.
-
The authors also provide a modified training method based on the Gauss–Seidel iteration method. This is a major breakthrough in this kind of research because complicated KV cache structures require huge computational cost. Over 10x acceleration is very important, and the insight from this training method can be a contribution to the community.
Critical weaknesses
My main concern is only one, but it is critical.
From Appendix E, the proposed method requires double the FLOPS for training compared with Vanilla Transformers. The main claim of the proposed method is better PPL compared with Vanilla Transformer; however, this comparison seems not to be fair with respect to FLOPS size. The authors should report PPL with FLOPS-matched experiments.
Minor
The main text has half-page room; I think the authors should move the experimental settings or some figures to the main text. For example, Figure 8 may be one.
Questions for authors
If authors provide FLOPS matched results and this is consistent with the authors' claim, I am willing to support the acceptance of this paper.
Minor
I could not read intuition of Figure 8 and introduction. This does not affect the understanding of the proposed architecture. However, better explanation and polish introduction may help reader.
Thank you for your review!
We agree that our architecture modifications are simple but proven to be effective and thank you for recognizing this too! We would like to highlight that these modifications addressed the key limitations that makes prior feedback architectures underperform compared to vanilla. For instance, LCKV performs worse than vanilla when more than 14 of the 22 vanilla layers are replaced with LCKV layers[1], whereas WhiteMatter becomes more expressive as more layers receive the added connections.
[1] Figure 8, Layer-Condensed KV Cache for Efficient Inference of Large Language Models
Please see General Response 1: WhiteMatter can improve performance over vanilla at matched training compute.
There are two reasons why the FLOPs savings in the main experiments are small.
First, faster convergence offered by an iteration schedule can manifest as either less compute at matched quality or higher quality at matched compute. The 12.5x speedup happens when we match quality across schedule, where Jacobi iteration needs 53 passes to reach the convergence criteria. This is not feasible in practice, so LCKV uses only 9 passes, which likely contributes to the performance gap between it and WhiteMatter.
Second, LCKV keeps vanilla warmup layers to maintain performance due to the limitations mentioned earlier. WhiteMatter addresses these limitations, so it can give all layers feedback connections to achieve higher performance, although it also remains compatible with warmup layers. If we match the number of feedback layers, the FLOPs difference would be larger.
Thank you for this feedback. We will make the distinction between convergence speed and training speed clearer.
Besides the use of multiple cyclic passes, the two methods also differ in how initial feedback signals are obtained. LRT performs a full sequence forward pass to initialize the recurrent memory before a subsequent interleaved pass. We instead initialize the KV directly from token embeddings so that the first forward pass can be cyclic, which we empirically found to be more cost effective.
LRT was published in EMNLP in October 2026, which is considered contemporaneous according to the Reviewer Guide.
Because of the difference in warmup layers and convergence quality, we do not claim an increase in training speed in the main experiment. However, we can see such an improvement when we control those.
We measured the training time per step of LCKV, WhiteMatter, and Feedback Transformer, excluding the LM head. For WhiteMatter, we use 4 warmup layers, similar to LCKV. We use cosine similarity between the gradient obtained from the iteration methods and the gradient from autoregressive execution as a proxy for convergence quality.
(Note: this measure is not ideal for cross model comparison because it is heavily influenced by architecture and weights. Comparing iteration methods on the same model is preferred.)
Passes X+Y means X no gradient passes followed by Y gradient passes.
With matched 4 warmup layers, WhiteMatter with cyclic iteration has higher gradient similarity than LCKV while using 24.7% less time. If LCKV use more passes for a closer gradient similarity, the saving becomes 52.3%.